Allow leaf sub-skills to emit agent findings within their domain

The original DO contract pinned all agent reasoning to the super-skill:
'Agent findings are emitted only by super-skills... Leaf sub-skills MUST
NOT emit agent findings'. This funnels all agent reasoning across all 6
domains through a single super-skill pass, which is the root structural
cause of the attention dilution we have been chasing in BCAppsBCQuality
PRs #28 and #30:

- T1 standalone run showed al-security-review finds rimd-on-read-only
  cleanly when run alone, but emits zero agent findings because the
  contract forbids it. So obvious things like case-without-else (no
  matching KB article yet) get dropped on the floor.
- The al-code-review self-review pass keeps producing 0-1 agent findings
  per PR because it is asked to reason across 6 domains in one pass.

The fix is to move agent reasoning into the leaves, bounded by each
leaf's domain. Each leaf now has both knowledge-backed and agent-finding
permissions within its own scope; the super-skill self-review pass
becomes a smaller, cross-cutting role.

skills/do.md
  - Replace the 'only by super-skills' / 'MUST NOT' clause with a
    two-tier model: leaf sub-skills MAY emit agent findings strictly
    within their declared domain; super-skills MAY emit agent findings
    for cross-cutting concerns that span domains.
  - Update the encoding rules: leaf agent findings have references:[]
    and an agent:-prefixed id, no from-sub-skill (the leaf's own report
    carries the finding under its own skill.id). Super-skills set
    from-sub-skill='agent' for their own self-review findings; when
    rolling up leaf agent findings, they set from-sub-skill=<leaf-id>.
  - Clarify that 'MUST validate against knowledge' applies to super-
    skill self-review candidates only - leaves already validated within
    their domain when they decided to emit.

microsoft/skills/review/al-{security,performance,privacy,style,upgrade,
                                                            ui}-review.md
  - New paragraph after the confidence rules instructing each leaf to
    surface domain-specific agent findings when no knowledge file
    covers a defect the agent recognises from general AL knowledge.
  - Bound the scope: 'The scope is strictly <domain>; defects outside
    this domain belong to other leaves and MUST NOT be emitted here.'
  - Same validation requirement: check the worklist for a matching
    knowledge file first; if one exists, upgrade to a knowledge-backed
    finding instead.

microsoft/skills/review/al-code-review.md
  - Rewrite the 'Agent self-review pass' subsection. The pass is now
    explicitly for cross-cutting concerns that no single leaf could
    have surfaced because they span multiple domains. Domain-specific
    reasoning belongs in the leaves, not duplicated here.
  - Update the rollup behavior to acknowledge leaf-emitted agent
    findings: they are rolled up like any other sub-skill finding, with
    from-sub-skill set to the leaf id, and are not re-validated by the
    super-skill (the leaf already validated within its own domain).
  - Drop the 'Leaf sub-skills MUST NOT emit agent findings' line.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Jesper Schulz-Wedde 2026-05-28 11:42:38 +02:00
parent 832504d428
commit 3a07ee82d2
8 changed files with 29 additions and 12 deletions

View file

@ -80,27 +80,27 @@ For each sub-skill in the worklist, executed one at a time per the discipline ab
### Agent self-review pass
After every sub-skill has produced its sub-result, perform a self-review pass against the same task input using the agent's built-in BC and AL knowledge. BCQuality is an **additive** knowledge layer: it augments the agent's review judgement, it does not replace it. The goal of this pass is to surface defects the agent recognises on its own — bugs, anti-patterns, error-handling gaps, AL idioms — that the leaf sub-skills did not catch because no BCQuality knowledge file covers them yet.
Each leaf sub-skill emits both knowledge-backed findings and, per its own contract, agent findings within its own domain. After every sub-skill has produced its sub-result, perform a super-skill self-review pass against the same task input. The goal of this pass is to surface defects the agent recognises on its own that **no single leaf could have surfaced** because they cross domain boundaries — architecture-level issues that touch performance and reliability at once, error-handling gaps that span security and UX, resource-lifecycle patterns that affect both correctness and performance, and similar cross-cutting concerns. BCQuality is an **additive** knowledge layer: it augments the agent's review judgement, it does not replace it.
This pass is mandatory. An empty agent-findings list is acceptable only when the diff is small enough that the leaves have provably exhausted the surface (in practice: PRs of ≤2 files with ≤30 changed lines and at least one sub-skill emitting findings). For larger diffs, returning an empty agent-findings list is a defect — the agent has built-in BC/AL knowledge that the leaves cannot supply, and refusing to apply it is the most common cause of parity loss against agents that do not have a BCQuality layer at all.
This pass is mandatory on real-size PRs. An empty agent-findings list at the super-skill level is acceptable only when the diff is small enough that nothing cross-cutting plausibly applies (in practice: PRs of ≤2 files with ≤30 changed lines). For larger diffs, the fact that leaves emitted findings does not exempt the super-skill — the leaves handle within-domain reasoning, the super-skill handles cross-domain reasoning, and a real-size PR with no cross-cutting candidates is rare.
Frame the pass by the domains the sub-skills already cover (performance, security, privacy, style, upgrade, UI) and by the cross-cutting concerns that span them (architecture, error handling, resource lifecycle). For each domain, ask whether the diff exhibits a pattern the agent recognises as a defect from general AL knowledge and that the corresponding sub-skill did not flag. The categories are anchors for completeness, not a script: a candidate from any category is in scope, and a candidate from no category is also in scope when the agent has independent grounds for it.
Frame the pass by cross-cutting concerns — architecture, error handling, resource lifecycle — and by the seams between leaf domains. Do not duplicate domain-specific reasoning that belongs in a leaf: a security-only concern is the security leaf's responsibility, not the super-skill's. The super-skill pass adds value where no individual leaf has the right scope.
For every candidate the agent identifies in this pass:
1. **Validate against BCQuality knowledge.** Check the candidate against the knowledge files the sub-skills have already loaded for this task (visible via their `references` and `suppressed` lists in `sub-results`).
- If a BCQuality knowledge file matches the candidate, upgrade it to a knowledge-backed finding: cite the file in `references`, set `id` to the file's path, set `from-sub-skill` to the sub-skill that owns that knowledge domain, and merge with or deduplicate against any sub-skill finding that already covers the same concern at the same location.
- If a BCQuality knowledge file **explicitly contradicts** the candidate (its `## Best Practice` or `## Anti Pattern` says the opposite of what the agent flagged), suppress the candidate and do not surface it.
- Otherwise the candidate has no BCQuality coverage; emit it as an agent finding.
- Otherwise the candidate has no BCQuality coverage; emit it as a super-skill agent finding.
2. **Emit agent finding.** Per DO's *Agent findings* rules:
- `from-sub-skill: "agent"`
- `from-sub-skill: "agent"` (the super-skill itself produced it)
- `references: []`
- `id` is a skill-defined slug prefixed with `agent:` (for example, `agent:missing-error-handling-on-http-call`).
- `confidence` capped at `medium`.
- `message` is non-empty and self-contained, describing both the issue and a concrete recommendation. A consumer rendering the finding has no knowledge-file footer to fall back on.
- `suggested-code` SHOULD be set when the fix is small and mechanical (e.g. removing a few unreachable lines, replacing a `Count() > 0` test with `not IsEmpty()`, declaring a missing `Label`, adding an `else` branch to a `case` over an enum). Omit it when the appropriate fix depends on context the agent cannot determine.
- `suggested-code` SHOULD be set when the fix is small and mechanical. Omit it when the appropriate fix depends on context the agent cannot determine.
Leaf sub-skills MUST NOT emit agent findings: their scope is bounded by the knowledge subset they evaluate. The self-review pass is a super-skill responsibility.
Leaf-level agent findings (those with `references: []` inside a sub-skill's report) are rolled up into the super-skill's top-level `findings[]` like any other sub-skill finding — they keep their `from-sub-skill: <leaf-id>` attribution and are not rewritten. They are not subject to the "MUST validate against knowledge" step above, because each leaf has already validated within its own domain.
### Suggested-code guidance

View file

@ -61,6 +61,8 @@ Set `confidence` to:
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
- `low` when the finding is an advisory derived only from applicability.
After evaluating each worklist entry, also consider whether the diff exhibits a performance defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly performance; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
Outcome selection:

View file

@ -61,6 +61,8 @@ Set `confidence` to:
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
- `low` when the finding is an advisory derived only from applicability.
After evaluating each worklist entry, also consider whether the diff exhibits a privacy defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly privacy; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
Outcome selection:

View file

@ -61,6 +61,8 @@ Set `confidence` to:
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
- `low` when the finding is an advisory derived only from applicability.
After evaluating each worklist entry, also consider whether the diff exhibits a security defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly security; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
Outcome selection:

View file

@ -59,6 +59,8 @@ Set `confidence` to:
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
- `low` when the finding is an advisory derived only from applicability.
After evaluating each worklist entry, also consider whether the diff exhibits a style defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly style; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
Outcome selection:

View file

@ -59,6 +59,8 @@ Set `confidence` to:
- `medium` when detection relies on heuristics (judging whether a caption is a noun phrase or a sentence phrase) or when any frontmatter dimension was `unknown`.
- `low` when the finding is an advisory derived only from applicability.
After evaluating each worklist entry, also consider whether the diff exhibits a UI defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly UI; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
Outcome selection:

View file

@ -61,6 +61,8 @@ Set `confidence` to:
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
- `low` when the finding is an advisory derived only from applicability.
After evaluating each worklist entry, also consider whether the diff exhibits a upgrade defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly upgrade; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
Outcome selection:

View file

@ -134,17 +134,22 @@ An empty `findings` array with `outcome: completed` means the skill ran and foun
When a super-skill rolls up a non-citation finding from a sub-skill (an `id` that is a slug, not a path), the super-skill MUST prefix the `id` with `<from-sub-skill>:` to avoid collisions across sub-skills (for example, a slug `missing-test` from `al-security-review` becomes `al-security-review:missing-test`). Citation-based findings are already globally unique through their repo-relative path and MUST NOT be rewritten.
**Agent findings.** A super-skill MAY emit findings that the agent identified through its own reasoning rather than from a BCQuality knowledge file. BCQuality is an **additive** knowledge layer: it augments the agent's pre-existing review judgement, it does not replace it. An agent finding is encoded by:
**Agent findings.** A skill MAY emit findings that the agent identified through its own reasoning rather than from a BCQuality knowledge file. BCQuality is an **additive** knowledge layer: it augments the agent's pre-existing review judgement, it does not replace it. An agent finding is encoded by:
- `from-sub-skill: "agent"` — the canonical marker. Use this exact value; do not invent equivalents.
- `references: []` — required. An agent finding has no knowledge-file citation by definition; if a citation existed, the finding would be a knowledge-backed finding instead.
- `id` — a skill-defined slug, prefixed with `agent:` (mirroring the `<from-sub-skill>:` rule). For example, `agent:obsolete-find-signature`.
- `confidence` — capped at `medium`. Without a knowledge-file citation there is no authoritative basis for `high` confidence.
- `message` — non-empty and self-contained. It MUST describe the issue and a concrete recommendation, since a consumer rendering the finding has no knowledge-file footer to fall back on.
- `from-sub-skill` — set by super-skills only. The literal string `"agent"` when the super-skill itself produced the finding from its own cross-cutting reasoning; or the producing leaf's `skill.id` when the super-skill is rolling up a leaf's agent finding. Absent on findings emitted directly by a leaf (the leaf's own report carries the finding under its `skill.id` already).
Agent findings are emitted **only by super-skills** (the `al-code-review` super-skill is the canonical example). Leaf sub-skills MUST NOT emit agent findings: a leaf's job is to evaluate one knowledge subset, and a finding it cannot cite from that subset is out of scope for it. Before emitting an agent finding, a super-skill MUST validate the candidate against the BCQuality knowledge it has already loaded for the task — if a knowledge file matches, the candidate is upgraded to a knowledge-backed finding (and merged or deduplicated against any sub-skill output that already covers the same concern); if a knowledge file explicitly contradicts the candidate, it is suppressed.
Agent findings may be emitted by both leaf sub-skills and super-skills, with different scope boundaries:
Consumers that render output MAY treat agent findings differently from knowledge-backed findings (for example, by labelling them and routing them to a separate review domain). The `from-sub-skill: "agent"` marker is the contract they rely on.
- A **leaf sub-skill** MAY emit agent findings strictly within its declared `domain`. al-security-review MAY surface an agent security finding that no knowledge file covers (for example, a `case` over a security-relevant enum with no `else` arm), but MUST NOT emit a style or performance agent finding — those are out of scope for the leaf and belong to other leaves or to the super-skill. The leaf's domain is the bounding box.
- A **super-skill** MAY emit agent findings of any kind, but its self-review pass is most useful for **cross-cutting** concerns that span multiple leaf domains (architecture-level smells, error-handling gaps that touch security and reliability, resource lifecycle issues). Domain-specific agent reasoning is the leaves' job; the super-skill should not duplicate it.
Before emitting an agent finding, a skill MUST validate the candidate against the BCQuality knowledge it has already loaded for the task — if a knowledge file matches, the candidate is upgraded to a knowledge-backed finding (and merged or deduplicated against any existing finding that already covers the same concern); if a knowledge file explicitly contradicts the candidate, it is suppressed. For a super-skill rolling up leaf reports, this validation is done against the union of knowledge files all leaves loaded, not just one leaf's set.
Consumers that render output MAY treat agent findings differently from knowledge-backed findings (for example, by labelling them and routing them to a separate review domain). The `references: []` marker, together with the `agent:` `id` prefix, is the contract they rely on; `from-sub-skill: "agent"` is the additional marker for super-skill-emitted agent findings.
**`findings[].severity`** — see the taxonomy below.
@ -167,7 +172,7 @@ The first reference is the **primary** reference: the knowledge file the finding
**`findings[].confidence`** — the skill's confidence that the finding is a true positive, given the evidence it evaluated. Not applicability confidence, not severity confidence. Values: `high`, `medium`, `low`.
**`findings[].from-sub-skill`** — optional. Set only by super-skills. The `skill.id` of the sub-skill that produced the finding, or the literal string `"agent"` for an agent finding the super-skill produced from its own reasoning. Absent on findings produced directly by a leaf skill.
**`findings[].from-sub-skill`** — optional. Set only by super-skills. The `skill.id` of the sub-skill that produced the finding, or the literal string `"agent"` for an agent finding the super-skill produced from its own cross-cutting reasoning. Absent on findings emitted directly by a leaf skill — including agent findings the leaf emits within its own domain, which appear in the leaf's own report without this field.
**`findings[].suggested-code`** — optional. A concrete code-replacement payload for the lines indicated by `location`. When present, the string MUST be a literal replacement for the source lines covered by `location.line` (or `location.range` if set) — i.e., what the file would contain after the fix, with no surrounding diff markers, fences, or commentary. Consumers MAY render it as a one-click suggestion in the delivery surface (for example, a GitHub ```` ```suggestion ```` block). Emit `suggested-code` only when the fix is small, mechanical, and unambiguous; omit it when the appropriate fix depends on context the skill cannot determine. The `suggested-code` payload supplements `message`; it does not replace the explanation in `message`.