mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-08-06 09:26:52 +01:00
Allow leaf sub-skills to emit agent findings within their domain
The original DO contract pinned all agent reasoning to the super-skill: 'Agent findings are emitted only by super-skills... Leaf sub-skills MUST NOT emit agent findings'. This funnels all agent reasoning across all 6 domains through a single super-skill pass, which is the root structural cause of the attention dilution we have been chasing in BCAppsBCQuality PRs #28 and #30: - T1 standalone run showed al-security-review finds rimd-on-read-only cleanly when run alone, but emits zero agent findings because the contract forbids it. So obvious things like case-without-else (no matching KB article yet) get dropped on the floor. - The al-code-review self-review pass keeps producing 0-1 agent findings per PR because it is asked to reason across 6 domains in one pass. The fix is to move agent reasoning into the leaves, bounded by each leaf's domain. Each leaf now has both knowledge-backed and agent-finding permissions within its own scope; the super-skill self-review pass becomes a smaller, cross-cutting role. skills/do.md - Replace the 'only by super-skills' / 'MUST NOT' clause with a two-tier model: leaf sub-skills MAY emit agent findings strictly within their declared domain; super-skills MAY emit agent findings for cross-cutting concerns that span domains. - Update the encoding rules: leaf agent findings have references:[] and an agent:-prefixed id, no from-sub-skill (the leaf's own report carries the finding under its own skill.id). Super-skills set from-sub-skill='agent' for their own self-review findings; when rolling up leaf agent findings, they set from-sub-skill=<leaf-id>. - Clarify that 'MUST validate against knowledge' applies to super- skill self-review candidates only - leaves already validated within their domain when they decided to emit. microsoft/skills/review/al-{security,performance,privacy,style,upgrade, ui}-review.md - New paragraph after the confidence rules instructing each leaf to surface domain-specific agent findings when no knowledge file covers a defect the agent recognises from general AL knowledge. - Bound the scope: 'The scope is strictly <domain>; defects outside this domain belong to other leaves and MUST NOT be emitted here.' - Same validation requirement: check the worklist for a matching knowledge file first; if one exists, upgrade to a knowledge-backed finding instead. microsoft/skills/review/al-code-review.md - Rewrite the 'Agent self-review pass' subsection. The pass is now explicitly for cross-cutting concerns that no single leaf could have surfaced because they span multiple domains. Domain-specific reasoning belongs in the leaves, not duplicated here. - Update the rollup behavior to acknowledge leaf-emitted agent findings: they are rolled up like any other sub-skill finding, with from-sub-skill set to the leaf id, and are not re-validated by the super-skill (the leaf already validated within its own domain). - Drop the 'Leaf sub-skills MUST NOT emit agent findings' line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
parent
832504d428
commit
3a07ee82d2
8 changed files with 29 additions and 12 deletions
|
|
@ -80,27 +80,27 @@ For each sub-skill in the worklist, executed one at a time per the discipline ab
|
|||
|
||||
### Agent self-review pass
|
||||
|
||||
After every sub-skill has produced its sub-result, perform a self-review pass against the same task input using the agent's built-in BC and AL knowledge. BCQuality is an **additive** knowledge layer: it augments the agent's review judgement, it does not replace it. The goal of this pass is to surface defects the agent recognises on its own — bugs, anti-patterns, error-handling gaps, AL idioms — that the leaf sub-skills did not catch because no BCQuality knowledge file covers them yet.
|
||||
Each leaf sub-skill emits both knowledge-backed findings and, per its own contract, agent findings within its own domain. After every sub-skill has produced its sub-result, perform a super-skill self-review pass against the same task input. The goal of this pass is to surface defects the agent recognises on its own that **no single leaf could have surfaced** because they cross domain boundaries — architecture-level issues that touch performance and reliability at once, error-handling gaps that span security and UX, resource-lifecycle patterns that affect both correctness and performance, and similar cross-cutting concerns. BCQuality is an **additive** knowledge layer: it augments the agent's review judgement, it does not replace it.
|
||||
|
||||
This pass is mandatory. An empty agent-findings list is acceptable only when the diff is small enough that the leaves have provably exhausted the surface (in practice: PRs of ≤2 files with ≤30 changed lines and at least one sub-skill emitting findings). For larger diffs, returning an empty agent-findings list is a defect — the agent has built-in BC/AL knowledge that the leaves cannot supply, and refusing to apply it is the most common cause of parity loss against agents that do not have a BCQuality layer at all.
|
||||
This pass is mandatory on real-size PRs. An empty agent-findings list at the super-skill level is acceptable only when the diff is small enough that nothing cross-cutting plausibly applies (in practice: PRs of ≤2 files with ≤30 changed lines). For larger diffs, the fact that leaves emitted findings does not exempt the super-skill — the leaves handle within-domain reasoning, the super-skill handles cross-domain reasoning, and a real-size PR with no cross-cutting candidates is rare.
|
||||
|
||||
Frame the pass by the domains the sub-skills already cover (performance, security, privacy, style, upgrade, UI) and by the cross-cutting concerns that span them (architecture, error handling, resource lifecycle). For each domain, ask whether the diff exhibits a pattern the agent recognises as a defect from general AL knowledge and that the corresponding sub-skill did not flag. The categories are anchors for completeness, not a script: a candidate from any category is in scope, and a candidate from no category is also in scope when the agent has independent grounds for it.
|
||||
Frame the pass by cross-cutting concerns — architecture, error handling, resource lifecycle — and by the seams between leaf domains. Do not duplicate domain-specific reasoning that belongs in a leaf: a security-only concern is the security leaf's responsibility, not the super-skill's. The super-skill pass adds value where no individual leaf has the right scope.
|
||||
|
||||
For every candidate the agent identifies in this pass:
|
||||
|
||||
1. **Validate against BCQuality knowledge.** Check the candidate against the knowledge files the sub-skills have already loaded for this task (visible via their `references` and `suppressed` lists in `sub-results`).
|
||||
- If a BCQuality knowledge file matches the candidate, upgrade it to a knowledge-backed finding: cite the file in `references`, set `id` to the file's path, set `from-sub-skill` to the sub-skill that owns that knowledge domain, and merge with or deduplicate against any sub-skill finding that already covers the same concern at the same location.
|
||||
- If a BCQuality knowledge file **explicitly contradicts** the candidate (its `## Best Practice` or `## Anti Pattern` says the opposite of what the agent flagged), suppress the candidate and do not surface it.
|
||||
- Otherwise the candidate has no BCQuality coverage; emit it as an agent finding.
|
||||
- Otherwise the candidate has no BCQuality coverage; emit it as a super-skill agent finding.
|
||||
2. **Emit agent finding.** Per DO's *Agent findings* rules:
|
||||
- `from-sub-skill: "agent"`
|
||||
- `from-sub-skill: "agent"` (the super-skill itself produced it)
|
||||
- `references: []`
|
||||
- `id` is a skill-defined slug prefixed with `agent:` (for example, `agent:missing-error-handling-on-http-call`).
|
||||
- `confidence` capped at `medium`.
|
||||
- `message` is non-empty and self-contained, describing both the issue and a concrete recommendation. A consumer rendering the finding has no knowledge-file footer to fall back on.
|
||||
- `suggested-code` SHOULD be set when the fix is small and mechanical (e.g. removing a few unreachable lines, replacing a `Count() > 0` test with `not IsEmpty()`, declaring a missing `Label`, adding an `else` branch to a `case` over an enum). Omit it when the appropriate fix depends on context the agent cannot determine.
|
||||
- `suggested-code` SHOULD be set when the fix is small and mechanical. Omit it when the appropriate fix depends on context the agent cannot determine.
|
||||
|
||||
Leaf sub-skills MUST NOT emit agent findings: their scope is bounded by the knowledge subset they evaluate. The self-review pass is a super-skill responsibility.
|
||||
Leaf-level agent findings (those with `references: []` inside a sub-skill's report) are rolled up into the super-skill's top-level `findings[]` like any other sub-skill finding — they keep their `from-sub-skill: <leaf-id>` attribution and are not rewritten. They are not subject to the "MUST validate against knowledge" step above, because each leaf has already validated within its own domain.
|
||||
|
||||
### Suggested-code guidance
|
||||
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
After evaluating each worklist entry, also consider whether the diff exhibits a performance defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly performance; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
After evaluating each worklist entry, also consider whether the diff exhibits a privacy defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly privacy; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
After evaluating each worklist entry, also consider whether the diff exhibits a security defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly security; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
|
|
|||
|
|
@ -59,6 +59,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
After evaluating each worklist entry, also consider whether the diff exhibits a style defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly style; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
|
|
|||
|
|
@ -59,6 +59,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics (judging whether a caption is a noun phrase or a sentence phrase) or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
After evaluating each worklist entry, also consider whether the diff exhibits a UI defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly UI; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
After evaluating each worklist entry, also consider whether the diff exhibits a upgrade defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly upgrade; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue