mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-08-06 09:26:52 +01:00
Strengthen al-code-review execution, propagate suggested-code to leaves, two new KB articles
Driven by a parity comparison between BCAppsBCQuality PR #27 and BCAppsCampAIRHack PR #162 on byte-identical content: | | BCQuality | AIRHack | |--|--|--| | Total findings | 6 | 10 | | Performance | 0 | 4 | | Security | 0 | 5 | Standalone runs of al-security-review and al-performance-review against the SAME diff produced the expected matches (rimd-on-read-only via inherent-permissions-minimal-grant; redundant-Get via avoid-redundant-get-when-record-already-loaded). The miss in the live run is therefore not a knowledge-coverage gap and not a worklist filtering issue. It is attention dilution inside the al-code-review super-skill, which the model collapses into one rolled-up generation pass on real-size PRs. Changes: microsoft/skills/review/al-code-review.md - New 'Execution discipline (mandatory)' subsection in the Action step that explicitly forbids collapsing leaves into one shared reasoning pass and requires each sub-skill to walk its Source -> Relevance -> Worklist -> Action steps as its own iteration before the next leaf starts. - Self-review pass is now described as the final, mandatory iteration with a concrete candidate-category checklist (architecture-level smells, error-handling gaps, magic constants, privacy/telemetry, resource lifecycle). Returning zero agent findings on a real-size diff is explicitly defined as a defect. microsoft/skills/review/al-{security,performance,privacy,style, upgrade,ui}-review.md - Each leaf skill now states that when an unambiguous .good.al companion exists, findings[].suggested-code should carry the literal replacement for the source lines. Closes the one-click-suggestion gap created when BCQ#19 only updated al-code-review. microsoft/knowledge/security/case-must-handle-unknown-enum-values.{md, bad.al, good.al} - New article: case over a security-sensitive enum (Authentication Type, Authorization Mode, Identity Provider, Permission Scope, Encryption Algorithm) MUST have an else arm. Without it, an unknown enum value silently falls through and the security context never initialises. The bad sample is lifted from the SharePoint Graph helper that triggered the parity finding. microsoft/knowledge/performance/instream-length-unreliable-for-bc- streams.{md,bad,good} - New article: InStream.Length returns 0 / partial for HTTP-response streams and some file-API streams, breaking size-threshold branching in upload code. Bad sample is the simple-vs-chunked Graph upload pattern; good sample materialises into a Temp Blob first. Companion change: microsoft/BCAppsBCQuality#28 extends the orchestrator's bootstrap prompt with the same per-iteration execution discipline and adds a CI warning when a >5-file PR returns zero agent findings (regression signal). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
parent
d8e8355259
commit
31b9949235
13 changed files with 236 additions and 3 deletions
|
|
@ -60,9 +60,18 @@ The worklist is the list of sub-skills judged relevant by the previous step. Eve
|
|||
|
||||
## Action
|
||||
|
||||
### Execution discipline (mandatory)
|
||||
|
||||
The Action step is a sequence of **discrete iterations**, not one combined generation. The contract requires the super-skill to invoke each sub-skill in turn and then perform a self-review pass. Concretely this means:
|
||||
|
||||
- Treat each sub-skill in the worklist as its own pass: read the sub-skill's instructions, apply its Source → Relevance → Worklist → Action steps to the orchestrator-supplied inputs, and produce that sub-skill's complete findings-report before moving on.
|
||||
- Do not collapse multiple sub-skills into one shared reasoning step. Each sub-skill has a distinct knowledge subset and a distinct evaluation procedure; sharing one rolled-up scan dilutes per-skill attention and causes leaves to silently underreport (this has been observed in production: leaf skills returned empty `findings[]` while their standalone runs against the same diff produced multiple matches).
|
||||
- The agent self-review pass is its own final iteration. Begin it only after every sub-skill in the worklist has completed and its sub-result is recorded.
|
||||
- Sub-skills are independent: re-walking the diff once per sub-skill is correct and expected. The output schema accommodates this — `sub-results` carries one entry per sub-skill, each a complete findings-report.
|
||||
|
||||
### Roll up sub-skill findings
|
||||
|
||||
For each sub-skill in the worklist:
|
||||
For each sub-skill in the worklist, executed one at a time per the discipline above:
|
||||
|
||||
1. Invoke the sub-skill with the orchestrator's inputs, passing only the subset each sub-skill declares in its `inputs`.
|
||||
2. Capture the sub-skill's complete findings-report verbatim and append it to `sub-results`.
|
||||
|
|
@ -71,7 +80,17 @@ For each sub-skill in the worklist:
|
|||
|
||||
### Agent self-review pass
|
||||
|
||||
After the sub-skill rollup, perform a self-review pass against the same task input using the agent's built-in BC and AL knowledge. BCQuality is an **additive** knowledge layer: it augments the agent's review judgement, it does not replace it. The goal of this pass is to surface defects the agent recognises on its own — bugs, anti-patterns, error-handling gaps, AL idioms — that the leaf sub-skills did not catch because no BCQuality knowledge file covers them yet.
|
||||
After every sub-skill has produced its sub-result, perform a self-review pass against the same task input using the agent's built-in BC and AL knowledge. BCQuality is an **additive** knowledge layer: it augments the agent's review judgement, it does not replace it. The goal of this pass is to surface defects the agent recognises on its own — bugs, anti-patterns, error-handling gaps, AL idioms — that the leaf sub-skills did not catch because no BCQuality knowledge file covers them yet.
|
||||
|
||||
This pass is mandatory. An empty agent-findings list is acceptable only when the diff is small enough that the leaves have provably exhausted the surface (in practice: PRs of ≤2 files with ≤30 changed lines and at least one sub-skill emitting findings). For larger diffs, returning an empty agent-findings list is a defect — the agent has built-in BC/AL knowledge that the leaves cannot supply, and refusing to apply it is the most common cause of parity loss against agents that do not have a BCQuality layer at all.
|
||||
|
||||
Before emitting the final report, walk these candidate categories explicitly against the diff and decide for each whether to emit a finding. This is a checklist, not an exhaustive list — it names patterns where the agent's general AL knowledge consistently outruns BCQuality coverage:
|
||||
|
||||
- **Architecture-level smells**: repeated `Record.Get` of the same key inside one call chain; large payloads streamed through memory when a streaming primitive exists; widely-scoped `Permissions = … = rimd` on codeunits that only read; `case` over an enum without an `else` branch (silent fall-through on unknown values); HTTP calls without `IsSuccessStatusCode` inspection; `tryFunction`-shaped procedures that swallow errors without surfacing them to telemetry.
|
||||
- **Error-handling gaps**: `Error()` built by `+` string concatenation; `if not Customer.Get(...) then Error(... + CustomerNo)`; failure paths that emit no telemetry; upgrade procedures that do not register `OnGetPerCompanyUpgradeTags` / `OnGetPerDatabaseUpgradeTags` event subscribers when they call `Set/HasUpgradeTag`.
|
||||
- **Magic constants** that encode a protocol or platform threshold (Graph 4 MB simple-upload cutoff, file-size limits, retry counts) without naming them through a `const`-suffixed Label.
|
||||
- **Privacy/telemetry surface**: PII bound into `Session.LogMessage` message text; placeholder telemetry event IDs (`'0000'`, `'TODO'`); `Locked = true` missing on Labels that carry URLs, JSON, or wire tokens.
|
||||
- **Resource lifecycle**: temporary records or HTTP message objects re-used across iterations without `Reset`/clearing; `Commit` inside a loop body.
|
||||
|
||||
For every candidate the agent identifies in this pass:
|
||||
|
||||
|
|
@ -85,7 +104,7 @@ For every candidate the agent identifies in this pass:
|
|||
- `id` is a skill-defined slug prefixed with `agent:` (for example, `agent:missing-error-handling-on-http-call`).
|
||||
- `confidence` capped at `medium`.
|
||||
- `message` is non-empty and self-contained, describing both the issue and a concrete recommendation. A consumer rendering the finding has no knowledge-file footer to fall back on.
|
||||
- `suggested-code` SHOULD be set when the fix is small and mechanical (e.g. removing a few unreachable lines, replacing a `Count() > 0` test with `not IsEmpty()`, declaring a missing `Label`). Omit it when the appropriate fix depends on context the agent cannot determine.
|
||||
- `suggested-code` SHOULD be set when the fix is small and mechanical (e.g. removing a few unreachable lines, replacing a `Count() > 0` test with `not IsEmpty()`, declaring a missing `Label`, adding an `else` branch to a `case` over an enum). Omit it when the appropriate fix depends on context the agent cannot determine.
|
||||
|
||||
Leaf sub-skills MUST NOT emit agent findings: their scope is bounded by the knowledge subset they evaluate. The self-review pass is a super-skill responsibility.
|
||||
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
||||
- `completed` — the skill evaluated every worklist item; default when the skill finishes normally, including when the resulting `findings` array is empty.
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
||||
- `completed` — the skill evaluated every worklist item; default when the skill finishes normally, including when the resulting `findings` array is empty.
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
||||
- `completed` — the skill evaluated every worklist item; default when the skill finishes normally, including when the resulting `findings` array is empty.
|
||||
|
|
|
|||
|
|
@ -59,6 +59,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
||||
- `completed` — the skill evaluated every worklist item.
|
||||
|
|
|
|||
|
|
@ -59,6 +59,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics (judging whether a caption is a noun phrase or a sentence phrase) or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
||||
- `completed` — the skill evaluated every worklist item.
|
||||
|
|
|
|||
|
|
@ -61,6 +61,8 @@ Set `confidence` to:
|
|||
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
||||
- `low` when the finding is an advisory derived only from applicability.
|
||||
|
||||
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
||||
|
||||
Outcome selection:
|
||||
|
||||
- `completed` — the skill evaluated every worklist item.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue