Require suggested-code or omission reason for mechanical findings

BCQ now wins on coverage in the synthetic complex benchmark, but only
2 of 25 rendered comments carried GitHub suggestion blocks while the
AIRHack reviewer emitted suggestions for every finding. Tighten the
DO contract and leaf-skill instructions so suggested-code is no longer
a soft affordance for mechanical fixes.

Changes:
- Add optional findings[].suggested-code-omission-reason to the DO
  schema.
- Document suggested-code as expected for small, local, mechanical
  findings, with examples (delete unreachable code, Count() > 0 ->
  not IsEmpty(), move local Label, add missing ToolTip/OptionCaption,
  replace string-concatenated Error, change permission token, add an
  obvious else/guard branch).
- Require omission reasons when a mechanical-looking finding omits
  suggested-code.
- Update all AL leaf skills and al-code-review guidance with the same
  stronger contract.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Jesper Schulz-Wedde 2026-05-28 15:00:10 +02:00
parent b342775971
commit d994f2ad0e
8 changed files with 38 additions and 10 deletions

View file

@ -61,7 +61,9 @@ Set `confidence` to:
After evaluating each worklist entry, also consider whether the diff exhibits a UI defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly UI; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
For every emitted finding, decide whether the fix is mechanical. A fix is mechanical when it is small, local, and unambiguous from the diff context (for example: delete unreachable lines; replace `Count() > 0` with `not IsEmpty()`; move a local `Label` to object scope; add a missing `ToolTip`, `OptionCaption`, or `DataClassification`; replace a string-concatenated `Error` with a Label-backed call; change an over-broad permission token; or add an obvious `else`/guard branch). For mechanical findings, emit `findings[].suggested-code` with the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. When a `.good.al` companion exists and the diff context matches the `.bad.al` shape, adapt the `.good.al` replacement into `suggested-code`.
Omit `suggested-code` only when the appropriate fix depends on context the skill cannot determine, when multiple defensible replacements exist, or when the fix spans non-contiguous code. If a finding is mechanical-looking but you omit `suggested-code`, set `findings[].suggested-code-omission-reason` to a short explanation. See `skills/do.md` for the full contract.
Outcome selection:
@ -101,3 +103,4 @@ Output conforms to the DO output contract. A populated example:
"suppressed": []
}
```