Follow-up to BCQuality#22. The previous contract made suggested-code
expected for mechanical findings, but still left too much judgement to
the model. In the synthetic benchmark, BCQ produced omission reasons
for 11 mechanical-looking findings instead of suggestions.
Add explicit must-suggest classes where omission is not allowed when
location points to the affected line/range and the replacement can be
expressed as one contiguous block:
- missing OptionCaption
- missing ToolTip
- formatter-only casing/spacing defects
- Count() existence/repeated-count fixes
- unreachable/dead contiguous blocks
- over-broad permission tokens with an obvious narrower token
- obvious DataClassification additions
- local Label moves when the target range includes both old and new
declaration locations
Update all AL leaf skills and al-code-review to reference these
must-suggest classes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
BCQ now wins on coverage in the synthetic complex benchmark, but only
2 of 25 rendered comments carried GitHub suggestion blocks while the
AIRHack reviewer emitted suggestions for every finding. Tighten the
DO contract and leaf-skill instructions so suggested-code is no longer
a soft affordance for mechanical fixes.
Changes:
- Add optional findings[].suggested-code-omission-reason to the DO
schema.
- Document suggested-code as expected for small, local, mechanical
findings, with examples (delete unreachable code, Count() > 0 ->
not IsEmpty(), move local Label, add missing ToolTip/OptionCaption,
replace string-concatenated Error, change permission token, add an
obvious else/guard branch).
- Require omission reasons when a mechanical-looking finding omits
suggested-code.
- Update all AL leaf skills and al-code-review guidance with the same
stronger contract.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The original DO contract pinned all agent reasoning to the super-skill:
'Agent findings are emitted only by super-skills... Leaf sub-skills MUST
NOT emit agent findings'. This funnels all agent reasoning across all 6
domains through a single super-skill pass, which is the root structural
cause of the attention dilution we have been chasing in BCAppsBCQuality
PRs #28 and #30:
- T1 standalone run showed al-security-review finds rimd-on-read-only
cleanly when run alone, but emits zero agent findings because the
contract forbids it. So obvious things like case-without-else (no
matching KB article yet) get dropped on the floor.
- The al-code-review self-review pass keeps producing 0-1 agent findings
per PR because it is asked to reason across 6 domains in one pass.
The fix is to move agent reasoning into the leaves, bounded by each
leaf's domain. Each leaf now has both knowledge-backed and agent-finding
permissions within its own scope; the super-skill self-review pass
becomes a smaller, cross-cutting role.
skills/do.md
- Replace the 'only by super-skills' / 'MUST NOT' clause with a
two-tier model: leaf sub-skills MAY emit agent findings strictly
within their declared domain; super-skills MAY emit agent findings
for cross-cutting concerns that span domains.
- Update the encoding rules: leaf agent findings have references:[]
and an agent:-prefixed id, no from-sub-skill (the leaf's own report
carries the finding under its own skill.id). Super-skills set
from-sub-skill='agent' for their own self-review findings; when
rolling up leaf agent findings, they set from-sub-skill=<leaf-id>.
- Clarify that 'MUST validate against knowledge' applies to super-
skill self-review candidates only - leaves already validated within
their domain when they decided to emit.
microsoft/skills/review/al-{security,performance,privacy,style,upgrade,
ui}-review.md
- New paragraph after the confidence rules instructing each leaf to
surface domain-specific agent findings when no knowledge file
covers a defect the agent recognises from general AL knowledge.
- Bound the scope: 'The scope is strictly <domain>; defects outside
this domain belong to other leaves and MUST NOT be emitted here.'
- Same validation requirement: check the worklist for a matching
knowledge file first; if one exists, upgrade to a knowledge-backed
finding instead.
microsoft/skills/review/al-code-review.md
- Rewrite the 'Agent self-review pass' subsection. The pass is now
explicitly for cross-cutting concerns that no single leaf could
have surfaced because they span multiple domains. Domain-specific
reasoning belongs in the leaves, not duplicated here.
- Update the rollup behavior to acknowledge leaf-emitted agent
findings: they are rolled up like any other sub-skill finding, with
from-sub-skill set to the leaf id, and are not re-validated by the
super-skill (the leaf already validated within its own domain).
- Drop the 'Leaf sub-skills MUST NOT emit agent findings' line.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds the contract field, skill instructions, and two knowledge articles
that BCAppsBCQuality's PR-review agent needs to match (and exceed) the
coverage of the embedded review agent in BCAppsCampAIRHack:
skills/do.md
- New optional findings[].suggested-code field. Documents what it MUST
contain (a literal line-replacement payload) and when to emit it.
microsoft/skills/review/al-code-review.md
- Instructs both the agent self-review pass and rolled-up sub-skill
findings to populate suggested-code when the fix is mechanical.
- Lists examples (dead code removal, Count > 0 -> IsEmpty, object-scope
Label) that map to issues observed in the parity comparison.
microsoft/knowledge/style/telemetry-event-id-stable-unique.{md,bad.al,good.al}
- New knowledge article: telemetry event IDs must be stable, unique,
and non-placeholder. Closes a gap surfaced by the parity comparison.
microsoft/knowledge/style/labels-declared-at-object-scope.{md,bad.al,good.al}
- New knowledge article: Labels must live in the object-level var
block, not in procedure-local var blocks. Closes the second gap.
microsoft/knowledge/privacy/no-pii-in-telemetry-message-string.md
- Adds an explicit note that changing DataClassification alone does not
make embedding PII into the message string acceptable, plus links to
the two adjacent privacy articles. Resolves the privacy advice the
parity comparison flagged as ambiguous.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Let super-skills surface findings the agent identifies on its own,
clearly tagged so consumers can render them differently from
knowledge-backed ones.
- skills/do.md: permit references:[] when from-sub-skill='agent';
define the agent-finding encoding (id 'agent:<slug>', confidence
capped at medium, self-contained message); restrict agent findings
to super-skills only.
- microsoft/skills/review/al-code-review.md: add a self-review pass
to Action that validates agent-identified candidates against
BCQuality (cite if matched, suppress if contradicted, surface as
agent finding otherwise). Add example finding.
- agent-consumption.md, README.md: describe the additive model and
the from-sub-skill: 'agent' marker so consumer orchestrators know
to render unbacked findings.
Strictly additive: existing knowledge-backed flow is unchanged and
backward compatible.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a new skill kind, 'entry-point', and its sole instance at
skills/entry.md. When an orchestrator points an agent at BCQuality,
the agent's first call is Entry: it receives a task context and
returns a dispatch record naming the action skill(s) to invoke.
Routing logic lives in Entry, not in the orchestrator.
Entry structurally follows DO's Source -> Relevance -> Worklist ->
Action pattern but the units it selects are action skills (not
knowledge files) and its output is a dispatch record (not a
findings-report).
Contract highlights:
- Inputs semantics in DO clarified as any-of: orchestrator supplies
whichever listed input types it has; skill must return
'not-applicable' if the subset is insufficient. This matches
the existing al-* canonical skills which declare
[pr-diff, file-path] as alternatives.
- Relevance admits candidates whose inputs intersect
inputs-available, not whose inputs are a subset.
- Dispatched inputs are the intersection, not the full
inputs-available set, to avoid leakage between skills.
- Super-skill precedence in Worklist supersedes a sub-skill only
when the goal is a broader match for the super than the sub.
When the goal specifically names a concern the sub handles
(e.g., 'performance review'), the sub wins and the super is
dropped with reason 'narrower-sub-skill-selected'.
- Skill layer precedence is defined here as custom > community >
microsoft, matching READ's rule for knowledge files.
- skipped[] carries 'superseded-by' for layer-precedence,
sub-skill, and super-skill drops, for traceability.
Propagate the concept through:
- skills/README.md: distinguish the runtime entry-point skill from
the three meta-skill contracts.
- skills/do.md: acknowledge entry-point alongside meta-skills as
the only kinds that live outside a layer.
- README.md: rewrite the Skills and Agent bootstrapping sections
so the bootstrap instruction is 'invoke /skills/entry.md first'.
- agent-consumption.md: update the Mermaid flow and step narrative
to show Entry dispatch, with READ and DO read on demand.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DO contract (skills/do.md)
- New 'sub-skills' optional frontmatter field on action skills: when
present and non-empty, the skill is a super-skill that composes
other action skills.
- New 'Composition (super-skills)' section covering section
interpretation, outcome rollup, summary aggregation, and suppression
scope.
- Output schema gains three optional fields: 'from-sub-skill' on each
finding, top-level 'sub-results[]' carrying nested findings-reports,
and top-level 'skipped-sub-skills[]'.
- Super-skills MUST NOT filter sub-skills by task content; leaves own
task-level applicability and signal via outcome.
- Findings from a failed sub-skill MUST NOT flow into the parent's
findings[] or counts, consistent with DO's rule that consumers
ignore a failed skill's findings. Reports are still preserved in
sub-results[] for traceability.
- Rolled-up non-citation finding ids MUST be prefixed with the sub-
skill id to prevent collisions across sub-skills. Citation-based
ids are already unique via repo path and are not rewritten.
- Outcome rollup rules updated: 'partial' covers S = {partial},
{partial, partial}, and {partial, failed}. Empty worklist rolls up
to 'not-applicable' with outcome-reason.
- Nested super-skills are not permitted in v1.
Reference skills (microsoft/skills/)
- al-code-review.md rewritten as the canonical super-skill: lists
al-performance-review and al-security-review as sub-skills, orch-
estrates invocation, aggregates output, and includes a worked
rolled-up JSON example plus the empty-corpus rollup.
- al-performance-review.md added as a leaf reference skill for the
performance knowledge domain.
- al-security-review.md added as a leaf reference skill for the
security knowledge domain.
- Both leaves retain the leaf-level rules validated in the prior
pass: partial-context message requirement, worklist-scoped
suppression, application-area semantics, and the platform-guarantee
threshold for blocker severity.
README updated to describe leaf vs super-skill and link all three
reference skills.
Two rubber-duck passes tightened the contract and caught schema
violations in the worked examples before commit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- skills/read.md (READ, Schema + Use) — knowledge-file contract:
frontmatter schema, required/optional sections (normative vs
non-normative), layer precedence with applicability-based conflict
detection, explicit frontmatter matching semantics (including
partial-context handling).
- skills/do.md (DO, Action Skill) — action-skill template:
frontmatter schema, required sections, four-step pattern
(Source -> Relevance -> Worklist -> Action), and the output contract
as a JSON schema with outcome, findings, structured references,
confidence, and mandatory suppression recording. Includes a worked
example.
- skills/write.md (WRITE, New Knowledge) — authoring guide:
atomicity, size, section guidance, field-by-field choices, file
naming, layer choice, pre-PR checklist. Defers to READ for the
format spec.
- README.md — link the three files from the meta-skills section and
update the output-contract paragraph to include outcome and
suppressed.
- agent-consumption.md — update step 5 (Agent emits structured output)
to match the richer DO contract.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
