mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-08-06 09:26:52 +01:00
6 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Jesper Schulz-Wedde
|
822cae1b27
|
Own the knowledge-index generator + index-aware review skills (#25)
* Make domain-skill knowledge discovery index-aware The 6 AL domain review skills and read.md now enumerate candidate articles from the BCQuality knowledge index (knowledge-index.json) instead of opening every file under the domain folder to read its frontmatter. The worklist selection predicate is unchanged (keywords intersect diff tokens, or topic matches a changed object type) - only the discovery source changes, so the same articles are selected. Full article bodies are read only for worklisted entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reconcile §Source wording with the lean knowledge index The BCQuality filter now emits a lean index whose per-article description is a one-line hint rather than the full verbatim Description. Update the six domain skills' §Source to say the index carries a one-line description hint (keywords, title, and a one-line description) instead of the full description. The worklist selection predicate is unchanged: keywords drive selection and the agent opens worklisted articles in full for their rule bodies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Own the knowledge-index generator in BCQuality The knowledge index is an acceleration of the skills' Source step, and its schema is part of that contract — so BCQuality should own the generator rather than each consumer re-implementing it. Add tools/Build-KnowledgeIndex.ps1 (the parser + lean-description shaping + emit, lifted verbatim from the BCAppsBCQuality filter prototype) and document the index in agent-consumption.md. Consumers prune their clone to policy, then call this script; the index stays in lockstep with the Source contract and every orchestrator gets the same faithful index for free. The worklist selection predicate is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Own knowledge-index generation in BCQuality (runtime + CI), not the consumer The index is now produced by BCQuality itself: Entry's preparation step rebuilds knowledge-index.json over the live, already-pruned clone at the start of every run, and a new CI workflow validates the generator's health (determinism, full coverage, selection-input integrity). Consumers no longer invoke or know about the index. Rebuilding over the pruned clone (vs shipping a committed full-corpus index) keeps the index exact for any consumer policy: it can never list a denied article, so policy-excluded rules cannot leak into discovery. READ now states the index is discovery-only -- a finding must cite an article opened in full, and rows whose file is absent are discarded before ranking. - skills/entry.md: new 'Preparation -- knowledge index' precondition - skills/read.md: index ownership + discovery-only invariant - microsoft/skills/review/*.md (6): 'BCQuality builds' (not 'the filter emits') - agent-consumption.md 5a: runtime+CI ownership rationale - .github/workflows/knowledge-index.yml + scripts/Test-KnowledgeIndex.ps1: generator guard - .gitignore: never commit the runtime index Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make runtime index build non-interactive and self-contained entry.md now gives the exact build command (pwsh ./tools/Build-KnowledgeIndex.ps1) so the agent's preparation step is unambiguous, and the generator's -BCQualityRoot parameter is optional (defaults to the clone root) so it runs in non-interactive -p mode without prompting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Resolve knowledge-index root to absolute path (cross-platform fix) Get-ChildItem.FullName is always absolute, so deriving the relative article path via Substring(\.Length) requires an absolute root. A relative root such as '.' (used by the CI guard's 'Test-KnowledgeIndex.ps1 -Root .') left the full path almost intact on Linux, producing bogus 'home/runner/.../knowledge' paths and failing the coverage check. Normalise both the generator's -BCQualityRoot and the test's -Root with Resolve-Path before use. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Jesper Schulz-Wedde <jesper.schulzwedde@microsoft.com> |
||
|
Jesper Schulz-Wedde
|
f07efa74a2 |
Tighten precision bar for agent findings and cap their severity
Agent findings (model-judgment findings not backed by a curated knowledge file) were producing too many false positives. Raise their emission bar and make them advisory: - skills/do.md: add a canonical 'Precision bar' for agent findings (emit only concrete, material defects an expert would agree on; steelman first; an explicit never-emit list; 'when in doubt, omit') and cap agent-finding severity at minor (advisory, non-gating), with a promotion note for severe-but-uncovered concerns. - al-code-review.md: decouple the mandatory self-review reasoning from output so emitting zero agent findings is a valid outcome; add the severity cap. - Six leaf review skills: add the per-domain precision bar and severity cap (style leaf gets domain-appropriate wording). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jesper Schulz-Wedde
|
d994f2ad0e |
Require suggested-code or omission reason for mechanical findings
BCQ now wins on coverage in the synthetic complex benchmark, but only 2 of 25 rendered comments carried GitHub suggestion blocks while the AIRHack reviewer emitted suggestions for every finding. Tighten the DO contract and leaf-skill instructions so suggested-code is no longer a soft affordance for mechanical fixes. Changes: - Add optional findings[].suggested-code-omission-reason to the DO schema. - Document suggested-code as expected for small, local, mechanical findings, with examples (delete unreachable code, Count() > 0 -> not IsEmpty(), move local Label, add missing ToolTip/OptionCaption, replace string-concatenated Error, change permission token, add an obvious else/guard branch). - Require omission reasons when a mechanical-looking finding omits suggested-code. - Update all AL leaf skills and al-code-review guidance with the same stronger contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jesper Schulz-Wedde
|
3a07ee82d2 |
Allow leaf sub-skills to emit agent findings within their domain
The original DO contract pinned all agent reasoning to the super-skill: 'Agent findings are emitted only by super-skills... Leaf sub-skills MUST NOT emit agent findings'. This funnels all agent reasoning across all 6 domains through a single super-skill pass, which is the root structural cause of the attention dilution we have been chasing in BCAppsBCQuality PRs #28 and #30: - T1 standalone run showed al-security-review finds rimd-on-read-only cleanly when run alone, but emits zero agent findings because the contract forbids it. So obvious things like case-without-else (no matching KB article yet) get dropped on the floor. - The al-code-review self-review pass keeps producing 0-1 agent findings per PR because it is asked to reason across 6 domains in one pass. The fix is to move agent reasoning into the leaves, bounded by each leaf's domain. Each leaf now has both knowledge-backed and agent-finding permissions within its own scope; the super-skill self-review pass becomes a smaller, cross-cutting role. skills/do.md - Replace the 'only by super-skills' / 'MUST NOT' clause with a two-tier model: leaf sub-skills MAY emit agent findings strictly within their declared domain; super-skills MAY emit agent findings for cross-cutting concerns that span domains. - Update the encoding rules: leaf agent findings have references:[] and an agent:-prefixed id, no from-sub-skill (the leaf's own report carries the finding under its own skill.id). Super-skills set from-sub-skill='agent' for their own self-review findings; when rolling up leaf agent findings, they set from-sub-skill=<leaf-id>. - Clarify that 'MUST validate against knowledge' applies to super- skill self-review candidates only - leaves already validated within their domain when they decided to emit. microsoft/skills/review/al-{security,performance,privacy,style,upgrade, ui}-review.md - New paragraph after the confidence rules instructing each leaf to surface domain-specific agent findings when no knowledge file covers a defect the agent recognises from general AL knowledge. - Bound the scope: 'The scope is strictly <domain>; defects outside this domain belong to other leaves and MUST NOT be emitted here.' - Same validation requirement: check the worklist for a matching knowledge file first; if one exists, upgrade to a knowledge-backed finding instead. microsoft/skills/review/al-code-review.md - Rewrite the 'Agent self-review pass' subsection. The pass is now explicitly for cross-cutting concerns that no single leaf could have surfaced because they span multiple domains. Domain-specific reasoning belongs in the leaves, not duplicated here. - Update the rollup behavior to acknowledge leaf-emitted agent findings: they are rolled up like any other sub-skill finding, with from-sub-skill set to the leaf id, and are not re-validated by the super-skill (the leaf already validated within its own domain). - Drop the 'Leaf sub-skills MUST NOT emit agent findings' line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jesper Schulz-Wedde
|
31b9949235 |
Strengthen al-code-review execution, propagate suggested-code to leaves, two new KB articles
Driven by a parity comparison between BCAppsBCQuality PR #27 and BCAppsCampAIRHack PR #162 on byte-identical content: | | BCQuality | AIRHack | |--|--|--| | Total findings | 6 | 10 | | Performance | 0 | 4 | | Security | 0 | 5 | Standalone runs of al-security-review and al-performance-review against the SAME diff produced the expected matches (rimd-on-read-only via inherent-permissions-minimal-grant; redundant-Get via avoid-redundant-get-when-record-already-loaded). The miss in the live run is therefore not a knowledge-coverage gap and not a worklist filtering issue. It is attention dilution inside the al-code-review super-skill, which the model collapses into one rolled-up generation pass on real-size PRs. Changes: microsoft/skills/review/al-code-review.md - New 'Execution discipline (mandatory)' subsection in the Action step that explicitly forbids collapsing leaves into one shared reasoning pass and requires each sub-skill to walk its Source -> Relevance -> Worklist -> Action steps as its own iteration before the next leaf starts. - Self-review pass is now described as the final, mandatory iteration with a concrete candidate-category checklist (architecture-level smells, error-handling gaps, magic constants, privacy/telemetry, resource lifecycle). Returning zero agent findings on a real-size diff is explicitly defined as a defect. microsoft/skills/review/al-{security,performance,privacy,style, upgrade,ui}-review.md - Each leaf skill now states that when an unambiguous .good.al companion exists, findings[].suggested-code should carry the literal replacement for the source lines. Closes the one-click-suggestion gap created when BCQ#19 only updated al-code-review. microsoft/knowledge/security/case-must-handle-unknown-enum-values.{md, bad.al, good.al} - New article: case over a security-sensitive enum (Authentication Type, Authorization Mode, Identity Provider, Permission Scope, Encryption Algorithm) MUST have an else arm. Without it, an unknown enum value silently falls through and the security context never initialises. The bad sample is lifted from the SharePoint Graph helper that triggered the parity finding. microsoft/knowledge/performance/instream-length-unreliable-for-bc- streams.{md,bad,good} - New article: InStream.Length returns 0 / partial for HTTP-response streams and some file-API streams, breaking size-threshold branching in upload code. Bad sample is the simple-vs-chunked Graph upload pattern; good sample materialises into a Temp Blob first. Companion change: microsoft/BCAppsBCQuality#28 extends the orchestrator's bootstrap prompt with the same per-iteration execution discipline and adds a CI warning when a >5-file PR returns zero agent findings (regression signal). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jesper Schulz-Wedde
|
287c041844 |
Expand review skills to match the 6-domain knowledge corpus
The knowledge corpus now covers performance, security, privacy, upgrade, style, and UI. Previously only two leaf reviewer skills existed (al-performance-review, al-security-review), so four of the six domains had knowledge with no skill sourcing from them. A community reader landing in privacy/, upgrade/, style/, or ui/ would see articles with no apparent consumer. Three changes: 1. Move existing review skills into `microsoft/skills/review/`. The `review/` subfolder groups all review-kind skills together and leaves room for future non-review action skills at the `microsoft/skills/` level. Updates references in README.md, agent-consumption.md, and skills/entry.md to the new paths. 2. Add four new leaf reviewer skills — al-privacy-review, al-upgrade-review, al-style-review, al-ui-review — each following the same DO template as al-performance-review/al-security-review but sourcing from the corresponding knowledge domain. al-upgrade-review and al-ui-review return `not-applicable` when the diff contains no upgrade surface or no page files, respectively. 3. Update al-code-review to compose all six leaf skills and retarget the dangling references in every populated JSON example (`use-setloadfields.md`, `no-plaintext-secrets-in-telemetry.md`, `avoid-implicit-commit.md` — none of which exist in the corpus) to real knowledge files: `call-setloadfields-before-filters.md`, `use-secrettext-for-credentials.md`, `never-hardcode-secrets-in-al.md`. Validator passes with 0 errors / 0 warnings. |