mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-08-08 02:16:52 +01:00
Complete AL review knowledge readiness (#108)
* Complete AL review knowledge readiness Fill telemetry and Query coverage, strengthen thin review domains, correct audited content defects, and add deterministic cheap-model evaluation and reference-integrity safeguards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9825b012-e653-496a-9310-c1f4b6f8ac27 * Generalize review fixture discovery Derive smoke cases from the leaf, domain, and paired-sample conventions so new leaves require no scoring-contract changes. Keep only exceptional selection/context overrides and fail when retrieval metadata cannot rank the selected article. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9825b012-e653-496a-9310-c1f4b6f8ac27 * Preserve published field IDs in sample Keep the existing Email and Contact Email field IDs unchanged, clarify that the sample represents an independent baseline, and use a local breaking-change rule for the generic smoke evaluation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9825b012-e653-496a-9310-c1f4b6f8ac27 * Clarify published field identity rules State explicitly that a published field keeps its ID, name, and type while a replacement is added as a separate field under an unused ID. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9825b012-e653-496a-9310-c1f4b6f8ac27 * Align field obsoletion sample baselines Use Email field ID 3 as the shared baseline so the bad example demonstrates a same-ID rename while the good example retains the original field and adds a separate replacement. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9825b012-e653-496a-9310-c1f4b6f8ac27 --------- Co-authored-by: Jesper Schulz-Wedde <jesper.schulzwedde@microsoft.com>
This commit is contained in:
parent
ae04938c03
commit
186d8a1314
105 changed files with 2229 additions and 212 deletions
11
skills/do.md
11
skills/do.md
|
|
@ -171,6 +171,8 @@ Consumers that render output MAY treat agent findings differently from knowledge
|
|||
|
||||
**`findings[].message`** — human-readable explanation of the finding. Single short paragraph. No markdown formatting assumptions.
|
||||
|
||||
**Applicability is not a finding.** Loading an article into the worklist only means its rule must be evaluated. If the changed code does not violate the article's normative guidance, emit nothing for that article. An `info` finding still requires a concrete observation defined by the article; skills MUST NOT use `info` to list guidance that merely happened to be relevant.
|
||||
|
||||
**`findings[].location`** — optional. When present:
|
||||
|
||||
- `file` MUST be a repo-relative path using forward slashes.
|
||||
|
|
@ -186,6 +188,15 @@ Findings without a `location` are permitted (for example, repository-wide observ
|
|||
|
||||
The first reference is the **primary** reference: the knowledge file the finding most directly cites. Additional references provide supporting context and are not ranked. `references` MAY be empty only for **agent findings** (see the `findings[].id` section above for the full encoding); any other finding MUST have at least one reference.
|
||||
|
||||
**Reference-integrity gate (mandatory).** A knowledge-backed finding may cite only a path copied verbatim from the current knowledge index or from a file discovered by the index fallback, and the skill must have opened that exact file in full before citing it. Never construct a plausible slug or infer a path from a topic name. Immediately before emitting the JSON document:
|
||||
|
||||
1. Verify every non-empty `references[].path` exists in the live checkout and was opened during this skill run.
|
||||
2. Verify every citation-based `findings[].id` exactly equals `references[0].path`.
|
||||
3. Remove any candidate that cannot satisfy both checks; it is not a knowledge-backed finding. Do not convert it into an agent finding merely to preserve it.
|
||||
4. If reference integrity cannot be checked reliably, return `outcome: "failed"` rather than emitting fabricated or unverified citations.
|
||||
|
||||
This gate applies independently to every leaf result and again to a super-skill's rolled-up result.
|
||||
|
||||
**`findings[].confidence`** — the skill's confidence that the finding is a true positive, given the evidence it evaluated. Not applicability confidence, not severity confidence. Values: `high`, `medium`, `low`.
|
||||
|
||||
**`findings[].from-sub-skill`** — optional. Set only by super-skills. The `skill.id` of the sub-skill that produced the finding, or the literal string `"agent"` for an agent finding the super-skill produced from its own cross-cutting reasoning. Absent on findings emitted directly by a leaf skill — including agent findings the leaf emits within its own domain, which appear in the leaf's own report without this field.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue