Generalize review fixture discovery

Derive smoke cases from the leaf, domain, and paired-sample conventions so new leaves require no scoring-contract changes. Keep only exceptional selection/context overrides and fail when retrieval metadata cannot rank the selected article.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 9825b012-e653-496a-9310-c1f4b6f8ac27
This commit is contained in:
Jesper Schulz-Wedde 2026-07-15 09:11:15 +02:00
parent e81632b4be
commit f76279057e
4 changed files with 115 additions and 241 deletions

View file

@ -1,6 +1,10 @@
# AL review evaluation
`review-fixtures.json` contains one positive and one clean control for every AL review leaf. Inputs are existing companion samples, but model-facing preparation hashes case IDs, neutralizes `Good`/`Bad` object-name tokens, and removes full-line sample comments so neither the article slug, domain, nor expected outcome reveals the answer.
The evaluation is convention-driven. For every `microsoft/skills/review/al-<domain>-review.md` leaf, the harness finds `microsoft/knowledge/<domain>/`, selects the first article (by filename) with both `.bad.al` and `.good.al` companions, and derives the expected positive and clean control automatically. Adding a conforming leaf requires no scoring-contract edit.
`review-fixtures.json` contains only global thresholds and optional exceptional overrides. An override may select a different article or add context when the generic convention cannot express a scenario. It should remain empty in the normal case.
Model-facing preparation hashes case IDs, neutralizes `Good`/`Bad` object-name tokens, and removes full-line sample comments so neither the article slug, domain, nor expected outcome reveals the answer.
## Validate the corpus
@ -8,7 +12,7 @@
pwsh ./tools/Test-ReviewFixtures.ps1 -Root .
```
This is credential-free and runs in CI. It validates fixture paths, expected article references, unique IDs, and positive/clean coverage for every registered leaf.
This credential-free check proves every registered leaf maps to a same-named knowledge domain with at least one complete AL sample pair and that all configured overrides are valid.
## Run a fast-model evaluation
@ -18,6 +22,8 @@ This is credential-free and runs in CI. It validates fixture paths, expected art
pwsh ./tools/Test-ReviewFixtures.ps1 -Root . -PrepareDirectory ./.evaluation-run
```
This is also the CI path. It derives all cases, builds the current index, requires the convention-selected article to rank naturally into the candidate cutoff, and prepares the neutral requests.
2. For a fast/small model, use one fresh invocation per `request-case-*.json`. Each request embeds the exact leaf instructions, that domain's candidate index rows with authoritative paths, and one opaque case. The model opens only matching articles and copies finding IDs from `candidateArticles[].path`. Save each response with the matching `result-case-*.json` name in the same directory.
`request-<domain>.json` files provide optional two-case leaf batches; save those as `result-<domain>.json`. Directory scoring prefers `result-case-*.json` when present and otherwise falls back to `result-*.json`. `review-request.json` is an optional all-domains stress test for larger models. Neither batch form is the preferred fast-model profile.
@ -47,4 +53,4 @@ This is credential-free and runs in CI. It validates fixture paths, expected art
For a single combined stress-test result, use `-ResultsPath` instead.
The committed gate requires full expected recall and no findings on clean controls. `allowedAdditional` in the manifest records known, independently valid overlaps without weakening the required primary finding.
The committed gate requires full expected recall, the exact convention-derived article ID, and no findings on clean controls.