mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-10-05 22:56:55 +01:00
Strengthen review contracts and add AL reliability guidance (#196)
* Strengthen review contracts and HTTP guidance - add outbound HttpClient transport and HTTP status review rules with paired fixtures`n- resolve layered action-skill overrides deterministically across enabled layers`n- validate findings reports and enforce measurable changed-fixture coverage * Add data handling and test isolation guidance - add SCM guidance for deriving base quantities through line unit-of-measure validation`n- add security guidance for parameterizing SetFilter with external text`n- add test isolation guidance for resetting per-test state before initialization guards`n- add web-service guidance for JSON null handling and invariant standard format 9`n- route and cover all five rules with paired evaluation fixtures * Fix findings report rollup validation * Validate findings report rollups * Enforce merged finding identity * Fix locationless finding deduplication * Reject conflicting merged corrections * Detect conflicting leaf corrections * Route HTTP error checks to canonical web-services knowledge Let the Error Handling leaf conditionally retrieve the existing HTTP owner articles, preserving applicability and exact-path provenance. Add deterministic source-contract and retrieval regressions without duplicating knowledge rules. Copilot-Session-Id: a92a7788-103e-4651-9b84-19e34caffb94 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: wenjiefan <wenjiefan@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Jesper Schulz-Wedde <jesper.schulzwedde@microsoft.com>
This commit is contained in:
parent
130d5de6c4
commit
4287233f80
41 changed files with 1974 additions and 38 deletions
|
|
@ -4,6 +4,15 @@ The evaluation is convention-driven. The harness discovers every `<layer>/skills
|
|||
|
||||
`review-fixtures.json` contains only global thresholds and optional exceptional overrides. An override may select a different `article`, add context when the generic convention cannot express a scenario, or use an `articles` array when one domain needs explicit regression coverage for several paired articles. Specify either `article` or `articles`, not both. The first selected article retains the stable `<domain>-bad` and `<domain>-good` manifest IDs; additional articles use slug-qualified IDs. Overrides should remain empty in the normal case.
|
||||
|
||||
CI also measures selected paired articles against every effective article that
|
||||
has both AL companions. A changed paired article must be selected by the
|
||||
domain convention or an override. When adding it would not provide a useful
|
||||
deterministic regression, add a narrow `coverageWaivers` entry with its exact
|
||||
article path and a non-empty reason. Waivers are reviewable exceptions, not a
|
||||
substitute for domain coverage. The generated coverage report includes totals
|
||||
and per-domain ratios; the ratio is informational, while changed-file coverage
|
||||
is mandatory.
|
||||
|
||||
Model-facing preparation hashes case IDs, neutralizes `Good`/`Bad` object-name tokens, and removes full-line sample comments so neither the article slug, domain, nor expected outcome reveals the answer.
|
||||
|
||||
The SCM `articles` override deliberately selects every rule in the initial
|
||||
|
|
@ -36,6 +45,13 @@ pwsh ./tools/Test-ReviewFixtures.ps1 -Root .
|
|||
|
||||
This credential-free check proves every selected leaf maps to a same-named knowledge domain with at least one complete AL sample pair and that all configured overrides are valid.
|
||||
|
||||
To reproduce the changed-file gate and emit the same measurable report as CI:
|
||||
|
||||
```powershell
|
||||
git diff --name-only origin/main...HEAD | Set-Content .changed-paths.txt
|
||||
pwsh ./tools/Test-ReviewFixtures.ps1 -Root . -ChangedPathsFile .changed-paths.txt -CoverageReportPath .coverage.json
|
||||
```
|
||||
|
||||
## Run a fast-model evaluation
|
||||
|
||||
1. Prepare neutral inputs:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue