Strengthen review contracts and add AL reliability guidance (#196)

* Strengthen review contracts and HTTP guidance

- add outbound HttpClient transport and HTTP status review rules with paired fixtures`n- resolve layered action-skill overrides deterministically across enabled layers`n- validate findings reports and enforce measurable changed-fixture coverage

* Add data handling and test isolation guidance

- add SCM guidance for deriving base quantities through line unit-of-measure validation`n- add security guidance for parameterizing SetFilter with external text`n- add test isolation guidance for resetting per-test state before initialization guards`n- add web-service guidance for JSON null handling and invariant standard format 9`n- route and cover all five rules with paired evaluation fixtures

* Fix findings report rollup validation

* Validate findings report rollups

* Enforce merged finding identity

* Fix locationless finding deduplication

* Reject conflicting merged corrections

* Detect conflicting leaf corrections

* Route HTTP error checks to canonical web-services knowledge

Let the Error Handling leaf conditionally retrieve the existing HTTP owner articles, preserving applicability and exact-path provenance. Add deterministic source-contract and retrieval regressions without duplicating knowledge rules.

Copilot-Session-Id: a92a7788-103e-4651-9b84-19e34caffb94

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: wenjiefan <wenjiefan@microsoft.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Jesper Schulz-Wedde <jesper.schulzwedde@microsoft.com>
This commit is contained in:
Stefano Demiliani 2026-09-29 13:03:39 +02:00 • committed by GitHub
parent 130d5de6c4
commit 4287233f80
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
41 changed files with 1974 additions and 38 deletions

View file

@ -4,6 +4,15 @@ The evaluation is convention-driven. The harness discovers every `<layer>/skills
`review-fixtures.json` contains only global thresholds and optional exceptional overrides. An override may select a different `article`, add context when the generic convention cannot express a scenario, or use an `articles` array when one domain needs explicit regression coverage for several paired articles. Specify either `article` or `articles`, not both. The first selected article retains the stable `<domain>-bad` and `<domain>-good` manifest IDs; additional articles use slug-qualified IDs. Overrides should remain empty in the normal case.
CI also measures selected paired articles against every effective article that
has both AL companions. A changed paired article must be selected by the
domain convention or an override. When adding it would not provide a useful
deterministic regression, add a narrow `coverageWaivers` entry with its exact
article path and a non-empty reason. Waivers are reviewable exceptions, not a
substitute for domain coverage. The generated coverage report includes totals
and per-domain ratios; the ratio is informational, while changed-file coverage
is mandatory.
Model-facing preparation hashes case IDs, neutralizes `Good`/`Bad` object-name tokens, and removes full-line sample comments so neither the article slug, domain, nor expected outcome reveals the answer.
The SCM `articles` override deliberately selects every rule in the initial
@ -36,6 +45,13 @@ pwsh ./tools/Test-ReviewFixtures.ps1 -Root .
This credential-free check proves every selected leaf maps to a same-named knowledge domain with at least one complete AL sample pair and that all configured overrides are valid.
To reproduce the changed-file gate and emit the same measurable report as CI:
```powershell
git diff --name-only origin/main...HEAD | Set-Content .changed-paths.txt
pwsh ./tools/Test-ReviewFixtures.ps1 -Root . -ChangedPathsFile .changed-paths.txt -CoverageReportPath .coverage.json
```
## Run a fast-model evaluation
1. Prepare neutral inputs:

View file

@ -3,6 +3,7 @@
"selection": "first-paired-al-article",
"minimumExpectedRecall": 1.0,
"minimumCleanRate": 1.0,
"coverageWaivers": [],
"overrides": {
"agents": {
"article": "wire-all-three-agent-interfaces"
@ -75,7 +76,14 @@
"reconcile-warehouse-adjustments-with-the-item-ledger",
"post-transfers-through-shipment-and-receipt-codeunits",
"use-date-aware-availability-for-promising",
"carry-out-requisition-actions-through-the-standard-workflow"
"carry-out-requisition-actions-through-the-standard-workflow",
"derive-base-quantities-through-the-line-unit-of-measure"
]
},
"security": {
"articles": [
"al-has-no-built-in-htmlencode",
"do-not-concatenate-external-text-into-setfilter"
]
},
"style": {
@ -88,14 +96,23 @@
"article": "telemetry-event-id-stable-unique"
},
"testing": {
"article": "ui-handlers-in-tests"
"articles": [
"ui-handlers-in-tests",
"reset-per-test-state-before-the-isinitialized-guard"
]
},
"upgrade": {
"article": "initvalue-does-not-update-existing-rows",
"context": "The extended table existed in the previous app version and already contains rows."
},
"web-services": {
"article": "expose-systemid-as-the-api-key"
"articles": [
"expose-systemid-as-the-api-key",
"handle-httpclient-platform-failure-before-response-access",
"check-http-status-before-consuming-response-body",
"check-json-null-before-converting-values",
"format-exchanged-values-with-standard-format-9"
]
}
}
}