Microsoft/bcquality
1
0
Fork 0
mirror of https://github.com/microsoft/BCQuality.git synced 2026-10-05 06:36:55 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions

Revert evaluation harness change for multiple articles per domain

Browse source 
The harness intentionally evaluates one paired article per domain. Keep it
as designed; how the retention pairs join privacy evaluation is left to the
maintainers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Jeremy Vyska 2026-09-30 15:49:13 +02:00
parent 07a171386d
commit f9333addcd
3 changed files with 15 additions and 44 deletions
Download patch file Download diff file Expand all files Collapse all files

2
evaluation/README.md
View file

@ -2,7 +2,7 @@
The evaluation is convention-driven. The harness discovers every `<layer>/skills/review/al-<domain>-review.md` leaf across the enabled `microsoft`, `community`, and `custom` layers. Duplicate domains resolve with `custom > community > microsoft` precedence. For each selected leaf, the harness finds paired knowledge across the same layers, applies the same precedence to duplicate article slugs, selects the first article (by filename) with both `.bad.al` and `.good.al` companions, and derives the expected positive and clean control automatically. Adding a conforming leaf requires no scoring-contract edit.
`review-fixtures.json` contains only global thresholds and optional exceptional overrides. An override may select a different article, add context, or list `additionalArticles` whose sample pairs become extra positive and clean cases for that domain, when the generic convention cannot express a scenario. It should remain empty in the normal case.
`review-fixtures.json` contains only global thresholds and optional exceptional overrides. An override may select a different article or add context when the generic convention cannot express a scenario. It should remain empty in the normal case.
Model-facing preparation hashes case IDs, neutralizes `Good`/`Bad` object-name tokens, and removes full-line sample comments so neither the article slug, domain, nor expected outcome reveals the answer.

6
evaluation/review-fixtures.json
View file

@ -23,11 +23,7 @@
"article": "use-isempty-for-existence-check"
},
"privacy": {
"article": "no-pii-in-telemetry-message-string",
"additionalArticles": [
"register-owned-log-tables-for-retention-policies",
"ship-a-default-retention-policy-setup"
]
"article": "no-pii-in-telemetry-message-string"
},
"style": {
"article": "label-comment-explains-placeholders"

51
tools/Test-ReviewFixtures.ps1
View file

@ -4,7 +4,7 @@
.DESCRIPTION
CI uses the static validation path to prove every registered AL review leaf
has at least one positive and one clean control, every fixture/reference exists, and
has one positive and one clean control, every fixture/reference exists, and
the manifest remains internally consistent.
For an actual model run, -PrepareDirectory copies inputs to neutral names and
@ -226,49 +226,24 @@ foreach ($domain in $leafDomains) {
continue
}
# The primary article keeps the domain-level case IDs; additionalArticles add
# further paired cases keyed by slug so existing case hashes stay stable.
$selections = [System.Collections.Generic.List[object]]::new()
$selections.Add([pscustomobject]@{ Article = $selectedArticle; IdPrefix = $domain }) | Out-Null
if ($override -and ($override.PSObject.Properties.Name -contains 'additionalArticles')) {
foreach ($additionalName in @($override.additionalArticles)) {
$additionalName = [string]$additionalName
if ($additionalName.EndsWith('.md')) {
$additionalName = [System.IO.Path]::GetFileNameWithoutExtension($additionalName)
}
if (@($selections | Where-Object { $_.Article.BaseName -eq $additionalName }).Count) {
$problems.Add("${domain}: additional article is already selected: $additionalName.md") | Out-Null
continue
}
$additionalArticle = $articles | Where-Object BaseName -eq $additionalName | Select-Object -First 1
if (-not $additionalArticle) {
$problems.Add("${domain}: additional article does not exist or lacks .good.al and .bad.al companion samples: $additionalName.md") | Out-Null
continue
}
$selections.Add([pscustomobject]@{ Article = $additionalArticle; IdPrefix = "$domain-$additionalName" }) | Out-Null
}
}
$articlePath = [string]$selectedArticle.ArticlePath
$sampleDirectory = (Split-Path -Parent $articlePath).Replace('\', '/')
$context = if ($override -and ($override.PSObject.Properties.Name -contains 'context')) {
[string]$override.context
} else {
$null
}
foreach ($selection in $selections) {
$articlePath = [string]$selection.Article.ArticlePath
$sampleDirectory = (Split-Path -Parent $articlePath).Replace('\', '/')
foreach ($kind in 'bad', 'good') {
$case = [pscustomobject]@{
id = "$($selection.IdPrefix)-$kind"
domain = $domain
input = "$sampleDirectory/$($selection.Article.BaseName).$kind.al"
expected = if ($kind -eq 'bad') { @($articlePath) } else { @() }
}
if ($context) {
$case | Add-Member -NotePropertyName context -NotePropertyValue $context
}
$caseList.Add($case) | Out-Null
foreach ($kind in 'bad', 'good') {
$case = [pscustomobject]@{
id = "$domain-$kind"
domain = $domain
input = "$sampleDirectory/$($selectedArticle.BaseName).$kind.al"
expected = if ($kind -eq 'bad') { @($articlePath) } else { @() }
}
if ($context) {
$case | Add-Member -NotePropertyName context -NotePropertyValue $context
}
$caseList.Add($case) | Out-Null
}
}
$cases = @($caseList)