This commit is contained in:
Wenjie Fan 2026-10-05 15:15:50 +02:00 • committed by GitHub
commit d28eb956e1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 400 additions and 45 deletions

View file

@ -168,8 +168,8 @@ pwsh .\tools\Test-ReviewContract.ps1 -Root .
The first command checks schema, sections, naming, sample references, and
skill registration. The second checks that every review leaf has a valid
positive/clean sample pair. The third checks the cross-surface findings-report
contract and its bounded range-normalization cases. None proves a model will
find every defect. See [evaluation](../evaluation/README.md) for optional
contract and its bounded citation-ID/range-normalization cases. None proves a
model will find every defect. See [evaluation](../evaluation/README.md) for optional
model-based scoring.
In the PR description, explain the mistake being prevented, supporting

View file

@ -59,9 +59,11 @@ only result.
5. Capture the exact Task return as the immutable raw audit payload and primary
transport. Preserve it unchanged in private artifacts or host logs. Before
the full DO acceptance gate, create a normalized candidate only for DO's
bounded optional-range case, record that normalization separately in private
telemetry, and accept the candidate only if the entire copy passes the
unchanged strict gate. Use `tools/Validate-FindingsReport.ps1`, passing the
bounded citation-ID and optional-range cases after strict JSON, full
structural schema, and reference-integrity checks. Record `normalizedIds`
and `removedRanges` separately in private telemetry, and accept the
candidate only if the entire copy passes the unchanged strict gate.
Use `tools/Validate-FindingsReport.ps1`, passing the
exact source paths and fully retrieved article paths; pass `-SkillKind super`
and `-ExpectedCompositionPath` for the final rolled-up report. The accepted report contains no undeclared
telemetry fields.

View file

@ -95,7 +95,7 @@ The Action step consists of **discrete leaf invocations**, not one combined gene
- **Isolate leaf invocations when the host supports it.** Each sub-skill SHOULD run in a fresh model call or child context containing only its assigned source paths, READ/DO contracts, the leaf instructions, the complete bounded domain catalog per READ, and articles that leaf worklists. Preserve each catalog row's exact `path`; the leaf must copy references from that catalog.
- **Keep run artifacts private.** Before dispatch, allocate a new GUID-named directory under the current session's artifact directory and a distinct scratch/report child directory for every leaf. Pass a leaf only its own assigned source paths and child directory, never the run root or sibling paths. A leaf MUST NOT discover, enumerate, read, modify, or delete sibling artifacts. Do not reuse a prior run directory, and do not clean up any run artifact until every leaf has finished and consolidation is complete.
- **Keep raw Task transport distinct from the accepted copy.** Capture the exact Task return as the immutable raw audit payload and primary transport. Preserve it unchanged in the leaf's private artifacts or host log. Then apply DO's bounded pre-gate range normalization, when eligible, and its full consumer acceptance gate. The report accepted for rollup is the exact return when no normalization occurred, or the normalized candidate copy when DO permits it; worker-side persistence of another report file is optional and redundant.
- **Keep raw Task transport distinct from the accepted copy.** Capture the exact Task return as the immutable raw audit payload and primary transport. Preserve it unchanged in the leaf's private artifacts or host log. Then apply DO's bounded citation-ID and optional-range normalization, when eligible, and its full consumer acceptance gate. The report accepted for rollup is the exact return when no normalization occurred, or the normalized candidate copy when DO permits it; worker-side persistence of another report file is optional and redundant.
- **Treat automatic output spills as host-owned.** If the host reports that a Task return was automatically spilled, the coordinator MAY read that file read-only only at the exact path returned by the tool. Never modify, delete, enumerate around, or reuse an automatic spill path. Never bypass a content-exclusion or access denial.
- Treat each sub-skill in the worklist as its own pass: read the sub-skill's instructions, apply its Source → Relevance → Worklist → Action steps to the orchestrator-supplied inputs, and produce that sub-skill's complete findings-report independently.
- Do not collapse multiple sub-skills into one shared reasoning step. Each sub-skill has a distinct knowledge subset and a distinct evaluation procedure; sharing one rolled-up scan dilutes per-skill attention and causes leaves to silently underreport (this has been observed in production: leaf skills returned empty `findings[]` while their standalone runs against the same diff produced multiple matches).
@ -108,7 +108,7 @@ The Action step consists of **discrete leaf invocations**, not one combined gene
For each sub-skill in the worklist:
1. Invoke the sub-skill with the orchestrator's inputs, passing only the subset each sub-skill declares in its `inputs`.
2. Capture the exact Task return as the immutable raw audit payload and primary transport. Preserve it unchanged in the leaf's private artifacts or host log before deriving a candidate. Apply only DO's bounded pre-gate normalization: when the complete raw report has no other defect, a finding has positive-integer `line`, `start-line`, and `end-line`, `start-line <= line <= end-line`, `start-line != line`, and no `suggested-code` field, copy the complete report and remove only that finding's optional `location.range`. Record the normalization separately in private run telemetry or artifacts, never in the findings-report. Validate the entire candidate through DO's existing strict acceptance gate. Accept the exact return when unchanged or the normalized candidate when it passes; otherwise record a separate failed validation result with no findings for rollup. Do not reconstruct JSON, infer fields, alter paths or references, clamp lines, normalize reversed or out-of-bounds ranges, remove a range associated with `suggested-code`, or salvage individual findings.
2. Capture the exact Task return as the immutable raw audit payload and primary transport. Preserve it unchanged in the leaf's private artifacts or host log before deriving a candidate. Apply only DO's bounded citation-ID and optional-range normalization procedure: strict JSON and the complete structural schema first, all citation safe-path/existence/full-body retrieval checks, then a deep copy with only exact primary-reference ID replacement and eligible range removal. Preserve agent IDs, all original source-bound checks, and immutable accepted nested reports. Record changed IDs and removed ranges privately by finding index, never in the findings-report. Validate the entire candidate through DO's full schema/semantic acceptance gate. Accept the exact return when unchanged or the normalized candidate when it passes; otherwise record a separate failed validation result with no findings for rollup. Do not reconstruct JSON, infer fields, alter paths or references, clamp lines, normalize reversed or out-of-bounds ranges, remove a range associated with `suggested-code`, or salvage individual findings.
3. Append the accepted findings-report, or the separate failed validation result, to `sub-results`. If its `outcome` is `failed`, stop here for this sub-skill: its findings are not reliable per the DO contract and MUST NOT be copied into the super-skill's top-level `findings[]` or counted in `summary.counts`.
4. Otherwise, compare each entry from the sub-skill's `findings[]` with findings already rolled up. Two findings are duplicates when they point to the same file and overlapping line/range and prescribe materially the same correction, even when their knowledge-file IDs differ. Merge duplicates instead of appending both: keep the more specific domain owner, preserve that finding's optional `domain` field verbatim (including its absence), use its reference as `references[0]` and therefore as `id`, append the other references as supporting references, keep the highest severity and confidence justified by either report, and preserve one self-contained message. Article and leaf ownership notes decide specificity; do not choose by execution order.
5. Append each non-duplicate finding, setting `from-sub-skill` to the sub-skill's `skill.id` and preserving its optional `domain` field verbatim, including its absence. For non-citation findings (those whose `id` is a skill-defined slug rather than a reference path), prefix `id` with `<from-sub-skill>:` to prevent collisions across sub-skills. Other finding fields are preserved.

View file

@ -122,7 +122,7 @@
"additionalProperties": false,
"required": ["id", "severity", "message", "references", "confidence"],
"properties": {
"id": { "type": "string", "minLength": 1, "pattern": "^[^#]+$" },
"id": { "type": "string", "minLength": 1 },
"severity": { "enum": ["blocker", "major", "minor", "info"] },
"message": { "type": "string", "minLength": 1 },
"location": { "$ref": "#/definitions/location" },
@ -143,6 +143,7 @@
},
"then": {
"properties": {
"id": { "pattern": "^(?:[a-z0-9]+(?:-[a-z0-9]+)*:)?agent:[a-z0-9]+(?:-[a-z0-9]+)*$" },
"confidence": { "enum": ["medium", "low"] },
"severity": { "enum": ["minor", "info"] }
}

View file

@ -182,8 +182,10 @@ Before emitting each leaf report or super-skill rollup:
`outcome-reason`, not unverified locations.
Validate the complete document against the schema and semantic rules before
returning it. Consumers MUST NOT strip ID suffixes, downgrade agent findings,
or clamp locations to make an invalid report pass the acceptance gate.
returning it. Consumers MUST NOT heuristically strip ID suffixes, downgrade
agent findings, or clamp locations to make an invalid report pass the acceptance gate. The
only ID exception is the exact primary-reference copy in the bounded consumer
procedure below; producers still MUST emit canonical IDs.
### Consumer acceptance gate
@ -193,36 +195,62 @@ creating any derived value. The accepted findings-report is either that exact
return or the bounded normalized candidate described below; the raw audit
payload never changes.
Before the full acceptance gate, a coordinator MAY create a normalized
Before the full acceptance gate, a coordinator or host MAY create a normalized
candidate copy only through this deterministic procedure:
1. Parse the exact return as strict JSON and provisionally check the complete
report without mutating it. Every acceptance rule below MUST already pass
except for one or more findings whose optional `location.range` has
`start-line != line`.
2. Each such finding is eligible only when `location.line`,
`location.range.start-line`, and `location.range.end-line` are positive
integers, `start-line <= line <= end-line`, and the finding does not contain
the `suggested-code` field. Field presence disqualifies normalization even
if its value is empty because suggested code may be bound to the reported
range.
3. Deep-copy the complete parsed report. In the candidate copy, remove only
`location.range` from every eligible finding. Retain `location.line` and
every other value unchanged. Do not add normalization metadata to the
findings-report.
4. Record each removed range separately in private run telemetry or artifacts,
associated with the immutable raw audit payload. This record is
runner-owned and is not part of the declared report schema.
1. Validate the exact return as strict JSON and against the complete structural
schema before forming a candidate. All required fields, types, enums,
conditional requirements, non-empty strings, positive-integer locations,
range shape, and additional-property restrictions MUST pass, including
nested reports. Do not repair JSON or coerce values. The structural schema
deliberately requires only a non-empty string for a cited ID: primary-path
equality (including rejecting fragments or scenario suffixes) is semantic,
not a global no-`#` pattern that would prevent this bounded procedure.
For `references: []`, the schema still enforces agent slug syntax and
severity/confidence caps; exact role/producer ownership is semantic.
2. Verify every cited path through all existing safe-path, live-snapshot
existence, and recorded full-body retrieval checks, including every
supporting citation. Reject absolute paths, backslashes, dot segments,
empty segments, URI escapes, fragments, query strings, and control
characters; do not resolve or rewrite them into valid paths. Unknown or
unretrieved references disqualify the complete report. An ID with an
`(^|:)agent:` marker or `from-sub-skill: "agent"` is an agent encoding,
never eligible for ID canonicalization; combining it with citations is
invalid.
3. Deep-copy the complete parsed report. Only for a knowledge-backed finding
with non-empty, fully verified `references`, set the candidate `id` exactly
to `references[0].path` when it differs (ordinal comparison). Never derive
an ID by trimming or parsing the raw ID. Already-canonical IDs are no-ops.
Provisionally validate the entire candidate, permitting only optional
`location.range` start mismatches. All original range endpoints MUST still
be checked against the source snapshot before removing any range.
4. In the same candidate copy, remove only an eligible `location.range`.
Eligibility requires `location.line`, `location.range.start-line`, and
`location.range.end-line` to be positive integers,
`start-line <= line <= end-line`, `start-line != line`, and the finding
does not contain the `suggested-code` field. Field presence disqualifies
range removal even if its value is empty. Valid uncited agent findings
remain eligible for this range-only operation; their IDs and caps never
change. Both operations apply only to the report's own `findings[]`:
accepted nested leaf reports are immutable, not re-normalized by rollup.
Record each changed ID's zero-based finding index, original ID, and
canonical ID, and each removed range with its finding index and original
endpoints, separately in private run telemetry or artifacts associated
with the exact raw payload. Do not add metadata to the findings-report.
5. Validate the entire normalized candidate with the existing full consumer
acceptance gate below. Only a candidate that passes every rule becomes the
accepted copy used for rollup. If any other validation defect exists, or
full validation fails, discard the candidate, preserve the raw payload, and
fail the complete leaf as before.
acceptance gate below, including the complete schema, reference equality,
role, source, counts, coverage, and composition checks without exceptions.
Only a candidate that passes every rule becomes the accepted copy used for
rollup. If any other validation defect exists, or full validation fails,
discard the candidate, preserve the raw payload, and fail the complete
report as before. Never accept or return a partially normalized subset.
This exception does not infer missing fields, alter references or paths, clamp
line numbers, repair JSON, normalize a reversed or out-of-bounds range, remove
a range from a finding containing `suggested-code`, or salvage arbitrary
individual findings.
It never canonicalizes agent/uncited IDs or changes messages, reference paths
or order, `location.line`, severity, confidence, or any other field.
Before accepting either the exact return or an eligible normalized candidate
as a findings-report, a coordinator or host MUST validate it deterministically:
@ -275,7 +303,8 @@ returned-and-skipped leaf IDs are invalid even without the artifact. Never
derive the expected composition from model output.
Pass
`-AllowBoundedNormalization` only when the host preserves the immutable raw
payload and records `removedRanges` in private telemetry as required above.
payload and records `normalizedIds` and `removedRanges` in private telemetry as
required above. These fields are returned beside `report`, never inside it.
Validation failure invalidates the complete return; consumers MUST NOT salvage
individual findings, infer missing fields, reconstruct JSON, clamp ranges,

View file

@ -128,7 +128,13 @@ foreach ($expected in @(
'For `references: []`, emit only `confidence: "medium"` or `"low"` and `severity: "minor"` or `"info"`.',
'Open the final source snapshot for every `location.file`.',
'1-based final-file line numbers within that file''s length, never diff/patch-relative line numbers.',
'Consumers MUST NOT strip ID suffixes, downgrade agent findings, or clamp locations',
'Consumers MUST NOT heuristically strip ID suffixes, downgrade agent findings, or clamp locations',
'complete structural schema before forming a candidate',
'set the candidate `id` exactly to `references[0].path`',
'Already-canonical IDs are no-ops.',
'Valid uncited agent findings remain eligible for this range-only operation',
'accepted nested leaf reports are immutable',
'zero-based finding index, original ID, and canonical ID',
'positive integers',
'start-line <= line <= end-line',
'does not contain the `suggested-code` field',
@ -356,7 +362,34 @@ try {
$fragmentReport.findings[0]
}
$fragmentFinding.id = "$articlePath#location"
Assert-ReportSchema $fragmentReport $false "$position citation id cannot append a fragment"
Assert-ReportSchema $fragmentReport $true "$position cited fragment defers equality to the semantic gate"
Set-Content -LiteralPath $reportPath -Value ($fragmentReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$kind = if ($position -eq 'leaf') { 'leaf' } else { 'super' }
Assert-ThrowsLike -Pattern '*PRIMARY_REFERENCE_MISMATCH*' -Action {
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SourceRoot $tmp -SkillKind $kind `
-SourcePaths $sourcePath -RetrievedArticlePaths $articlePath
}
if ($position -eq 'nested-leaf') {
Assert-ThrowsLike -Pattern '*PRIMARY_REFERENCE_MISMATCH*' -Action {
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SourceRoot $tmp -SkillKind super `
-SourcePaths $sourcePath -RetrievedArticlePaths $articlePath -AllowBoundedNormalization
}
}
else {
$acceptedFragment = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SourceRoot $tmp -SkillKind $kind `
-SourcePaths $sourcePath -RetrievedArticlePaths $articlePath -AllowBoundedNormalization
Assert-True ($acceptedFragment.normalizedIds.Count -eq 1) "$position cited fragment is replaced only in the candidate"
Assert-True ($acceptedFragment.report.findings[0].id -ceq $articlePath) 'canonical id is the exact primary path'
}
}
$citedRootAgent = $citationSuper | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$citedRootAgent.findings[0].'from-sub-skill' = 'agent'
$citedRootAgent.findings[0].id = 'raw-cited-scenario'
Set-Content -LiteralPath $reportPath -Value ($citedRootAgent | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-ThrowsLike -Pattern '*AGENT_REFERENCE_INVALID*' -Action {
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SourceRoot $tmp -SkillKind super `
-SourcePaths $sourcePath -RetrievedArticlePaths $articlePath -AllowBoundedNormalization
}
$duplicateLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
@ -983,8 +1016,13 @@ try {
Set-Content -LiteralPath $reportPath -Value ($mismatchedCitation | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-ThrowsLike -Pattern '*PRIMARY_REFERENCE_MISMATCH*' -Action {
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SourceRoot $tmp `
-SourcePaths $sourcePath -RetrievedArticlePaths $articlePath -AllowBoundedNormalization
-SourcePaths $sourcePath -RetrievedArticlePaths $articlePath
}
$acceptedCitation = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SourceRoot $tmp `
-SourcePaths $sourcePath -RetrievedArticlePaths $articlePath -AllowBoundedNormalization
Assert-True ($acceptedCitation.normalizedIds.Count -eq 1 -and $acceptedCitation.removedRanges.Count -eq 0) `
'ID-only normalization preserves an aligned range'
Assert-True ($acceptedCitation.report.findings[0].id -ceq $articlePath) 'ID-only candidate uses the primary reference'
$finalSourcePath = 'src/final-snapshot.al'
Set-Content -LiteralPath (Join-Path $tmp $finalSourcePath) -Value (1..22 | ForEach-Object { "line $_" }) -Encoding utf8NoBOM
@ -1031,6 +1069,259 @@ try {
Assert-True ($normalized.removedRanges.Count -eq 1) 'normalization records one removed range'
Assert-True (-not ($normalized.report.findings[0].location.PSObject.Properties.Name -contains 'range')) `
'accepted normalized report removes only the optional range'
# Minimal finding fixtures copied verbatim in value from smoke 37310924454 / synthetic__privacy-015.
# Source leaf SHA256: 26aead0958e6ffe60c947f740b95ecf016783116a88d1254e87cea5c54c10107.
# Final handoff SHA256: c313a4ad40d270f4f77821ac36e375baaf107b8944fb8673e1d27d42c1e8b054.
$privacyFindings = @'
[
{
"id": "privacy-notice-consent-for-external-data-transfer-ai-context",
"severity": "major",
"message": "The AI service request sends task and user context to an external service without checking approval for a dedicated privacy notice. No path should issue an external data request without integration-specific approval.",
"location": {"file": "src/AIContextBuilder.Codeunit.al", "line": 25, "range": {"start-line": 23, "end-line": 25}},
"references": [{"path": "microsoft/knowledge/privacy/privacy-notice-consent-for-external-data-transfer.md"}],
"confidence": "high",
"domain": "Privacy",
"suggested-code-omission-reason": "The fix requires adding and registering a dedicated notice identifier and placing the approval check in the appropriate transaction context."
},
{
"id": "privacy-notice-consent-for-external-data-transfer-customer-export",
"severity": "major",
"message": "The customer exporter posts names, email addresses, phone numbers, and addresses to a partner without checking approval for a dedicated privacy notice. Consent for another service does not authorize this external transfer.",
"location": {"file": "src/CustomerDataExporter.Codeunit.al", "line": 24, "range": {"start-line": 20, "end-line": 24}},
"references": [{"path": "microsoft/knowledge/privacy/privacy-notice-consent-for-external-data-transfer.md"}],
"confidence": "high",
"domain": "Privacy",
"suggested-code-omission-reason": "The fix requires adding and registering a dedicated notice identifier and placing the approval check in the appropriate transaction context."
},
{
"id": "privacy-notice-consent-for-external-data-transfer-crm-sync",
"severity": "major",
"message": "The CRM sync posts customer email addresses, names, phone numbers, and addresses to an external service without checking approval for a dedicated privacy notice. No path should issue the request without approval.",
"location": {"file": "src/ExternalCRMSync.Codeunit.al", "line": 23, "range": {"start-line": 19, "end-line": 23}},
"references": [{"path": "microsoft/knowledge/privacy/privacy-notice-consent-for-external-data-transfer.md"}],
"confidence": "high",
"domain": "Privacy",
"suggested-code-omission-reason": "The fix requires adding and registering a dedicated notice identifier and placing the approval check in the appropriate transaction context."
},
{
"id": "privacy-notice-consent-for-external-data-transfer-email",
"severity": "major",
"message": "The email dispatcher sends recipient addresses, subjects, and message bodies to Microsoft Graph without an approval check for the integration's privacy notice. No external data request should proceed without approval.",
"location": {"file": "src/OutboxEmailDispatcher.Codeunit.al", "line": 23, "range": {"start-line": 18, "end-line": 23}},
"references": [{"path": "microsoft/knowledge/privacy/privacy-notice-consent-for-external-data-transfer.md"}],
"confidence": "high",
"domain": "Privacy",
"suggested-code-omission-reason": "The fix requires determining the integration notice and placing its approval check in the appropriate transaction context before posting."
}
]
'@ | ConvertFrom-Json -DateKind String
$privacyReport = $validReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json -DateKind String
$privacyReport.skill.id = 'al-privacy-review'
$privacyReport.findings = $privacyFindings
$privacyReport.summary.counts.minor = 0
$privacyReport.summary.counts.major = 4
$privacyReport.summary.coverage.'worklist-size' = 2
$privacyReport.summary.coverage.'items-evaluated' = 2
$privacyPath = $privacyFindings[0].references[0].path
$privacySources = @($privacyFindings | ForEach-Object { $_.location.file })
$sourceLengths = @(34, 29, 42, 57)
for ($index = 0; $index -lt $privacyFindings.Count; $index++) {
# Only source bounds are exercised here, not the AL behavior or a model.
Set-Content -LiteralPath (Join-Path $tmp $privacySources[$index]) `
-Value (1..$sourceLengths[$index] | ForEach-Object { "line $_" }) -Encoding utf8NoBOM
}
function Assert-PrivacyReport {
param(
[object] $Candidate,
[string] $ErrorPattern,
[string[]] $RetrievedPaths = @($privacyPath),
[switch] $Strict
)
$json = $Candidate | ConvertTo-Json -Depth 30
# Deliberate whitespace, escapes, CRLF, and BOM must survive acceptance and rejection byte-for-byte.
$json = " `r`n" + ($json.Replace('Privacy', '\u0050rivacy') -replace '\r?\n', "`r`n") + "`r`n "
Set-Content -LiteralPath $reportPath -Value $json -NoNewline -Encoding utf8BOM
$before = [Convert]::ToBase64String([IO.File]::ReadAllBytes($reportPath))
$objectBefore = $Candidate | ConvertTo-Json -Depth 30 -Compress
$invoke = {
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SourceRoot $tmp `
-SourcePaths $privacySources -RetrievedArticlePaths $RetrievedPaths -AllowBoundedNormalization:(-not $Strict)
}
try {
if ($ErrorPattern) {
Assert-ThrowsLike -Pattern $ErrorPattern -Action $invoke
}
else {
& $invoke
}
}
finally {
Assert-True ([Convert]::ToBase64String([IO.File]::ReadAllBytes($reportPath)) -ceq $before) `
'exact raw bytes are immutable on acceptance and rejection'
Assert-True (($Candidate | ConvertTo-Json -Depth 30 -Compress) -ceq $objectBefore) `
'the caller-owned report is not mutated'
}
}
Assert-PrivacyReport $privacyReport '*PRIMARY_REFERENCE_MISMATCH*' -Strict
$acceptedPrivacy = Assert-PrivacyReport $privacyReport
Assert-True ($acceptedPrivacy.normalized -and $acceptedPrivacy.normalizedIds.Count -eq 4 -and
$acceptedPrivacy.removedRanges.Count -eq 4) 'all four smoke findings require combined ID and range normalization'
$canonicalPrivacy = $privacyReport | ConvertTo-Json -Depth 30 | ConvertFrom-Json -DateKind String
for ($index = 0; $index -lt $privacyFindings.Count; $index++) {
$canonicalPrivacy.findings[$index].id = $privacyPath
$canonicalPrivacy.findings[$index].location.PSObject.Properties.Remove('range')
$idRecord = $acceptedPrivacy.normalizedIds[$index]
$rangeRecord = $acceptedPrivacy.removedRanges[$index]
Assert-True ($idRecord.findingIndex -eq $index -and $idRecord.originalId -ceq $privacyFindings[$index].id -and
$idRecord.canonicalId -ceq $privacyPath) 'private ID telemetry identifies the exact original and canonical IDs'
Assert-True ($rangeRecord.findingIndex -eq $index -and
$rangeRecord.startLine -eq $privacyFindings[$index].location.range.'start-line' -and
$rangeRecord.endLine -eq $privacyFindings[$index].location.range.'end-line') 'private range telemetry preserves original endpoints'
}
Assert-True (($acceptedPrivacy.report | ConvertTo-Json -Depth 30 -Compress) -ceq
($canonicalPrivacy | ConvertTo-Json -Depth 30 -Compress)) 'the entire candidate differs only in the two permitted fields'
Assert-ReportSchema $acceptedPrivacy.report $true 'accepted smoke report has no undeclared telemetry fields'
$canonicalNoOp = Assert-PrivacyReport $canonicalPrivacy
Assert-True (-not $canonicalNoOp.normalized -and $canonicalNoOp.normalizedIds.Count -eq 0 -and
$canonicalNoOp.removedRanges.Count -eq 0) 'already-canonical candidate is an idempotent no-op'
$multipleReferences = $privacyReport | ConvertTo-Json -Depth 30 | ConvertFrom-Json
$multipleReferences.findings[0].references += [pscustomobject]@{ path = $articlePath; sha = ('a' * 40) }
$acceptedMultiple = Assert-PrivacyReport $multipleReferences -RetrievedPaths @($privacyPath, $articlePath)
Assert-True ($acceptedMultiple.report.findings[0].id -ceq $privacyPath -and
($acceptedMultiple.report.findings[0].references | ConvertTo-Json -Compress) -ceq
($multipleReferences.findings[0].references | ConvertTo-Json -Compress)) 'primary selection preserves citation order, paths, and SHA'
Assert-PrivacyReport $multipleReferences '*REFERENCE_NOT_RETRIEVED*'
Assert-PrivacyReport $privacyReport '*REFERENCE_NOT_RETRIEVED*' -RetrievedPaths @()
foreach ($referenceCase in @(
@{ Path = 'microsoft/knowledge/privacy/unknown-article.md'; Error = '*REFERENCE_MISSING*' }
@{ Path = 'microsoft/knowledge/privacy/../privacy/privacy-notice-consent-for-external-data-transfer.md'; Error = '*REFERENCE_PATH_INVALID*' }
@{ Path = 'microsoft/knowledge/privacy/./privacy-notice-consent-for-external-data-transfer.md'; Error = '*REFERENCE_PATH_INVALID*' }
@{ Path = 'microsoft/knowledge//privacy/privacy-notice-consent-for-external-data-transfer.md'; Error = '*REFERENCE_PATH_INVALID*' }
@{ Path = '/microsoft/knowledge/privacy/article.md'; Error = '*REFERENCE_PATH_INVALID*' }
@{ Path = 'microsoft/knowledge/privacy/article%2e.md'; Error = '*REFERENCE_PATH_INVALID*' }
@{ Path = 'microsoft/knowledge/privacy/article#fragment.md'; Error = '*REFERENCE_PATH_INVALID*' }
@{ Path = 'microsoft/knowledge/privacy/article?query.md'; Error = '*REFERENCE_PATH_INVALID*' }
@{ Path = 'microsoft\knowledge\privacy\article.md'; Error = '*Invalid findings-report JSON or schema*' }
)) {
foreach ($referenceIndex in 0, 1) {
$badReference = $multipleReferences | ConvertTo-Json -Depth 30 | ConvertFrom-Json
$badReference.findings[0].references[$referenceIndex].path = $referenceCase.Path
Assert-PrivacyReport $badReference $referenceCase.Error -RetrievedPaths @($privacyPath, $articlePath, $referenceCase.Path)
}
}
foreach ($rawId in @("$privacyPath#AI", "${privacyPath}:AI", 'unrelated-scenario', $privacyPath.ToUpperInvariant())) {
$idVariant = $privacyReport | ConvertTo-Json -Depth 30 | ConvertFrom-Json
$idVariant.findings[0].id = $rawId
$acceptedVariant = Assert-PrivacyReport $idVariant
Assert-True ($acceptedVariant.report.findings[0].id -ceq $privacyPath) 'ID canonicalization copies the path, not a parsed or trimmed ID'
}
foreach ($rawId in @('', $null, 42, $true, @('scenario'), @{ value = 'scenario' })) {
$badId = $privacyReport | ConvertTo-Json -Depth 30 | ConvertFrom-Json
$badId.findings[0].id = $rawId
Assert-PrivacyReport $badId '*Invalid findings-report JSON or schema*'
}
foreach ($rawId in 'agent:ai-context', 'al-privacy-review:agent:ai-context') {
$citedAgent = $privacyReport | ConvertTo-Json -Depth 30 | ConvertFrom-Json
$citedAgent.findings[0].id = $rawId
Assert-PrivacyReport $citedAgent '*AGENT_REFERENCE_INVALID*'
}
$agentReport = $privacyReport | ConvertTo-Json -Depth 30 | ConvertFrom-Json
$agentReport.findings = @($agentReport.findings[0])
$agentReport.findings[0].id = 'agent:ai-context'
$agentReport.findings[0].references = @()
$agentReport.findings[0].severity = 'minor'
$agentReport.findings[0].confidence = 'medium'
$agentReport.summary.counts.major = 0
$agentReport.summary.counts.minor = 1
$acceptedAgent = Assert-PrivacyReport $agentReport -RetrievedPaths @()
Assert-True ($acceptedAgent.normalizedIds.Count -eq 0 -and $acceptedAgent.removedRanges.Count -eq 1 -and
$acceptedAgent.report.findings[0].id -ceq 'agent:ai-context') 'valid uncited agent supports range-only normalization without ID changes'
foreach ($rawId in 'scenario-id', 'agent:AI', 'agent:ai#fragment', 'al-privacy-review:agent:ai-context') {
$badAgent = $agentReport | ConvertTo-Json -Depth 30 | ConvertFrom-Json
$badAgent.findings[0].id = $rawId
$expectedError = if ($rawId -ceq 'al-privacy-review:agent:ai-context') { '*AGENT_ID_INVALID*' } else { '*Invalid findings-report JSON or schema*' }
Assert-PrivacyReport $badAgent $expectedError
}
foreach ($severity in 'blocker', 'major', 'minor', 'info') {
foreach ($confidence in 'high', 'medium', 'low') {
$agentCaps = $agentReport | ConvertTo-Json -Depth 30 | ConvertFrom-Json
$agentCaps.findings[0].severity = $severity
$agentCaps.findings[0].confidence = $confidence
$agentCaps.summary.counts.minor = 0
$agentCaps.summary.counts.$severity = 1
if ($severity -in @('blocker', 'major') -or $confidence -eq 'high') {
Assert-PrivacyReport $agentCaps '*Invalid findings-report JSON or schema*'
}
else {
$acceptedCaps = Assert-PrivacyReport $agentCaps
Assert-True ($acceptedCaps.report.findings[0].severity -ceq $severity -and
$acceptedCaps.report.findings[0].confidence -ceq $confidence) 'agent caps are preserved, never downgraded'
}
}
}
foreach ($defect in @(
@{ Edit = { param($r) $r.summary.counts.major = 3 }; Error = '*COUNT_MISMATCH*' }
@{ Edit = { param($r) $r.summary.coverage.'items-evaluated' = 1 }; Error = '*COMPLETED_COVERAGE_INCOMPLETE*' }
@{ Edit = { param($r) $r.findings[3].location.line = 58 }; Error = '*SOURCE_LINE_INVALID*' }
@{ Edit = { param($r) $r.findings[3].location.range.'end-line' = 58 }; Error = '*SOURCE_RANGE_INVALID*' }
@{ Edit = { param($r) $r.findings[3].location.range.'end-line' = 17 }; Error = '*SOURCE_RANGE_INVALID*' }
@{ Edit = { param($r) $r.findings[3].location.range.'start-line' = 24; $r.findings[3].location.range.'end-line' = 25 }; Error = '*RANGE_START_MISMATCH*' }
@{ Edit = { param($r) $r.findings[3].location.range.'end-line' = 22 }; Error = '*RANGE_START_MISMATCH*' }
@{ Edit = { param($r) $r.findings[3].location.line = 23.5 }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].location.range.'start-line' = 0 }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].location.range.'end-line' = '23' }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].location.range.PSObject.Properties.Remove('end-line') }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].location.range | Add-Member 'extra' 1 }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].message = '' }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].confidence = 'certain' }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].severity = 'critical' }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].PSObject.Properties.Remove('id') }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].references = $null }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].references[0].path = $null }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].location.file = 'src/missing.al' }; Error = '*SOURCE_MISSING*' }
@{ Edit = { param($r) $r.findings[3].location.file = 'src/codeunit.al' }; Error = '*SOURCE_OUT_OF_SCOPE*' }
@{ Edit = { param($r) $r.findings[3] | Add-Member 'from-sub-skill' 'al-privacy-review' }; Error = '*LEAF_PRODUCER_INVALID*' }
@{ Edit = { param($r) $r.findings[3] | Add-Member 'extra' 'not permitted' }; Error = '*Invalid findings-report JSON or schema*' }
@{ Edit = { param($r) $r.findings[3].references[0] | Add-Member 'sha' 'not-a-sha' }; Error = '*Invalid findings-report JSON or schema*' }
)) {
$defectiveReport = $privacyReport | ConvertTo-Json -Depth 30 | ConvertFrom-Json
& $defect.Edit $defectiveReport
Assert-PrivacyReport $defectiveReport $defect.Error
}
foreach ($suggestion in @('exit;', '', $null, 1)) {
$suggestedRange = $privacyReport | ConvertTo-Json -Depth 30 | ConvertFrom-Json
$suggestedRange.findings[3] | Add-Member 'suggested-code' $suggestion
$expectedError = if ($suggestion -ceq 'exit;') { '*RANGE_START_MISMATCH*' } else { '*Invalid findings-report JSON or schema*' }
Assert-PrivacyReport $suggestedRange $expectedError
}
$suggestedIdOnly = $canonicalPrivacy | ConvertTo-Json -Depth 30 | ConvertFrom-Json
$suggestedIdOnly.findings[0].id = 'scenario-with-safe-suggestion'
$suggestedIdOnly.findings[0] | Add-Member 'suggested-code' 'exit;'
$acceptedSuggestion = Assert-PrivacyReport $suggestedIdOnly
Assert-True ($acceptedSuggestion.normalizedIds.Count -eq 1 -and $acceptedSuggestion.removedRanges.Count -eq 0 -and
$acceptedSuggestion.report.findings[0].'suggested-code' -ceq 'exit;') 'ID-only normalization never rewrites suggested code'
foreach ($invalidJson in @(
'{"findings": [],}'
'{"findings": [/* no repair */]}'
'{"message": "Unescaped "quote""}'
'[]'
)) {
Set-Content -LiteralPath $reportPath -Value $invalidJson -NoNewline -Encoding utf8NoBOM
Assert-ThrowsLike -Pattern '*Invalid findings-report JSON or schema*' -Action {
& $validator -ReportPath $reportPath -BCQualityRoot $Root -AllowBoundedNormalization
}
Assert-True ([IO.File]::ReadAllText($reportPath) -ceq $invalidJson) 'strict JSON and structural failures are never reconstructed'
}
}
finally {
Remove-Item -LiteralPath $tmp -Recurse -Force -ErrorAction SilentlyContinue

View file

@ -425,8 +425,16 @@ function Get-SemanticErrors {
Add-Error 'SUPER_PRODUCER_REQUIRED' $findingPath 'A super-skill finding must identify its producer in from-sub-skill.'
}
if (-not $references.Count) {
if ($finding.id -cnotmatch '(^|:)agent:[a-z0-9]+(?:-[a-z0-9]+)*$') {
Add-Error 'AGENT_ID_INVALID' "$findingPath.id" 'An agent finding id must contain an agent: slug marker.'
$agentPrefix = if ($CurrentSkillKind -ceq 'leaf' -or
($hasProducer -and $finding.'from-sub-skill' -ceq 'agent')) {
'^agent:'
}
else {
$producer = if ($hasProducer) { $finding.'from-sub-skill' } else { '' }
'^' + [regex]::Escape([string]$producer) + ':agent:'
}
if ($finding.id -cnotmatch ($agentPrefix + '[a-z0-9]+(?:-[a-z0-9]+)*$')) {
Add-Error 'AGENT_ID_INVALID' "$findingPath.id" 'An agent finding id must use its role-specific agent: prefix.'
}
if ($finding.confidence -ceq 'high') {
Add-Error 'AGENT_CONFIDENCE_INVALID' "$findingPath.confidence" 'Agent confidence cannot be high.'
@ -436,11 +444,16 @@ function Get-SemanticErrors {
}
}
else {
if ($finding.id -cmatch '(^|:)agent:' -or
($hasProducer -and $finding.'from-sub-skill' -ceq 'agent')) {
Add-Error 'AGENT_REFERENCE_INVALID' "$findingPath.references" 'An agent finding must not contain citations.'
}
if ($finding.id -cne $references[0].path) {
Add-Error 'PRIMARY_REFERENCE_MISMATCH' "$findingPath.id" 'Finding id must equal the primary reference path.'
}
foreach ($reference in $references) {
if ($reference.path -cnotmatch '^(microsoft|community|custom)/knowledge/.+\.md$') {
if ($reference.path -cnotmatch '^(microsoft|community|custom)/knowledge/.+\.md$' -or
$reference.path -cmatch '(^|/)\.{1,2}(/|$)|//|[\\:#?%*\x00-\x1F\x7F]') {
Add-Error 'REFERENCE_PATH_INVALID' "$findingPath.references" "Invalid knowledge path '$($reference.path)'."
continue
}
@ -719,14 +732,29 @@ function Get-SemanticErrors {
$errors = @(Get-SemanticErrors $report)
$normalized = $false
$normalizedIds = [Collections.Generic.List[object]]::new()
$removedRanges = [Collections.Generic.List[object]]::new()
if ($errors.Count -and $AllowBoundedNormalization) {
$otherErrors = @($errors | Where-Object Code -CNE 'RANGE_START_MISMATCH')
$rangeErrors = @($errors | Where-Object Code -CEQ 'RANGE_START_MISMATCH')
if (-not $otherErrors.Count -and $rangeErrors.Count) {
$referenceErrors = @($errors | Where-Object Code -CIN @(
'REFERENCE_PATH_INVALID', 'REFERENCE_MISSING', 'REFERENCE_NOT_RETRIEVED', 'AGENT_REFERENCE_INVALID'
))
if (-not $referenceErrors.Count) {
$candidate = $report | ConvertTo-Json -Depth 100 | ConvertFrom-Json -Depth 100 -DateKind String
$eligible = $true
foreach ($finding in @($candidate.findings)) {
for ($index = 0; $index -lt $candidate.findings.Count; $index++) {
$finding = $candidate.findings[$index]
if (@($finding.references).Count -and $finding.id -cne $finding.references[0].path) {
$normalizedIds.Add([pscustomobject]@{
findingIndex = $index
originalId = $finding.id
canonicalId = $finding.references[0].path
}) | Out-Null
$finding.id = $finding.references[0].path
}
}
# Check all original endpoints before any range can be removed.
$eligible = -not @(Get-SemanticErrors $candidate -PermitRangeStartMismatch).Count
for ($index = 0; $eligible -and $index -lt $candidate.findings.Count; $index++) {
$finding = $candidate.findings[$index]
if (-not (Test-HasProperty $finding 'location') -or
-not (Test-HasProperty $finding.location 'range') -or
$finding.location.range.'start-line' -eq $finding.location.line) {
@ -740,6 +768,7 @@ if ($errors.Count -and $AllowBoundedNormalization) {
break
}
$removedRanges.Add([pscustomobject]@{
findingIndex = $index
findingId = $finding.id
file = $finding.location.file
line = $finding.location.line
@ -748,7 +777,9 @@ if ($errors.Count -and $AllowBoundedNormalization) {
}) | Out-Null
$finding.location.PSObject.Properties.Remove('range')
}
if ($eligible -and -not @(Get-SemanticErrors $candidate).Count) {
if ($eligible -and
($candidate | ConvertTo-Json -Depth 100 | Test-Json -SchemaFile $schemaPath -ErrorAction Stop) -and
-not @(Get-SemanticErrors $candidate).Count) {
$report = $candidate
$normalized = $true
$errors = @()
@ -764,5 +795,6 @@ if ($errors.Count) {
return [pscustomobject][ordered]@{
normalized = $normalized
report = $report
normalizedIds = @($normalizedIds)
removedRanges = @($removedRanges)
}