diff --git a/docs/standalone-runner.md b/docs/standalone-runner.md index d3186ef..74b2f8e 100644 --- a/docs/standalone-runner.md +++ b/docs/standalone-runner.md @@ -88,7 +88,8 @@ For example, if style is selected and security was disabled: { "superSkill": { "id": "al-code-review", "version": 1 }, "subSkills": [{ "id": "al-style-review", "version": 1 }], - "skipped": [{ "id": "al-security-review", "version": 1, "reason": "configuration" }] + "skipped": [{ "id": "al-security-review", "version": 1, "reason": "configuration" }], + "acceptedResults": [] } ``` @@ -101,10 +102,28 @@ Keep paths, layers, and other resolver metadata if useful for private audit. Do not add the artifact to the findings-report or overwrite it to hide an unfinished invocation. Preserve it with the run's raw payloads. +Initialize `acceptedResults` before dispatch. After accepting a leaf, save its +exact accepted copy (including any permitted normalization) in an immutable +host-owned file outside worker/composer write access. Append only its capture +to `acceptedResults`, for example: + +```json +{"id": "al-style-review", "version": 1, "reportPath": "accepted/style.json"} +``` + +`reportPath` may be absolute or relative to the composition artifact's directory. +Capture a host-created failed validation report the same way. Never construct +these captures from the composed `sub-results` or permit the composing model +to supply or alter them. Keep the original selection and exclusions unchanged. + The gate rejects repeated or unexpected leaf IDs, wrong selected versions, -reordered results, and fabricated or missing exclusions. If selected leaves -remain unfinished, the report must be `partial` with a non-failed returned -report, otherwise `failed`; identify unfinished leaf IDs in `outcome-reason`. +reordered results, fabricated or missing exclusions, and uncaptured or altered +leaf content. JSON property order is immaterial; array order, field presence, +types, and values must match. Every captured leaf must appear in `sub-results`. +If selected leaves remain unfinished, the report must be `partial` with a non-failed returned +report, otherwise `failed`; name every unfinished leaf ID exactly in +`outcome-reason`. Top-level `from-sub-skill: "agent"` findings are rejected +while selected leaf results are missing. Do not fabricate leaf reports or use configuration skips for budget exhaustion. Wait for started invocations to finish; omit the self-review if the selected composition remains incomplete. Coverage still sums the non-failed leaf @@ -112,6 +131,7 @@ knowledge worklists, not the number of selected leaf slots. Calls without the expected artifact remain supported for structural/semantic validation, but cannot certify that a composed review covered its selection. +They also cannot bind nested leaves to the host's accepted outputs. They still reject duplicate leaf IDs and returned-and-skipped conflicts. ## Runner-owned choices diff --git a/microsoft/skills/review/al-code-review.md b/microsoft/skills/review/al-code-review.md index 8699d7d..30bf659 100644 --- a/microsoft/skills/review/al-code-review.md +++ b/microsoft/skills/review/al-code-review.md @@ -78,6 +78,14 @@ skill's version, and record any input-incompatible exclusions with `reason: "not-applicable"`. Do not derive this artifact from leaf reports or change it merely because execution later runs out of budget. Pass its path as `-ExpectedCompositionPath` for final super-skill validation. +Initialize `acceptedResults` to `[]`. After each leaf's acceptance gate, the +host saves its exact accepted copy (or host-created failed validation result) +in an immutable private file and appends its `id`, `version`, and `reportPath` +to that array. The host alone owns these captures; neither workers nor the +composing model may write them. Never derive them from composed `sub-results`. +Keep the pre-dispatch selection and exclusions unchanged. Final validation +requires every nested leaf to match its captured JSON content exactly and +every captured result to be included. Property order is immaterial. ## Action diff --git a/skills/do.md b/skills/do.md index 16fbdd9..e5aef5b 100644 --- a/skills/do.md +++ b/skills/do.md @@ -228,16 +228,31 @@ scope and the leaf's recorded set of fully retrieved article paths. Pass a super-skill's rolled-up report. Prepare that private artifact before leaf dispatch, after layer resolution and input compatibility checks. It contains `superSkill` (`id`, `version`), ordered selected `subSkills` (each with `id`, -`version`), and `skipped` (each with `id`, `version`, `reason`). Reasons are -`configuration` or `not-applicable`; budget exhaustion is not a skip reason. +`version`), `skipped` (each with `id`, `version`, `reason`), and an initially +empty `acceptedResults` array. Reasons are `configuration` or `not-applicable`; +budget exhaustion is not a skip reason. Additional resolver metadata may be retained in the artifact, not the report. +After each leaf passes its acceptance gate, the host saves the exact accepted +copy in a private immutable file and appends an `acceptedResults` entry with +`id`, `version`, and `reportPath`. Capture host-created failed validation +results the same way. Paths may be absolute or relative to the composition +artifact's directory. Capture the normalized accepted copy when normalization +was permitted, not the invalid raw return. Do not expose these files or write +access to the artifact to leaf workers or the composing model. Only the host +may append captures; the pre-dispatch selection and exclusions remain fixed. The validator binds the super-skill and leaf identities and versions, checks -selected order, and requires exact agreement on exclusions. Every returned -leaf must be unique; a selected leaf cannot be reclassified as skipped by the -report. Without the artifact, validation remains structural and semantic but +selected order, and requires exact agreement on exclusions. Each nested leaf +must exactly match its host-captured accepted JSON content, ignoring object +property order but preserving array order, types, values, and field presence. +Every captured leaf must be included; uncaptured or altered leaves are invalid. +Every returned leaf must be unique; a selected leaf cannot be reclassified as skipped by the +report. When selected leaves are missing, `outcome-reason` must name every +missing ID exactly and top-level `from-sub-skill: "agent"` findings are forbidden. +Without the artifact, validation remains structural and semantic but cannot prove composition completeness, selected versions, order, or legitimate -exclusions. Duplicate or both returned-and-skipped leaf IDs are invalid even -without the artifact. Never derive the expected composition from model output. +exclusions, nor bind leaves to accepted host outputs. Duplicate or both +returned-and-skipped leaf IDs are invalid even without the artifact. Never +derive the expected composition from model output. Pass `-AllowBoundedNormalization` only when the host preserves the immutable raw payload and records `removedRanges` in private telemetry as required above. diff --git a/tools/Test-ReviewContract.ps1 b/tools/Test-ReviewContract.ps1 index 118fc3a..a567e4c 100644 --- a/tools/Test-ReviewContract.ps1 +++ b/tools/Test-ReviewContract.ps1 @@ -294,6 +294,18 @@ try { } $compositionPath = Join-Path $tmp 'composition.json' + function Save-AcceptedLeafReports { + param([object[]] $LeafReports) + + foreach ($leafReport in $LeafReports) { + $leafPath = Join-Path $tmp "$([guid]::NewGuid()).json" + Set-Content -LiteralPath $leafPath -Value ($leafReport | ConvertTo-Json -Depth 100) -Encoding utf8NoBOM + $accepted = & $validator -ReportPath $leafPath -BCQualityRoot $Root + Assert-True (-not $accepted.normalized) 'host captures a validated leaf report' + @{ id = $leafReport.skill.id; version = $leafReport.skill.version; reportPath = $leafPath } + } + } + $expectedComposition = [ordered]@{ superSkill = @{ id = 'al-code-review'; version = 1 } subSkills = @( @@ -301,6 +313,7 @@ try { @{ id = 'al-security-review'; version = 1 } ) skipped = @() + acceptedResults = @(Save-AcceptedLeafReports @($completedLeaf, $completedSecurityLeaf)) } Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM Set-Content -LiteralPath $reportPath -Value ($validSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM @@ -308,6 +321,9 @@ try { -ExpectedCompositionPath $compositionPath Assert-True (-not $acceptedBoundSuper.normalized) 'complete composition matches the expected worklist' + $incompleteComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $incompleteComposition.acceptedResults = @($incompleteComposition.acceptedResults[0]) + Set-Content -LiteralPath $compositionPath -Value ($incompleteComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM $missingLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json $missingLeafReport.'sub-results' = @($completedLeaf) $missingLeafReport.summary.coverage.'worklist-size' = 1 @@ -340,6 +356,103 @@ try { } } + $genericMissingReason = $missingLeafReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $genericMissingReason.'outcome-reason' = 'Budget expired.' + Assert-CompositionReport $genericMissingReason '*SUPER_MISSING_LEAF_REASON*' + $genericMissingReason.'outcome-reason' = 'prefix-al-security-review-suffix was not evaluated.' + Assert-CompositionReport $genericMissingReason '*SUPER_MISSING_LEAF_REASON*' + + foreach ($fabricatedOutcome in 'completed', 'not-applicable', 'no-knowledge') { + $fabricatedLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $fabricatedLeafReport.'sub-results'[1].outcome = $fabricatedOutcome + if ($fabricatedOutcome -cne 'completed') { + $fabricatedLeafReport.'sub-results'[1].summary.coverage.'worklist-size' = 0 + $fabricatedLeafReport.'sub-results'[1].summary.coverage.'items-evaluated' = 0 + $fabricatedLeafReport.summary.coverage.'worklist-size' = 1 + $fabricatedLeafReport.summary.coverage.'items-evaluated' = 1 + } + Assert-CompositionReport $fabricatedLeafReport '*SUPER_LEAF_NOT_ACCEPTED*' + } + Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM + Assert-CompositionReport $missingLeafReport '*SUPER_ACCEPTED_LEAF_MISSING*' + $alteredLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $alteredLeafReport.'sub-results'[1].summary.coverage.'worklist-size' = 2 + $alteredLeafReport.'sub-results'[1].summary.coverage.'items-evaluated' = 2 + $alteredLeafReport.summary.coverage.'worklist-size' = 3 + $alteredLeafReport.summary.coverage.'items-evaluated' = 3 + Assert-CompositionReport $alteredLeafReport '*SUPER_LEAF_CONTENT_MISMATCH*' + $reorderedProperties = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $reorderedProperties.'sub-results'[0].skill = [pscustomobject]@{ version = 1; id = 'al-style-review' } + Assert-CompositionReport $reorderedProperties + foreach ($alteredOutcome in 'not-applicable', 'no-knowledge') { + $alteredOutcomeReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $alteredOutcomeReport.'sub-results'[1].outcome = $alteredOutcome + $alteredOutcomeReport.'sub-results'[1].summary.coverage.'worklist-size' = 0 + $alteredOutcomeReport.'sub-results'[1].summary.coverage.'items-evaluated' = 0 + $alteredOutcomeReport.summary.coverage.'worklist-size' = 1 + $alteredOutcomeReport.summary.coverage.'items-evaluated' = 1 + Assert-CompositionReport $alteredOutcomeReport '*SUPER_LEAF_CONTENT_MISMATCH*' + } + $relativeCaptureComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json + foreach ($capture in $relativeCaptureComposition.acceptedResults) { + $capture.reportPath = Split-Path -Leaf $capture.reportPath + } + Set-Content -LiteralPath $compositionPath -Value ($relativeCaptureComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM + Assert-CompositionReport $validSuperReport + $uncapturedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $uncapturedComposition.PSObject.Properties.Remove('acceptedResults') + Set-Content -LiteralPath $compositionPath -Value ($uncapturedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM + Assert-CompositionReport $validSuperReport '*Invalid expected composition*' + $duplicateCaptureComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $duplicateCaptureComposition.acceptedResults = @($duplicateCaptureComposition.acceptedResults[0], $duplicateCaptureComposition.acceptedResults[0]) + Set-Content -LiteralPath $compositionPath -Value ($duplicateCaptureComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM + Assert-CompositionReport $validSuperReport '*Invalid expected composition*' + + $capturedFindingLeaf = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $capturedFindingLeaf.findings = @(@{ + id = 'agent:leaf-issue' + severity = 'minor' + confidence = 'medium' + message = 'Preserve this accepted leaf finding.' + references = @() + }) + $secondCapturedFinding = $capturedFindingLeaf.findings[0] | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $secondCapturedFinding.id = 'agent:second-leaf-issue' + $capturedFindingLeaf.findings = @($capturedFindingLeaf.findings[0], $secondCapturedFinding) + $capturedFindingLeaf.summary.counts.minor = 2 + $findingComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $findingComposition.acceptedResults = @(Save-AcceptedLeafReports @($capturedFindingLeaf, $completedSecurityLeaf)) + Set-Content -LiteralPath $compositionPath -Value ($findingComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM + $capturedFindingReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $capturedFindingReport.'sub-results'[0] = $capturedFindingLeaf + $capturedFindingReport.findings = @($capturedFindingLeaf.findings | ConvertTo-Json -Depth 20 | ConvertFrom-Json) + foreach ($finding in $capturedFindingReport.findings) { + $finding.id = "al-style-review:$($finding.id)" + $finding | Add-Member -NotePropertyName 'from-sub-skill' -NotePropertyValue 'al-style-review' + } + $capturedFindingReport.summary.counts.minor = 2 + Assert-CompositionReport $capturedFindingReport + $reorderedFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $reorderedFindingReport.'sub-results'[0].findings = @( + $reorderedFindingReport.'sub-results'[0].findings[1] + $reorderedFindingReport.'sub-results'[0].findings[0] + ) + Assert-CompositionReport $reorderedFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*' + $addedLeafFieldReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $addedLeafFieldReport.'sub-results'[0] | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'A composer-added field.' + Assert-CompositionReport $addedLeafFieldReport '*SUPER_LEAF_CONTENT_MISMATCH*' + $alteredFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $alteredFindingReport.'sub-results'[0].findings[0].message = 'A fabricated replacement message.' + $alteredFindingReport.findings[0].message = 'A fabricated replacement message.' + Assert-CompositionReport $alteredFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*' + $removedFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $removedFindingReport.'sub-results'[0].findings = @() + $removedFindingReport.'sub-results'[0].summary.counts.minor = 0 + $removedFindingReport.findings = @() + $removedFindingReport.summary.counts.minor = 0 + Assert-CompositionReport $removedFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*' + Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM + foreach ($case in @( @{ Pattern = '*SUPER_IDENTITY_MISMATCH*'; Change = { param($candidate) $candidate.skill.id = 'al-other-review' } } @{ Pattern = '*SUPER_IDENTITY_MISMATCH*'; Change = { param($candidate) $candidate.skill.version = 2 } } @@ -359,6 +472,9 @@ try { ) Assert-CompositionReport $fabricatedSkip '*SUPER_UNEXPECTED_SKIP*' + $emptyAcceptedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $emptyAcceptedComposition.acceptedResults = @() + Set-Content -LiteralPath $compositionPath -Value ($emptyAcceptedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM $noResults = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json $noResults.'sub-results' = @() $noResults.summary.coverage.'worklist-size' = 0 @@ -366,9 +482,37 @@ try { $noResults.outcome = 'not-applicable' Assert-CompositionReport $noResults '*SUPER_OUTCOME_MISMATCH*' $noResults.outcome = 'failed' - $noResults | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'No selected leaf could be evaluated.' + $noResults | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'al-style-review and al-security-review could not be evaluated.' Assert-CompositionReport $noResults + $oneMissingId = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $oneMissingId.'outcome-reason' = 'al-security-review could not be evaluated.' + Assert-CompositionReport $oneMissingId '*SUPER_MISSING_LEAF_REASON*' + foreach ($baseReport in @($missingLeafReport, $noResults, $validSuperReport)) { + $capturedComposition = if ($baseReport.outcome -ceq 'completed') { $expectedComposition } else { $incompleteComposition } + Set-Content -LiteralPath $compositionPath -Value ($capturedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM + $selfReviewReport = $baseReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $selfReviewReport.findings = @(@{ + id = 'agent:cross-domain-gap' + domain = 'Agent' + severity = 'minor' + confidence = 'medium' + message = 'A cross-domain issue needs attention.' + references = @() + 'from-sub-skill' = 'agent' + }) + $selfReviewReport.summary.counts.minor = 1 + if ($baseReport.outcome -ceq 'completed') { + Assert-CompositionReport $selfReviewReport + } + elseif ($baseReport.outcome -ceq 'failed') { + Assert-CompositionReport $selfReviewReport '*Invalid findings-report JSON or schema*' + } + else { + Assert-CompositionReport $selfReviewReport '*SUPER_AGENT_REVIEW_INCOMPLETE*' + } + } + $allFailed = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json $allFailed.'sub-results' = @($completedLeaf, $completedSecurityLeaf) | ConvertTo-Json -Depth 20 | ConvertFrom-Json foreach ($leaf in $allFailed.'sub-results') { @@ -376,11 +520,15 @@ try { $leaf.summary.coverage.'items-evaluated' = 0 $leaf | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'Invocation failed.' } + $failedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $failedComposition.acceptedResults = @(Save-AcceptedLeafReports $allFailed.'sub-results') + Set-Content -LiteralPath $compositionPath -Value ($failedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM Assert-CompositionReport $allFailed $skipComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json $skipComposition.subSkills = @($skipComposition.subSkills[0]) $skipComposition.skipped = @(@{ id = 'al-security-review'; version = 1; reason = 'not-applicable' }) + $skipComposition.acceptedResults = @($skipComposition.acceptedResults[0]) Set-Content -LiteralPath $compositionPath -Value ($skipComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM $validSkippedReport = $fabricatedSkip | ConvertTo-Json -Depth 20 | ConvertFrom-Json $validSkippedReport.outcome = 'completed' @@ -406,6 +554,7 @@ try { @{ id = $_.id; version = $_.version; reason = 'configuration' } }) $allSkippedComposition.subSkills = @() + $allSkippedComposition.acceptedResults = @() Set-Content -LiteralPath $compositionPath -Value ($allSkippedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM $allSkippedReport = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json $allSkippedReport.outcome = 'not-applicable' @@ -464,7 +613,6 @@ try { $declaredSkill = @($fixtureIndex.skills | Where-Object path -CEQ $declaredPath)[0] @{ id = $_.id; version = $declaredSkill.version; reason = $_.reason; declaredPath = $declaredPath } }) - Set-Content -LiteralPath $compositionPath -Value ($resolved | ConvertTo-Json -Depth 30) -Encoding utf8NoBOM $resolvedReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json $resolvedReport.'sub-results' = @($resolved.subSkills | ForEach-Object { $leafReport = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json @@ -477,6 +625,8 @@ try { $resolvedReport | Add-Member -NotePropertyName 'skipped-sub-skills' -NotePropertyValue @( $resolved.skipped | ForEach-Object { @{ skill = @{ id = $_.id; version = $_.version }; reason = $_.reason } } ) + $resolved | Add-Member -NotePropertyName 'acceptedResults' -NotePropertyValue @(Save-AcceptedLeafReports $resolvedReport.'sub-results') + Set-Content -LiteralPath $compositionPath -Value ($resolved | ConvertTo-Json -Depth 30) -Encoding utf8NoBOM Assert-CompositionReport $resolvedReport } Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM @@ -643,6 +793,9 @@ try { Set-Content -LiteralPath $reportPath -Value ($partialSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM $acceptedPartialSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super Assert-True (-not $acceptedPartialSuper.normalized) 'partial super-skill excludes failed coverage from its rollup' + $partialComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json + $partialComposition.acceptedResults = @(Save-AcceptedLeafReports @($completedLeaf, $failedLeaf)) + Set-Content -LiteralPath $compositionPath -Value ($partialComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM Assert-CompositionReport $partialSuperReport $failedLeafLeakage = $partialSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json diff --git a/tools/Validate-FindingsReport.ps1 b/tools/Validate-FindingsReport.ps1 index cf42c5b..d5e5dce 100644 --- a/tools/Validate-FindingsReport.ps1 +++ b/tools/Validate-FindingsReport.ps1 @@ -38,6 +38,7 @@ catch { $expectedComposition = $null $expectedLeaves = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal) $expectedSkips = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal) +$acceptedLeaves = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal) if ($ExpectedCompositionPath) { if ($SkillKind -cne 'super') { throw 'Expected composition is supported only for super-skill reports.' @@ -55,10 +56,22 @@ if ($ExpectedCompositionPath) { } $compositionSchema = @{ type = 'object' - required = @('superSkill', 'subSkills', 'skipped') + required = @('superSkill', 'subSkills', 'skipped', 'acceptedResults') properties = @{ superSkill = $identitySchema subSkills = @{ type = 'array'; items = $identitySchema } + acceptedResults = @{ + type = 'array' + items = @{ + type = 'object' + required = @('id', 'version', 'reportPath') + properties = @{ + id = $identitySchema.properties.id + version = $identitySchema.properties.version + reportPath = @{ type = 'string'; minLength = 1 } + } + } + } skipped = @{ type = 'array' items = @{ type = 'object'; required = @('id', 'version', 'reason'); properties = $skipProperties } @@ -84,6 +97,31 @@ if ($ExpectedCompositionPath) { } $expectedSkips.Add([string]$skip.id, $skip) } + $compositionDirectory = Split-Path -Parent (Resolve-Path -LiteralPath $ExpectedCompositionPath).Path + foreach ($accepted in @($expectedComposition.acceptedResults)) { + if (-not $expectedLeaves.ContainsKey([string]$accepted.id) -or + $accepted.version -ne $expectedLeaves[$accepted.id].version -or + $acceptedLeaves.ContainsKey([string]$accepted.id)) { + throw "Accepted result '$($accepted.id)' must uniquely match a selected leaf and version." + } + $acceptedPath = if ([IO.Path]::IsPathRooted($accepted.reportPath)) { + $accepted.reportPath + } + else { + Join-Path $compositionDirectory $accepted.reportPath + } + $acceptedRaw = Get-Content -LiteralPath $acceptedPath -Raw + if (-not ($acceptedRaw | Test-Json -SchemaFile $schemaPath -ErrorAction Stop)) { + throw "Accepted result '$($accepted.id)' does not satisfy the report schema." + } + $acceptedReport = $acceptedRaw | ConvertFrom-Json -Depth 100 + if ($acceptedReport.skill.id -cne $accepted.id -or $acceptedReport.skill.version -ne $accepted.version -or + $acceptedReport.PSObject.Properties.Name -ccontains 'sub-results' -or + $acceptedReport.PSObject.Properties.Name -ccontains 'skipped-sub-skills') { + throw "Accepted result '$($accepted.id)' must be a leaf report with the captured identity." + } + $acceptedLeaves.Add([string]$accepted.id, $acceptedReport) + } } catch { throw "Invalid expected composition: $($_.Exception.Message)" @@ -105,6 +143,39 @@ function Test-HasProperty { return $null -ne $Object -and $Object.PSObject.Properties.Name -ccontains $Name } +function Test-JsonContentEqual { + param([object] $First, [object] $Second) + + if ($null -eq $First -or $null -eq $Second) { + return $null -eq $First -and $null -eq $Second + } + if ($First -is [pscustomobject] -or $Second -is [pscustomobject]) { + if ($First -isnot [pscustomobject] -or $Second -isnot [pscustomobject] -or + @($First.PSObject.Properties).Count -ne @($Second.PSObject.Properties).Count) { + return $false + } + foreach ($property in $First.PSObject.Properties) { + if (-not (Test-HasProperty $Second $property.Name) -or + -not (Test-JsonContentEqual $property.Value $Second.PSObject.Properties[$property.Name].Value)) { + return $false + } + } + return $true + } + if ($First -is [array] -or $Second -is [array]) { + if ($First -isnot [array] -or $Second -isnot [array] -or $First.Count -ne $Second.Count) { + return $false + } + for ($index = 0; $index -lt $First.Count; $index++) { + if (-not (Test-JsonContentEqual $First[$index] $Second[$index])) { + return $false + } + } + return $true + } + return $First.GetType() -eq $Second.GetType() -and $First -ceq $Second +} + function Get-SourceLineCount { param([string] $Path) @@ -441,6 +512,14 @@ function Get-SemanticErrors { Add-Error 'SUPER_LEAF_VERSION_MISMATCH' "$ReportPathPrefix.sub-results[$index].skill.version" ` "Unexpected version for '$($identity.id)'." } + if (-not $acceptedLeaves.ContainsKey([string]$identity.id)) { + Add-Error 'SUPER_LEAF_NOT_ACCEPTED' "$ReportPathPrefix.sub-results[$index]" ` + "Leaf '$($identity.id)' has no host-captured accepted result." + } + elseif (-not (Test-JsonContentEqual $subResults[$index] $acceptedLeaves[$identity.id])) { + Add-Error 'SUPER_LEAF_CONTENT_MISMATCH' "$ReportPathPrefix.sub-results[$index]" ` + "Leaf '$($identity.id)' differs from its host-captured accepted result." + } $slot = [Array]::IndexOf($orderedIds, $identity.id) if ($slot -le $previousSlot) { Add-Error 'SUPER_SUB_RESULT_ORDER' "$ReportPathPrefix.sub-results[$index].skill" ` @@ -451,6 +530,16 @@ function Get-SemanticErrors { foreach ($leaf in @($expectedComposition.subSkills)) { if (-not $producerIds.Contains([string]$leaf.id)) { $missingResults++ + if ($acceptedLeaves.ContainsKey([string]$leaf.id)) { + Add-Error 'SUPER_ACCEPTED_LEAF_MISSING' "$ReportPathPrefix.sub-results" ` + "Host-captured accepted leaf '$($leaf.id)' must be included." + } + $reason = if (Test-HasProperty $Current 'outcome-reason') { $Current.'outcome-reason' } else { '' } + $idPattern = '(?