mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-10-05 14:46:55 +01:00
Bind composed leaf reports to host-accepted results
This commit is contained in:
parent
11fbd227f4
commit
aecd081778
5 changed files with 303 additions and 14 deletions
|
|
@ -88,7 +88,8 @@ For example, if style is selected and security was disabled:
|
||||||
{
|
{
|
||||||
"superSkill": { "id": "al-code-review", "version": 1 },
|
"superSkill": { "id": "al-code-review", "version": 1 },
|
||||||
"subSkills": [{ "id": "al-style-review", "version": 1 }],
|
"subSkills": [{ "id": "al-style-review", "version": 1 }],
|
||||||
"skipped": [{ "id": "al-security-review", "version": 1, "reason": "configuration" }]
|
"skipped": [{ "id": "al-security-review", "version": 1, "reason": "configuration" }],
|
||||||
|
"acceptedResults": []
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -101,10 +102,28 @@ Keep paths, layers, and other resolver metadata if useful for private audit.
|
||||||
Do not add the artifact to the findings-report or overwrite it to hide an
|
Do not add the artifact to the findings-report or overwrite it to hide an
|
||||||
unfinished invocation. Preserve it with the run's raw payloads.
|
unfinished invocation. Preserve it with the run's raw payloads.
|
||||||
|
|
||||||
|
Initialize `acceptedResults` before dispatch. After accepting a leaf, save its
|
||||||
|
exact accepted copy (including any permitted normalization) in an immutable
|
||||||
|
host-owned file outside worker/composer write access. Append only its capture
|
||||||
|
to `acceptedResults`, for example:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"id": "al-style-review", "version": 1, "reportPath": "accepted/style.json"}
|
||||||
|
```
|
||||||
|
|
||||||
|
`reportPath` may be absolute or relative to the composition artifact's directory.
|
||||||
|
Capture a host-created failed validation report the same way. Never construct
|
||||||
|
these captures from the composed `sub-results` or permit the composing model
|
||||||
|
to supply or alter them. Keep the original selection and exclusions unchanged.
|
||||||
|
|
||||||
The gate rejects repeated or unexpected leaf IDs, wrong selected versions,
|
The gate rejects repeated or unexpected leaf IDs, wrong selected versions,
|
||||||
reordered results, and fabricated or missing exclusions. If selected leaves
|
reordered results, fabricated or missing exclusions, and uncaptured or altered
|
||||||
remain unfinished, the report must be `partial` with a non-failed returned
|
leaf content. JSON property order is immaterial; array order, field presence,
|
||||||
report, otherwise `failed`; identify unfinished leaf IDs in `outcome-reason`.
|
types, and values must match. Every captured leaf must appear in `sub-results`.
|
||||||
|
If selected leaves remain unfinished, the report must be `partial` with a non-failed returned
|
||||||
|
report, otherwise `failed`; name every unfinished leaf ID exactly in
|
||||||
|
`outcome-reason`. Top-level `from-sub-skill: "agent"` findings are rejected
|
||||||
|
while selected leaf results are missing.
|
||||||
Do not fabricate leaf reports or use configuration skips for budget exhaustion.
|
Do not fabricate leaf reports or use configuration skips for budget exhaustion.
|
||||||
Wait for started invocations to finish; omit the self-review if the selected
|
Wait for started invocations to finish; omit the self-review if the selected
|
||||||
composition remains incomplete. Coverage still sums the non-failed leaf
|
composition remains incomplete. Coverage still sums the non-failed leaf
|
||||||
|
|
@ -112,6 +131,7 @@ knowledge worklists, not the number of selected leaf slots.
|
||||||
|
|
||||||
Calls without the expected artifact remain supported for structural/semantic
|
Calls without the expected artifact remain supported for structural/semantic
|
||||||
validation, but cannot certify that a composed review covered its selection.
|
validation, but cannot certify that a composed review covered its selection.
|
||||||
|
They also cannot bind nested leaves to the host's accepted outputs.
|
||||||
They still reject duplicate leaf IDs and returned-and-skipped conflicts.
|
They still reject duplicate leaf IDs and returned-and-skipped conflicts.
|
||||||
|
|
||||||
## Runner-owned choices
|
## Runner-owned choices
|
||||||
|
|
|
||||||
|
|
@ -78,6 +78,14 @@ skill's version, and record any input-incompatible exclusions with
|
||||||
`reason: "not-applicable"`. Do not derive this artifact from leaf reports or
|
`reason: "not-applicable"`. Do not derive this artifact from leaf reports or
|
||||||
change it merely because execution later runs out of budget. Pass its path as
|
change it merely because execution later runs out of budget. Pass its path as
|
||||||
`-ExpectedCompositionPath` for final super-skill validation.
|
`-ExpectedCompositionPath` for final super-skill validation.
|
||||||
|
Initialize `acceptedResults` to `[]`. After each leaf's acceptance gate, the
|
||||||
|
host saves its exact accepted copy (or host-created failed validation result)
|
||||||
|
in an immutable private file and appends its `id`, `version`, and `reportPath`
|
||||||
|
to that array. The host alone owns these captures; neither workers nor the
|
||||||
|
composing model may write them. Never derive them from composed `sub-results`.
|
||||||
|
Keep the pre-dispatch selection and exclusions unchanged. Final validation
|
||||||
|
requires every nested leaf to match its captured JSON content exactly and
|
||||||
|
every captured result to be included. Property order is immaterial.
|
||||||
|
|
||||||
## Action
|
## Action
|
||||||
|
|
||||||
|
|
|
||||||
29
skills/do.md
29
skills/do.md
|
|
@ -228,16 +228,31 @@ scope and the leaf's recorded set of fully retrieved article paths. Pass
|
||||||
a super-skill's rolled-up report. Prepare that private artifact before leaf
|
a super-skill's rolled-up report. Prepare that private artifact before leaf
|
||||||
dispatch, after layer resolution and input compatibility checks. It contains
|
dispatch, after layer resolution and input compatibility checks. It contains
|
||||||
`superSkill` (`id`, `version`), ordered selected `subSkills` (each with `id`,
|
`superSkill` (`id`, `version`), ordered selected `subSkills` (each with `id`,
|
||||||
`version`), and `skipped` (each with `id`, `version`, `reason`). Reasons are
|
`version`), `skipped` (each with `id`, `version`, `reason`), and an initially
|
||||||
`configuration` or `not-applicable`; budget exhaustion is not a skip reason.
|
empty `acceptedResults` array. Reasons are `configuration` or `not-applicable`;
|
||||||
|
budget exhaustion is not a skip reason.
|
||||||
Additional resolver metadata may be retained in the artifact, not the report.
|
Additional resolver metadata may be retained in the artifact, not the report.
|
||||||
|
After each leaf passes its acceptance gate, the host saves the exact accepted
|
||||||
|
copy in a private immutable file and appends an `acceptedResults` entry with
|
||||||
|
`id`, `version`, and `reportPath`. Capture host-created failed validation
|
||||||
|
results the same way. Paths may be absolute or relative to the composition
|
||||||
|
artifact's directory. Capture the normalized accepted copy when normalization
|
||||||
|
was permitted, not the invalid raw return. Do not expose these files or write
|
||||||
|
access to the artifact to leaf workers or the composing model. Only the host
|
||||||
|
may append captures; the pre-dispatch selection and exclusions remain fixed.
|
||||||
The validator binds the super-skill and leaf identities and versions, checks
|
The validator binds the super-skill and leaf identities and versions, checks
|
||||||
selected order, and requires exact agreement on exclusions. Every returned
|
selected order, and requires exact agreement on exclusions. Each nested leaf
|
||||||
leaf must be unique; a selected leaf cannot be reclassified as skipped by the
|
must exactly match its host-captured accepted JSON content, ignoring object
|
||||||
report. Without the artifact, validation remains structural and semantic but
|
property order but preserving array order, types, values, and field presence.
|
||||||
|
Every captured leaf must be included; uncaptured or altered leaves are invalid.
|
||||||
|
Every returned leaf must be unique; a selected leaf cannot be reclassified as skipped by the
|
||||||
|
report. When selected leaves are missing, `outcome-reason` must name every
|
||||||
|
missing ID exactly and top-level `from-sub-skill: "agent"` findings are forbidden.
|
||||||
|
Without the artifact, validation remains structural and semantic but
|
||||||
cannot prove composition completeness, selected versions, order, or legitimate
|
cannot prove composition completeness, selected versions, order, or legitimate
|
||||||
exclusions. Duplicate or both returned-and-skipped leaf IDs are invalid even
|
exclusions, nor bind leaves to accepted host outputs. Duplicate or both
|
||||||
without the artifact. Never derive the expected composition from model output.
|
returned-and-skipped leaf IDs are invalid even without the artifact. Never
|
||||||
|
derive the expected composition from model output.
|
||||||
Pass
|
Pass
|
||||||
`-AllowBoundedNormalization` only when the host preserves the immutable raw
|
`-AllowBoundedNormalization` only when the host preserves the immutable raw
|
||||||
payload and records `removedRanges` in private telemetry as required above.
|
payload and records `removedRanges` in private telemetry as required above.
|
||||||
|
|
|
||||||
|
|
@ -294,6 +294,18 @@ try {
|
||||||
}
|
}
|
||||||
|
|
||||||
$compositionPath = Join-Path $tmp 'composition.json'
|
$compositionPath = Join-Path $tmp 'composition.json'
|
||||||
|
function Save-AcceptedLeafReports {
|
||||||
|
param([object[]] $LeafReports)
|
||||||
|
|
||||||
|
foreach ($leafReport in $LeafReports) {
|
||||||
|
$leafPath = Join-Path $tmp "$([guid]::NewGuid()).json"
|
||||||
|
Set-Content -LiteralPath $leafPath -Value ($leafReport | ConvertTo-Json -Depth 100) -Encoding utf8NoBOM
|
||||||
|
$accepted = & $validator -ReportPath $leafPath -BCQualityRoot $Root
|
||||||
|
Assert-True (-not $accepted.normalized) 'host captures a validated leaf report'
|
||||||
|
@{ id = $leafReport.skill.id; version = $leafReport.skill.version; reportPath = $leafPath }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
$expectedComposition = [ordered]@{
|
$expectedComposition = [ordered]@{
|
||||||
superSkill = @{ id = 'al-code-review'; version = 1 }
|
superSkill = @{ id = 'al-code-review'; version = 1 }
|
||||||
subSkills = @(
|
subSkills = @(
|
||||||
|
|
@ -301,6 +313,7 @@ try {
|
||||||
@{ id = 'al-security-review'; version = 1 }
|
@{ id = 'al-security-review'; version = 1 }
|
||||||
)
|
)
|
||||||
skipped = @()
|
skipped = @()
|
||||||
|
acceptedResults = @(Save-AcceptedLeafReports @($completedLeaf, $completedSecurityLeaf))
|
||||||
}
|
}
|
||||||
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
Set-Content -LiteralPath $reportPath -Value ($validSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
Set-Content -LiteralPath $reportPath -Value ($validSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
|
|
@ -308,6 +321,9 @@ try {
|
||||||
-ExpectedCompositionPath $compositionPath
|
-ExpectedCompositionPath $compositionPath
|
||||||
Assert-True (-not $acceptedBoundSuper.normalized) 'complete composition matches the expected worklist'
|
Assert-True (-not $acceptedBoundSuper.normalized) 'complete composition matches the expected worklist'
|
||||||
|
|
||||||
|
$incompleteComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$incompleteComposition.acceptedResults = @($incompleteComposition.acceptedResults[0])
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($incompleteComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
$missingLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
$missingLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
$missingLeafReport.'sub-results' = @($completedLeaf)
|
$missingLeafReport.'sub-results' = @($completedLeaf)
|
||||||
$missingLeafReport.summary.coverage.'worklist-size' = 1
|
$missingLeafReport.summary.coverage.'worklist-size' = 1
|
||||||
|
|
@ -340,6 +356,103 @@ try {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
$genericMissingReason = $missingLeafReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$genericMissingReason.'outcome-reason' = 'Budget expired.'
|
||||||
|
Assert-CompositionReport $genericMissingReason '*SUPER_MISSING_LEAF_REASON*'
|
||||||
|
$genericMissingReason.'outcome-reason' = 'prefix-al-security-review-suffix was not evaluated.'
|
||||||
|
Assert-CompositionReport $genericMissingReason '*SUPER_MISSING_LEAF_REASON*'
|
||||||
|
|
||||||
|
foreach ($fabricatedOutcome in 'completed', 'not-applicable', 'no-knowledge') {
|
||||||
|
$fabricatedLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$fabricatedLeafReport.'sub-results'[1].outcome = $fabricatedOutcome
|
||||||
|
if ($fabricatedOutcome -cne 'completed') {
|
||||||
|
$fabricatedLeafReport.'sub-results'[1].summary.coverage.'worklist-size' = 0
|
||||||
|
$fabricatedLeafReport.'sub-results'[1].summary.coverage.'items-evaluated' = 0
|
||||||
|
$fabricatedLeafReport.summary.coverage.'worklist-size' = 1
|
||||||
|
$fabricatedLeafReport.summary.coverage.'items-evaluated' = 1
|
||||||
|
}
|
||||||
|
Assert-CompositionReport $fabricatedLeafReport '*SUPER_LEAF_NOT_ACCEPTED*'
|
||||||
|
}
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
|
Assert-CompositionReport $missingLeafReport '*SUPER_ACCEPTED_LEAF_MISSING*'
|
||||||
|
$alteredLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$alteredLeafReport.'sub-results'[1].summary.coverage.'worklist-size' = 2
|
||||||
|
$alteredLeafReport.'sub-results'[1].summary.coverage.'items-evaluated' = 2
|
||||||
|
$alteredLeafReport.summary.coverage.'worklist-size' = 3
|
||||||
|
$alteredLeafReport.summary.coverage.'items-evaluated' = 3
|
||||||
|
Assert-CompositionReport $alteredLeafReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||||
|
$reorderedProperties = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$reorderedProperties.'sub-results'[0].skill = [pscustomobject]@{ version = 1; id = 'al-style-review' }
|
||||||
|
Assert-CompositionReport $reorderedProperties
|
||||||
|
foreach ($alteredOutcome in 'not-applicable', 'no-knowledge') {
|
||||||
|
$alteredOutcomeReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$alteredOutcomeReport.'sub-results'[1].outcome = $alteredOutcome
|
||||||
|
$alteredOutcomeReport.'sub-results'[1].summary.coverage.'worklist-size' = 0
|
||||||
|
$alteredOutcomeReport.'sub-results'[1].summary.coverage.'items-evaluated' = 0
|
||||||
|
$alteredOutcomeReport.summary.coverage.'worklist-size' = 1
|
||||||
|
$alteredOutcomeReport.summary.coverage.'items-evaluated' = 1
|
||||||
|
Assert-CompositionReport $alteredOutcomeReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||||
|
}
|
||||||
|
$relativeCaptureComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
foreach ($capture in $relativeCaptureComposition.acceptedResults) {
|
||||||
|
$capture.reportPath = Split-Path -Leaf $capture.reportPath
|
||||||
|
}
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($relativeCaptureComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
|
Assert-CompositionReport $validSuperReport
|
||||||
|
$uncapturedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$uncapturedComposition.PSObject.Properties.Remove('acceptedResults')
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($uncapturedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
|
Assert-CompositionReport $validSuperReport '*Invalid expected composition*'
|
||||||
|
$duplicateCaptureComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$duplicateCaptureComposition.acceptedResults = @($duplicateCaptureComposition.acceptedResults[0], $duplicateCaptureComposition.acceptedResults[0])
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($duplicateCaptureComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
|
Assert-CompositionReport $validSuperReport '*Invalid expected composition*'
|
||||||
|
|
||||||
|
$capturedFindingLeaf = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$capturedFindingLeaf.findings = @(@{
|
||||||
|
id = 'agent:leaf-issue'
|
||||||
|
severity = 'minor'
|
||||||
|
confidence = 'medium'
|
||||||
|
message = 'Preserve this accepted leaf finding.'
|
||||||
|
references = @()
|
||||||
|
})
|
||||||
|
$secondCapturedFinding = $capturedFindingLeaf.findings[0] | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$secondCapturedFinding.id = 'agent:second-leaf-issue'
|
||||||
|
$capturedFindingLeaf.findings = @($capturedFindingLeaf.findings[0], $secondCapturedFinding)
|
||||||
|
$capturedFindingLeaf.summary.counts.minor = 2
|
||||||
|
$findingComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$findingComposition.acceptedResults = @(Save-AcceptedLeafReports @($capturedFindingLeaf, $completedSecurityLeaf))
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($findingComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
|
$capturedFindingReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$capturedFindingReport.'sub-results'[0] = $capturedFindingLeaf
|
||||||
|
$capturedFindingReport.findings = @($capturedFindingLeaf.findings | ConvertTo-Json -Depth 20 | ConvertFrom-Json)
|
||||||
|
foreach ($finding in $capturedFindingReport.findings) {
|
||||||
|
$finding.id = "al-style-review:$($finding.id)"
|
||||||
|
$finding | Add-Member -NotePropertyName 'from-sub-skill' -NotePropertyValue 'al-style-review'
|
||||||
|
}
|
||||||
|
$capturedFindingReport.summary.counts.minor = 2
|
||||||
|
Assert-CompositionReport $capturedFindingReport
|
||||||
|
$reorderedFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$reorderedFindingReport.'sub-results'[0].findings = @(
|
||||||
|
$reorderedFindingReport.'sub-results'[0].findings[1]
|
||||||
|
$reorderedFindingReport.'sub-results'[0].findings[0]
|
||||||
|
)
|
||||||
|
Assert-CompositionReport $reorderedFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||||
|
$addedLeafFieldReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$addedLeafFieldReport.'sub-results'[0] | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'A composer-added field.'
|
||||||
|
Assert-CompositionReport $addedLeafFieldReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||||
|
$alteredFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$alteredFindingReport.'sub-results'[0].findings[0].message = 'A fabricated replacement message.'
|
||||||
|
$alteredFindingReport.findings[0].message = 'A fabricated replacement message.'
|
||||||
|
Assert-CompositionReport $alteredFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||||
|
$removedFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$removedFindingReport.'sub-results'[0].findings = @()
|
||||||
|
$removedFindingReport.'sub-results'[0].summary.counts.minor = 0
|
||||||
|
$removedFindingReport.findings = @()
|
||||||
|
$removedFindingReport.summary.counts.minor = 0
|
||||||
|
Assert-CompositionReport $removedFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
|
|
||||||
foreach ($case in @(
|
foreach ($case in @(
|
||||||
@{ Pattern = '*SUPER_IDENTITY_MISMATCH*'; Change = { param($candidate) $candidate.skill.id = 'al-other-review' } }
|
@{ Pattern = '*SUPER_IDENTITY_MISMATCH*'; Change = { param($candidate) $candidate.skill.id = 'al-other-review' } }
|
||||||
@{ Pattern = '*SUPER_IDENTITY_MISMATCH*'; Change = { param($candidate) $candidate.skill.version = 2 } }
|
@{ Pattern = '*SUPER_IDENTITY_MISMATCH*'; Change = { param($candidate) $candidate.skill.version = 2 } }
|
||||||
|
|
@ -359,6 +472,9 @@ try {
|
||||||
)
|
)
|
||||||
Assert-CompositionReport $fabricatedSkip '*SUPER_UNEXPECTED_SKIP*'
|
Assert-CompositionReport $fabricatedSkip '*SUPER_UNEXPECTED_SKIP*'
|
||||||
|
|
||||||
|
$emptyAcceptedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$emptyAcceptedComposition.acceptedResults = @()
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($emptyAcceptedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
$noResults = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
$noResults = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
$noResults.'sub-results' = @()
|
$noResults.'sub-results' = @()
|
||||||
$noResults.summary.coverage.'worklist-size' = 0
|
$noResults.summary.coverage.'worklist-size' = 0
|
||||||
|
|
@ -366,9 +482,37 @@ try {
|
||||||
$noResults.outcome = 'not-applicable'
|
$noResults.outcome = 'not-applicable'
|
||||||
Assert-CompositionReport $noResults '*SUPER_OUTCOME_MISMATCH*'
|
Assert-CompositionReport $noResults '*SUPER_OUTCOME_MISMATCH*'
|
||||||
$noResults.outcome = 'failed'
|
$noResults.outcome = 'failed'
|
||||||
$noResults | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'No selected leaf could be evaluated.'
|
$noResults | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'al-style-review and al-security-review could not be evaluated.'
|
||||||
Assert-CompositionReport $noResults
|
Assert-CompositionReport $noResults
|
||||||
|
|
||||||
|
$oneMissingId = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$oneMissingId.'outcome-reason' = 'al-security-review could not be evaluated.'
|
||||||
|
Assert-CompositionReport $oneMissingId '*SUPER_MISSING_LEAF_REASON*'
|
||||||
|
foreach ($baseReport in @($missingLeafReport, $noResults, $validSuperReport)) {
|
||||||
|
$capturedComposition = if ($baseReport.outcome -ceq 'completed') { $expectedComposition } else { $incompleteComposition }
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($capturedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
|
$selfReviewReport = $baseReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$selfReviewReport.findings = @(@{
|
||||||
|
id = 'agent:cross-domain-gap'
|
||||||
|
domain = 'Agent'
|
||||||
|
severity = 'minor'
|
||||||
|
confidence = 'medium'
|
||||||
|
message = 'A cross-domain issue needs attention.'
|
||||||
|
references = @()
|
||||||
|
'from-sub-skill' = 'agent'
|
||||||
|
})
|
||||||
|
$selfReviewReport.summary.counts.minor = 1
|
||||||
|
if ($baseReport.outcome -ceq 'completed') {
|
||||||
|
Assert-CompositionReport $selfReviewReport
|
||||||
|
}
|
||||||
|
elseif ($baseReport.outcome -ceq 'failed') {
|
||||||
|
Assert-CompositionReport $selfReviewReport '*Invalid findings-report JSON or schema*'
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
Assert-CompositionReport $selfReviewReport '*SUPER_AGENT_REVIEW_INCOMPLETE*'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
$allFailed = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
$allFailed = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
$allFailed.'sub-results' = @($completedLeaf, $completedSecurityLeaf) | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
$allFailed.'sub-results' = @($completedLeaf, $completedSecurityLeaf) | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
foreach ($leaf in $allFailed.'sub-results') {
|
foreach ($leaf in $allFailed.'sub-results') {
|
||||||
|
|
@ -376,11 +520,15 @@ try {
|
||||||
$leaf.summary.coverage.'items-evaluated' = 0
|
$leaf.summary.coverage.'items-evaluated' = 0
|
||||||
$leaf | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'Invocation failed.'
|
$leaf | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'Invocation failed.'
|
||||||
}
|
}
|
||||||
|
$failedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$failedComposition.acceptedResults = @(Save-AcceptedLeafReports $allFailed.'sub-results')
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($failedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
Assert-CompositionReport $allFailed
|
Assert-CompositionReport $allFailed
|
||||||
|
|
||||||
$skipComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
$skipComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
$skipComposition.subSkills = @($skipComposition.subSkills[0])
|
$skipComposition.subSkills = @($skipComposition.subSkills[0])
|
||||||
$skipComposition.skipped = @(@{ id = 'al-security-review'; version = 1; reason = 'not-applicable' })
|
$skipComposition.skipped = @(@{ id = 'al-security-review'; version = 1; reason = 'not-applicable' })
|
||||||
|
$skipComposition.acceptedResults = @($skipComposition.acceptedResults[0])
|
||||||
Set-Content -LiteralPath $compositionPath -Value ($skipComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
Set-Content -LiteralPath $compositionPath -Value ($skipComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
$validSkippedReport = $fabricatedSkip | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
$validSkippedReport = $fabricatedSkip | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
$validSkippedReport.outcome = 'completed'
|
$validSkippedReport.outcome = 'completed'
|
||||||
|
|
@ -406,6 +554,7 @@ try {
|
||||||
@{ id = $_.id; version = $_.version; reason = 'configuration' }
|
@{ id = $_.id; version = $_.version; reason = 'configuration' }
|
||||||
})
|
})
|
||||||
$allSkippedComposition.subSkills = @()
|
$allSkippedComposition.subSkills = @()
|
||||||
|
$allSkippedComposition.acceptedResults = @()
|
||||||
Set-Content -LiteralPath $compositionPath -Value ($allSkippedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
Set-Content -LiteralPath $compositionPath -Value ($allSkippedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
$allSkippedReport = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
$allSkippedReport = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
$allSkippedReport.outcome = 'not-applicable'
|
$allSkippedReport.outcome = 'not-applicable'
|
||||||
|
|
@ -464,7 +613,6 @@ try {
|
||||||
$declaredSkill = @($fixtureIndex.skills | Where-Object path -CEQ $declaredPath)[0]
|
$declaredSkill = @($fixtureIndex.skills | Where-Object path -CEQ $declaredPath)[0]
|
||||||
@{ id = $_.id; version = $declaredSkill.version; reason = $_.reason; declaredPath = $declaredPath }
|
@{ id = $_.id; version = $declaredSkill.version; reason = $_.reason; declaredPath = $declaredPath }
|
||||||
})
|
})
|
||||||
Set-Content -LiteralPath $compositionPath -Value ($resolved | ConvertTo-Json -Depth 30) -Encoding utf8NoBOM
|
|
||||||
$resolvedReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
$resolvedReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
$resolvedReport.'sub-results' = @($resolved.subSkills | ForEach-Object {
|
$resolvedReport.'sub-results' = @($resolved.subSkills | ForEach-Object {
|
||||||
$leafReport = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
$leafReport = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
|
@ -477,6 +625,8 @@ try {
|
||||||
$resolvedReport | Add-Member -NotePropertyName 'skipped-sub-skills' -NotePropertyValue @(
|
$resolvedReport | Add-Member -NotePropertyName 'skipped-sub-skills' -NotePropertyValue @(
|
||||||
$resolved.skipped | ForEach-Object { @{ skill = @{ id = $_.id; version = $_.version }; reason = $_.reason } }
|
$resolved.skipped | ForEach-Object { @{ skill = @{ id = $_.id; version = $_.version }; reason = $_.reason } }
|
||||||
)
|
)
|
||||||
|
$resolved | Add-Member -NotePropertyName 'acceptedResults' -NotePropertyValue @(Save-AcceptedLeafReports $resolvedReport.'sub-results')
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($resolved | ConvertTo-Json -Depth 30) -Encoding utf8NoBOM
|
||||||
Assert-CompositionReport $resolvedReport
|
Assert-CompositionReport $resolvedReport
|
||||||
}
|
}
|
||||||
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
|
|
@ -643,6 +793,9 @@ try {
|
||||||
Set-Content -LiteralPath $reportPath -Value ($partialSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
Set-Content -LiteralPath $reportPath -Value ($partialSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
$acceptedPartialSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super
|
$acceptedPartialSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super
|
||||||
Assert-True (-not $acceptedPartialSuper.normalized) 'partial super-skill excludes failed coverage from its rollup'
|
Assert-True (-not $acceptedPartialSuper.normalized) 'partial super-skill excludes failed coverage from its rollup'
|
||||||
|
$partialComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
$partialComposition.acceptedResults = @(Save-AcceptedLeafReports @($completedLeaf, $failedLeaf))
|
||||||
|
Set-Content -LiteralPath $compositionPath -Value ($partialComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||||
Assert-CompositionReport $partialSuperReport
|
Assert-CompositionReport $partialSuperReport
|
||||||
|
|
||||||
$failedLeafLeakage = $partialSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
$failedLeafLeakage = $partialSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||||
|
|
|
||||||
|
|
@ -38,6 +38,7 @@ catch {
|
||||||
$expectedComposition = $null
|
$expectedComposition = $null
|
||||||
$expectedLeaves = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
|
$expectedLeaves = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
|
||||||
$expectedSkips = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
|
$expectedSkips = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
|
||||||
|
$acceptedLeaves = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
|
||||||
if ($ExpectedCompositionPath) {
|
if ($ExpectedCompositionPath) {
|
||||||
if ($SkillKind -cne 'super') {
|
if ($SkillKind -cne 'super') {
|
||||||
throw 'Expected composition is supported only for super-skill reports.'
|
throw 'Expected composition is supported only for super-skill reports.'
|
||||||
|
|
@ -55,10 +56,22 @@ if ($ExpectedCompositionPath) {
|
||||||
}
|
}
|
||||||
$compositionSchema = @{
|
$compositionSchema = @{
|
||||||
type = 'object'
|
type = 'object'
|
||||||
required = @('superSkill', 'subSkills', 'skipped')
|
required = @('superSkill', 'subSkills', 'skipped', 'acceptedResults')
|
||||||
properties = @{
|
properties = @{
|
||||||
superSkill = $identitySchema
|
superSkill = $identitySchema
|
||||||
subSkills = @{ type = 'array'; items = $identitySchema }
|
subSkills = @{ type = 'array'; items = $identitySchema }
|
||||||
|
acceptedResults = @{
|
||||||
|
type = 'array'
|
||||||
|
items = @{
|
||||||
|
type = 'object'
|
||||||
|
required = @('id', 'version', 'reportPath')
|
||||||
|
properties = @{
|
||||||
|
id = $identitySchema.properties.id
|
||||||
|
version = $identitySchema.properties.version
|
||||||
|
reportPath = @{ type = 'string'; minLength = 1 }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
skipped = @{
|
skipped = @{
|
||||||
type = 'array'
|
type = 'array'
|
||||||
items = @{ type = 'object'; required = @('id', 'version', 'reason'); properties = $skipProperties }
|
items = @{ type = 'object'; required = @('id', 'version', 'reason'); properties = $skipProperties }
|
||||||
|
|
@ -84,6 +97,31 @@ if ($ExpectedCompositionPath) {
|
||||||
}
|
}
|
||||||
$expectedSkips.Add([string]$skip.id, $skip)
|
$expectedSkips.Add([string]$skip.id, $skip)
|
||||||
}
|
}
|
||||||
|
$compositionDirectory = Split-Path -Parent (Resolve-Path -LiteralPath $ExpectedCompositionPath).Path
|
||||||
|
foreach ($accepted in @($expectedComposition.acceptedResults)) {
|
||||||
|
if (-not $expectedLeaves.ContainsKey([string]$accepted.id) -or
|
||||||
|
$accepted.version -ne $expectedLeaves[$accepted.id].version -or
|
||||||
|
$acceptedLeaves.ContainsKey([string]$accepted.id)) {
|
||||||
|
throw "Accepted result '$($accepted.id)' must uniquely match a selected leaf and version."
|
||||||
|
}
|
||||||
|
$acceptedPath = if ([IO.Path]::IsPathRooted($accepted.reportPath)) {
|
||||||
|
$accepted.reportPath
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
Join-Path $compositionDirectory $accepted.reportPath
|
||||||
|
}
|
||||||
|
$acceptedRaw = Get-Content -LiteralPath $acceptedPath -Raw
|
||||||
|
if (-not ($acceptedRaw | Test-Json -SchemaFile $schemaPath -ErrorAction Stop)) {
|
||||||
|
throw "Accepted result '$($accepted.id)' does not satisfy the report schema."
|
||||||
|
}
|
||||||
|
$acceptedReport = $acceptedRaw | ConvertFrom-Json -Depth 100
|
||||||
|
if ($acceptedReport.skill.id -cne $accepted.id -or $acceptedReport.skill.version -ne $accepted.version -or
|
||||||
|
$acceptedReport.PSObject.Properties.Name -ccontains 'sub-results' -or
|
||||||
|
$acceptedReport.PSObject.Properties.Name -ccontains 'skipped-sub-skills') {
|
||||||
|
throw "Accepted result '$($accepted.id)' must be a leaf report with the captured identity."
|
||||||
|
}
|
||||||
|
$acceptedLeaves.Add([string]$accepted.id, $acceptedReport)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
catch {
|
catch {
|
||||||
throw "Invalid expected composition: $($_.Exception.Message)"
|
throw "Invalid expected composition: $($_.Exception.Message)"
|
||||||
|
|
@ -105,6 +143,39 @@ function Test-HasProperty {
|
||||||
return $null -ne $Object -and $Object.PSObject.Properties.Name -ccontains $Name
|
return $null -ne $Object -and $Object.PSObject.Properties.Name -ccontains $Name
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function Test-JsonContentEqual {
|
||||||
|
param([object] $First, [object] $Second)
|
||||||
|
|
||||||
|
if ($null -eq $First -or $null -eq $Second) {
|
||||||
|
return $null -eq $First -and $null -eq $Second
|
||||||
|
}
|
||||||
|
if ($First -is [pscustomobject] -or $Second -is [pscustomobject]) {
|
||||||
|
if ($First -isnot [pscustomobject] -or $Second -isnot [pscustomobject] -or
|
||||||
|
@($First.PSObject.Properties).Count -ne @($Second.PSObject.Properties).Count) {
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
foreach ($property in $First.PSObject.Properties) {
|
||||||
|
if (-not (Test-HasProperty $Second $property.Name) -or
|
||||||
|
-not (Test-JsonContentEqual $property.Value $Second.PSObject.Properties[$property.Name].Value)) {
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return $true
|
||||||
|
}
|
||||||
|
if ($First -is [array] -or $Second -is [array]) {
|
||||||
|
if ($First -isnot [array] -or $Second -isnot [array] -or $First.Count -ne $Second.Count) {
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
for ($index = 0; $index -lt $First.Count; $index++) {
|
||||||
|
if (-not (Test-JsonContentEqual $First[$index] $Second[$index])) {
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return $true
|
||||||
|
}
|
||||||
|
return $First.GetType() -eq $Second.GetType() -and $First -ceq $Second
|
||||||
|
}
|
||||||
|
|
||||||
function Get-SourceLineCount {
|
function Get-SourceLineCount {
|
||||||
param([string] $Path)
|
param([string] $Path)
|
||||||
|
|
||||||
|
|
@ -441,6 +512,14 @@ function Get-SemanticErrors {
|
||||||
Add-Error 'SUPER_LEAF_VERSION_MISMATCH' "$ReportPathPrefix.sub-results[$index].skill.version" `
|
Add-Error 'SUPER_LEAF_VERSION_MISMATCH' "$ReportPathPrefix.sub-results[$index].skill.version" `
|
||||||
"Unexpected version for '$($identity.id)'."
|
"Unexpected version for '$($identity.id)'."
|
||||||
}
|
}
|
||||||
|
if (-not $acceptedLeaves.ContainsKey([string]$identity.id)) {
|
||||||
|
Add-Error 'SUPER_LEAF_NOT_ACCEPTED' "$ReportPathPrefix.sub-results[$index]" `
|
||||||
|
"Leaf '$($identity.id)' has no host-captured accepted result."
|
||||||
|
}
|
||||||
|
elseif (-not (Test-JsonContentEqual $subResults[$index] $acceptedLeaves[$identity.id])) {
|
||||||
|
Add-Error 'SUPER_LEAF_CONTENT_MISMATCH' "$ReportPathPrefix.sub-results[$index]" `
|
||||||
|
"Leaf '$($identity.id)' differs from its host-captured accepted result."
|
||||||
|
}
|
||||||
$slot = [Array]::IndexOf($orderedIds, $identity.id)
|
$slot = [Array]::IndexOf($orderedIds, $identity.id)
|
||||||
if ($slot -le $previousSlot) {
|
if ($slot -le $previousSlot) {
|
||||||
Add-Error 'SUPER_SUB_RESULT_ORDER' "$ReportPathPrefix.sub-results[$index].skill" `
|
Add-Error 'SUPER_SUB_RESULT_ORDER' "$ReportPathPrefix.sub-results[$index].skill" `
|
||||||
|
|
@ -451,6 +530,16 @@ function Get-SemanticErrors {
|
||||||
foreach ($leaf in @($expectedComposition.subSkills)) {
|
foreach ($leaf in @($expectedComposition.subSkills)) {
|
||||||
if (-not $producerIds.Contains([string]$leaf.id)) {
|
if (-not $producerIds.Contains([string]$leaf.id)) {
|
||||||
$missingResults++
|
$missingResults++
|
||||||
|
if ($acceptedLeaves.ContainsKey([string]$leaf.id)) {
|
||||||
|
Add-Error 'SUPER_ACCEPTED_LEAF_MISSING' "$ReportPathPrefix.sub-results" `
|
||||||
|
"Host-captured accepted leaf '$($leaf.id)' must be included."
|
||||||
|
}
|
||||||
|
$reason = if (Test-HasProperty $Current 'outcome-reason') { $Current.'outcome-reason' } else { '' }
|
||||||
|
$idPattern = '(?<![a-zA-Z0-9_-])' + [regex]::Escape($leaf.id) + '(?![a-zA-Z0-9_-])'
|
||||||
|
if ($reason -cnotmatch $idPattern) {
|
||||||
|
Add-Error 'SUPER_MISSING_LEAF_REASON' "$ReportPathPrefix.outcome-reason" `
|
||||||
|
"The reason must name missing selected leaf '$($leaf.id)'."
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
foreach ($skip in $skips) {
|
foreach ($skip in $skips) {
|
||||||
|
|
@ -507,6 +596,10 @@ function Get-SemanticErrors {
|
||||||
$finding = $findings[$index]
|
$finding = $findings[$index]
|
||||||
$producerId = [string]$finding.'from-sub-skill'
|
$producerId = [string]$finding.'from-sub-skill'
|
||||||
if ($producerId -ceq 'agent') {
|
if ($producerId -ceq 'agent') {
|
||||||
|
if ($missingResults -gt 0) {
|
||||||
|
Add-Error 'SUPER_AGENT_REVIEW_INCOMPLETE' "$ReportPathPrefix.findings[$index]" `
|
||||||
|
'Super-skill self-review is forbidden while selected leaf results are missing.'
|
||||||
|
}
|
||||||
if (@($finding.references).Count -or
|
if (@($finding.references).Count -or
|
||||||
$finding.id -cnotmatch '^agent:[a-z0-9]+(?:-[a-z0-9]+)*$' -or
|
$finding.id -cnotmatch '^agent:[a-z0-9]+(?:-[a-z0-9]+)*$' -or
|
||||||
-not (Test-HasProperty $finding 'domain') -or
|
-not (Test-HasProperty $finding 'domain') -or
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue