mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-10-05 06:36:55 +01:00
Validate composed reviews against the expected leaf worklist (#204)
* Validate composed reviews against the expected leaf worklist * Bind composed leaf reports to host-accepted results * Preserve literal strings in composed report acceptance
This commit is contained in:
parent
0867171b1a
commit
206c5feff4
6 changed files with 784 additions and 26 deletions
|
|
@ -154,7 +154,8 @@ not a deployable or compiled application.
|
|||
## Before opening a PR
|
||||
|
||||
From your BCQuality checkout, use the existing validators. The Python
|
||||
validator needs Python and PyYAML; the fixture harness needs PowerShell 7.
|
||||
validator needs Python and PyYAML; the fixture harness needs PowerShell 7.5 or later
|
||||
so findings-report parsing preserves timestamp-shaped JSON strings verbatim.
|
||||
If PyYAML is not installed in your development environment, install it with
|
||||
`python -m pip install pyyaml`.
|
||||
|
||||
|
|
|
|||
|
|
@ -53,7 +53,9 @@ only result.
|
|||
with `tools/Resolve-SkillWorklist.ps1`, passing the enabled layers and
|
||||
disabled skill paths from the task context. Execute every resolved leaf as
|
||||
a discrete invocation. Leaves are independent and may be scheduled serially
|
||||
or concurrently.
|
||||
or concurrently. Before dispatch, preserve the final ordered selection and
|
||||
legitimate configuration/input-compatibility exclusions as a private expected
|
||||
composition artifact; see [composition acceptance](#composition-acceptance).
|
||||
5. Capture the exact Task return as the immutable raw audit payload and primary
|
||||
transport. Preserve it unchanged in private artifacts or host logs. Before
|
||||
the full DO acceptance gate, create a normalized candidate only for DO's
|
||||
|
|
@ -61,7 +63,7 @@ only result.
|
|||
telemetry, and accept the candidate only if the entire copy passes the
|
||||
unchanged strict gate. Use `tools/Validate-FindingsReport.ps1`, passing the
|
||||
exact source paths and fully retrieved article paths; pass `-SkillKind super`
|
||||
for the final rolled-up report. The accepted report contains no undeclared
|
||||
and `-ExpectedCompositionPath` for the final rolled-up report. The accepted report contains no undeclared
|
||||
telemetry fields.
|
||||
6. Collect each accepted findings-report into `sub-results` in the declared
|
||||
`sub-skills` order, not completion order. Run the super-skill self-review
|
||||
|
|
@ -74,6 +76,67 @@ The runner must never inspect the diff to skip a review domain. A leaf decides
|
|||
its own task-level applicability and reports `not-applicable` or
|
||||
`no-knowledge`.
|
||||
|
||||
## Composition acceptance
|
||||
|
||||
The findings-report validator requires PowerShell 7.5 or later to preserve
|
||||
literal JSON strings with `ConvertFrom-Json -DateKind String`.
|
||||
|
||||
A report can be internally consistent while omitting a selected review. Bind
|
||||
the final acceptance gate to the host's selection, not just the returned
|
||||
reports. The private JSON input to `-ExpectedCompositionPath` follows the
|
||||
[DO consumer acceptance contract](../skills/do.md#consumer-acceptance-gate).
|
||||
For example, if style is selected and security was disabled:
|
||||
|
||||
```json
|
||||
{
|
||||
"superSkill": { "id": "al-code-review", "version": 1 },
|
||||
"subSkills": [{ "id": "al-style-review", "version": 1 }],
|
||||
"skipped": [{ "id": "al-security-review", "version": 1, "reason": "configuration" }],
|
||||
"acceptedResults": []
|
||||
}
|
||||
```
|
||||
|
||||
Build this artifact from `Resolve-SkillWorklist.ps1` and the input-compatibility
|
||||
decision before dispatch. Resolver `skipped` entries have no version: enrich
|
||||
them from the declared skill's indexed version. Move an input-incompatible
|
||||
selected slot to `skipped` with its selected version and `not-applicable`
|
||||
reason; never use source content or later model output to make that decision.
|
||||
Keep paths, layers, and other resolver metadata if useful for private audit.
|
||||
Do not add the artifact to the findings-report or overwrite it to hide an
|
||||
unfinished invocation. Preserve it with the run's raw payloads.
|
||||
|
||||
Initialize `acceptedResults` before dispatch. After accepting a leaf, save its
|
||||
exact accepted copy (including any permitted normalization) in an immutable
|
||||
host-owned file outside worker/composer write access. Append only its capture
|
||||
to `acceptedResults`, for example:
|
||||
|
||||
```json
|
||||
{"id": "al-style-review", "version": 1, "reportPath": "accepted/style.json"}
|
||||
```
|
||||
|
||||
`reportPath` may be absolute or relative to the composition artifact's directory.
|
||||
Capture a host-created failed validation report the same way. Never construct
|
||||
these captures from the composed `sub-results` or permit the composing model
|
||||
to supply or alter them. Keep the original selection and exclusions unchanged.
|
||||
|
||||
The gate rejects repeated or unexpected leaf IDs, wrong selected versions,
|
||||
reordered results, fabricated or missing exclusions, and uncaptured or altered
|
||||
leaf content. JSON property order is immaterial; array order, field presence,
|
||||
types, and values must match. Every captured leaf must appear in `sub-results`.
|
||||
If selected leaves remain unfinished, the report must be `partial` with a non-failed returned
|
||||
report, otherwise `failed`; name every unfinished leaf ID exactly in
|
||||
`outcome-reason`. Top-level `from-sub-skill: "agent"` findings are rejected
|
||||
while selected leaf results are missing.
|
||||
Do not fabricate leaf reports or use configuration skips for budget exhaustion.
|
||||
Wait for started invocations to finish; omit the self-review if the selected
|
||||
composition remains incomplete. Coverage still sums the non-failed leaf
|
||||
knowledge worklists, not the number of selected leaf slots.
|
||||
|
||||
Calls without the expected artifact remain supported for structural/semantic
|
||||
validation, but cannot certify that a composed review covered its selection.
|
||||
They also cannot bind nested leaves to the host's accepted outputs.
|
||||
They still reject duplicate leaf IDs and returned-and-skipped conflicts.
|
||||
|
||||
## Runner-owned choices
|
||||
|
||||
Keep these settings and behaviors outside BCQuality:
|
||||
|
|
@ -113,6 +176,8 @@ A compatible runner:
|
|||
only part of the review is reliable;
|
||||
- orders `sub-results` by the declared worklist and orders rendered findings
|
||||
deterministically;
|
||||
- validates composition against the host-owned selection, and reports selected
|
||||
leaves left unfinished as incomplete rather than clean or configured away;
|
||||
- calculates top-level severity counts from deduplicated top-level findings,
|
||||
not by summing leaf counts;
|
||||
- preserves knowledge paths verbatim and verifies references before publishing;
|
||||
|
|
|
|||
|
|
@ -71,6 +71,22 @@ Sub-skills that fail either check are not invoked and are recorded in `skipped-s
|
|||
|
||||
The worklist is the list of sub-skills judged relevant by the previous step. Every sub-skill in the worklist will be invoked in the Action step.
|
||||
|
||||
Before dispatch, preserve that selection in a private host-owned expected
|
||||
composition artifact using DO's input contract. Start from the resolver's
|
||||
ordered selection; enrich configuration skips with the declared indexed
|
||||
skill's version, and record any input-incompatible exclusions with
|
||||
`reason: "not-applicable"`. Do not derive this artifact from leaf reports or
|
||||
change it merely because execution later runs out of budget. Pass its path as
|
||||
`-ExpectedCompositionPath` for final super-skill validation.
|
||||
Initialize `acceptedResults` to `[]`. After each leaf's acceptance gate, the
|
||||
host saves its exact accepted copy (or host-created failed validation result)
|
||||
in an immutable private file and appends its `id`, `version`, and `reportPath`
|
||||
to that array. The host alone owns these captures; neither workers nor the
|
||||
composing model may write them. Never derive them from composed `sub-results`.
|
||||
Keep the pre-dispatch selection and exclusions unchanged. Final validation
|
||||
requires every nested leaf to match its captured JSON content exactly and
|
||||
every captured result to be included. Property order is immaterial.
|
||||
|
||||
## Action
|
||||
|
||||
### Execution discipline (mandatory)
|
||||
|
|
@ -85,7 +101,7 @@ The Action step consists of **discrete leaf invocations**, not one combined gene
|
|||
- Do not collapse multiple sub-skills into one shared reasoning step. Each sub-skill has a distinct knowledge subset and a distinct evaluation procedure; sharing one rolled-up scan dilutes per-skill attention and causes leaves to silently underreport (this has been observed in production: leaf skills returned empty `findings[]` while their standalone runs against the same diff produced multiple matches).
|
||||
- The agent self-review pass is its own final iteration. Begin it only after every sub-skill in the worklist has completed and its sub-result is recorded.
|
||||
- Sub-skills are independent: re-walking the diff once per sub-skill is correct and expected. The output schema accommodates this — `sub-results` carries one entry per sub-skill, each a complete findings-report, in the frontmatter `sub-skills` order regardless of completion order.
|
||||
- When isolated calls are unavailable and the current model cannot finish every leaf within its budget, return `partial` with completed `sub-results` and name the first unevaluated sub-skill in `outcome-reason`. Never silently mark the remaining leaves clean.
|
||||
- When the execution budget prevents invoking every selected leaf, wait for started invocations to finish and preserve their accepted `sub-results`. Return `partial` if any returned report is non-`failed`, otherwise `failed`, and name the unfinished leaf IDs in `outcome-reason`. Do not run the self-review on incomplete composition, invent sub-results, or record budget exhaustion as a configured/input-incompatible skip. Never silently mark the remaining leaves clean.
|
||||
|
||||
### Roll up sub-skill findings
|
||||
|
||||
|
|
@ -138,7 +154,9 @@ Calculate `summary.counts` from the final top-level `findings[]`, after failed s
|
|||
Derive `outcome` using the DO rollup rules. `outcome-reason` is populated for `partial` and `failed` and SHOULD summarize per-sub-skill state, for example: *"al-security-review failed (tool timeout); al-performance-review completed."*
|
||||
|
||||
Before emitting the rollup, apply DO's consumer acceptance gate to every nested
|
||||
and top-level finding. A leaf's nested report is its accepted exact return or
|
||||
and top-level finding, and validate the final report with
|
||||
`-SkillKind super -ExpectedCompositionPath <host-owned-json>` against the
|
||||
selection preserved before dispatch. A leaf's nested report is its accepted exact return or
|
||||
its accepted normalized candidate copy; its exact Task return remains the
|
||||
separate immutable raw audit payload. Treat an invalid sub-result as failed and
|
||||
exclude all of its findings from the top-level rollup. Never reconstruct it
|
||||
|
|
|
|||
56
skills/do.md
56
skills/do.md
|
|
@ -224,7 +224,36 @@ as a findings-report, a coordinator or host MUST validate it deterministically:
|
|||
|
||||
Hosts SHOULD execute `tools/Validate-FindingsReport.ps1` with the exact source
|
||||
scope and the leaf's recorded set of fully retrieved article paths. Pass
|
||||
`-SkillKind super` when validating a super-skill's rolled-up report. Pass
|
||||
`-SkillKind super -ExpectedCompositionPath <host-owned-json>` when validating
|
||||
a super-skill's rolled-up report. Prepare that private artifact before leaf
|
||||
dispatch, after layer resolution and input compatibility checks. It contains
|
||||
`superSkill` (`id`, `version`), ordered selected `subSkills` (each with `id`,
|
||||
`version`), `skipped` (each with `id`, `version`, `reason`), and an initially
|
||||
empty `acceptedResults` array. Reasons are `configuration` or `not-applicable`;
|
||||
budget exhaustion is not a skip reason.
|
||||
Additional resolver metadata may be retained in the artifact, not the report.
|
||||
After each leaf passes its acceptance gate, the host saves the exact accepted
|
||||
copy in a private immutable file and appends an `acceptedResults` entry with
|
||||
`id`, `version`, and `reportPath`. Capture host-created failed validation
|
||||
results the same way. Paths may be absolute or relative to the composition
|
||||
artifact's directory. Capture the normalized accepted copy when normalization
|
||||
was permitted, not the invalid raw return. Do not expose these files or write
|
||||
access to the artifact to leaf workers or the composing model. Only the host
|
||||
may append captures; the pre-dispatch selection and exclusions remain fixed.
|
||||
The validator binds the super-skill and leaf identities and versions, checks
|
||||
selected order, and requires exact agreement on exclusions. Each nested leaf
|
||||
must exactly match its host-captured accepted JSON content, ignoring object
|
||||
property order but preserving array order, types, values, and field presence.
|
||||
Every captured leaf must be included; uncaptured or altered leaves are invalid.
|
||||
Every returned leaf must be unique; a selected leaf cannot be reclassified as skipped by the
|
||||
report. When selected leaves are missing, `outcome-reason` must name every
|
||||
missing ID exactly and top-level `from-sub-skill: "agent"` findings are forbidden.
|
||||
Without the artifact, validation remains structural and semantic but
|
||||
cannot prove composition completeness, selected versions, order, or legitimate
|
||||
exclusions, nor bind leaves to accepted host outputs. Duplicate or both
|
||||
returned-and-skipped leaf IDs are invalid even without the artifact. Never
|
||||
derive the expected composition from model output.
|
||||
Pass
|
||||
`-AllowBoundedNormalization` only when the host preserves the immutable raw
|
||||
payload and records `removedRanges` in private telemetry as required above.
|
||||
|
||||
|
|
@ -330,7 +359,7 @@ Omit `suggested-code` only when the appropriate fix depends on context the skill
|
|||
- `reference` — the suppressed file (same object shape as `findings[].references`).
|
||||
- `reason` — `layer-precedence` when another layer won under READ's precedence rules; `configuration` when the consumer disabled the file's layer.
|
||||
|
||||
**`sub-results`** — super-skills only. Array of complete findings-reports, one per sub-skill that was invoked (i.e., every sub-skill not listed in `skipped-sub-skills`). Each entry MUST itself conform to this output contract. Entries MUST appear in the worklist's declared order, regardless of invocation or completion order. Leaf skills MUST NOT emit `sub-results`.
|
||||
**`sub-results`** — super-skills only. Array of complete findings-reports, one per invoked sub-skill, with no duplicate skill IDs. Each entry MUST itself conform to this output contract. Entries MUST appear in the worklist's declared order, regardless of invocation or completion order. A selected leaf left uninvoked by budget exhaustion has no fabricated sub-result and is not a configured or input-incompatible skip; its absence requires the incomplete-composition outcome below. Leaf skills MUST NOT emit `sub-results`.
|
||||
|
||||
**`skipped-sub-skills`** — super-skills only. Array of sub-skills that were declared in frontmatter but not invoked. `reason` is `configuration` when the orchestrator disabled the sub-skill, or `not-applicable` when the super-skill's Relevance step ruled it out.
|
||||
|
||||
|
|
@ -356,10 +385,14 @@ orchestrator.
|
|||
|
||||
Each leaf invocation MUST remain a discrete evaluation with its own complete
|
||||
findings-report. An orchestrator MAY execute independent leaves serially or
|
||||
concurrently, but MUST invoke every worklisted leaf, preserve `sub-results` in
|
||||
the declared worklist order, and wait for every invocation to finish before
|
||||
performing any super-skill self-review or final rollup. Scheduling MUST NOT
|
||||
change relevance, coverage, failure, reference-integrity, or output semantics.
|
||||
concurrently, but MUST attempt every worklisted leaf, preserve `sub-results`
|
||||
in the declared worklist order, and wait for every started invocation to
|
||||
finish before final rollup. If its execution budget prevents dispatching
|
||||
remaining leaves, preserve the unfinished selection in the host-owned expected
|
||||
composition and use the incomplete-composition outcome below. Do not perform
|
||||
the super-skill self-review until every selected leaf has returned. Scheduling
|
||||
MUST NOT change relevance, coverage, failure, reference-integrity, or output
|
||||
semantics.
|
||||
|
||||
Orchestrators SHOULD generate `skill-index.json` with
|
||||
`tools/Build-SkillIndex.ps1` instead of parsing Markdown. Each declared
|
||||
|
|
@ -390,7 +423,16 @@ The five required sections still apply. Their meaning shifts from knowledge file
|
|||
|
||||
### Outcome rollup
|
||||
|
||||
A super-skill's `outcome` is derived from its sub-skills' outcomes. Let S be the multiset of sub-skill outcomes for sub-skills in the worklist (skipped sub-skills do not contribute):
|
||||
A super-skill's `outcome` is derived from its selected worklist and returned
|
||||
sub-skills' outcomes. If selected leaves have no returned report, the outcome
|
||||
is `partial` when at least one returned report is non-`failed`, or `failed`
|
||||
when no non-`failed` report is available. It MUST NOT be `completed`,
|
||||
`not-applicable`, or `no-knowledge`. Name unfinished leaf IDs in
|
||||
`outcome-reason`, preserve the valid returned reports, and never invent
|
||||
successful or failed invocations for leaves that were not invoked.
|
||||
|
||||
When every selected leaf has returned, let S be the multiset of their outcomes
|
||||
(skipped sub-skills do not contribute):
|
||||
|
||||
- `failed` — every element of S is `failed`.
|
||||
- `partial` — S contains at least one `partial`, OR S contains at least one `failed` alongside at least one non-`failed` outcome.
|
||||
|
|
|
|||
|
|
@ -269,6 +269,8 @@ try {
|
|||
& $validator -ReportPath $reportPath -BCQualityRoot $Root
|
||||
}
|
||||
|
||||
$completedSecurityLeaf = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$completedSecurityLeaf.skill.id = 'al-security-review'
|
||||
$validSuperReport = [ordered]@{
|
||||
skill = [ordered]@{ id = 'al-code-review'; version = 1 }
|
||||
outcome = 'completed'
|
||||
|
|
@ -278,12 +280,410 @@ try {
|
|||
}
|
||||
findings = @()
|
||||
suppressed = @()
|
||||
'sub-results' = @($completedLeaf, $completedLeaf)
|
||||
'sub-results' = @($completedLeaf, $completedSecurityLeaf)
|
||||
}
|
||||
Set-Content -LiteralPath $reportPath -Value ($validSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$acceptedSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super
|
||||
Assert-True (-not $acceptedSuper.normalized) 'valid super-skill report is accepted'
|
||||
|
||||
$duplicateLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$duplicateLeafReport.'sub-results' = @($completedLeaf, $completedLeaf)
|
||||
Set-Content -LiteralPath $reportPath -Value ($duplicateLeafReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Assert-ThrowsLike -Pattern '*SUPER_DUPLICATE_SUB_RESULT*' -Action {
|
||||
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super
|
||||
}
|
||||
|
||||
$compositionPath = Join-Path $tmp 'composition.json'
|
||||
function Save-AcceptedLeafReports {
|
||||
param([object[]] $LeafReports)
|
||||
|
||||
foreach ($leafReport in $LeafReports) {
|
||||
$leafPath = Join-Path $tmp "$([guid]::NewGuid()).json"
|
||||
Set-Content -LiteralPath $leafPath -Value ($leafReport | ConvertTo-Json -Depth 100) -Encoding utf8NoBOM
|
||||
$accepted = & $validator -ReportPath $leafPath -BCQualityRoot $Root
|
||||
Assert-True (-not $accepted.normalized) 'host captures a validated leaf report'
|
||||
@{ id = $leafReport.skill.id; version = $leafReport.skill.version; reportPath = $leafPath }
|
||||
}
|
||||
}
|
||||
|
||||
$expectedComposition = [ordered]@{
|
||||
superSkill = @{ id = 'al-code-review'; version = 1 }
|
||||
subSkills = @(
|
||||
@{ id = 'al-style-review'; version = 1 }
|
||||
@{ id = 'al-security-review'; version = 1 }
|
||||
)
|
||||
skipped = @()
|
||||
acceptedResults = @(Save-AcceptedLeafReports @($completedLeaf, $completedSecurityLeaf))
|
||||
}
|
||||
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Set-Content -LiteralPath $reportPath -Value ($validSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$acceptedBoundSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super `
|
||||
-ExpectedCompositionPath $compositionPath
|
||||
Assert-True (-not $acceptedBoundSuper.normalized) 'complete composition matches the expected worklist'
|
||||
|
||||
$incompleteComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$incompleteComposition.acceptedResults = @($incompleteComposition.acceptedResults[0])
|
||||
Set-Content -LiteralPath $compositionPath -Value ($incompleteComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$missingLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$missingLeafReport.'sub-results' = @($completedLeaf)
|
||||
$missingLeafReport.summary.coverage.'worklist-size' = 1
|
||||
$missingLeafReport.summary.coverage.'items-evaluated' = 1
|
||||
Set-Content -LiteralPath $reportPath -Value ($missingLeafReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Assert-ThrowsLike -Pattern '*SUPER_OUTCOME_MISMATCH*' -Action {
|
||||
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super -ExpectedCompositionPath $compositionPath
|
||||
}
|
||||
$missingLeafReport.outcome = 'partial'
|
||||
$missingLeafReport | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'al-security-review was not evaluated before the budget expired.'
|
||||
Set-Content -LiteralPath $reportPath -Value ($missingLeafReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$acceptedIncompleteSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super `
|
||||
-ExpectedCompositionPath $compositionPath
|
||||
Assert-True ($acceptedIncompleteSuper.report.outcome -ceq 'partial') 'unfinished selected leaves require a truthful partial outcome'
|
||||
|
||||
function Assert-CompositionReport {
|
||||
param([object] $Candidate, [string] $ErrorPattern)
|
||||
|
||||
Set-Content -LiteralPath $reportPath -Value ($Candidate | ConvertTo-Json -Depth 30) -Encoding utf8NoBOM
|
||||
if ($ErrorPattern) {
|
||||
Assert-ThrowsLike -Pattern $ErrorPattern -Action {
|
||||
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super `
|
||||
-ExpectedCompositionPath $compositionPath -AllowBoundedNormalization
|
||||
}
|
||||
}
|
||||
else {
|
||||
$accepted = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super `
|
||||
-ExpectedCompositionPath $compositionPath
|
||||
Assert-True (-not $accepted.normalized) 'valid expected composition is accepted without repairs'
|
||||
}
|
||||
}
|
||||
|
||||
$genericMissingReason = $missingLeafReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$genericMissingReason.'outcome-reason' = 'Budget expired.'
|
||||
Assert-CompositionReport $genericMissingReason '*SUPER_MISSING_LEAF_REASON*'
|
||||
$genericMissingReason.'outcome-reason' = 'prefix-al-security-review-suffix was not evaluated.'
|
||||
Assert-CompositionReport $genericMissingReason '*SUPER_MISSING_LEAF_REASON*'
|
||||
|
||||
foreach ($fabricatedOutcome in 'completed', 'not-applicable', 'no-knowledge') {
|
||||
$fabricatedLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$fabricatedLeafReport.'sub-results'[1].outcome = $fabricatedOutcome
|
||||
if ($fabricatedOutcome -cne 'completed') {
|
||||
$fabricatedLeafReport.'sub-results'[1].summary.coverage.'worklist-size' = 0
|
||||
$fabricatedLeafReport.'sub-results'[1].summary.coverage.'items-evaluated' = 0
|
||||
$fabricatedLeafReport.summary.coverage.'worklist-size' = 1
|
||||
$fabricatedLeafReport.summary.coverage.'items-evaluated' = 1
|
||||
}
|
||||
Assert-CompositionReport $fabricatedLeafReport '*SUPER_LEAF_NOT_ACCEPTED*'
|
||||
}
|
||||
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Assert-CompositionReport $missingLeafReport '*SUPER_ACCEPTED_LEAF_MISSING*'
|
||||
$alteredLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$alteredLeafReport.'sub-results'[1].summary.coverage.'worklist-size' = 2
|
||||
$alteredLeafReport.'sub-results'[1].summary.coverage.'items-evaluated' = 2
|
||||
$alteredLeafReport.summary.coverage.'worklist-size' = 3
|
||||
$alteredLeafReport.summary.coverage.'items-evaluated' = 3
|
||||
Assert-CompositionReport $alteredLeafReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||
$reorderedProperties = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$reorderedProperties.'sub-results'[0].skill = [pscustomobject]@{ version = 1; id = 'al-style-review' }
|
||||
Assert-CompositionReport $reorderedProperties
|
||||
foreach ($alteredOutcome in 'not-applicable', 'no-knowledge') {
|
||||
$alteredOutcomeReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$alteredOutcomeReport.'sub-results'[1].outcome = $alteredOutcome
|
||||
$alteredOutcomeReport.'sub-results'[1].summary.coverage.'worklist-size' = 0
|
||||
$alteredOutcomeReport.'sub-results'[1].summary.coverage.'items-evaluated' = 0
|
||||
$alteredOutcomeReport.summary.coverage.'worklist-size' = 1
|
||||
$alteredOutcomeReport.summary.coverage.'items-evaluated' = 1
|
||||
Assert-CompositionReport $alteredOutcomeReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||
}
|
||||
$relativeCaptureComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
foreach ($capture in $relativeCaptureComposition.acceptedResults) {
|
||||
$capture.reportPath = Split-Path -Leaf $capture.reportPath
|
||||
}
|
||||
Set-Content -LiteralPath $compositionPath -Value ($relativeCaptureComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Assert-CompositionReport $validSuperReport
|
||||
$uncapturedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$uncapturedComposition.PSObject.Properties.Remove('acceptedResults')
|
||||
Set-Content -LiteralPath $compositionPath -Value ($uncapturedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Assert-CompositionReport $validSuperReport '*Invalid expected composition*'
|
||||
$duplicateCaptureComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$duplicateCaptureComposition.acceptedResults = @($duplicateCaptureComposition.acceptedResults[0], $duplicateCaptureComposition.acceptedResults[0])
|
||||
Set-Content -LiteralPath $compositionPath -Value ($duplicateCaptureComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Assert-CompositionReport $validSuperReport '*Invalid expected composition*'
|
||||
|
||||
$capturedFindingLeaf = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$capturedFindingLeaf.findings = @(@{
|
||||
id = 'agent:leaf-issue'
|
||||
severity = 'minor'
|
||||
confidence = 'medium'
|
||||
message = 'Preserve this accepted leaf finding.'
|
||||
references = @()
|
||||
})
|
||||
$secondCapturedFinding = $capturedFindingLeaf.findings[0] | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$secondCapturedFinding.id = 'agent:second-leaf-issue'
|
||||
$capturedFindingLeaf.findings = @($capturedFindingLeaf.findings[0], $secondCapturedFinding)
|
||||
$capturedFindingLeaf.summary.counts.minor = 2
|
||||
$findingComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$findingComposition.acceptedResults = @(Save-AcceptedLeafReports @($capturedFindingLeaf, $completedSecurityLeaf))
|
||||
Set-Content -LiteralPath $compositionPath -Value ($findingComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$capturedFindingReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$capturedFindingReport.'sub-results'[0] = $capturedFindingLeaf
|
||||
$capturedFindingReport.findings = @($capturedFindingLeaf.findings | ConvertTo-Json -Depth 20 | ConvertFrom-Json)
|
||||
foreach ($finding in $capturedFindingReport.findings) {
|
||||
$finding.id = "al-style-review:$($finding.id)"
|
||||
$finding | Add-Member -NotePropertyName 'from-sub-skill' -NotePropertyValue 'al-style-review'
|
||||
}
|
||||
$capturedFindingReport.summary.counts.minor = 2
|
||||
Assert-CompositionReport $capturedFindingReport
|
||||
$reorderedFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$reorderedFindingReport.'sub-results'[0].findings = @(
|
||||
$reorderedFindingReport.'sub-results'[0].findings[1]
|
||||
$reorderedFindingReport.'sub-results'[0].findings[0]
|
||||
)
|
||||
Assert-CompositionReport $reorderedFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||
$addedLeafFieldReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$addedLeafFieldReport.'sub-results'[0] | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'A composer-added field.'
|
||||
Assert-CompositionReport $addedLeafFieldReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||
$alteredFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$alteredFindingReport.'sub-results'[0].findings[0].message = 'A fabricated replacement message.'
|
||||
$alteredFindingReport.findings[0].message = 'A fabricated replacement message.'
|
||||
Assert-CompositionReport $alteredFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||
$removedFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$removedFindingReport.'sub-results'[0].findings = @()
|
||||
$removedFindingReport.'sub-results'[0].summary.counts.minor = 0
|
||||
$removedFindingReport.findings = @()
|
||||
$removedFindingReport.summary.counts.minor = 0
|
||||
Assert-CompositionReport $removedFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||
|
||||
$capturedCorrectionLeaf = $capturedFindingLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$capturedCorrectionLeaf.findings[0] | Add-Member -NotePropertyName 'suggested-code' -NotePropertyValue 'exit(1);'
|
||||
$correctionComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$correctionComposition.acceptedResults = @(Save-AcceptedLeafReports @($capturedCorrectionLeaf, $completedSecurityLeaf))
|
||||
Set-Content -LiteralPath $compositionPath -Value ($correctionComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$capturedCorrectionReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$capturedCorrectionReport.'sub-results'[0] = $capturedCorrectionLeaf
|
||||
$capturedCorrectionReport.findings[0] | Add-Member -NotePropertyName 'suggested-code' -NotePropertyValue 'exit(1);'
|
||||
Assert-CompositionReport $capturedCorrectionReport
|
||||
$correctionCapturePath = $correctionComposition.acceptedResults[0].reportPath
|
||||
$immutableCorrectionCapture = [IO.File]::ReadAllText($correctionCapturePath)
|
||||
foreach ($codePoint in @(0x0000, 0x00AD, 0x200B, 0xFEFF)) {
|
||||
$alteredCorrectionReport = $capturedCorrectionReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$alteredCorrection = 'ex' + [char]$codePoint + 'it(1);'
|
||||
$alteredCorrectionReport.'sub-results'[0].findings[0].'suggested-code' = $alteredCorrection
|
||||
$alteredCorrectionReport.findings[0].'suggested-code' = $alteredCorrection
|
||||
Assert-CompositionReport $alteredCorrectionReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||
$rolledOnlyCorrectionReport = $capturedCorrectionReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$rolledOnlyCorrectionReport.findings[0].'suggested-code' = $alteredCorrection
|
||||
Assert-CompositionReport $rolledOnlyCorrectionReport '*SUPER_FINDING_MISMATCH*'
|
||||
Assert-True ([string]::Equals([IO.File]::ReadAllText($correctionCapturePath), $immutableCorrectionCapture, [StringComparison]::Ordinal)) `
|
||||
'rejecting altered corrections leaves the immutable host capture unchanged'
|
||||
}
|
||||
$timestampReason = '2026-10-02T09:00:00Z'
|
||||
$timestampLeaf = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$timestampLeaf | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue $timestampReason
|
||||
$timestampComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$timestampComposition.acceptedResults = @(Save-AcceptedLeafReports @($timestampLeaf, $completedSecurityLeaf))
|
||||
Set-Content -LiteralPath $compositionPath -Value ($timestampComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$timestampReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$timestampReport.'sub-results'[0] = $timestampLeaf
|
||||
foreach ($alteredTimestamp in @('2026-10-02T09:00:00.000Z', '2026-10-02T09:00:00+00:00')) {
|
||||
$timestampLeaf.'outcome-reason' = $alteredTimestamp
|
||||
Assert-CompositionReport $timestampReport '*SUPER_LEAF_CONTENT_MISMATCH*'
|
||||
}
|
||||
$timestampLeaf.'outcome-reason' = $timestampReason
|
||||
Set-Content -LiteralPath $reportPath -Value ($timestampReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$acceptedTimestampReport = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super `
|
||||
-ExpectedCompositionPath $compositionPath
|
||||
$acceptedReason = $acceptedTimestampReport.report.'sub-results'[0].'outcome-reason'
|
||||
Assert-True ($acceptedReason -is [string] -and [string]::Equals($acceptedReason, $timestampReason, [StringComparison]::Ordinal)) `
|
||||
'accepted timestamp-shaped JSON text remains the original literal string'
|
||||
$normalizedTimestampReport = $validReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$normalizedTimestampReport | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue '2026-10-02T09:00:00.000Z'
|
||||
$normalizedTimestampReport.findings[0].location.range.'start-line' = 1
|
||||
Set-Content -LiteralPath $reportPath -Value ($normalizedTimestampReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$acceptedNormalizedTimestamp = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SourceRoot $tmp `
|
||||
-SourcePaths $sourcePath -RetrievedArticlePaths $articlePath -AllowBoundedNormalization
|
||||
Assert-True $acceptedNormalizedTimestamp.normalized 'bounded normalization still applies to an eligible range'
|
||||
Assert-True ($acceptedNormalizedTimestamp.report.'outcome-reason' -is [string] -and
|
||||
[string]::Equals($acceptedNormalizedTimestamp.report.'outcome-reason', $normalizedTimestampReport.'outcome-reason', [StringComparison]::Ordinal)) `
|
||||
'bounded normalization preserves unrelated timestamp-shaped text exactly'
|
||||
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
|
||||
foreach ($case in @(
|
||||
@{ Pattern = '*SUPER_IDENTITY_MISMATCH*'; Change = { param($candidate) $candidate.skill.id = 'al-other-review' } }
|
||||
@{ Pattern = '*SUPER_IDENTITY_MISMATCH*'; Change = { param($candidate) $candidate.skill.version = 2 } }
|
||||
@{ Pattern = '*SUPER_LEAF_VERSION_MISMATCH*'; Change = { param($candidate) $candidate.'sub-results'[0].skill.version = 2 } }
|
||||
@{ Pattern = '*SUPER_UNEXPECTED_SUB_RESULT*'; Change = { param($candidate) $candidate.'sub-results'[0].skill.id = 'al-other-review' } }
|
||||
@{ Pattern = '*SUPER_SUB_RESULT_ORDER*'; Change = { param($candidate) $candidate.'sub-results' = @($candidate.'sub-results'[1], $candidate.'sub-results'[0]) } }
|
||||
@{ Pattern = '*SUPER_DUPLICATE_SUB_RESULT*'; Change = { param($candidate) $candidate.'sub-results' = @($candidate.'sub-results'[0], $candidate.'sub-results'[0]) } }
|
||||
)) {
|
||||
$candidate = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
& $case.Change $candidate
|
||||
Assert-CompositionReport $candidate $case.Pattern
|
||||
}
|
||||
|
||||
$fabricatedSkip = $missingLeafReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$fabricatedSkip | Add-Member -NotePropertyName 'skipped-sub-skills' -NotePropertyValue @(
|
||||
@{ skill = @{ id = 'al-security-review'; version = 1 }; reason = 'configuration' }
|
||||
)
|
||||
Assert-CompositionReport $fabricatedSkip '*SUPER_UNEXPECTED_SKIP*'
|
||||
|
||||
$emptyAcceptedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$emptyAcceptedComposition.acceptedResults = @()
|
||||
Set-Content -LiteralPath $compositionPath -Value ($emptyAcceptedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$noResults = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$noResults.'sub-results' = @()
|
||||
$noResults.summary.coverage.'worklist-size' = 0
|
||||
$noResults.summary.coverage.'items-evaluated' = 0
|
||||
$noResults.outcome = 'not-applicable'
|
||||
Assert-CompositionReport $noResults '*SUPER_OUTCOME_MISMATCH*'
|
||||
$noResults.outcome = 'failed'
|
||||
$noResults | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'al-style-review and al-security-review could not be evaluated.'
|
||||
Assert-CompositionReport $noResults
|
||||
|
||||
$oneMissingId = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$oneMissingId.'outcome-reason' = 'al-security-review could not be evaluated.'
|
||||
Assert-CompositionReport $oneMissingId '*SUPER_MISSING_LEAF_REASON*'
|
||||
foreach ($baseReport in @($missingLeafReport, $noResults, $validSuperReport)) {
|
||||
$capturedComposition = if ($baseReport.outcome -ceq 'completed') { $expectedComposition } else { $incompleteComposition }
|
||||
Set-Content -LiteralPath $compositionPath -Value ($capturedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$selfReviewReport = $baseReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$selfReviewReport.findings = @(@{
|
||||
id = 'agent:cross-domain-gap'
|
||||
domain = 'Agent'
|
||||
severity = 'minor'
|
||||
confidence = 'medium'
|
||||
message = 'A cross-domain issue needs attention.'
|
||||
references = @()
|
||||
'from-sub-skill' = 'agent'
|
||||
})
|
||||
$selfReviewReport.summary.counts.minor = 1
|
||||
if ($baseReport.outcome -ceq 'completed') {
|
||||
Assert-CompositionReport $selfReviewReport
|
||||
}
|
||||
elseif ($baseReport.outcome -ceq 'failed') {
|
||||
Assert-CompositionReport $selfReviewReport '*Invalid findings-report JSON or schema*'
|
||||
}
|
||||
else {
|
||||
Assert-CompositionReport $selfReviewReport '*SUPER_AGENT_REVIEW_INCOMPLETE*'
|
||||
}
|
||||
}
|
||||
|
||||
$allFailed = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$allFailed.'sub-results' = @($completedLeaf, $completedSecurityLeaf) | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
foreach ($leaf in $allFailed.'sub-results') {
|
||||
$leaf.outcome = 'failed'
|
||||
$leaf.summary.coverage.'items-evaluated' = 0
|
||||
$leaf | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'Invocation failed.'
|
||||
}
|
||||
$failedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$failedComposition.acceptedResults = @(Save-AcceptedLeafReports $allFailed.'sub-results')
|
||||
Set-Content -LiteralPath $compositionPath -Value ($failedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Assert-CompositionReport $allFailed
|
||||
|
||||
$skipComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$skipComposition.subSkills = @($skipComposition.subSkills[0])
|
||||
$skipComposition.skipped = @(@{ id = 'al-security-review'; version = 1; reason = 'not-applicable' })
|
||||
$skipComposition.acceptedResults = @($skipComposition.acceptedResults[0])
|
||||
Set-Content -LiteralPath $compositionPath -Value ($skipComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$validSkippedReport = $fabricatedSkip | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$validSkippedReport.outcome = 'completed'
|
||||
$validSkippedReport.PSObject.Properties.Remove('outcome-reason')
|
||||
$validSkippedReport.'skipped-sub-skills'[0].reason = 'not-applicable'
|
||||
Assert-CompositionReport $validSkippedReport
|
||||
Assert-CompositionReport $missingLeafReport '*SUPER_SKIP_MISSING*'
|
||||
$wrongSkip = $validSkippedReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$wrongSkip.'skipped-sub-skills'[0].reason = 'configuration'
|
||||
Assert-CompositionReport $wrongSkip '*SUPER_SKIP_MISMATCH*'
|
||||
$wrongSkip.'skipped-sub-skills'[0].reason = 'not-applicable'
|
||||
$wrongSkip.'skipped-sub-skills'[0].skill.version = 2
|
||||
Assert-CompositionReport $wrongSkip '*SUPER_SKIP_MISMATCH*'
|
||||
$duplicateSkip = $validSkippedReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$duplicateSkip.'skipped-sub-skills' = @($duplicateSkip.'skipped-sub-skills'[0], $duplicateSkip.'skipped-sub-skills'[0])
|
||||
Assert-CompositionReport $duplicateSkip '*SUPER_SKIP_CONFLICT*'
|
||||
$returnedAndSkipped = $validSkippedReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$returnedAndSkipped.'skipped-sub-skills'[0].skill.id = 'al-style-review'
|
||||
Assert-CompositionReport $returnedAndSkipped '*SUPER_SKIP_CONFLICT*'
|
||||
|
||||
$allSkippedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$allSkippedComposition.skipped = @($allSkippedComposition.subSkills | ForEach-Object {
|
||||
@{ id = $_.id; version = $_.version; reason = 'configuration' }
|
||||
})
|
||||
$allSkippedComposition.subSkills = @()
|
||||
$allSkippedComposition.acceptedResults = @()
|
||||
Set-Content -LiteralPath $compositionPath -Value ($allSkippedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$allSkippedReport = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$allSkippedReport.outcome = 'not-applicable'
|
||||
$allSkippedReport.PSObject.Properties.Remove('outcome-reason')
|
||||
$allSkippedReport | Add-Member -NotePropertyName 'skipped-sub-skills' -NotePropertyValue @(
|
||||
$allSkippedComposition.skipped | ForEach-Object { @{ skill = @{ id = $_.id; version = $_.version }; reason = $_.reason } }
|
||||
)
|
||||
Assert-CompositionReport $allSkippedReport
|
||||
|
||||
$invalidComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$invalidComposition.skipped = @(@{ id = 'al-style-review'; version = 1; reason = 'configuration' })
|
||||
Set-Content -LiteralPath $compositionPath -Value ($invalidComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Assert-CompositionReport $validSuperReport '*Invalid expected composition*'
|
||||
$invalidComposition.skipped = @()
|
||||
$invalidComposition.subSkills[0].version = '1'
|
||||
Set-Content -LiteralPath $compositionPath -Value ($invalidComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Assert-CompositionReport $validSuperReport '*Invalid expected composition*'
|
||||
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
|
||||
$layerFixtureRoot = Join-Path $tmp 'layered-skills'
|
||||
foreach ($layer in 'microsoft', 'community', 'custom') {
|
||||
$layerDirectory = Join-Path $layerFixtureRoot $layer
|
||||
New-Item -ItemType Directory -Path $layerDirectory -Force | Out-Null
|
||||
$sourceSkills = Join-Path $Root "$layer/skills"
|
||||
if (Test-Path -LiteralPath $sourceSkills -PathType Container) {
|
||||
Copy-Item -LiteralPath $sourceSkills -Destination (Join-Path $layerDirectory 'skills') -Recurse
|
||||
}
|
||||
}
|
||||
$customSkillDirectory = Join-Path $layerFixtureRoot 'custom/skills/review'
|
||||
New-Item -ItemType Directory -Path $customSkillDirectory -Force | Out-Null
|
||||
$customSkillPath = 'custom/skills/review/company-style-review.md'
|
||||
$styleSkillText = Get-Content -LiteralPath (Join-Path $Root 'microsoft/skills/review/al-style-review.md') -Raw
|
||||
Set-Content -LiteralPath (Join-Path $layerFixtureRoot $customSkillPath) `
|
||||
-Value ($styleSkillText -replace '(?m)^version: 1\r?$', 'version: 7') -Encoding utf8NoBOM
|
||||
$fixtureIndexPath = Join-Path $tmp 'layered-skill-index.json'
|
||||
& (Join-Path $Root 'tools/Build-SkillIndex.ps1') -BCQualityRoot $layerFixtureRoot -IndexPath $fixtureIndexPath | Out-Null
|
||||
$fixtureIndex = Get-Content -LiteralPath $fixtureIndexPath -Raw | ConvertFrom-Json
|
||||
foreach ($selection in @(
|
||||
@{ Disabled = @(); ExpectedLayer = 'custom'; ExpectedVersion = 7 }
|
||||
@{ Disabled = @($customSkillPath); ExpectedLayer = 'microsoft'; ExpectedVersion = 1 }
|
||||
@{ Disabled = @($customSkillPath, 'microsoft/skills/review/al-style-review.md'); ExpectedLayer = $null }
|
||||
)) {
|
||||
$resolved = & (Join-Path $Root 'tools/Resolve-SkillWorklist.ps1') -BCQualityRoot $layerFixtureRoot `
|
||||
-IndexPath $fixtureIndexPath -SuperSkillPath 'microsoft/skills/review/al-code-review.md' `
|
||||
-DisabledSkills $selection.Disabled
|
||||
$styleSlots = @($resolved.subSkills | Where-Object id -CEQ 'al-style-review')
|
||||
if ($selection.ExpectedLayer) {
|
||||
Assert-True ($styleSlots.Count -eq 1 -and $styleSlots[0].layer -ceq $selection.ExpectedLayer -and
|
||||
$styleSlots[0].version -eq $selection.ExpectedVersion) 'resolver-selected override or fallback is authoritative'
|
||||
}
|
||||
else {
|
||||
Assert-True ($styleSlots.Count -eq 0) 'fully disabled slot is not selected'
|
||||
}
|
||||
$resolved.skipped = @($resolved.skipped | ForEach-Object {
|
||||
$declaredPath = $_.declaredPath
|
||||
$declaredSkill = @($fixtureIndex.skills | Where-Object path -CEQ $declaredPath)[0]
|
||||
@{ id = $_.id; version = $declaredSkill.version; reason = $_.reason; declaredPath = $declaredPath }
|
||||
})
|
||||
$resolvedReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$resolvedReport.'sub-results' = @($resolved.subSkills | ForEach-Object {
|
||||
$leafReport = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$leafReport.skill.id = $_.id
|
||||
$leafReport.skill.version = $_.version
|
||||
$leafReport
|
||||
})
|
||||
$resolvedReport.summary.coverage.'worklist-size' = $resolved.subSkills.Count
|
||||
$resolvedReport.summary.coverage.'items-evaluated' = $resolved.subSkills.Count
|
||||
$resolvedReport | Add-Member -NotePropertyName 'skipped-sub-skills' -NotePropertyValue @(
|
||||
$resolved.skipped | ForEach-Object { @{ skill = @{ id = $_.id; version = $_.version }; reason = $_.reason } }
|
||||
)
|
||||
$resolved | Add-Member -NotePropertyName 'acceptedResults' -NotePropertyValue @(Save-AcceptedLeafReports $resolvedReport.'sub-results')
|
||||
Set-Content -LiteralPath $compositionPath -Value ($resolved | ConvertTo-Json -Depth 30) -Encoding utf8NoBOM
|
||||
Assert-CompositionReport $resolvedReport
|
||||
}
|
||||
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
|
||||
$styleFindingLeaf = $validReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$securityFindingLeaf = $validReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$securityFindingLeaf.skill.id = 'al-security-review'
|
||||
|
|
@ -446,6 +846,10 @@ try {
|
|||
Set-Content -LiteralPath $reportPath -Value ($partialSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
$acceptedPartialSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super
|
||||
Assert-True (-not $acceptedPartialSuper.normalized) 'partial super-skill excludes failed coverage from its rollup'
|
||||
$partialComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$partialComposition.acceptedResults = @(Save-AcceptedLeafReports @($completedLeaf, $failedLeaf))
|
||||
Set-Content -LiteralPath $compositionPath -Value ($partialComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
|
||||
Assert-CompositionReport $partialSuperReport
|
||||
|
||||
$failedLeafLeakage = $partialSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
|
||||
$failedLeafLeakage.summary.counts.minor = 1
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
#Requires -Version 7.5
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Validates a BCQuality findings-report against its structural and semantic contract.
|
||||
|
|
@ -12,6 +13,7 @@ param(
|
|||
[string[]] $RetrievedArticlePaths = @(),
|
||||
[ValidateSet('leaf', 'super')]
|
||||
[string] $SkillKind = 'leaf',
|
||||
[string] $ExpectedCompositionPath,
|
||||
[switch] $AllowBoundedNormalization
|
||||
)
|
||||
|
||||
|
|
@ -28,12 +30,105 @@ try {
|
|||
if (-not ($raw | Test-Json -SchemaFile $schemaPath -ErrorAction Stop)) {
|
||||
throw 'Report does not satisfy schemas/findings-report.schema.json.'
|
||||
}
|
||||
$report = $raw | ConvertFrom-Json -Depth 100
|
||||
$report = $raw | ConvertFrom-Json -Depth 100 -DateKind String
|
||||
}
|
||||
catch {
|
||||
throw "Invalid findings-report JSON or schema: $($_.Exception.Message)"
|
||||
}
|
||||
|
||||
$expectedComposition = $null
|
||||
$expectedLeaves = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
|
||||
$expectedSkips = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
|
||||
$acceptedLeaves = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
|
||||
if ($ExpectedCompositionPath) {
|
||||
if ($SkillKind -cne 'super') {
|
||||
throw 'Expected composition is supported only for super-skill reports.'
|
||||
}
|
||||
$contractSchema = Get-Content -LiteralPath $schemaPath -Raw | ConvertFrom-Json -AsHashtable
|
||||
$identitySchema = @{
|
||||
type = 'object'
|
||||
required = @('id', 'version')
|
||||
properties = $contractSchema.definitions.skillReference.properties
|
||||
}
|
||||
$skipProperties = @{
|
||||
id = $identitySchema.properties.id
|
||||
version = $identitySchema.properties.version
|
||||
reason = @{ enum = @('configuration', 'not-applicable') }
|
||||
}
|
||||
$compositionSchema = @{
|
||||
type = 'object'
|
||||
required = @('superSkill', 'subSkills', 'skipped', 'acceptedResults')
|
||||
properties = @{
|
||||
superSkill = $identitySchema
|
||||
subSkills = @{ type = 'array'; items = $identitySchema }
|
||||
acceptedResults = @{
|
||||
type = 'array'
|
||||
items = @{
|
||||
type = 'object'
|
||||
required = @('id', 'version', 'reportPath')
|
||||
properties = @{
|
||||
id = $identitySchema.properties.id
|
||||
version = $identitySchema.properties.version
|
||||
reportPath = @{ type = 'string'; minLength = 1 }
|
||||
}
|
||||
}
|
||||
}
|
||||
skipped = @{
|
||||
type = 'array'
|
||||
items = @{ type = 'object'; required = @('id', 'version', 'reason'); properties = $skipProperties }
|
||||
}
|
||||
}
|
||||
} | ConvertTo-Json -Depth 20
|
||||
try {
|
||||
$compositionRaw = Get-Content -LiteralPath $ExpectedCompositionPath -Raw
|
||||
if (-not ($compositionRaw | Test-Json -Schema $compositionSchema -ErrorAction Stop)) {
|
||||
throw 'Expected composition does not satisfy its input contract.'
|
||||
}
|
||||
$expectedComposition = $compositionRaw | ConvertFrom-Json -Depth 100 -DateKind String
|
||||
$expectedIds = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal)
|
||||
foreach ($leaf in @($expectedComposition.subSkills)) {
|
||||
if (-not $expectedIds.Add([string]$leaf.id)) {
|
||||
throw "Duplicate expected skill id '$($leaf.id)'."
|
||||
}
|
||||
$expectedLeaves.Add([string]$leaf.id, $leaf)
|
||||
}
|
||||
foreach ($skip in @($expectedComposition.skipped)) {
|
||||
if (-not $expectedIds.Add([string]$skip.id)) {
|
||||
throw "Duplicate or selected-and-skipped expected skill id '$($skip.id)'."
|
||||
}
|
||||
$expectedSkips.Add([string]$skip.id, $skip)
|
||||
}
|
||||
$compositionDirectory = Split-Path -Parent (Resolve-Path -LiteralPath $ExpectedCompositionPath).Path
|
||||
foreach ($accepted in @($expectedComposition.acceptedResults)) {
|
||||
if (-not $expectedLeaves.ContainsKey([string]$accepted.id) -or
|
||||
$accepted.version -ne $expectedLeaves[$accepted.id].version -or
|
||||
$acceptedLeaves.ContainsKey([string]$accepted.id)) {
|
||||
throw "Accepted result '$($accepted.id)' must uniquely match a selected leaf and version."
|
||||
}
|
||||
$acceptedPath = if ([IO.Path]::IsPathRooted($accepted.reportPath)) {
|
||||
$accepted.reportPath
|
||||
}
|
||||
else {
|
||||
Join-Path $compositionDirectory $accepted.reportPath
|
||||
}
|
||||
$acceptedRaw = Get-Content -LiteralPath $acceptedPath -Raw
|
||||
if (-not ($acceptedRaw | Test-Json -SchemaFile $schemaPath -ErrorAction Stop)) {
|
||||
throw "Accepted result '$($accepted.id)' does not satisfy the report schema."
|
||||
}
|
||||
$acceptedReport = $acceptedRaw | ConvertFrom-Json -Depth 100 -DateKind String
|
||||
if ($acceptedReport.skill.id -cne $accepted.id -or $acceptedReport.skill.version -ne $accepted.version -or
|
||||
$acceptedReport.PSObject.Properties.Name -ccontains 'sub-results' -or
|
||||
$acceptedReport.PSObject.Properties.Name -ccontains 'skipped-sub-skills') {
|
||||
throw "Accepted result '$($accepted.id)' must be a leaf report with the captured identity."
|
||||
}
|
||||
$acceptedLeaves.Add([string]$accepted.id, $acceptedReport)
|
||||
}
|
||||
}
|
||||
catch {
|
||||
throw "Invalid expected composition: $($_.Exception.Message)"
|
||||
}
|
||||
}
|
||||
|
||||
$retrieved = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal)
|
||||
foreach ($path in $RetrievedArticlePaths) {
|
||||
$retrieved.Add($path) | Out-Null
|
||||
|
|
@ -49,6 +144,43 @@ function Test-HasProperty {
|
|||
return $null -ne $Object -and $Object.PSObject.Properties.Name -ccontains $Name
|
||||
}
|
||||
|
||||
function Test-JsonContentEqual {
|
||||
param([object] $First, [object] $Second)
|
||||
|
||||
if ($null -eq $First -or $null -eq $Second) {
|
||||
return $null -eq $First -and $null -eq $Second
|
||||
}
|
||||
if ($First -is [pscustomobject] -or $Second -is [pscustomobject]) {
|
||||
if ($First -isnot [pscustomobject] -or $Second -isnot [pscustomobject] -or
|
||||
@($First.PSObject.Properties).Count -ne @($Second.PSObject.Properties).Count) {
|
||||
return $false
|
||||
}
|
||||
foreach ($property in $First.PSObject.Properties) {
|
||||
if (-not (Test-HasProperty $Second $property.Name) -or
|
||||
-not (Test-JsonContentEqual $property.Value $Second.PSObject.Properties[$property.Name].Value)) {
|
||||
return $false
|
||||
}
|
||||
}
|
||||
return $true
|
||||
}
|
||||
if ($First -is [array] -or $Second -is [array]) {
|
||||
if ($First -isnot [array] -or $Second -isnot [array] -or $First.Count -ne $Second.Count) {
|
||||
return $false
|
||||
}
|
||||
for ($index = 0; $index -lt $First.Count; $index++) {
|
||||
if (-not (Test-JsonContentEqual $First[$index] $Second[$index])) {
|
||||
return $false
|
||||
}
|
||||
}
|
||||
return $true
|
||||
}
|
||||
if ($First -is [string] -or $Second -is [string]) {
|
||||
return $First -is [string] -and $Second -is [string] -and
|
||||
[string]::Equals($First, $Second, [StringComparison]::Ordinal)
|
||||
}
|
||||
return $First.GetType() -eq $Second.GetType() -and $First -ceq $Second
|
||||
}
|
||||
|
||||
function Get-SourceLineCount {
|
||||
param([string] $Path)
|
||||
|
||||
|
|
@ -79,8 +211,14 @@ function Get-SemanticErrors {
|
|||
}
|
||||
|
||||
function Get-DerivedSuperOutcome {
|
||||
param([object[]] $SubResults)
|
||||
param([object[]] $SubResults, [int] $MissingResults = 0)
|
||||
|
||||
if ($MissingResults -gt 0) {
|
||||
if (@($SubResults | Where-Object outcome -CNE 'failed').Count) {
|
||||
return 'partial'
|
||||
}
|
||||
return 'failed'
|
||||
}
|
||||
if (-not $SubResults.Count) {
|
||||
return 'not-applicable'
|
||||
}
|
||||
|
|
@ -138,9 +276,10 @@ function Get-SemanticErrors {
|
|||
$firstHasCode = Test-HasProperty $First 'suggested-code'
|
||||
$secondHasCode = Test-HasProperty $Second 'suggested-code'
|
||||
if ($firstHasCode -or $secondHasCode) {
|
||||
return $firstHasCode -and $secondHasCode -and $First.'suggested-code' -ceq $Second.'suggested-code'
|
||||
return $firstHasCode -and $secondHasCode -and
|
||||
[string]::Equals($First.'suggested-code', $Second.'suggested-code', [StringComparison]::Ordinal)
|
||||
}
|
||||
return $First.message -ceq $Second.message
|
||||
return [string]::Equals($First.message, $Second.message, [StringComparison]::Ordinal)
|
||||
}
|
||||
|
||||
function Test-ReferencesInclude {
|
||||
|
|
@ -181,7 +320,7 @@ function Get-SemanticErrors {
|
|||
$RolledFinding.id -cne $expectedId -or
|
||||
$RolledFinding.severity -cne $LeafFinding.severity -or
|
||||
$RolledFinding.confidence -cne $LeafFinding.confidence -or
|
||||
$RolledFinding.message -cne $LeafFinding.message -or
|
||||
-not [string]::Equals($RolledFinding.message, $LeafFinding.message, [StringComparison]::Ordinal) -or
|
||||
$rolledReferences.Count -ne $leafReferences.Count -or
|
||||
-not (Test-ReferencesInclude $rolledReferences $leafReferences)) {
|
||||
return $false
|
||||
|
|
@ -190,7 +329,7 @@ function Get-SemanticErrors {
|
|||
$rolledHasProperty = Test-HasProperty $RolledFinding $name
|
||||
$leafHasProperty = Test-HasProperty $LeafFinding $name
|
||||
if ($rolledHasProperty -ne $leafHasProperty -or
|
||||
($rolledHasProperty -and $RolledFinding.$name -cne $LeafFinding.$name)) {
|
||||
($rolledHasProperty -and -not [string]::Equals($RolledFinding.$name, $LeafFinding.$name, [StringComparison]::Ordinal))) {
|
||||
return $false
|
||||
}
|
||||
}
|
||||
|
|
@ -206,7 +345,7 @@ function Get-SemanticErrors {
|
|||
$sameCorrection = Test-SameCorrection $RolledFinding $LeafFinding
|
||||
$correctionsConflict = (Test-HasProperty $RolledFinding 'suggested-code') -and
|
||||
(Test-HasProperty $LeafFinding 'suggested-code') -and
|
||||
$RolledFinding.'suggested-code' -cne $LeafFinding.'suggested-code'
|
||||
-not [string]::Equals($RolledFinding.'suggested-code', $LeafFinding.'suggested-code', [StringComparison]::Ordinal)
|
||||
$explicitCrossRuleMerge = $leafReferences.Count -and
|
||||
$rolledReferences.Count -gt $leafReferences.Count -and
|
||||
-not $correctionsConflict -and
|
||||
|
|
@ -342,20 +481,105 @@ function Get-SemanticErrors {
|
|||
|
||||
if ($CurrentSkillKind -ceq 'super' -and $hasSubResults) {
|
||||
$subResults = @($Current.'sub-results')
|
||||
$producerIds = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal)
|
||||
for ($index = 0; $index -lt $subResults.Count; $index++) {
|
||||
if (-not $producerIds.Add([string]$subResults[$index].skill.id)) {
|
||||
Add-Error 'SUPER_DUPLICATE_SUB_RESULT' "$ReportPathPrefix.sub-results[$index].skill.id" `
|
||||
'A leaf may appear only once in sub-results.'
|
||||
}
|
||||
Test-Report $subResults[$index] "$ReportPathPrefix.sub-results[$index]" 'leaf'
|
||||
}
|
||||
|
||||
$expectedOutcome = Get-DerivedSuperOutcome $subResults
|
||||
$skippedIds = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal)
|
||||
$skips = if ($hasSkippedSubSkills) { @($Current.'skipped-sub-skills') } else { @() }
|
||||
foreach ($skip in $skips) {
|
||||
if (-not $skippedIds.Add([string]$skip.skill.id) -or $producerIds.Contains([string]$skip.skill.id)) {
|
||||
Add-Error 'SUPER_SKIP_CONFLICT' "$ReportPathPrefix.skipped-sub-skills" `
|
||||
"Skill '$($skip.skill.id)' is duplicated or both returned and skipped."
|
||||
}
|
||||
}
|
||||
|
||||
$missingResults = 0
|
||||
if ($null -ne $expectedComposition) {
|
||||
if ($Current.skill.id -cne $expectedComposition.superSkill.id -or
|
||||
$Current.skill.version -ne $expectedComposition.superSkill.version) {
|
||||
Add-Error 'SUPER_IDENTITY_MISMATCH' "$ReportPathPrefix.skill" 'Super-skill identity differs from expected composition.'
|
||||
}
|
||||
$previousSlot = -1
|
||||
$orderedIds = @($expectedComposition.subSkills | ForEach-Object { $_.id })
|
||||
for ($index = 0; $index -lt $subResults.Count; $index++) {
|
||||
$identity = $subResults[$index].skill
|
||||
if (-not $expectedLeaves.ContainsKey([string]$identity.id)) {
|
||||
Add-Error 'SUPER_UNEXPECTED_SUB_RESULT' "$ReportPathPrefix.sub-results[$index].skill" `
|
||||
"Skill '$($identity.id)' was not selected."
|
||||
continue
|
||||
}
|
||||
if ($identity.version -ne $expectedLeaves[$identity.id].version) {
|
||||
Add-Error 'SUPER_LEAF_VERSION_MISMATCH' "$ReportPathPrefix.sub-results[$index].skill.version" `
|
||||
"Unexpected version for '$($identity.id)'."
|
||||
}
|
||||
if (-not $acceptedLeaves.ContainsKey([string]$identity.id)) {
|
||||
Add-Error 'SUPER_LEAF_NOT_ACCEPTED' "$ReportPathPrefix.sub-results[$index]" `
|
||||
"Leaf '$($identity.id)' has no host-captured accepted result."
|
||||
}
|
||||
elseif (-not (Test-JsonContentEqual $subResults[$index] $acceptedLeaves[$identity.id])) {
|
||||
Add-Error 'SUPER_LEAF_CONTENT_MISMATCH' "$ReportPathPrefix.sub-results[$index]" `
|
||||
"Leaf '$($identity.id)' differs from its host-captured accepted result."
|
||||
}
|
||||
$slot = [Array]::IndexOf($orderedIds, $identity.id)
|
||||
if ($slot -le $previousSlot) {
|
||||
Add-Error 'SUPER_SUB_RESULT_ORDER' "$ReportPathPrefix.sub-results[$index].skill" `
|
||||
'Sub-results must preserve the selected worklist order.'
|
||||
}
|
||||
$previousSlot = $slot
|
||||
}
|
||||
foreach ($leaf in @($expectedComposition.subSkills)) {
|
||||
if (-not $producerIds.Contains([string]$leaf.id)) {
|
||||
$missingResults++
|
||||
if ($acceptedLeaves.ContainsKey([string]$leaf.id)) {
|
||||
Add-Error 'SUPER_ACCEPTED_LEAF_MISSING' "$ReportPathPrefix.sub-results" `
|
||||
"Host-captured accepted leaf '$($leaf.id)' must be included."
|
||||
}
|
||||
$reason = if (Test-HasProperty $Current 'outcome-reason') { $Current.'outcome-reason' } else { '' }
|
||||
$idPattern = '(?<![a-zA-Z0-9_-])' + [regex]::Escape($leaf.id) + '(?![a-zA-Z0-9_-])'
|
||||
if ($reason -cnotmatch $idPattern) {
|
||||
Add-Error 'SUPER_MISSING_LEAF_REASON' "$ReportPathPrefix.outcome-reason" `
|
||||
"The reason must name missing selected leaf '$($leaf.id)'."
|
||||
}
|
||||
}
|
||||
}
|
||||
foreach ($skip in $skips) {
|
||||
if (-not $expectedSkips.ContainsKey([string]$skip.skill.id)) {
|
||||
Add-Error 'SUPER_UNEXPECTED_SKIP' "$ReportPathPrefix.skipped-sub-skills" `
|
||||
"Skill '$($skip.skill.id)' was not excluded by the coordinator."
|
||||
continue
|
||||
}
|
||||
$expectedSkip = $expectedSkips[$skip.skill.id]
|
||||
if ($skip.skill.version -ne $expectedSkip.version -or $skip.reason -cne $expectedSkip.reason) {
|
||||
Add-Error 'SUPER_SKIP_MISMATCH' "$ReportPathPrefix.skipped-sub-skills" `
|
||||
"Skip identity or reason differs for '$($skip.skill.id)'."
|
||||
}
|
||||
}
|
||||
foreach ($skip in @($expectedComposition.skipped)) {
|
||||
if (-not $skippedIds.Contains([string]$skip.id)) {
|
||||
Add-Error 'SUPER_SKIP_MISSING' "$ReportPathPrefix.skipped-sub-skills" `
|
||||
"Expected exclusion '$($skip.id)' is missing."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$expectedOutcome = Get-DerivedSuperOutcome $subResults $missingResults
|
||||
if ($Current.outcome -cne $expectedOutcome) {
|
||||
Add-Error 'SUPER_OUTCOME_MISMATCH' "$ReportPathPrefix.outcome" "Expected '$expectedOutcome' from sub-results."
|
||||
}
|
||||
|
||||
$includedSubResults = @($subResults | Where-Object outcome -CNE 'failed')
|
||||
$expectedWorklistSize = ($includedSubResults | Measure-Object -Property { $_.summary.coverage.'worklist-size' } -Sum).Sum
|
||||
$expectedItemsEvaluated = ($includedSubResults | Measure-Object -Property { $_.summary.coverage.'items-evaluated' } -Sum).Sum
|
||||
if ($null -eq $expectedWorklistSize) { $expectedWorklistSize = 0 }
|
||||
if ($null -eq $expectedItemsEvaluated) { $expectedItemsEvaluated = 0 }
|
||||
$expectedWorklistSize = 0
|
||||
$expectedItemsEvaluated = 0
|
||||
foreach ($subResult in $includedSubResults) {
|
||||
$expectedWorklistSize += $subResult.summary.coverage.'worklist-size'
|
||||
$expectedItemsEvaluated += $subResult.summary.coverage.'items-evaluated'
|
||||
}
|
||||
if ($worklistSize -ne $expectedWorklistSize -or $itemsEvaluated -ne $expectedItemsEvaluated) {
|
||||
Add-Error 'SUPER_COVERAGE_MISMATCH' "$ReportPathPrefix.summary.coverage" `
|
||||
"Expected worklist-size $expectedWorklistSize and items-evaluated $expectedItemsEvaluated from non-failed sub-results."
|
||||
|
|
@ -378,6 +602,10 @@ function Get-SemanticErrors {
|
|||
$finding = $findings[$index]
|
||||
$producerId = [string]$finding.'from-sub-skill'
|
||||
if ($producerId -ceq 'agent') {
|
||||
if ($missingResults -gt 0) {
|
||||
Add-Error 'SUPER_AGENT_REVIEW_INCOMPLETE' "$ReportPathPrefix.findings[$index]" `
|
||||
'Super-skill self-review is forbidden while selected leaf results are missing.'
|
||||
}
|
||||
if (@($finding.references).Count -or
|
||||
$finding.id -cnotmatch '^agent:[a-z0-9]+(?:-[a-z0-9]+)*$' -or
|
||||
-not (Test-HasProperty $finding 'domain') -or
|
||||
|
|
@ -496,7 +724,7 @@ if ($errors.Count -and $AllowBoundedNormalization) {
|
|||
$otherErrors = @($errors | Where-Object Code -CNE 'RANGE_START_MISMATCH')
|
||||
$rangeErrors = @($errors | Where-Object Code -CEQ 'RANGE_START_MISMATCH')
|
||||
if (-not $otherErrors.Count -and $rangeErrors.Count) {
|
||||
$candidate = $report | ConvertTo-Json -Depth 100 | ConvertFrom-Json -Depth 100
|
||||
$candidate = $report | ConvertTo-Json -Depth 100 | ConvertFrom-Json -Depth 100 -DateKind String
|
||||
$eligible = $true
|
||||
foreach ($finding in @($candidate.findings)) {
|
||||
if (-not (Test-HasProperty $finding 'location') -or
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue