Microsoft/bcquality
1
0
Fork 0
mirror of https://github.com/microsoft/BCQuality.git synced 2026-10-05 06:36:55 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions

Validate composed reviews against the expected leaf worklist (#204)

Browse source 
* Validate composed reviews against the expected leaf worklist

* Bind composed leaf reports to host-accepted results

* Preserve literal strings in composed report acceptance
This commit is contained in:
Stefano Demiliani 2026-10-02 10:27:27 +02:00 • committed by GitHub
parent 0867171b1a
commit 206c5feff4
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 784 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

3
docs/contributing.md
View file

@ -154,7 +154,8 @@ not a deployable or compiled application.
## Before opening a PR ## Before opening a PR
From your BCQuality checkout, use the existing validators. The Python From your BCQuality checkout, use the existing validators. The Python
validator needs Python and PyYAML; the fixture harness needs PowerShell 7. validator needs Python and PyYAML; the fixture harness needs PowerShell 7.5 or later
so findings-report parsing preserves timestamp-shaped JSON strings verbatim.
If PyYAML is not installed in your development environment, install it with If PyYAML is not installed in your development environment, install it with
`python -m pip install pyyaml`. `python -m pip install pyyaml`.

69
docs/standalone-runner.md
View file

@ -53,7 +53,9 @@ only result.
with `tools/Resolve-SkillWorklist.ps1`, passing the enabled layers and with `tools/Resolve-SkillWorklist.ps1`, passing the enabled layers and
disabled skill paths from the task context. Execute every resolved leaf as disabled skill paths from the task context. Execute every resolved leaf as
a discrete invocation. Leaves are independent and may be scheduled serially a discrete invocation. Leaves are independent and may be scheduled serially
or concurrently. or concurrently. Before dispatch, preserve the final ordered selection and
legitimate configuration/input-compatibility exclusions as a private expected
composition artifact; see [composition acceptance](#composition-acceptance).
5. Capture the exact Task return as the immutable raw audit payload and primary 5. Capture the exact Task return as the immutable raw audit payload and primary
transport. Preserve it unchanged in private artifacts or host logs. Before transport. Preserve it unchanged in private artifacts or host logs. Before
the full DO acceptance gate, create a normalized candidate only for DO's the full DO acceptance gate, create a normalized candidate only for DO's
@ -61,7 +63,7 @@ only result.
telemetry, and accept the candidate only if the entire copy passes the telemetry, and accept the candidate only if the entire copy passes the
unchanged strict gate. Use `tools/Validate-FindingsReport.ps1`, passing the unchanged strict gate. Use `tools/Validate-FindingsReport.ps1`, passing the
exact source paths and fully retrieved article paths; pass `-SkillKind super` exact source paths and fully retrieved article paths; pass `-SkillKind super`
for the final rolled-up report. The accepted report contains no undeclared and `-ExpectedCompositionPath` for the final rolled-up report. The accepted report contains no undeclared
telemetry fields. telemetry fields.
6. Collect each accepted findings-report into `sub-results` in the declared 6. Collect each accepted findings-report into `sub-results` in the declared
`sub-skills` order, not completion order. Run the super-skill self-review `sub-skills` order, not completion order. Run the super-skill self-review
@ -74,6 +76,67 @@ The runner must never inspect the diff to skip a review domain. A leaf decides
its own task-level applicability and reports `not-applicable` or its own task-level applicability and reports `not-applicable` or
`no-knowledge`. `no-knowledge`.
## Composition acceptance
The findings-report validator requires PowerShell 7.5 or later to preserve
literal JSON strings with `ConvertFrom-Json -DateKind String`.
A report can be internally consistent while omitting a selected review. Bind
the final acceptance gate to the host's selection, not just the returned
reports. The private JSON input to `-ExpectedCompositionPath` follows the
[DO consumer acceptance contract](../skills/do.md#consumer-acceptance-gate).
For example, if style is selected and security was disabled:
```json
{
"superSkill": { "id": "al-code-review", "version": 1 },
"subSkills": [{ "id": "al-style-review", "version": 1 }],
"skipped": [{ "id": "al-security-review", "version": 1, "reason": "configuration" }],
"acceptedResults": []
}
```
Build this artifact from `Resolve-SkillWorklist.ps1` and the input-compatibility
decision before dispatch. Resolver `skipped` entries have no version: enrich
them from the declared skill's indexed version. Move an input-incompatible
selected slot to `skipped` with its selected version and `not-applicable`
reason; never use source content or later model output to make that decision.
Keep paths, layers, and other resolver metadata if useful for private audit.
Do not add the artifact to the findings-report or overwrite it to hide an
unfinished invocation. Preserve it with the run's raw payloads.
Initialize `acceptedResults` before dispatch. After accepting a leaf, save its
exact accepted copy (including any permitted normalization) in an immutable
host-owned file outside worker/composer write access. Append only its capture
to `acceptedResults`, for example:
```json
{"id": "al-style-review", "version": 1, "reportPath": "accepted/style.json"}
```
`reportPath` may be absolute or relative to the composition artifact's directory.
Capture a host-created failed validation report the same way. Never construct
these captures from the composed `sub-results` or permit the composing model
to supply or alter them. Keep the original selection and exclusions unchanged.
The gate rejects repeated or unexpected leaf IDs, wrong selected versions,
reordered results, fabricated or missing exclusions, and uncaptured or altered
leaf content. JSON property order is immaterial; array order, field presence,
types, and values must match. Every captured leaf must appear in `sub-results`.
If selected leaves remain unfinished, the report must be `partial` with a non-failed returned
report, otherwise `failed`; name every unfinished leaf ID exactly in
`outcome-reason`. Top-level `from-sub-skill: "agent"` findings are rejected
while selected leaf results are missing.
Do not fabricate leaf reports or use configuration skips for budget exhaustion.
Wait for started invocations to finish; omit the self-review if the selected
composition remains incomplete. Coverage still sums the non-failed leaf
knowledge worklists, not the number of selected leaf slots.
Calls without the expected artifact remain supported for structural/semantic
validation, but cannot certify that a composed review covered its selection.
They also cannot bind nested leaves to the host's accepted outputs.
They still reject duplicate leaf IDs and returned-and-skipped conflicts.
## Runner-owned choices ## Runner-owned choices
Keep these settings and behaviors outside BCQuality: Keep these settings and behaviors outside BCQuality:
@ -113,6 +176,8 @@ A compatible runner:
only part of the review is reliable; only part of the review is reliable;
- orders `sub-results` by the declared worklist and orders rendered findings - orders `sub-results` by the declared worklist and orders rendered findings
deterministically; deterministically;
- validates composition against the host-owned selection, and reports selected
leaves left unfinished as incomplete rather than clean or configured away;
- calculates top-level severity counts from deduplicated top-level findings, - calculates top-level severity counts from deduplicated top-level findings,
not by summing leaf counts; not by summing leaf counts;
- preserves knowledge paths verbatim and verifies references before publishing; - preserves knowledge paths verbatim and verifies references before publishing;

22
microsoft/skills/review/al-code-review.md
View file

@ -71,6 +71,22 @@ Sub-skills that fail either check are not invoked and are recorded in `skipped-s
The worklist is the list of sub-skills judged relevant by the previous step. Every sub-skill in the worklist will be invoked in the Action step. The worklist is the list of sub-skills judged relevant by the previous step. Every sub-skill in the worklist will be invoked in the Action step.
Before dispatch, preserve that selection in a private host-owned expected
composition artifact using DO's input contract. Start from the resolver's
ordered selection; enrich configuration skips with the declared indexed
skill's version, and record any input-incompatible exclusions with
`reason: "not-applicable"`. Do not derive this artifact from leaf reports or
change it merely because execution later runs out of budget. Pass its path as
`-ExpectedCompositionPath` for final super-skill validation.
Initialize `acceptedResults` to `[]`. After each leaf's acceptance gate, the
host saves its exact accepted copy (or host-created failed validation result)
in an immutable private file and appends its `id`, `version`, and `reportPath`
to that array. The host alone owns these captures; neither workers nor the
composing model may write them. Never derive them from composed `sub-results`.
Keep the pre-dispatch selection and exclusions unchanged. Final validation
requires every nested leaf to match its captured JSON content exactly and
every captured result to be included. Property order is immaterial.
## Action ## Action
### Execution discipline (mandatory) ### Execution discipline (mandatory)
@ -85,7 +101,7 @@ The Action step consists of **discrete leaf invocations**, not one combined gene
- Do not collapse multiple sub-skills into one shared reasoning step. Each sub-skill has a distinct knowledge subset and a distinct evaluation procedure; sharing one rolled-up scan dilutes per-skill attention and causes leaves to silently underreport (this has been observed in production: leaf skills returned empty `findings[]` while their standalone runs against the same diff produced multiple matches). - Do not collapse multiple sub-skills into one shared reasoning step. Each sub-skill has a distinct knowledge subset and a distinct evaluation procedure; sharing one rolled-up scan dilutes per-skill attention and causes leaves to silently underreport (this has been observed in production: leaf skills returned empty `findings[]` while their standalone runs against the same diff produced multiple matches).
- The agent self-review pass is its own final iteration. Begin it only after every sub-skill in the worklist has completed and its sub-result is recorded. - The agent self-review pass is its own final iteration. Begin it only after every sub-skill in the worklist has completed and its sub-result is recorded.
- Sub-skills are independent: re-walking the diff once per sub-skill is correct and expected. The output schema accommodates this — `sub-results` carries one entry per sub-skill, each a complete findings-report, in the frontmatter `sub-skills` order regardless of completion order. - Sub-skills are independent: re-walking the diff once per sub-skill is correct and expected. The output schema accommodates this — `sub-results` carries one entry per sub-skill, each a complete findings-report, in the frontmatter `sub-skills` order regardless of completion order.
- When isolated calls are unavailable and the current model cannot finish every leaf within its budget, return `partial` with completed `sub-results` and name the first unevaluated sub-skill in `outcome-reason`. Never silently mark the remaining leaves clean. - When the execution budget prevents invoking every selected leaf, wait for started invocations to finish and preserve their accepted `sub-results`. Return `partial` if any returned report is non-`failed`, otherwise `failed`, and name the unfinished leaf IDs in `outcome-reason`. Do not run the self-review on incomplete composition, invent sub-results, or record budget exhaustion as a configured/input-incompatible skip. Never silently mark the remaining leaves clean.
### Roll up sub-skill findings ### Roll up sub-skill findings
@ -138,7 +154,9 @@ Calculate `summary.counts` from the final top-level `findings[]`, after failed s
Derive `outcome` using the DO rollup rules. `outcome-reason` is populated for `partial` and `failed` and SHOULD summarize per-sub-skill state, for example: *"al-security-review failed (tool timeout); al-performance-review completed."* Derive `outcome` using the DO rollup rules. `outcome-reason` is populated for `partial` and `failed` and SHOULD summarize per-sub-skill state, for example: *"al-security-review failed (tool timeout); al-performance-review completed."*
Before emitting the rollup, apply DO's consumer acceptance gate to every nested Before emitting the rollup, apply DO's consumer acceptance gate to every nested
and top-level finding. A leaf's nested report is its accepted exact return or and top-level finding, and validate the final report with
`-SkillKind super -ExpectedCompositionPath <host-owned-json>` against the
selection preserved before dispatch. A leaf's nested report is its accepted exact return or
its accepted normalized candidate copy; its exact Task return remains the its accepted normalized candidate copy; its exact Task return remains the
separate immutable raw audit payload. Treat an invalid sub-result as failed and separate immutable raw audit payload. Treat an invalid sub-result as failed and
exclude all of its findings from the top-level rollup. Never reconstruct it exclude all of its findings from the top-level rollup. Never reconstruct it

56
skills/do.md
View file

@ -224,7 +224,36 @@ as a findings-report, a coordinator or host MUST validate it deterministically:
Hosts SHOULD execute `tools/Validate-FindingsReport.ps1` with the exact source Hosts SHOULD execute `tools/Validate-FindingsReport.ps1` with the exact source
scope and the leaf's recorded set of fully retrieved article paths. Pass scope and the leaf's recorded set of fully retrieved article paths. Pass
`-SkillKind super` when validating a super-skill's rolled-up report. Pass `-SkillKind super -ExpectedCompositionPath <host-owned-json>` when validating
a super-skill's rolled-up report. Prepare that private artifact before leaf
dispatch, after layer resolution and input compatibility checks. It contains
`superSkill` (`id`, `version`), ordered selected `subSkills` (each with `id`,
`version`), `skipped` (each with `id`, `version`, `reason`), and an initially
empty `acceptedResults` array. Reasons are `configuration` or `not-applicable`;
budget exhaustion is not a skip reason.
Additional resolver metadata may be retained in the artifact, not the report.
After each leaf passes its acceptance gate, the host saves the exact accepted
copy in a private immutable file and appends an `acceptedResults` entry with
`id`, `version`, and `reportPath`. Capture host-created failed validation
results the same way. Paths may be absolute or relative to the composition
artifact's directory. Capture the normalized accepted copy when normalization
was permitted, not the invalid raw return. Do not expose these files or write
access to the artifact to leaf workers or the composing model. Only the host
may append captures; the pre-dispatch selection and exclusions remain fixed.
The validator binds the super-skill and leaf identities and versions, checks
selected order, and requires exact agreement on exclusions. Each nested leaf
must exactly match its host-captured accepted JSON content, ignoring object
property order but preserving array order, types, values, and field presence.
Every captured leaf must be included; uncaptured or altered leaves are invalid.
Every returned leaf must be unique; a selected leaf cannot be reclassified as skipped by the
report. When selected leaves are missing, `outcome-reason` must name every
missing ID exactly and top-level `from-sub-skill: "agent"` findings are forbidden.
Without the artifact, validation remains structural and semantic but
cannot prove composition completeness, selected versions, order, or legitimate
exclusions, nor bind leaves to accepted host outputs. Duplicate or both
returned-and-skipped leaf IDs are invalid even without the artifact. Never
derive the expected composition from model output.
Pass
`-AllowBoundedNormalization` only when the host preserves the immutable raw `-AllowBoundedNormalization` only when the host preserves the immutable raw
payload and records `removedRanges` in private telemetry as required above. payload and records `removedRanges` in private telemetry as required above.
@ -330,7 +359,7 @@ Omit `suggested-code` only when the appropriate fix depends on context the skill
- `reference` — the suppressed file (same object shape as `findings[].references`). - `reference` — the suppressed file (same object shape as `findings[].references`).
- `reason` — `layer-precedence` when another layer won under READ's precedence rules; `configuration` when the consumer disabled the file's layer. - `reason` — `layer-precedence` when another layer won under READ's precedence rules; `configuration` when the consumer disabled the file's layer.
**`sub-results`** — super-skills only. Array of complete findings-reports, one per sub-skill that was invoked (i.e., every sub-skill not listed in `skipped-sub-skills`). Each entry MUST itself conform to this output contract. Entries MUST appear in the worklist's declared order, regardless of invocation or completion order. Leaf skills MUST NOT emit `sub-results`. **`sub-results`** — super-skills only. Array of complete findings-reports, one per invoked sub-skill, with no duplicate skill IDs. Each entry MUST itself conform to this output contract. Entries MUST appear in the worklist's declared order, regardless of invocation or completion order. A selected leaf left uninvoked by budget exhaustion has no fabricated sub-result and is not a configured or input-incompatible skip; its absence requires the incomplete-composition outcome below. Leaf skills MUST NOT emit `sub-results`.
**`skipped-sub-skills`** — super-skills only. Array of sub-skills that were declared in frontmatter but not invoked. `reason` is `configuration` when the orchestrator disabled the sub-skill, or `not-applicable` when the super-skill's Relevance step ruled it out. **`skipped-sub-skills`** — super-skills only. Array of sub-skills that were declared in frontmatter but not invoked. `reason` is `configuration` when the orchestrator disabled the sub-skill, or `not-applicable` when the super-skill's Relevance step ruled it out.
@ -356,10 +385,14 @@ orchestrator.
Each leaf invocation MUST remain a discrete evaluation with its own complete Each leaf invocation MUST remain a discrete evaluation with its own complete
findings-report. An orchestrator MAY execute independent leaves serially or findings-report. An orchestrator MAY execute independent leaves serially or
concurrently, but MUST invoke every worklisted leaf, preserve `sub-results` in concurrently, but MUST attempt every worklisted leaf, preserve `sub-results`
the declared worklist order, and wait for every invocation to finish before in the declared worklist order, and wait for every started invocation to
performing any super-skill self-review or final rollup. Scheduling MUST NOT finish before final rollup. If its execution budget prevents dispatching
change relevance, coverage, failure, reference-integrity, or output semantics. remaining leaves, preserve the unfinished selection in the host-owned expected
composition and use the incomplete-composition outcome below. Do not perform
the super-skill self-review until every selected leaf has returned. Scheduling
MUST NOT change relevance, coverage, failure, reference-integrity, or output
semantics.
Orchestrators SHOULD generate `skill-index.json` with Orchestrators SHOULD generate `skill-index.json` with
`tools/Build-SkillIndex.ps1` instead of parsing Markdown. Each declared `tools/Build-SkillIndex.ps1` instead of parsing Markdown. Each declared
@ -390,7 +423,16 @@ The five required sections still apply. Their meaning shifts from knowledge file
### Outcome rollup ### Outcome rollup
A super-skill's `outcome` is derived from its sub-skills' outcomes. Let S be the multiset of sub-skill outcomes for sub-skills in the worklist (skipped sub-skills do not contribute): A super-skill's `outcome` is derived from its selected worklist and returned
sub-skills' outcomes. If selected leaves have no returned report, the outcome
is `partial` when at least one returned report is non-`failed`, or `failed`
when no non-`failed` report is available. It MUST NOT be `completed`,
`not-applicable`, or `no-knowledge`. Name unfinished leaf IDs in
`outcome-reason`, preserve the valid returned reports, and never invent
successful or failed invocations for leaves that were not invoked.
When every selected leaf has returned, let S be the multiset of their outcomes
(skipped sub-skills do not contribute):
- `failed` — every element of S is `failed`. - `failed` — every element of S is `failed`.
- `partial` — S contains at least one `partial`, OR S contains at least one `failed` alongside at least one non-`failed` outcome. - `partial` — S contains at least one `partial`, OR S contains at least one `failed` alongside at least one non-`failed` outcome.

406
tools/Test-ReviewContract.ps1
View file

@ -269,6 +269,8 @@ try {
& $validator -ReportPath $reportPath -BCQualityRoot $Root & $validator -ReportPath $reportPath -BCQualityRoot $Root
} }
$completedSecurityLeaf = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$completedSecurityLeaf.skill.id = 'al-security-review'
$validSuperReport = [ordered]@{ $validSuperReport = [ordered]@{
skill = [ordered]@{ id = 'al-code-review'; version = 1 } skill = [ordered]@{ id = 'al-code-review'; version = 1 }
outcome = 'completed' outcome = 'completed'
@ -278,12 +280,410 @@ try {
} }
findings = @() findings = @()
suppressed = @() suppressed = @()
'sub-results' = @($completedLeaf, $completedLeaf) 'sub-results' = @($completedLeaf, $completedSecurityLeaf)
} }
Set-Content -LiteralPath $reportPath -Value ($validSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM Set-Content -LiteralPath $reportPath -Value ($validSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$acceptedSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super $acceptedSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super
Assert-True (-not $acceptedSuper.normalized) 'valid super-skill report is accepted' Assert-True (-not $acceptedSuper.normalized) 'valid super-skill report is accepted'
$duplicateLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$duplicateLeafReport.'sub-results' = @($completedLeaf, $completedLeaf)
Set-Content -LiteralPath $reportPath -Value ($duplicateLeafReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-ThrowsLike -Pattern '*SUPER_DUPLICATE_SUB_RESULT*' -Action {
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super
}
$compositionPath = Join-Path $tmp 'composition.json'
function Save-AcceptedLeafReports {
param([object[]] $LeafReports)
foreach ($leafReport in $LeafReports) {
$leafPath = Join-Path $tmp "$([guid]::NewGuid()).json"
Set-Content -LiteralPath $leafPath -Value ($leafReport | ConvertTo-Json -Depth 100) -Encoding utf8NoBOM
$accepted = & $validator -ReportPath $leafPath -BCQualityRoot $Root
Assert-True (-not $accepted.normalized) 'host captures a validated leaf report'
@{ id = $leafReport.skill.id; version = $leafReport.skill.version; reportPath = $leafPath }
}
}
$expectedComposition = [ordered]@{
superSkill = @{ id = 'al-code-review'; version = 1 }
subSkills = @(
@{ id = 'al-style-review'; version = 1 }
@{ id = 'al-security-review'; version = 1 }
)
skipped = @()
acceptedResults = @(Save-AcceptedLeafReports @($completedLeaf, $completedSecurityLeaf))
}
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Set-Content -LiteralPath $reportPath -Value ($validSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$acceptedBoundSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super `
-ExpectedCompositionPath $compositionPath
Assert-True (-not $acceptedBoundSuper.normalized) 'complete composition matches the expected worklist'
$incompleteComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$incompleteComposition.acceptedResults = @($incompleteComposition.acceptedResults[0])
Set-Content -LiteralPath $compositionPath -Value ($incompleteComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$missingLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$missingLeafReport.'sub-results' = @($completedLeaf)
$missingLeafReport.summary.coverage.'worklist-size' = 1
$missingLeafReport.summary.coverage.'items-evaluated' = 1
Set-Content -LiteralPath $reportPath -Value ($missingLeafReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-ThrowsLike -Pattern '*SUPER_OUTCOME_MISMATCH*' -Action {
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super -ExpectedCompositionPath $compositionPath
}
$missingLeafReport.outcome = 'partial'
$missingLeafReport | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'al-security-review was not evaluated before the budget expired.'
Set-Content -LiteralPath $reportPath -Value ($missingLeafReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$acceptedIncompleteSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super `
-ExpectedCompositionPath $compositionPath
Assert-True ($acceptedIncompleteSuper.report.outcome -ceq 'partial') 'unfinished selected leaves require a truthful partial outcome'
function Assert-CompositionReport {
param([object] $Candidate, [string] $ErrorPattern)
Set-Content -LiteralPath $reportPath -Value ($Candidate | ConvertTo-Json -Depth 30) -Encoding utf8NoBOM
if ($ErrorPattern) {
Assert-ThrowsLike -Pattern $ErrorPattern -Action {
& $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super `
-ExpectedCompositionPath $compositionPath -AllowBoundedNormalization
}
}
else {
$accepted = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super `
-ExpectedCompositionPath $compositionPath
Assert-True (-not $accepted.normalized) 'valid expected composition is accepted without repairs'
}
}
$genericMissingReason = $missingLeafReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$genericMissingReason.'outcome-reason' = 'Budget expired.'
Assert-CompositionReport $genericMissingReason '*SUPER_MISSING_LEAF_REASON*'
$genericMissingReason.'outcome-reason' = 'prefix-al-security-review-suffix was not evaluated.'
Assert-CompositionReport $genericMissingReason '*SUPER_MISSING_LEAF_REASON*'
foreach ($fabricatedOutcome in 'completed', 'not-applicable', 'no-knowledge') {
$fabricatedLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$fabricatedLeafReport.'sub-results'[1].outcome = $fabricatedOutcome
if ($fabricatedOutcome -cne 'completed') {
$fabricatedLeafReport.'sub-results'[1].summary.coverage.'worklist-size' = 0
$fabricatedLeafReport.'sub-results'[1].summary.coverage.'items-evaluated' = 0
$fabricatedLeafReport.summary.coverage.'worklist-size' = 1
$fabricatedLeafReport.summary.coverage.'items-evaluated' = 1
}
Assert-CompositionReport $fabricatedLeafReport '*SUPER_LEAF_NOT_ACCEPTED*'
}
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-CompositionReport $missingLeafReport '*SUPER_ACCEPTED_LEAF_MISSING*'
$alteredLeafReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$alteredLeafReport.'sub-results'[1].summary.coverage.'worklist-size' = 2
$alteredLeafReport.'sub-results'[1].summary.coverage.'items-evaluated' = 2
$alteredLeafReport.summary.coverage.'worklist-size' = 3
$alteredLeafReport.summary.coverage.'items-evaluated' = 3
Assert-CompositionReport $alteredLeafReport '*SUPER_LEAF_CONTENT_MISMATCH*'
$reorderedProperties = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$reorderedProperties.'sub-results'[0].skill = [pscustomobject]@{ version = 1; id = 'al-style-review' }
Assert-CompositionReport $reorderedProperties
foreach ($alteredOutcome in 'not-applicable', 'no-knowledge') {
$alteredOutcomeReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$alteredOutcomeReport.'sub-results'[1].outcome = $alteredOutcome
$alteredOutcomeReport.'sub-results'[1].summary.coverage.'worklist-size' = 0
$alteredOutcomeReport.'sub-results'[1].summary.coverage.'items-evaluated' = 0
$alteredOutcomeReport.summary.coverage.'worklist-size' = 1
$alteredOutcomeReport.summary.coverage.'items-evaluated' = 1
Assert-CompositionReport $alteredOutcomeReport '*SUPER_LEAF_CONTENT_MISMATCH*'
}
$relativeCaptureComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
foreach ($capture in $relativeCaptureComposition.acceptedResults) {
$capture.reportPath = Split-Path -Leaf $capture.reportPath
}
Set-Content -LiteralPath $compositionPath -Value ($relativeCaptureComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-CompositionReport $validSuperReport
$uncapturedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$uncapturedComposition.PSObject.Properties.Remove('acceptedResults')
Set-Content -LiteralPath $compositionPath -Value ($uncapturedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-CompositionReport $validSuperReport '*Invalid expected composition*'
$duplicateCaptureComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$duplicateCaptureComposition.acceptedResults = @($duplicateCaptureComposition.acceptedResults[0], $duplicateCaptureComposition.acceptedResults[0])
Set-Content -LiteralPath $compositionPath -Value ($duplicateCaptureComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-CompositionReport $validSuperReport '*Invalid expected composition*'
$capturedFindingLeaf = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$capturedFindingLeaf.findings = @(@{
id = 'agent:leaf-issue'
severity = 'minor'
confidence = 'medium'
message = 'Preserve this accepted leaf finding.'
references = @()
})
$secondCapturedFinding = $capturedFindingLeaf.findings[0] | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$secondCapturedFinding.id = 'agent:second-leaf-issue'
$capturedFindingLeaf.findings = @($capturedFindingLeaf.findings[0], $secondCapturedFinding)
$capturedFindingLeaf.summary.counts.minor = 2
$findingComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$findingComposition.acceptedResults = @(Save-AcceptedLeafReports @($capturedFindingLeaf, $completedSecurityLeaf))
Set-Content -LiteralPath $compositionPath -Value ($findingComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$capturedFindingReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$capturedFindingReport.'sub-results'[0] = $capturedFindingLeaf
$capturedFindingReport.findings = @($capturedFindingLeaf.findings | ConvertTo-Json -Depth 20 | ConvertFrom-Json)
foreach ($finding in $capturedFindingReport.findings) {
$finding.id = "al-style-review:$($finding.id)"
$finding | Add-Member -NotePropertyName 'from-sub-skill' -NotePropertyValue 'al-style-review'
}
$capturedFindingReport.summary.counts.minor = 2
Assert-CompositionReport $capturedFindingReport
$reorderedFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$reorderedFindingReport.'sub-results'[0].findings = @(
$reorderedFindingReport.'sub-results'[0].findings[1]
$reorderedFindingReport.'sub-results'[0].findings[0]
)
Assert-CompositionReport $reorderedFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*'
$addedLeafFieldReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$addedLeafFieldReport.'sub-results'[0] | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'A composer-added field.'
Assert-CompositionReport $addedLeafFieldReport '*SUPER_LEAF_CONTENT_MISMATCH*'
$alteredFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$alteredFindingReport.'sub-results'[0].findings[0].message = 'A fabricated replacement message.'
$alteredFindingReport.findings[0].message = 'A fabricated replacement message.'
Assert-CompositionReport $alteredFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*'
$removedFindingReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$removedFindingReport.'sub-results'[0].findings = @()
$removedFindingReport.'sub-results'[0].summary.counts.minor = 0
$removedFindingReport.findings = @()
$removedFindingReport.summary.counts.minor = 0
Assert-CompositionReport $removedFindingReport '*SUPER_LEAF_CONTENT_MISMATCH*'
$capturedCorrectionLeaf = $capturedFindingLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$capturedCorrectionLeaf.findings[0] | Add-Member -NotePropertyName 'suggested-code' -NotePropertyValue 'exit(1);'
$correctionComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$correctionComposition.acceptedResults = @(Save-AcceptedLeafReports @($capturedCorrectionLeaf, $completedSecurityLeaf))
Set-Content -LiteralPath $compositionPath -Value ($correctionComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$capturedCorrectionReport = $capturedFindingReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$capturedCorrectionReport.'sub-results'[0] = $capturedCorrectionLeaf
$capturedCorrectionReport.findings[0] | Add-Member -NotePropertyName 'suggested-code' -NotePropertyValue 'exit(1);'
Assert-CompositionReport $capturedCorrectionReport
$correctionCapturePath = $correctionComposition.acceptedResults[0].reportPath
$immutableCorrectionCapture = [IO.File]::ReadAllText($correctionCapturePath)
foreach ($codePoint in @(0x0000, 0x00AD, 0x200B, 0xFEFF)) {
$alteredCorrectionReport = $capturedCorrectionReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$alteredCorrection = 'ex' + [char]$codePoint + 'it(1);'
$alteredCorrectionReport.'sub-results'[0].findings[0].'suggested-code' = $alteredCorrection
$alteredCorrectionReport.findings[0].'suggested-code' = $alteredCorrection
Assert-CompositionReport $alteredCorrectionReport '*SUPER_LEAF_CONTENT_MISMATCH*'
$rolledOnlyCorrectionReport = $capturedCorrectionReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$rolledOnlyCorrectionReport.findings[0].'suggested-code' = $alteredCorrection
Assert-CompositionReport $rolledOnlyCorrectionReport '*SUPER_FINDING_MISMATCH*'
Assert-True ([string]::Equals([IO.File]::ReadAllText($correctionCapturePath), $immutableCorrectionCapture, [StringComparison]::Ordinal)) `
'rejecting altered corrections leaves the immutable host capture unchanged'
}
$timestampReason = '2026-10-02T09:00:00Z'
$timestampLeaf = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$timestampLeaf | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue $timestampReason
$timestampComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$timestampComposition.acceptedResults = @(Save-AcceptedLeafReports @($timestampLeaf, $completedSecurityLeaf))
Set-Content -LiteralPath $compositionPath -Value ($timestampComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$timestampReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$timestampReport.'sub-results'[0] = $timestampLeaf
foreach ($alteredTimestamp in @('2026-10-02T09:00:00.000Z', '2026-10-02T09:00:00+00:00')) {
$timestampLeaf.'outcome-reason' = $alteredTimestamp
Assert-CompositionReport $timestampReport '*SUPER_LEAF_CONTENT_MISMATCH*'
}
$timestampLeaf.'outcome-reason' = $timestampReason
Set-Content -LiteralPath $reportPath -Value ($timestampReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$acceptedTimestampReport = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super `
-ExpectedCompositionPath $compositionPath
$acceptedReason = $acceptedTimestampReport.report.'sub-results'[0].'outcome-reason'
Assert-True ($acceptedReason -is [string] -and [string]::Equals($acceptedReason, $timestampReason, [StringComparison]::Ordinal)) `
'accepted timestamp-shaped JSON text remains the original literal string'
$normalizedTimestampReport = $validReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$normalizedTimestampReport | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue '2026-10-02T09:00:00.000Z'
$normalizedTimestampReport.findings[0].location.range.'start-line' = 1
Set-Content -LiteralPath $reportPath -Value ($normalizedTimestampReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$acceptedNormalizedTimestamp = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SourceRoot $tmp `
-SourcePaths $sourcePath -RetrievedArticlePaths $articlePath -AllowBoundedNormalization
Assert-True $acceptedNormalizedTimestamp.normalized 'bounded normalization still applies to an eligible range'
Assert-True ($acceptedNormalizedTimestamp.report.'outcome-reason' -is [string] -and
[string]::Equals($acceptedNormalizedTimestamp.report.'outcome-reason', $normalizedTimestampReport.'outcome-reason', [StringComparison]::Ordinal)) `
'bounded normalization preserves unrelated timestamp-shaped text exactly'
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
foreach ($case in @(
@{ Pattern = '*SUPER_IDENTITY_MISMATCH*'; Change = { param($candidate) $candidate.skill.id = 'al-other-review' } }
@{ Pattern = '*SUPER_IDENTITY_MISMATCH*'; Change = { param($candidate) $candidate.skill.version = 2 } }
@{ Pattern = '*SUPER_LEAF_VERSION_MISMATCH*'; Change = { param($candidate) $candidate.'sub-results'[0].skill.version = 2 } }
@{ Pattern = '*SUPER_UNEXPECTED_SUB_RESULT*'; Change = { param($candidate) $candidate.'sub-results'[0].skill.id = 'al-other-review' } }
@{ Pattern = '*SUPER_SUB_RESULT_ORDER*'; Change = { param($candidate) $candidate.'sub-results' = @($candidate.'sub-results'[1], $candidate.'sub-results'[0]) } }
@{ Pattern = '*SUPER_DUPLICATE_SUB_RESULT*'; Change = { param($candidate) $candidate.'sub-results' = @($candidate.'sub-results'[0], $candidate.'sub-results'[0]) } }
)) {
$candidate = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
& $case.Change $candidate
Assert-CompositionReport $candidate $case.Pattern
}
$fabricatedSkip = $missingLeafReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$fabricatedSkip | Add-Member -NotePropertyName 'skipped-sub-skills' -NotePropertyValue @(
@{ skill = @{ id = 'al-security-review'; version = 1 }; reason = 'configuration' }
)
Assert-CompositionReport $fabricatedSkip '*SUPER_UNEXPECTED_SKIP*'
$emptyAcceptedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$emptyAcceptedComposition.acceptedResults = @()
Set-Content -LiteralPath $compositionPath -Value ($emptyAcceptedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$noResults = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$noResults.'sub-results' = @()
$noResults.summary.coverage.'worklist-size' = 0
$noResults.summary.coverage.'items-evaluated' = 0
$noResults.outcome = 'not-applicable'
Assert-CompositionReport $noResults '*SUPER_OUTCOME_MISMATCH*'
$noResults.outcome = 'failed'
$noResults | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'al-style-review and al-security-review could not be evaluated.'
Assert-CompositionReport $noResults
$oneMissingId = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$oneMissingId.'outcome-reason' = 'al-security-review could not be evaluated.'
Assert-CompositionReport $oneMissingId '*SUPER_MISSING_LEAF_REASON*'
foreach ($baseReport in @($missingLeafReport, $noResults, $validSuperReport)) {
$capturedComposition = if ($baseReport.outcome -ceq 'completed') { $expectedComposition } else { $incompleteComposition }
Set-Content -LiteralPath $compositionPath -Value ($capturedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$selfReviewReport = $baseReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$selfReviewReport.findings = @(@{
id = 'agent:cross-domain-gap'
domain = 'Agent'
severity = 'minor'
confidence = 'medium'
message = 'A cross-domain issue needs attention.'
references = @()
'from-sub-skill' = 'agent'
})
$selfReviewReport.summary.counts.minor = 1
if ($baseReport.outcome -ceq 'completed') {
Assert-CompositionReport $selfReviewReport
}
elseif ($baseReport.outcome -ceq 'failed') {
Assert-CompositionReport $selfReviewReport '*Invalid findings-report JSON or schema*'
}
else {
Assert-CompositionReport $selfReviewReport '*SUPER_AGENT_REVIEW_INCOMPLETE*'
}
}
$allFailed = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$allFailed.'sub-results' = @($completedLeaf, $completedSecurityLeaf) | ConvertTo-Json -Depth 20 | ConvertFrom-Json
foreach ($leaf in $allFailed.'sub-results') {
$leaf.outcome = 'failed'
$leaf.summary.coverage.'items-evaluated' = 0
$leaf | Add-Member -NotePropertyName 'outcome-reason' -NotePropertyValue 'Invocation failed.'
}
$failedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$failedComposition.acceptedResults = @(Save-AcceptedLeafReports $allFailed.'sub-results')
Set-Content -LiteralPath $compositionPath -Value ($failedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-CompositionReport $allFailed
$skipComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$skipComposition.subSkills = @($skipComposition.subSkills[0])
$skipComposition.skipped = @(@{ id = 'al-security-review'; version = 1; reason = 'not-applicable' })
$skipComposition.acceptedResults = @($skipComposition.acceptedResults[0])
Set-Content -LiteralPath $compositionPath -Value ($skipComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$validSkippedReport = $fabricatedSkip | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$validSkippedReport.outcome = 'completed'
$validSkippedReport.PSObject.Properties.Remove('outcome-reason')
$validSkippedReport.'skipped-sub-skills'[0].reason = 'not-applicable'
Assert-CompositionReport $validSkippedReport
Assert-CompositionReport $missingLeafReport '*SUPER_SKIP_MISSING*'
$wrongSkip = $validSkippedReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$wrongSkip.'skipped-sub-skills'[0].reason = 'configuration'
Assert-CompositionReport $wrongSkip '*SUPER_SKIP_MISMATCH*'
$wrongSkip.'skipped-sub-skills'[0].reason = 'not-applicable'
$wrongSkip.'skipped-sub-skills'[0].skill.version = 2
Assert-CompositionReport $wrongSkip '*SUPER_SKIP_MISMATCH*'
$duplicateSkip = $validSkippedReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$duplicateSkip.'skipped-sub-skills' = @($duplicateSkip.'skipped-sub-skills'[0], $duplicateSkip.'skipped-sub-skills'[0])
Assert-CompositionReport $duplicateSkip '*SUPER_SKIP_CONFLICT*'
$returnedAndSkipped = $validSkippedReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$returnedAndSkipped.'skipped-sub-skills'[0].skill.id = 'al-style-review'
Assert-CompositionReport $returnedAndSkipped '*SUPER_SKIP_CONFLICT*'
$allSkippedComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$allSkippedComposition.skipped = @($allSkippedComposition.subSkills | ForEach-Object {
@{ id = $_.id; version = $_.version; reason = 'configuration' }
})
$allSkippedComposition.subSkills = @()
$allSkippedComposition.acceptedResults = @()
Set-Content -LiteralPath $compositionPath -Value ($allSkippedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$allSkippedReport = $noResults | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$allSkippedReport.outcome = 'not-applicable'
$allSkippedReport.PSObject.Properties.Remove('outcome-reason')
$allSkippedReport | Add-Member -NotePropertyName 'skipped-sub-skills' -NotePropertyValue @(
$allSkippedComposition.skipped | ForEach-Object { @{ skill = @{ id = $_.id; version = $_.version }; reason = $_.reason } }
)
Assert-CompositionReport $allSkippedReport
$invalidComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$invalidComposition.skipped = @(@{ id = 'al-style-review'; version = 1; reason = 'configuration' })
Set-Content -LiteralPath $compositionPath -Value ($invalidComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-CompositionReport $validSuperReport '*Invalid expected composition*'
$invalidComposition.skipped = @()
$invalidComposition.subSkills[0].version = '1'
Set-Content -LiteralPath $compositionPath -Value ($invalidComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-CompositionReport $validSuperReport '*Invalid expected composition*'
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$layerFixtureRoot = Join-Path $tmp 'layered-skills'
foreach ($layer in 'microsoft', 'community', 'custom') {
$layerDirectory = Join-Path $layerFixtureRoot $layer
New-Item -ItemType Directory -Path $layerDirectory -Force | Out-Null
$sourceSkills = Join-Path $Root "$layer/skills"
if (Test-Path -LiteralPath $sourceSkills -PathType Container) {
Copy-Item -LiteralPath $sourceSkills -Destination (Join-Path $layerDirectory 'skills') -Recurse
}
}
$customSkillDirectory = Join-Path $layerFixtureRoot 'custom/skills/review'
New-Item -ItemType Directory -Path $customSkillDirectory -Force | Out-Null
$customSkillPath = 'custom/skills/review/company-style-review.md'
$styleSkillText = Get-Content -LiteralPath (Join-Path $Root 'microsoft/skills/review/al-style-review.md') -Raw
Set-Content -LiteralPath (Join-Path $layerFixtureRoot $customSkillPath) `
-Value ($styleSkillText -replace '(?m)^version: 1\r?$', 'version: 7') -Encoding utf8NoBOM
$fixtureIndexPath = Join-Path $tmp 'layered-skill-index.json'
& (Join-Path $Root 'tools/Build-SkillIndex.ps1') -BCQualityRoot $layerFixtureRoot -IndexPath $fixtureIndexPath | Out-Null
$fixtureIndex = Get-Content -LiteralPath $fixtureIndexPath -Raw | ConvertFrom-Json
foreach ($selection in @(
@{ Disabled = @(); ExpectedLayer = 'custom'; ExpectedVersion = 7 }
@{ Disabled = @($customSkillPath); ExpectedLayer = 'microsoft'; ExpectedVersion = 1 }
@{ Disabled = @($customSkillPath, 'microsoft/skills/review/al-style-review.md'); ExpectedLayer = $null }
)) {
$resolved = & (Join-Path $Root 'tools/Resolve-SkillWorklist.ps1') -BCQualityRoot $layerFixtureRoot `
-IndexPath $fixtureIndexPath -SuperSkillPath 'microsoft/skills/review/al-code-review.md' `
-DisabledSkills $selection.Disabled
$styleSlots = @($resolved.subSkills | Where-Object id -CEQ 'al-style-review')
if ($selection.ExpectedLayer) {
Assert-True ($styleSlots.Count -eq 1 -and $styleSlots[0].layer -ceq $selection.ExpectedLayer -and
$styleSlots[0].version -eq $selection.ExpectedVersion) 'resolver-selected override or fallback is authoritative'
}
else {
Assert-True ($styleSlots.Count -eq 0) 'fully disabled slot is not selected'
}
$resolved.skipped = @($resolved.skipped | ForEach-Object {
$declaredPath = $_.declaredPath
$declaredSkill = @($fixtureIndex.skills | Where-Object path -CEQ $declaredPath)[0]
@{ id = $_.id; version = $declaredSkill.version; reason = $_.reason; declaredPath = $declaredPath }
})
$resolvedReport = $validSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$resolvedReport.'sub-results' = @($resolved.subSkills | ForEach-Object {
$leafReport = $completedLeaf | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$leafReport.skill.id = $_.id
$leafReport.skill.version = $_.version
$leafReport
})
$resolvedReport.summary.coverage.'worklist-size' = $resolved.subSkills.Count
$resolvedReport.summary.coverage.'items-evaluated' = $resolved.subSkills.Count
$resolvedReport | Add-Member -NotePropertyName 'skipped-sub-skills' -NotePropertyValue @(
$resolved.skipped | ForEach-Object { @{ skill = @{ id = $_.id; version = $_.version }; reason = $_.reason } }
)
$resolved | Add-Member -NotePropertyName 'acceptedResults' -NotePropertyValue @(Save-AcceptedLeafReports $resolvedReport.'sub-results')
Set-Content -LiteralPath $compositionPath -Value ($resolved | ConvertTo-Json -Depth 30) -Encoding utf8NoBOM
Assert-CompositionReport $resolvedReport
}
Set-Content -LiteralPath $compositionPath -Value ($expectedComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$styleFindingLeaf = $validReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json $styleFindingLeaf = $validReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$securityFindingLeaf = $validReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json $securityFindingLeaf = $validReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$securityFindingLeaf.skill.id = 'al-security-review' $securityFindingLeaf.skill.id = 'al-security-review'
@ -446,6 +846,10 @@ try {
Set-Content -LiteralPath $reportPath -Value ($partialSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM Set-Content -LiteralPath $reportPath -Value ($partialSuperReport | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
$acceptedPartialSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super $acceptedPartialSuper = & $validator -ReportPath $reportPath -BCQualityRoot $Root -SkillKind super
Assert-True (-not $acceptedPartialSuper.normalized) 'partial super-skill excludes failed coverage from its rollup' Assert-True (-not $acceptedPartialSuper.normalized) 'partial super-skill excludes failed coverage from its rollup'
$partialComposition = $expectedComposition | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$partialComposition.acceptedResults = @(Save-AcceptedLeafReports @($completedLeaf, $failedLeaf))
Set-Content -LiteralPath $compositionPath -Value ($partialComposition | ConvertTo-Json -Depth 20) -Encoding utf8NoBOM
Assert-CompositionReport $partialSuperReport
$failedLeafLeakage = $partialSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json $failedLeafLeakage = $partialSuperReport | ConvertTo-Json -Depth 20 | ConvertFrom-Json
$failedLeafLeakage.summary.counts.minor = 1 $failedLeafLeakage.summary.counts.minor = 1

254
tools/Validate-FindingsReport.ps1
View file

@ -1,3 +1,4 @@
#Requires -Version 7.5
<# <#
.SYNOPSIS .SYNOPSIS
Validates a BCQuality findings-report against its structural and semantic contract. Validates a BCQuality findings-report against its structural and semantic contract.
@ -12,6 +13,7 @@ param(
[string[]] $RetrievedArticlePaths = @(), [string[]] $RetrievedArticlePaths = @(),
[ValidateSet('leaf', 'super')] [ValidateSet('leaf', 'super')]
[string] $SkillKind = 'leaf', [string] $SkillKind = 'leaf',
[string] $ExpectedCompositionPath,
[switch] $AllowBoundedNormalization [switch] $AllowBoundedNormalization
) )
@ -28,12 +30,105 @@ try {
if (-not ($raw | Test-Json -SchemaFile $schemaPath -ErrorAction Stop)) { if (-not ($raw | Test-Json -SchemaFile $schemaPath -ErrorAction Stop)) {
throw 'Report does not satisfy schemas/findings-report.schema.json.' throw 'Report does not satisfy schemas/findings-report.schema.json.'
} }
$report = $raw | ConvertFrom-Json -Depth 100 $report = $raw | ConvertFrom-Json -Depth 100 -DateKind String
} }
catch { catch {
throw "Invalid findings-report JSON or schema: $($_.Exception.Message)" throw "Invalid findings-report JSON or schema: $($_.Exception.Message)"
} }
$expectedComposition = $null
$expectedLeaves = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
$expectedSkips = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
$acceptedLeaves = [Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal)
if ($ExpectedCompositionPath) {
if ($SkillKind -cne 'super') {
throw 'Expected composition is supported only for super-skill reports.'
}
$contractSchema = Get-Content -LiteralPath $schemaPath -Raw | ConvertFrom-Json -AsHashtable
$identitySchema = @{
type = 'object'
required = @('id', 'version')
properties = $contractSchema.definitions.skillReference.properties
}
$skipProperties = @{
id = $identitySchema.properties.id
version = $identitySchema.properties.version
reason = @{ enum = @('configuration', 'not-applicable') }
}
$compositionSchema = @{
type = 'object'
required = @('superSkill', 'subSkills', 'skipped', 'acceptedResults')
properties = @{
superSkill = $identitySchema
subSkills = @{ type = 'array'; items = $identitySchema }
acceptedResults = @{
type = 'array'
items = @{
type = 'object'
required = @('id', 'version', 'reportPath')
properties = @{
id = $identitySchema.properties.id
version = $identitySchema.properties.version
reportPath = @{ type = 'string'; minLength = 1 }
}
}
}
skipped = @{
type = 'array'
items = @{ type = 'object'; required = @('id', 'version', 'reason'); properties = $skipProperties }
}
}
} | ConvertTo-Json -Depth 20
try {
$compositionRaw = Get-Content -LiteralPath $ExpectedCompositionPath -Raw
if (-not ($compositionRaw | Test-Json -Schema $compositionSchema -ErrorAction Stop)) {
throw 'Expected composition does not satisfy its input contract.'
}
$expectedComposition = $compositionRaw | ConvertFrom-Json -Depth 100 -DateKind String
$expectedIds = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal)
foreach ($leaf in @($expectedComposition.subSkills)) {
if (-not $expectedIds.Add([string]$leaf.id)) {
throw "Duplicate expected skill id '$($leaf.id)'."
}
$expectedLeaves.Add([string]$leaf.id, $leaf)
}
foreach ($skip in @($expectedComposition.skipped)) {
if (-not $expectedIds.Add([string]$skip.id)) {
throw "Duplicate or selected-and-skipped expected skill id '$($skip.id)'."
}
$expectedSkips.Add([string]$skip.id, $skip)
}
$compositionDirectory = Split-Path -Parent (Resolve-Path -LiteralPath $ExpectedCompositionPath).Path
foreach ($accepted in @($expectedComposition.acceptedResults)) {
if (-not $expectedLeaves.ContainsKey([string]$accepted.id) -or
$accepted.version -ne $expectedLeaves[$accepted.id].version -or
$acceptedLeaves.ContainsKey([string]$accepted.id)) {
throw "Accepted result '$($accepted.id)' must uniquely match a selected leaf and version."
}
$acceptedPath = if ([IO.Path]::IsPathRooted($accepted.reportPath)) {
$accepted.reportPath
}
else {
Join-Path $compositionDirectory $accepted.reportPath
}
$acceptedRaw = Get-Content -LiteralPath $acceptedPath -Raw
if (-not ($acceptedRaw | Test-Json -SchemaFile $schemaPath -ErrorAction Stop)) {
throw "Accepted result '$($accepted.id)' does not satisfy the report schema."
}
$acceptedReport = $acceptedRaw | ConvertFrom-Json -Depth 100 -DateKind String
if ($acceptedReport.skill.id -cne $accepted.id -or $acceptedReport.skill.version -ne $accepted.version -or
$acceptedReport.PSObject.Properties.Name -ccontains 'sub-results' -or
$acceptedReport.PSObject.Properties.Name -ccontains 'skipped-sub-skills') {
throw "Accepted result '$($accepted.id)' must be a leaf report with the captured identity."
}
$acceptedLeaves.Add([string]$accepted.id, $acceptedReport)
}
}
catch {
throw "Invalid expected composition: $($_.Exception.Message)"
}
}
$retrieved = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) $retrieved = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal)
foreach ($path in $RetrievedArticlePaths) { foreach ($path in $RetrievedArticlePaths) {
$retrieved.Add($path) | Out-Null $retrieved.Add($path) | Out-Null
@ -49,6 +144,43 @@ function Test-HasProperty {
return $null -ne $Object -and $Object.PSObject.Properties.Name -ccontains $Name return $null -ne $Object -and $Object.PSObject.Properties.Name -ccontains $Name
} }
function Test-JsonContentEqual {
param([object] $First, [object] $Second)
if ($null -eq $First -or $null -eq $Second) {
return $null -eq $First -and $null -eq $Second
}
if ($First -is [pscustomobject] -or $Second -is [pscustomobject]) {
if ($First -isnot [pscustomobject] -or $Second -isnot [pscustomobject] -or
@($First.PSObject.Properties).Count -ne @($Second.PSObject.Properties).Count) {
return $false
}
foreach ($property in $First.PSObject.Properties) {
if (-not (Test-HasProperty $Second $property.Name) -or
-not (Test-JsonContentEqual $property.Value $Second.PSObject.Properties[$property.Name].Value)) {
return $false
}
}
return $true
}
if ($First -is [array] -or $Second -is [array]) {
if ($First -isnot [array] -or $Second -isnot [array] -or $First.Count -ne $Second.Count) {
return $false
}
for ($index = 0; $index -lt $First.Count; $index++) {
if (-not (Test-JsonContentEqual $First[$index] $Second[$index])) {
return $false
}
}
return $true
}
if ($First -is [string] -or $Second -is [string]) {
return $First -is [string] -and $Second -is [string] -and
[string]::Equals($First, $Second, [StringComparison]::Ordinal)
}
return $First.GetType() -eq $Second.GetType() -and $First -ceq $Second
}
function Get-SourceLineCount { function Get-SourceLineCount {
param([string] $Path) param([string] $Path)
@ -79,8 +211,14 @@ function Get-SemanticErrors {
} }
function Get-DerivedSuperOutcome { function Get-DerivedSuperOutcome {
param([object[]] $SubResults) param([object[]] $SubResults, [int] $MissingResults = 0)
if ($MissingResults -gt 0) {
if (@($SubResults | Where-Object outcome -CNE 'failed').Count) {
return 'partial'
}
return 'failed'
}
if (-not $SubResults.Count) { if (-not $SubResults.Count) {
return 'not-applicable' return 'not-applicable'
} }
@ -138,9 +276,10 @@ function Get-SemanticErrors {
$firstHasCode = Test-HasProperty $First 'suggested-code' $firstHasCode = Test-HasProperty $First 'suggested-code'
$secondHasCode = Test-HasProperty $Second 'suggested-code' $secondHasCode = Test-HasProperty $Second 'suggested-code'
if ($firstHasCode -or $secondHasCode) { if ($firstHasCode -or $secondHasCode) {
return $firstHasCode -and $secondHasCode -and $First.'suggested-code' -ceq $Second.'suggested-code' return $firstHasCode -and $secondHasCode -and
[string]::Equals($First.'suggested-code', $Second.'suggested-code', [StringComparison]::Ordinal)
} }
return $First.message -ceq $Second.message return [string]::Equals($First.message, $Second.message, [StringComparison]::Ordinal)
} }
function Test-ReferencesInclude { function Test-ReferencesInclude {
@ -181,7 +320,7 @@ function Get-SemanticErrors {
$RolledFinding.id -cne $expectedId -or $RolledFinding.id -cne $expectedId -or
$RolledFinding.severity -cne $LeafFinding.severity -or $RolledFinding.severity -cne $LeafFinding.severity -or
$RolledFinding.confidence -cne $LeafFinding.confidence -or $RolledFinding.confidence -cne $LeafFinding.confidence -or
$RolledFinding.message -cne $LeafFinding.message -or -not [string]::Equals($RolledFinding.message, $LeafFinding.message, [StringComparison]::Ordinal) -or
$rolledReferences.Count -ne $leafReferences.Count -or $rolledReferences.Count -ne $leafReferences.Count -or
-not (Test-ReferencesInclude $rolledReferences $leafReferences)) { -not (Test-ReferencesInclude $rolledReferences $leafReferences)) {
return $false return $false
@ -190,7 +329,7 @@ function Get-SemanticErrors {
$rolledHasProperty = Test-HasProperty $RolledFinding $name $rolledHasProperty = Test-HasProperty $RolledFinding $name
$leafHasProperty = Test-HasProperty $LeafFinding $name $leafHasProperty = Test-HasProperty $LeafFinding $name
if ($rolledHasProperty -ne $leafHasProperty -or if ($rolledHasProperty -ne $leafHasProperty -or
($rolledHasProperty -and $RolledFinding.$name -cne $LeafFinding.$name)) { ($rolledHasProperty -and -not [string]::Equals($RolledFinding.$name, $LeafFinding.$name, [StringComparison]::Ordinal))) {
return $false return $false
} }
} }
@ -206,7 +345,7 @@ function Get-SemanticErrors {
$sameCorrection = Test-SameCorrection $RolledFinding $LeafFinding $sameCorrection = Test-SameCorrection $RolledFinding $LeafFinding
$correctionsConflict = (Test-HasProperty $RolledFinding 'suggested-code') -and $correctionsConflict = (Test-HasProperty $RolledFinding 'suggested-code') -and
(Test-HasProperty $LeafFinding 'suggested-code') -and (Test-HasProperty $LeafFinding 'suggested-code') -and
$RolledFinding.'suggested-code' -cne $LeafFinding.'suggested-code' -not [string]::Equals($RolledFinding.'suggested-code', $LeafFinding.'suggested-code', [StringComparison]::Ordinal)
$explicitCrossRuleMerge = $leafReferences.Count -and $explicitCrossRuleMerge = $leafReferences.Count -and
$rolledReferences.Count -gt $leafReferences.Count -and $rolledReferences.Count -gt $leafReferences.Count -and
-not $correctionsConflict -and -not $correctionsConflict -and
@ -342,20 +481,105 @@ function Get-SemanticErrors {
if ($CurrentSkillKind -ceq 'super' -and $hasSubResults) { if ($CurrentSkillKind -ceq 'super' -and $hasSubResults) {
$subResults = @($Current.'sub-results') $subResults = @($Current.'sub-results')
$producerIds = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal)
for ($index = 0; $index -lt $subResults.Count; $index++) { for ($index = 0; $index -lt $subResults.Count; $index++) {
if (-not $producerIds.Add([string]$subResults[$index].skill.id)) {
Add-Error 'SUPER_DUPLICATE_SUB_RESULT' "$ReportPathPrefix.sub-results[$index].skill.id" `
'A leaf may appear only once in sub-results.'
}
Test-Report $subResults[$index] "$ReportPathPrefix.sub-results[$index]" 'leaf' Test-Report $subResults[$index] "$ReportPathPrefix.sub-results[$index]" 'leaf'
} }
$expectedOutcome = Get-DerivedSuperOutcome $subResults $skippedIds = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal)
$skips = if ($hasSkippedSubSkills) { @($Current.'skipped-sub-skills') } else { @() }
foreach ($skip in $skips) {
if (-not $skippedIds.Add([string]$skip.skill.id) -or $producerIds.Contains([string]$skip.skill.id)) {
Add-Error 'SUPER_SKIP_CONFLICT' "$ReportPathPrefix.skipped-sub-skills" `
"Skill '$($skip.skill.id)' is duplicated or both returned and skipped."
}
}
$missingResults = 0
if ($null -ne $expectedComposition) {
if ($Current.skill.id -cne $expectedComposition.superSkill.id -or
$Current.skill.version -ne $expectedComposition.superSkill.version) {
Add-Error 'SUPER_IDENTITY_MISMATCH' "$ReportPathPrefix.skill" 'Super-skill identity differs from expected composition.'
}
$previousSlot = -1
$orderedIds = @($expectedComposition.subSkills | ForEach-Object { $_.id })
for ($index = 0; $index -lt $subResults.Count; $index++) {
$identity = $subResults[$index].skill
if (-not $expectedLeaves.ContainsKey([string]$identity.id)) {
Add-Error 'SUPER_UNEXPECTED_SUB_RESULT' "$ReportPathPrefix.sub-results[$index].skill" `
"Skill '$($identity.id)' was not selected."
continue
}
if ($identity.version -ne $expectedLeaves[$identity.id].version) {
Add-Error 'SUPER_LEAF_VERSION_MISMATCH' "$ReportPathPrefix.sub-results[$index].skill.version" `
"Unexpected version for '$($identity.id)'."
}
if (-not $acceptedLeaves.ContainsKey([string]$identity.id)) {
Add-Error 'SUPER_LEAF_NOT_ACCEPTED' "$ReportPathPrefix.sub-results[$index]" `
"Leaf '$($identity.id)' has no host-captured accepted result."
}
elseif (-not (Test-JsonContentEqual $subResults[$index] $acceptedLeaves[$identity.id])) {
Add-Error 'SUPER_LEAF_CONTENT_MISMATCH' "$ReportPathPrefix.sub-results[$index]" `
"Leaf '$($identity.id)' differs from its host-captured accepted result."
}
$slot = [Array]::IndexOf($orderedIds, $identity.id)
if ($slot -le $previousSlot) {
Add-Error 'SUPER_SUB_RESULT_ORDER' "$ReportPathPrefix.sub-results[$index].skill" `
'Sub-results must preserve the selected worklist order.'
}
$previousSlot = $slot
}
foreach ($leaf in @($expectedComposition.subSkills)) {
if (-not $producerIds.Contains([string]$leaf.id)) {
$missingResults++
if ($acceptedLeaves.ContainsKey([string]$leaf.id)) {
Add-Error 'SUPER_ACCEPTED_LEAF_MISSING' "$ReportPathPrefix.sub-results" `
"Host-captured accepted leaf '$($leaf.id)' must be included."
}
$reason = if (Test-HasProperty $Current 'outcome-reason') { $Current.'outcome-reason' } else { '' }
$idPattern = '(?<![a-zA-Z0-9_-])' + [regex]::Escape($leaf.id) + '(?![a-zA-Z0-9_-])'
if ($reason -cnotmatch $idPattern) {
Add-Error 'SUPER_MISSING_LEAF_REASON' "$ReportPathPrefix.outcome-reason" `
"The reason must name missing selected leaf '$($leaf.id)'."
}
}
}
foreach ($skip in $skips) {
if (-not $expectedSkips.ContainsKey([string]$skip.skill.id)) {
Add-Error 'SUPER_UNEXPECTED_SKIP' "$ReportPathPrefix.skipped-sub-skills" `
"Skill '$($skip.skill.id)' was not excluded by the coordinator."
continue
}
$expectedSkip = $expectedSkips[$skip.skill.id]
if ($skip.skill.version -ne $expectedSkip.version -or $skip.reason -cne $expectedSkip.reason) {
Add-Error 'SUPER_SKIP_MISMATCH' "$ReportPathPrefix.skipped-sub-skills" `
"Skip identity or reason differs for '$($skip.skill.id)'."
}
}
foreach ($skip in @($expectedComposition.skipped)) {
if (-not $skippedIds.Contains([string]$skip.id)) {
Add-Error 'SUPER_SKIP_MISSING' "$ReportPathPrefix.skipped-sub-skills" `
"Expected exclusion '$($skip.id)' is missing."
}
}
}
$expectedOutcome = Get-DerivedSuperOutcome $subResults $missingResults
if ($Current.outcome -cne $expectedOutcome) { if ($Current.outcome -cne $expectedOutcome) {
Add-Error 'SUPER_OUTCOME_MISMATCH' "$ReportPathPrefix.outcome" "Expected '$expectedOutcome' from sub-results." Add-Error 'SUPER_OUTCOME_MISMATCH' "$ReportPathPrefix.outcome" "Expected '$expectedOutcome' from sub-results."
} }
$includedSubResults = @($subResults | Where-Object outcome -CNE 'failed') $includedSubResults = @($subResults | Where-Object outcome -CNE 'failed')
$expectedWorklistSize = ($includedSubResults | Measure-Object -Property { $_.summary.coverage.'worklist-size' } -Sum).Sum $expectedWorklistSize = 0
$expectedItemsEvaluated = ($includedSubResults | Measure-Object -Property { $_.summary.coverage.'items-evaluated' } -Sum).Sum $expectedItemsEvaluated = 0
if ($null -eq $expectedWorklistSize) { $expectedWorklistSize = 0 } foreach ($subResult in $includedSubResults) {
if ($null -eq $expectedItemsEvaluated) { $expectedItemsEvaluated = 0 } $expectedWorklistSize += $subResult.summary.coverage.'worklist-size'
$expectedItemsEvaluated += $subResult.summary.coverage.'items-evaluated'
}
if ($worklistSize -ne $expectedWorklistSize -or $itemsEvaluated -ne $expectedItemsEvaluated) { if ($worklistSize -ne $expectedWorklistSize -or $itemsEvaluated -ne $expectedItemsEvaluated) {
Add-Error 'SUPER_COVERAGE_MISMATCH' "$ReportPathPrefix.summary.coverage" ` Add-Error 'SUPER_COVERAGE_MISMATCH' "$ReportPathPrefix.summary.coverage" `
"Expected worklist-size $expectedWorklistSize and items-evaluated $expectedItemsEvaluated from non-failed sub-results." "Expected worklist-size $expectedWorklistSize and items-evaluated $expectedItemsEvaluated from non-failed sub-results."
@ -378,6 +602,10 @@ function Get-SemanticErrors {
$finding = $findings[$index] $finding = $findings[$index]
$producerId = [string]$finding.'from-sub-skill' $producerId = [string]$finding.'from-sub-skill'
if ($producerId -ceq 'agent') { if ($producerId -ceq 'agent') {
if ($missingResults -gt 0) {
Add-Error 'SUPER_AGENT_REVIEW_INCOMPLETE' "$ReportPathPrefix.findings[$index]" `
'Super-skill self-review is forbidden while selected leaf results are missing.'
}
if (@($finding.references).Count -or if (@($finding.references).Count -or
$finding.id -cnotmatch '^agent:[a-z0-9]+(?:-[a-z0-9]+)*$' -or $finding.id -cnotmatch '^agent:[a-z0-9]+(?:-[a-z0-9]+)*$' -or
-not (Test-HasProperty $finding 'domain') -or -not (Test-HasProperty $finding 'domain') -or
@ -496,7 +724,7 @@ if ($errors.Count -and $AllowBoundedNormalization) {
$otherErrors = @($errors | Where-Object Code -CNE 'RANGE_START_MISMATCH') $otherErrors = @($errors | Where-Object Code -CNE 'RANGE_START_MISMATCH')
$rangeErrors = @($errors | Where-Object Code -CEQ 'RANGE_START_MISMATCH') $rangeErrors = @($errors | Where-Object Code -CEQ 'RANGE_START_MISMATCH')
if (-not $otherErrors.Count -and $rangeErrors.Count) { if (-not $otherErrors.Count -and $rangeErrors.Count) {
$candidate = $report | ConvertTo-Json -Depth 100 | ConvertFrom-Json -Depth 100 $candidate = $report | ConvertTo-Json -Depth 100 | ConvertFrom-Json -Depth 100 -DateKind String
$eligible = $true $eligible = $true
foreach ($finding in @($candidate.findings)) { foreach ($finding in @($candidate.findings)) {
if (-not (Test-HasProperty $finding 'location') -or if (-not (Test-HasProperty $finding 'location') -or