mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-10-05 06:36:55 +01:00
Validate composed reviews against the expected leaf worklist (#204)
* Validate composed reviews against the expected leaf worklist * Bind composed leaf reports to host-accepted results * Preserve literal strings in composed report acceptance
This commit is contained in:
parent
0867171b1a
commit
206c5feff4
6 changed files with 784 additions and 26 deletions
56
skills/do.md
56
skills/do.md
|
|
@ -224,7 +224,36 @@ as a findings-report, a coordinator or host MUST validate it deterministically:
|
|||
|
||||
Hosts SHOULD execute `tools/Validate-FindingsReport.ps1` with the exact source
|
||||
scope and the leaf's recorded set of fully retrieved article paths. Pass
|
||||
`-SkillKind super` when validating a super-skill's rolled-up report. Pass
|
||||
`-SkillKind super -ExpectedCompositionPath <host-owned-json>` when validating
|
||||
a super-skill's rolled-up report. Prepare that private artifact before leaf
|
||||
dispatch, after layer resolution and input compatibility checks. It contains
|
||||
`superSkill` (`id`, `version`), ordered selected `subSkills` (each with `id`,
|
||||
`version`), `skipped` (each with `id`, `version`, `reason`), and an initially
|
||||
empty `acceptedResults` array. Reasons are `configuration` or `not-applicable`;
|
||||
budget exhaustion is not a skip reason.
|
||||
Additional resolver metadata may be retained in the artifact, not the report.
|
||||
After each leaf passes its acceptance gate, the host saves the exact accepted
|
||||
copy in a private immutable file and appends an `acceptedResults` entry with
|
||||
`id`, `version`, and `reportPath`. Capture host-created failed validation
|
||||
results the same way. Paths may be absolute or relative to the composition
|
||||
artifact's directory. Capture the normalized accepted copy when normalization
|
||||
was permitted, not the invalid raw return. Do not expose these files or write
|
||||
access to the artifact to leaf workers or the composing model. Only the host
|
||||
may append captures; the pre-dispatch selection and exclusions remain fixed.
|
||||
The validator binds the super-skill and leaf identities and versions, checks
|
||||
selected order, and requires exact agreement on exclusions. Each nested leaf
|
||||
must exactly match its host-captured accepted JSON content, ignoring object
|
||||
property order but preserving array order, types, values, and field presence.
|
||||
Every captured leaf must be included; uncaptured or altered leaves are invalid.
|
||||
Every returned leaf must be unique; a selected leaf cannot be reclassified as skipped by the
|
||||
report. When selected leaves are missing, `outcome-reason` must name every
|
||||
missing ID exactly and top-level `from-sub-skill: "agent"` findings are forbidden.
|
||||
Without the artifact, validation remains structural and semantic but
|
||||
cannot prove composition completeness, selected versions, order, or legitimate
|
||||
exclusions, nor bind leaves to accepted host outputs. Duplicate or both
|
||||
returned-and-skipped leaf IDs are invalid even without the artifact. Never
|
||||
derive the expected composition from model output.
|
||||
Pass
|
||||
`-AllowBoundedNormalization` only when the host preserves the immutable raw
|
||||
payload and records `removedRanges` in private telemetry as required above.
|
||||
|
||||
|
|
@ -330,7 +359,7 @@ Omit `suggested-code` only when the appropriate fix depends on context the skill
|
|||
- `reference` — the suppressed file (same object shape as `findings[].references`).
|
||||
- `reason` — `layer-precedence` when another layer won under READ's precedence rules; `configuration` when the consumer disabled the file's layer.
|
||||
|
||||
**`sub-results`** — super-skills only. Array of complete findings-reports, one per sub-skill that was invoked (i.e., every sub-skill not listed in `skipped-sub-skills`). Each entry MUST itself conform to this output contract. Entries MUST appear in the worklist's declared order, regardless of invocation or completion order. Leaf skills MUST NOT emit `sub-results`.
|
||||
**`sub-results`** — super-skills only. Array of complete findings-reports, one per invoked sub-skill, with no duplicate skill IDs. Each entry MUST itself conform to this output contract. Entries MUST appear in the worklist's declared order, regardless of invocation or completion order. A selected leaf left uninvoked by budget exhaustion has no fabricated sub-result and is not a configured or input-incompatible skip; its absence requires the incomplete-composition outcome below. Leaf skills MUST NOT emit `sub-results`.
|
||||
|
||||
**`skipped-sub-skills`** — super-skills only. Array of sub-skills that were declared in frontmatter but not invoked. `reason` is `configuration` when the orchestrator disabled the sub-skill, or `not-applicable` when the super-skill's Relevance step ruled it out.
|
||||
|
||||
|
|
@ -356,10 +385,14 @@ orchestrator.
|
|||
|
||||
Each leaf invocation MUST remain a discrete evaluation with its own complete
|
||||
findings-report. An orchestrator MAY execute independent leaves serially or
|
||||
concurrently, but MUST invoke every worklisted leaf, preserve `sub-results` in
|
||||
the declared worklist order, and wait for every invocation to finish before
|
||||
performing any super-skill self-review or final rollup. Scheduling MUST NOT
|
||||
change relevance, coverage, failure, reference-integrity, or output semantics.
|
||||
concurrently, but MUST attempt every worklisted leaf, preserve `sub-results`
|
||||
in the declared worklist order, and wait for every started invocation to
|
||||
finish before final rollup. If its execution budget prevents dispatching
|
||||
remaining leaves, preserve the unfinished selection in the host-owned expected
|
||||
composition and use the incomplete-composition outcome below. Do not perform
|
||||
the super-skill self-review until every selected leaf has returned. Scheduling
|
||||
MUST NOT change relevance, coverage, failure, reference-integrity, or output
|
||||
semantics.
|
||||
|
||||
Orchestrators SHOULD generate `skill-index.json` with
|
||||
`tools/Build-SkillIndex.ps1` instead of parsing Markdown. Each declared
|
||||
|
|
@ -390,7 +423,16 @@ The five required sections still apply. Their meaning shifts from knowledge file
|
|||
|
||||
### Outcome rollup
|
||||
|
||||
A super-skill's `outcome` is derived from its sub-skills' outcomes. Let S be the multiset of sub-skill outcomes for sub-skills in the worklist (skipped sub-skills do not contribute):
|
||||
A super-skill's `outcome` is derived from its selected worklist and returned
|
||||
sub-skills' outcomes. If selected leaves have no returned report, the outcome
|
||||
is `partial` when at least one returned report is non-`failed`, or `failed`
|
||||
when no non-`failed` report is available. It MUST NOT be `completed`,
|
||||
`not-applicable`, or `no-knowledge`. Name unfinished leaf IDs in
|
||||
`outcome-reason`, preserve the valid returned reports, and never invent
|
||||
successful or failed invocations for leaves that were not invoked.
|
||||
|
||||
When every selected leaf has returned, let S be the multiset of their outcomes
|
||||
(skipped sub-skills do not contribute):
|
||||
|
||||
- `failed` — every element of S is `failed`.
|
||||
- `partial` — S contains at least one `partial`, OR S contains at least one `failed` alongside at least one non-`failed` outcome.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue