mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-08-06 17:36:53 +01:00
Own the knowledge-index generator + index-aware review skills (#25)
* Make domain-skill knowledge discovery index-aware The 6 AL domain review skills and read.md now enumerate candidate articles from the BCQuality knowledge index (knowledge-index.json) instead of opening every file under the domain folder to read its frontmatter. The worklist selection predicate is unchanged (keywords intersect diff tokens, or topic matches a changed object type) - only the discovery source changes, so the same articles are selected. Full article bodies are read only for worklisted entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reconcile §Source wording with the lean knowledge index The BCQuality filter now emits a lean index whose per-article description is a one-line hint rather than the full verbatim Description. Update the six domain skills' §Source to say the index carries a one-line description hint (keywords, title, and a one-line description) instead of the full description. The worklist selection predicate is unchanged: keywords drive selection and the agent opens worklisted articles in full for their rule bodies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Own the knowledge-index generator in BCQuality The knowledge index is an acceleration of the skills' Source step, and its schema is part of that contract — so BCQuality should own the generator rather than each consumer re-implementing it. Add tools/Build-KnowledgeIndex.ps1 (the parser + lean-description shaping + emit, lifted verbatim from the BCAppsBCQuality filter prototype) and document the index in agent-consumption.md. Consumers prune their clone to policy, then call this script; the index stays in lockstep with the Source contract and every orchestrator gets the same faithful index for free. The worklist selection predicate is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Own knowledge-index generation in BCQuality (runtime + CI), not the consumer The index is now produced by BCQuality itself: Entry's preparation step rebuilds knowledge-index.json over the live, already-pruned clone at the start of every run, and a new CI workflow validates the generator's health (determinism, full coverage, selection-input integrity). Consumers no longer invoke or know about the index. Rebuilding over the pruned clone (vs shipping a committed full-corpus index) keeps the index exact for any consumer policy: it can never list a denied article, so policy-excluded rules cannot leak into discovery. READ now states the index is discovery-only -- a finding must cite an article opened in full, and rows whose file is absent are discarded before ranking. - skills/entry.md: new 'Preparation -- knowledge index' precondition - skills/read.md: index ownership + discovery-only invariant - microsoft/skills/review/*.md (6): 'BCQuality builds' (not 'the filter emits') - agent-consumption.md 5a: runtime+CI ownership rationale - .github/workflows/knowledge-index.yml + scripts/Test-KnowledgeIndex.ps1: generator guard - .gitignore: never commit the runtime index Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make runtime index build non-interactive and self-contained entry.md now gives the exact build command (pwsh ./tools/Build-KnowledgeIndex.ps1) so the agent's preparation step is unambiguous, and the generator's -BCQualityRoot parameter is optional (defaults to the clone root) so it runs in non-interactive -p mode without prompting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Resolve knowledge-index root to absolute path (cross-platform fix) Get-ChildItem.FullName is always absolute, so deriving the relative article path via Substring(\.Length) requires an absolute root. A relative root such as '.' (used by the CI guard's 'Test-KnowledgeIndex.ps1 -Root .') left the full path almost intact on Linux, producing bogus 'home/runner/.../knowledge' paths and failing the coverage check. Normalise both the generator's -BCQualityRoot and the test's -Root with Resolve-Path before use. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Jesper Schulz-Wedde <jesper.schulzwedde@microsoft.com>
This commit is contained in:
Jesper Schulz-Wedde
GitHub
parent
b19889ec46
commit
822cae1b27
13 changed files with 402 additions and 13 deletions
93
.github/scripts/Test-KnowledgeIndex.ps1
vendored
Normal file
93
.github/scripts/Test-KnowledgeIndex.ps1
vendored
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
<#
|
||||
.SYNOPSIS
|
||||
CI guard for the knowledge-index generator (tools/Build-KnowledgeIndex.ps1).
|
||||
|
||||
.DESCRIPTION
|
||||
BCQuality owns the knowledge index, so BCQuality CI — not each consumer —
|
||||
proves the generator is healthy. This script does NOT ship a committed
|
||||
index that consumers trust at runtime (the index is rebuilt over each
|
||||
consumer's already-pruned clone by Entry's preparation step, which keeps it
|
||||
exact for any policy). Instead it asserts the generator itself is sound:
|
||||
|
||||
1. Determinism — building twice yields byte-identical output once the
|
||||
volatile `generatedAt` header is normalized.
|
||||
2. Coverage — every `*/knowledge/**/*.md` article appears exactly once;
|
||||
every indexed path exists; no duplicates; no article is dropped.
|
||||
3. Selection-input integrity — every parsed article row carries the
|
||||
non-empty `domain` + `keywords` the worklist predicate selects on, and
|
||||
every article parses (an unparseable article is an invalid file).
|
||||
|
||||
Exit code 0 = healthy; non-zero = a problem CI must block on.
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string] $Root = (Resolve-Path (Join-Path $PSScriptRoot '..' '..'))
|
||||
)
|
||||
|
||||
Set-StrictMode -Version Latest
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Normalise to an absolute path so Substring-based relative-path derivation
|
||||
# below matches the absolute FullName the generator emits (CI passes -Root .).
|
||||
$Root = (Resolve-Path -LiteralPath $Root).Path
|
||||
|
||||
$generator = Join-Path $Root 'tools/Build-KnowledgeIndex.ps1'
|
||||
if (-not (Test-Path $generator)) { throw "Generator not found: $generator" }
|
||||
|
||||
$tmp = Join-Path ([System.IO.Path]::GetTempPath()) ("kbindex_" + [guid]::NewGuid().ToString('N'))
|
||||
New-Item -ItemType Directory -Force -Path $tmp | Out-Null
|
||||
$idxA = Join-Path $tmp 'a.json'
|
||||
$idxB = Join-Path $tmp 'b.json'
|
||||
|
||||
$problems = [System.Collections.Generic.List[string]]::new()
|
||||
|
||||
& $generator -BCQualityRoot $Root -IndexPath $idxA | Out-Null
|
||||
& $generator -BCQualityRoot $Root -IndexPath $idxB | Out-Null
|
||||
|
||||
# 1. Determinism (ignoring the volatile generatedAt timestamp).
|
||||
$norm = { param($p) ((Get-Content -LiteralPath $p -Raw) -replace '"generatedAt":"[^"]*"', '"generatedAt":"<n>"') }
|
||||
if ((& $norm $idxA) -ne (& $norm $idxB)) {
|
||||
$problems.Add('Non-deterministic: two builds differ beyond generatedAt.') | Out-Null
|
||||
}
|
||||
|
||||
$index = Get-Content -LiteralPath $idxA -Raw | ConvertFrom-Json
|
||||
$rows = @($index.articles)
|
||||
|
||||
# 2. Coverage: one row per knowledge .md, every path real, no duplicates.
|
||||
$onDisk = @(
|
||||
foreach ($layer in 'microsoft', 'community', 'custom') {
|
||||
$kb = Join-Path $Root (Join-Path $layer 'knowledge')
|
||||
if (Test-Path $kb) {
|
||||
Get-ChildItem -LiteralPath $kb -Recurse -File -Filter '*.md' |
|
||||
ForEach-Object { ($_.FullName.Substring($Root.Length).TrimStart([char]'/', [char]'\') -replace '\\', '/') }
|
||||
}
|
||||
}
|
||||
)
|
||||
if ($rows.Count -ne $onDisk.Count) {
|
||||
$problems.Add("Coverage mismatch: index has $($rows.Count) rows, disk has $($onDisk.Count) knowledge .md files.") | Out-Null
|
||||
}
|
||||
$rowPaths = @($rows | ForEach-Object { $_.path })
|
||||
$dupes = @($rowPaths | Group-Object | Where-Object Count -gt 1 | ForEach-Object { $_.Name })
|
||||
if ($dupes.Count) { $problems.Add("Duplicate index rows: $($dupes -join ', ')") | Out-Null }
|
||||
$missingOnDisk = @($rowPaths | Where-Object { -not (Test-Path (Join-Path $Root $_)) })
|
||||
if ($missingOnDisk.Count) { $problems.Add("Indexed paths absent on disk: $($missingOnDisk -join ', ')") | Out-Null }
|
||||
$missingInIndex = @($onDisk | Where-Object { $rowPaths -notcontains $_ })
|
||||
if ($missingInIndex.Count) { $problems.Add("Articles missing from index: $($missingInIndex -join ', ')") | Out-Null }
|
||||
|
||||
# 3. Selection-input integrity: parsed rows must carry domain + keywords.
|
||||
$unparsed = @($rows | Where-Object { -not $_.parsed } | ForEach-Object { $_.path })
|
||||
if ($unparsed.Count) { $problems.Add("Unparseable (invalid) articles: $($unparsed -join ', ')") | Out-Null }
|
||||
foreach ($r in $rows | Where-Object { $_.parsed }) {
|
||||
if ([string]::IsNullOrWhiteSpace([string]$r.domain)) { $problems.Add("Empty domain: $($r.path)") | Out-Null }
|
||||
if (-not @($r.keywords).Where({ "$_".Trim() }).Count) { $problems.Add("Empty keywords: $($r.path)") | Out-Null }
|
||||
}
|
||||
|
||||
Remove-Item -Recurse -Force $tmp -ErrorAction SilentlyContinue
|
||||
|
||||
if ($problems.Count) {
|
||||
Write-Host "Knowledge-index check FAILED ($($problems.Count) problem(s)):" -ForegroundColor Red
|
||||
$problems | ForEach-Object { Write-Host " - $_" -ForegroundColor Red }
|
||||
exit 1
|
||||
}
|
||||
Write-Host "Knowledge-index check PASSED: $($rows.Count) articles, deterministic, full coverage, selection inputs intact." -ForegroundColor Green
|
||||
exit 0
|
||||
Loading…
Add table
Add a link
Reference in a new issue