mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-08-06 09:26:52 +01:00
Adds the contract field, skill instructions, and two knowledge articles
that BCAppsBCQuality's PR-review agent needs to match (and exceed) the
coverage of the embedded review agent in BCAppsCampAIRHack:
skills/do.md
- New optional findings[].suggested-code field. Documents what it MUST
contain (a literal line-replacement payload) and when to emit it.
microsoft/skills/review/al-code-review.md
- Instructs both the agent self-review pass and rolled-up sub-skill
findings to populate suggested-code when the fix is mechanical.
- Lists examples (dead code removal, Count > 0 -> IsEmpty, object-scope
Label) that map to issues observed in the parity comparison.
microsoft/knowledge/style/telemetry-event-id-stable-unique.{md,bad.al,good.al}
- New knowledge article: telemetry event IDs must be stable, unique,
and non-placeholder. Closes a gap surfaced by the parity comparison.
microsoft/knowledge/style/labels-declared-at-object-scope.{md,bad.al,good.al}
- New knowledge article: Labels must live in the object-level var
block, not in procedure-local var blocks. Closes the second gap.
microsoft/knowledge/privacy/no-pii-in-telemetry-message-string.md
- Adds an explicit note that changing DataClassification alone does not
make embedding PII into the message string acceptable, plus links to
the two adjacent privacy articles. Resolves the privacy advice the
parity comparison flagged as ambiguous.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
264 lines
18 KiB
Markdown
264 lines
18 KiB
Markdown
---
|
|
kind: meta-skill
|
|
id: do
|
|
version: 1
|
|
title: Action Skill — the template every action skill follows
|
|
---
|
|
|
|
# DO
|
|
|
|
An action skill is a markdown file that tells an agent how to do one concrete job — review a pull request, audit telemetry usage, generate a skeleton — using knowledge files from BCQuality. This document is the template every action skill follows. Orchestrators rely on the template to consume any skill without skill-specific parsing.
|
|
|
|
This contract is stable. Changes require a PR approved by both maintainers.
|
|
|
|
## What an action skill is
|
|
|
|
An action skill is a single markdown file with YAML frontmatter. It lives inside a layer:
|
|
|
|
- `/microsoft/skills/` — platform-endorsed action skills.
|
|
- `/community/skills/` — community-contributed action skills.
|
|
- `/custom/skills/` — partner or customer action skills (typically in a consumer repo, not in BCQuality itself).
|
|
|
|
Action skills do not live at the repo root. The files in `/skills/` — the three meta-skill contracts (READ, DO, WRITE) and the entry-point skill (`entry.md`, `kind: entry-point`) — are the only skills that sit outside a layer. The entry-point skill structurally follows this same four-step pattern but produces a dispatch record rather than a findings-report; see `skills/entry.md` for its contract.
|
|
|
|
## Frontmatter schema
|
|
|
|
```yaml
|
|
---
|
|
kind: action-skill
|
|
id: al-code-review
|
|
version: 1
|
|
title: AL code review
|
|
description: Reviews AL source changes against performance and security guidance.
|
|
inputs: [pr-diff, object-list]
|
|
outputs: [findings-report]
|
|
bc-version: [26..28]
|
|
technologies: [al]
|
|
countries: [w1]
|
|
application-area: [all]
|
|
---
|
|
```
|
|
|
|
`kind`, `id`, `version`, `title`, `description`, `inputs`, `outputs` are required and specific to action skills.
|
|
|
|
`bc-version`, `technologies`, `countries`, `application-area` are optional filters that let an orchestrator pre-select applicable skills for a task. They follow the same semantics as in READ.
|
|
|
|
`inputs` is a list of abstract input types the skill **accepts**. Standard values: `pr-diff`, `object-list`, `file-path`, `repository`, `telemetry-query`. Semantics are any-of: the orchestrator supplies whichever listed input types it has, and the skill is invoked with a non-empty subset of its declared `inputs`. A skill that cannot proceed with the supplied subset MUST return `outcome: "not-applicable"`. `outputs` is always a single-element list naming the output kind; today only `findings-report` is defined.
|
|
|
|
`sub-skills` is an optional field. When present and non-empty, the skill is a **super-skill** that composes other action skills; see *Composition* below. Values are repo-relative paths to action-skill files.
|
|
|
|
## Required sections
|
|
|
|
Every action skill MUST contain these five sections, in order:
|
|
|
|
- `## Source` — declares which folders and tags to search for knowledge.
|
|
- `## Relevance` — declares how to filter the candidates.
|
|
- `## Worklist` — declares how to narrow filtered candidates to the set that applies to this task.
|
|
- `## Action` — declares what the skill does with the narrowed set.
|
|
- `## Output` — declares the shape of the produced output; typically a reference to the contract below.
|
|
|
|
## The four-step pattern
|
|
|
|
**Source.** List the folders and tag filters to collect candidates from. Sources span layers: an action skill sources from the same `domain` subfolder across every enabled layer. Example: *"Source from `/*/knowledge/performance/` and `/*/knowledge/security/`."*
|
|
|
|
**Relevance.** Apply frontmatter filters to the candidates. Typical filters: match `bc-version` against the target environment, match `technologies` against the languages in scope, match `countries` and `application-area` against the consuming codebase's context. The exact matching rules are defined in READ (*Frontmatter matching semantics*). Files that do not match are discarded.
|
|
|
|
**Worklist.** Narrow the relevant candidates to the subset that applies to the current task. This is where the task-specific signal enters: the objects changed in the PR, the queries being audited, the skeleton being generated. Typical moves: match `keywords` against task vocabulary, match file topics against changed objects, deduplicate by concern.
|
|
|
|
**Action.** Execute the skill's work against the worklist. Evaluate each item in the worklist against the task input and emit findings. The action step is where skill behavior differs; the preceding three steps are uniform.
|
|
|
|
## Output contract
|
|
|
|
Every action skill emits a single JSON document that conforms to this schema:
|
|
|
|
```json
|
|
{
|
|
"skill": { "id": "string", "version": 1 },
|
|
"outcome": "completed | not-applicable | no-knowledge | partial | failed",
|
|
"outcome-reason": "string",
|
|
"summary": {
|
|
"counts": { "blocker": 0, "major": 0, "minor": 0, "info": 0 },
|
|
"coverage": { "worklist-size": 0, "items-evaluated": 0 }
|
|
},
|
|
"findings": [
|
|
{
|
|
"id": "string",
|
|
"severity": "blocker | major | minor | info",
|
|
"message": "string",
|
|
"location": {
|
|
"file": "string",
|
|
"line": 0,
|
|
"range": { "start-line": 0, "end-line": 0 }
|
|
},
|
|
"references": [
|
|
{ "path": "string", "sha": "string" }
|
|
],
|
|
"confidence": "high | medium | low",
|
|
"from-sub-skill": "string",
|
|
"suggested-code": "string"
|
|
}
|
|
],
|
|
"suppressed": [
|
|
{
|
|
"reference": { "path": "string", "sha": "string" },
|
|
"reason": "layer-precedence | configuration"
|
|
}
|
|
],
|
|
"sub-results": [
|
|
{ "...full nested findings-report..." : null }
|
|
],
|
|
"skipped-sub-skills": [
|
|
{
|
|
"skill": { "id": "string", "version": 1 },
|
|
"reason": "configuration | not-applicable"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### Field semantics
|
|
|
|
**`outcome`** (required) —
|
|
|
|
- `completed` — the skill ran end-to-end; `findings` reflects the full result (including the empty set).
|
|
- `not-applicable` — the skill's frontmatter filters did not match the task context; the skill declined to run.
|
|
- `no-knowledge` — the skill ran but found no applicable knowledge files; `findings` MUST be empty.
|
|
- `partial` — the skill evaluated part of its worklist but did not finish. `summary.coverage` reflects the evaluated subset. Set `outcome-reason` to explain.
|
|
- `failed` — the skill encountered an error and produced no reliable findings. Set `outcome-reason`. Consumers SHOULD ignore `findings` on a failed outcome.
|
|
|
|
`outcome-reason` is optional for `completed`, `not-applicable`, and `no-knowledge`; required for `partial` and `failed`.
|
|
|
|
An empty `findings` array with `outcome: completed` means the skill ran and found nothing to flag. Orchestrators MUST NOT conflate this with `not-applicable` or `no-knowledge`.
|
|
|
|
**`findings[].id`** — a stable identifier for the rule or concern that produced the finding. For citation-based findings (any finding with a non-empty `references`), `id` MUST equal `references[0].path` — the primary knowledge file's repo-relative path. For skills that detect concerns without a direct citation, `id` is a skill-defined slug (kebab-case, stable across versions of the skill). The same `id` produced in two runs MUST refer to the same concern; consumers MAY deduplicate findings by `id`.
|
|
|
|
When a super-skill rolls up a non-citation finding from a sub-skill (an `id` that is a slug, not a path), the super-skill MUST prefix the `id` with `<from-sub-skill>:` to avoid collisions across sub-skills (for example, a slug `missing-test` from `al-security-review` becomes `al-security-review:missing-test`). Citation-based findings are already globally unique through their repo-relative path and MUST NOT be rewritten.
|
|
|
|
**Agent findings.** A super-skill MAY emit findings that the agent identified through its own reasoning rather than from a BCQuality knowledge file. BCQuality is an **additive** knowledge layer: it augments the agent's pre-existing review judgement, it does not replace it. An agent finding is encoded by:
|
|
|
|
- `from-sub-skill: "agent"` — the canonical marker. Use this exact value; do not invent equivalents.
|
|
- `references: []` — required. An agent finding has no knowledge-file citation by definition; if a citation existed, the finding would be a knowledge-backed finding instead.
|
|
- `id` — a skill-defined slug, prefixed with `agent:` (mirroring the `<from-sub-skill>:` rule). For example, `agent:obsolete-find-signature`.
|
|
- `confidence` — capped at `medium`. Without a knowledge-file citation there is no authoritative basis for `high` confidence.
|
|
- `message` — non-empty and self-contained. It MUST describe the issue and a concrete recommendation, since a consumer rendering the finding has no knowledge-file footer to fall back on.
|
|
|
|
Agent findings are emitted **only by super-skills** (the `al-code-review` super-skill is the canonical example). Leaf sub-skills MUST NOT emit agent findings: a leaf's job is to evaluate one knowledge subset, and a finding it cannot cite from that subset is out of scope for it. Before emitting an agent finding, a super-skill MUST validate the candidate against the BCQuality knowledge it has already loaded for the task — if a knowledge file matches, the candidate is upgraded to a knowledge-backed finding (and merged or deduplicated against any sub-skill output that already covers the same concern); if a knowledge file explicitly contradicts the candidate, it is suppressed.
|
|
|
|
Consumers that render output MAY treat agent findings differently from knowledge-backed findings (for example, by labelling them and routing them to a separate review domain). The `from-sub-skill: "agent"` marker is the contract they rely on.
|
|
|
|
**`findings[].severity`** — see the taxonomy below.
|
|
|
|
**`findings[].message`** — human-readable explanation of the finding. Single short paragraph. No markdown formatting assumptions.
|
|
|
|
**`findings[].location`** — optional. When present:
|
|
|
|
- `file` MUST be a repo-relative path using forward slashes.
|
|
- `line` is the primary line number, 1-based.
|
|
- `range` is optional and describes a contiguous line span; `start-line` and `end-line` are 1-based and inclusive. `start-line` MUST equal `line` when both are present.
|
|
|
|
Findings without a `location` are permitted (for example, repository-wide observations).
|
|
|
|
**`findings[].references`** — array of knowledge-file references. Each reference is an object:
|
|
|
|
- `path` (required) — repo-relative path to the knowledge file, forward slashes.
|
|
- `sha` (optional) — commit SHA the skill read when producing the finding. Consumers SHOULD include `sha` when the skill was invoked with a specific repo state.
|
|
|
|
The first reference is the **primary** reference: the knowledge file the finding most directly cites. Additional references provide supporting context and are not ranked. `references` MAY be empty only for **agent findings** (see the `findings[].id` section above for the full encoding); any other finding MUST have at least one reference.
|
|
|
|
**`findings[].confidence`** — the skill's confidence that the finding is a true positive, given the evidence it evaluated. Not applicability confidence, not severity confidence. Values: `high`, `medium`, `low`.
|
|
|
|
**`findings[].from-sub-skill`** — optional. Set only by super-skills. The `skill.id` of the sub-skill that produced the finding, or the literal string `"agent"` for an agent finding the super-skill produced from its own reasoning. Absent on findings produced directly by a leaf skill.
|
|
|
|
**`findings[].suggested-code`** — optional. A concrete code-replacement payload for the lines indicated by `location`. When present, the string MUST be a literal replacement for the source lines covered by `location.line` (or `location.range` if set) — i.e., what the file would contain after the fix, with no surrounding diff markers, fences, or commentary. Consumers MAY render it as a one-click suggestion in the delivery surface (for example, a GitHub ```` ```suggestion ```` block). Emit `suggested-code` only when the fix is small, mechanical, and unambiguous; omit it when the appropriate fix depends on context the skill cannot determine. The `suggested-code` payload supplements `message`; it does not replace the explanation in `message`.
|
|
|
|
**`suppressed`** — MUST list every knowledge file that was discarded due to layer precedence or consumer configuration, whenever that file would otherwise have contributed to the worklist. Each entry contains:
|
|
|
|
- `reference` — the suppressed file (same object shape as `findings[].references`).
|
|
- `reason` — `layer-precedence` when another layer won under READ's precedence rules; `configuration` when the consumer disabled the file's layer.
|
|
|
|
**`sub-results`** — super-skills only. Array of complete findings-reports, one per sub-skill that was invoked (i.e., every sub-skill not listed in `skipped-sub-skills`). Each entry MUST itself conform to this output contract. Leaf skills MUST NOT emit `sub-results`.
|
|
|
|
**`skipped-sub-skills`** — super-skills only. Array of sub-skills that were declared in frontmatter but not invoked. `reason` is `configuration` when the orchestrator disabled the sub-skill, or `not-applicable` when the super-skill's Relevance step ruled it out.
|
|
|
|
Severity taxonomy:
|
|
|
|
- `blocker` — violates platform-level guarantees; the work cannot proceed as-is.
|
|
- `major` — significant defect; should be fixed before merge.
|
|
- `minor` — quality concern; worth flagging but not a gate.
|
|
- `info` — observation or context; not actionable on its own.
|
|
|
|
## Composition (super-skills)
|
|
|
|
A **super-skill** is an action skill whose frontmatter declares a non-empty `sub-skills: [...]`. A super-skill does not evaluate knowledge files directly; it invokes other action skills and composes their output.
|
|
|
|
Composition is flat: a super-skill MAY list only leaf skills (skills without their own `sub-skills`). Nested super-skills are not permitted in v1.
|
|
|
|
### Section interpretation for super-skills
|
|
|
|
The five required sections still apply. Their meaning shifts from knowledge files to sub-skills:
|
|
|
|
- `## Source` — names the sub-skills invoked (mirrors `sub-skills` in frontmatter).
|
|
- `## Relevance` — rules for deciding which sub-skills apply to the current task. A sub-skill is relevant when its declared `inputs` are satisfied by the orchestrator's provided inputs and the orchestrator has not disabled it via configuration. The super-skill MUST NOT filter sub-skills by task content (for example, by inspecting the diff or the file). Task-level applicability is the sub-skill's own responsibility; sub-skills signal non-applicability by returning `outcome: "not-applicable"` or `outcome: "no-knowledge"`.
|
|
- `## Worklist` — the final list of sub-skills to invoke; the rest go to `skipped-sub-skills`.
|
|
- `## Action` — invoke each worklisted sub-skill with the appropriate subset of inputs, collect its findings-report verbatim into `sub-results`, and copy its `findings[]` into the super-skill's top-level `findings[]` with `from-sub-skill` set. Findings from a sub-skill with `outcome: "failed"` MUST NOT be copied into the super-skill's top-level `findings[]` and MUST NOT contribute to the super-skill's `summary.counts` (their report is still preserved in `sub-results` for traceability, consistent with DO's rule that consumers ignore a failed skill's findings).
|
|
- `## Output` — the super-skill's output contract, including `sub-results` and, if any, `skipped-sub-skills`.
|
|
|
|
### Outcome rollup
|
|
|
|
A super-skill's `outcome` is derived from its sub-skills' outcomes. Let S be the multiset of sub-skill outcomes for sub-skills in the worklist (skipped sub-skills do not contribute):
|
|
|
|
- `failed` — every element of S is `failed`.
|
|
- `partial` — S contains at least one `partial`, OR S contains at least one `failed` alongside at least one non-`failed` outcome.
|
|
- `not-applicable` — every element of S is `not-applicable`.
|
|
- `no-knowledge` — every element of S is `no-knowledge` or `not-applicable`, and at least one is `no-knowledge`.
|
|
- `completed` — otherwise (every element of S is `completed`, `no-knowledge`, or `not-applicable`, with at least one `completed`).
|
|
|
|
When the worklist is empty (every sub-skill was skipped), `outcome` is `not-applicable`; `outcome-reason` SHOULD describe the skip reasons, for example *"all sub-skills disabled by configuration"* or *"no sub-skill accepted the supplied inputs"*.
|
|
|
|
`outcome-reason` is required for `partial` and `failed` and SHOULD summarize per-sub-skill state.
|
|
|
|
### Rolled-up summary
|
|
|
|
`summary.counts` is the sum of sub-skill counts. `summary.coverage.worklist-size` and `items-evaluated` are the sums across invoked sub-skills.
|
|
|
|
### Suppression scope
|
|
|
|
A super-skill's top-level `suppressed[]` remains knowledge-file-only and is typically empty. Knowledge-file suppression is reported by the leaf sub-skill inside its own entry in `sub-results`. Sub-skills the super-skill chose not to invoke belong in `skipped-sub-skills`, never in `suppressed`.
|
|
|
|
## Worked example
|
|
|
|
A minimal action skill that cites applicable guidance for a changed AL file, without generating findings of its own:
|
|
|
|
```yaml
|
|
---
|
|
kind: action-skill
|
|
id: cite-applicable-guidance
|
|
version: 1
|
|
title: Cite applicable guidance
|
|
description: Lists knowledge files relevant to a changed AL file.
|
|
inputs: [file-path]
|
|
outputs: [findings-report]
|
|
technologies: [al]
|
|
---
|
|
```
|
|
|
|
```markdown
|
|
## Source
|
|
All files under `/*/knowledge/` across enabled layers.
|
|
|
|
## Relevance
|
|
Filter by `technologies: [al]` and `bc-version` matching the target environment.
|
|
|
|
## Worklist
|
|
Intersect `keywords` with tokens derived from the target file's object name and changed members.
|
|
|
|
## Action
|
|
For each worklist entry, emit one finding with severity `info`, a message naming the concern, and a reference object pointing to the knowledge file.
|
|
|
|
## Output
|
|
Conforms to the DO output contract.
|
|
```
|
|
|
|
## How orchestrators consume output
|
|
|
|
An orchestrator invokes an action skill with an input appropriate to the skill's declared `inputs`, receives the JSON output, and maps findings to its delivery surface (PR comments, build gates, IDE diagnostics). The orchestrator MUST NOT interpret skill-specific fields beyond the schema above. Skills that need richer semantics MUST encode them within the schema (for example, by adding structured `message` text) rather than extending the output shape.
|