mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-08-09 02:41:37 +01:00
3a07ee82d2
The original DO contract pinned all agent reasoning to the super-skill: 'Agent findings are emitted only by super-skills... Leaf sub-skills MUST NOT emit agent findings'. This funnels all agent reasoning across all 6 domains through a single super-skill pass, which is the root structural cause of the attention dilution we have been chasing in BCAppsBCQuality PRs #28 and #30: - T1 standalone run showed al-security-review finds rimd-on-read-only cleanly when run alone, but emits zero agent findings because the contract forbids it. So obvious things like case-without-else (no matching KB article yet) get dropped on the floor. - The al-code-review self-review pass keeps producing 0-1 agent findings per PR because it is asked to reason across 6 domains in one pass. The fix is to move agent reasoning into the leaves, bounded by each leaf's domain. Each leaf now has both knowledge-backed and agent-finding permissions within its own scope; the super-skill self-review pass becomes a smaller, cross-cutting role. skills/do.md - Replace the 'only by super-skills' / 'MUST NOT' clause with a two-tier model: leaf sub-skills MAY emit agent findings strictly within their declared domain; super-skills MAY emit agent findings for cross-cutting concerns that span domains. - Update the encoding rules: leaf agent findings have references:[] and an agent:-prefixed id, no from-sub-skill (the leaf's own report carries the finding under its own skill.id). Super-skills set from-sub-skill='agent' for their own self-review findings; when rolling up leaf agent findings, they set from-sub-skill=<leaf-id>. - Clarify that 'MUST validate against knowledge' applies to super- skill self-review candidates only - leaves already validated within their domain when they decided to emit. microsoft/skills/review/al-{security,performance,privacy,style,upgrade, ui}-review.md - New paragraph after the confidence rules instructing each leaf to surface domain-specific agent findings when no knowledge file covers a defect the agent recognises from general AL knowledge. - Bound the scope: 'The scope is strictly <domain>; defects outside this domain belong to other leaves and MUST NOT be emitted here.' - Same validation requirement: check the worklist for a matching knowledge file first; if one exists, upgrade to a knowledge-backed finding instead. microsoft/skills/review/al-code-review.md - Rewrite the 'Agent self-review pass' subsection. The pass is now explicitly for cross-cutting concerns that no single leaf could have surfaced because they span multiple domains. Domain-specific reasoning belongs in the leaves, not duplicated here. - Update the rollup behavior to acknowledge leaf-emitted agent findings: they are rolled up like any other sub-skill finding, with from-sub-skill set to the leaf id, and are not re-validated by the super-skill (the leaf already validated within its own domain). - Drop the 'Leaf sub-skills MUST NOT emit agent findings' line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
106 lines
8.7 KiB
Markdown
106 lines
8.7 KiB
Markdown
---
|
|
kind: action-skill
|
|
id: al-privacy-review
|
|
version: 1
|
|
title: AL privacy review
|
|
description: Reviews AL source changes against privacy and data-classification guidance from BCQuality.
|
|
inputs: [pr-diff, file-path]
|
|
outputs: [findings-report]
|
|
bc-version: [all]
|
|
technologies: [al]
|
|
countries: [w1]
|
|
application-area: [all]
|
|
---
|
|
|
|
# AL privacy review
|
|
|
|
Reviews AL source changes against the `privacy` knowledge domain in BCQuality and emits a findings report. This is a leaf action skill: it invokes no sub-skills. It is one of the skills composed by `al-code-review`.
|
|
|
|
An orchestrator invokes this skill with either a `pr-diff` (the standard PR-review entry point) or a `file-path` (single-file review). The skill produces a single JSON document conforming to the DO output contract.
|
|
|
|
## Source
|
|
|
|
Collect all knowledge files under `*/knowledge/privacy/**/*.md`, across every enabled layer (`/microsoft/`, `/community/`, `/custom/`). Relevance trims the result to the subset that applies.
|
|
|
|
## Relevance
|
|
|
|
Apply the frontmatter matching rules defined in READ (*Frontmatter matching semantics*) against the task context:
|
|
|
|
- `bc-version` — the target BC version from the PR branch's `app.json` or the orchestrator-supplied version. If unavailable, the dimension is `unknown`.
|
|
- `technologies` — `[al]`.
|
|
- `countries` — the countries declared in the consuming app's `app.json`. Default to the orchestrator's configured context; if absent, `unknown`.
|
|
- `application-area` — the union of application areas declared by the changed objects. Pass the actual set; do not substitute `[all]`. If the area cannot be determined from the changes, the dimension is `unknown`.
|
|
|
|
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when the orchestrator's configuration permits them; findings derived from those files MUST have `confidence` no higher than `medium`, AND the finding's `message` MUST name the dimension or dimensions that were unknown.
|
|
|
|
## Worklist
|
|
|
|
Narrow the relevant files to the subset that applies to the changes under review. Exclude test codeunits, test libraries, test helper code, files under test/Test/Tests paths, and objects with `Subtype = Test`; test data is synthetic and does not ship to customers. For each relevant file, compute overlap against:
|
|
|
|
- The changed AL object names and types — especially tables and tableextensions (for `DataClassification` on fields), codeunits that call `Error`, `Session.LogMessage`, or `FeatureTelemetry`, codeunits performing outgoing HTTP requests with customer data, migration codeunits, and objects reading or writing `IsolatedStorage`.
|
|
- The changed procedures and triggers, weighted toward those that call `Error`, `Session.LogMessage`, `StrSubstNo`, `GetLastErrorText`, `FeatureTelemetry.LogUsage`/`LogUptake`/`LogError`, `HttpClient.Post`/`Get`, `IsolatedStorage.Set`/`SetEncrypted`/`Get`, or `PrivacyNotice.GetPrivacyNoticeApprovalState`.
|
|
- Tokens extracted from the diff that relate to privacy (`DataClassification`, `CustomerContent`, `EndUserIdentifiableInformation`, `EndUserPseudonymousIdentifiers`, `SystemMetadata`, `ToBeClassified`, `PrivacyNotice`, `GetLastErrorText`, `TelemetryScope`, `FeatureTelemetry`, `CustomDimensions`, `LogUsage`, `LogUptake`, `LogError`, `HybridSL`, `HybridGP`, `HybridBC`).
|
|
|
|
A file enters the candidate worklist when its `keywords` intersect the extracted tokens or its topic (derived from filename and Description) matches a changed object type.
|
|
|
|
Once the candidate worklist is known, resolve layer-precedence conflicts per READ. Drop lower-precedence files whose normative guidance (`## Best Practice` or `## Anti Pattern`) directly contradicts a higher-precedence candidate, and record each dropped file in `suppressed` with `reason: "layer-precedence"`. Files that would have been candidates but are hidden because their layer is disabled in consumer configuration are recorded with `reason: "configuration"`. Files that never became candidates are NOT recorded in `suppressed`.
|
|
|
|
When the post-conflict worklist is empty because no applicable privacy knowledge exists, or because configuration suppressed every candidate, emit `outcome: "no-knowledge"`. When the worklist is empty because no applicable privacy knowledge matched the changes, emit `outcome: "completed"` with an empty `findings` array.
|
|
|
|
## Action
|
|
|
|
For each worklist entry, evaluate the diff against the file's `## Best Practice` and `## Anti Pattern` sections. Emit findings as follows:
|
|
|
|
- When the diff contains a clear match for an Anti Pattern, emit a finding with severity `major` or `blocker`, a message summarizing the anti-pattern, `location` pointing to the offending line or range, and a `references` entry pointing to the knowledge file. Use `blocker` only when the knowledge file states the anti-pattern violates a platform-level guarantee (for example, documented telemetry-classification rules or GDPR-adjacent data-handling requirements). When the file does not make such a claim, the ceiling is `major`.
|
|
- When the diff contains code that contradicts a Best Practice without being a full anti-pattern, emit `minor` with the same reference shape.
|
|
- When the skill cannot detect a violation but the file is clearly applicable to the change, emit `info` citing the file. Repository-wide observations MAY omit `location`.
|
|
|
|
Set `confidence` to:
|
|
|
|
- `high` when the detection is based on an unambiguous pattern match (identifier, syntax, object type).
|
|
- `medium` when detection relies on heuristics or when any frontmatter dimension was `unknown`.
|
|
- `low` when the finding is an advisory derived only from applicability.
|
|
|
|
After evaluating each worklist entry, also consider whether the diff exhibits a privacy defect the agent recognises from its general AL knowledge that no knowledge file in the worklist covers. Such candidates are agent findings within this skill's domain — emit them with `references: []`, an `id` slug prefixed with `agent:`, `confidence` capped at `medium`, and a `message` that is self-contained (describing both the issue and a concrete recommendation, since there is no knowledge-file footer for the consumer to fall back on). The scope is strictly privacy; defects outside this domain belong to other leaves and MUST NOT be emitted here. Before emitting, check the worklist for a knowledge file that matches the candidate — if one exists, upgrade the candidate to a knowledge-backed finding instead. See `skills/do.md` for the full contract.
|
|
|
|
When the knowledge file ships an unambiguous `.good.al` companion that names exactly the correction the finding requires (and the diff context makes the substitution mechanical), set `findings[].suggested-code` to the literal replacement for the source lines indicated by `location`. The payload must be a verbatim replacement — no diff markers, no fences, no commentary — that the consumer can render as a one-click suggestion. Skip the field when the appropriate fix depends on context the skill cannot determine, or when more than one defensible replacement exists. See `skills/do.md` for the full contract.
|
|
|
|
Outcome selection:
|
|
|
|
- `completed` — the skill evaluated every worklist item; default when the skill finishes normally, including when the resulting `findings` array is empty.
|
|
- `no-knowledge` — no applicable privacy knowledge survived Source, Relevance, configuration filtering, and conflict resolution. `findings` is empty.
|
|
- `not-applicable` — the task context lacks an AL dimension (no AL changes in the diff, or `technologies` filter rejected the task).
|
|
- `partial` — a time or token budget was hit before the worklist was exhausted. `summary.coverage` reflects the evaluated subset; `outcome-reason` explains the cause.
|
|
- `failed` — an unrecoverable error occurred. `outcome-reason` is required.
|
|
|
|
## Output
|
|
|
|
Output conforms to the DO output contract. A populated example:
|
|
|
|
```json
|
|
{
|
|
"skill": { "id": "al-privacy-review", "version": 1 },
|
|
"outcome": "completed",
|
|
"summary": {
|
|
"counts": { "blocker": 0, "major": 1, "minor": 0, "info": 0 },
|
|
"coverage": { "worklist-size": 1, "items-evaluated": 1 }
|
|
},
|
|
"findings": [
|
|
{
|
|
"id": "microsoft/knowledge/privacy/strsubstno-prebuild-breaks-error-telemetry-classification.md",
|
|
"severity": "major",
|
|
"message": "Error receives a pre-built Text produced by StrSubstNo with customer name and email as arguments. Per the referenced guidance the platform cannot classify or strip PII from an opaque Text and will export the full message to telemetry.",
|
|
"location": {
|
|
"file": "src/Sales/CustomerValidation.Codeunit.al",
|
|
"line": 64,
|
|
"range": { "start-line": 60, "end-line": 64 }
|
|
},
|
|
"references": [
|
|
{ "path": "microsoft/knowledge/privacy/strsubstno-prebuild-breaks-error-telemetry-classification.md" }
|
|
],
|
|
"confidence": "high"
|
|
}
|
|
],
|
|
"suppressed": []
|
|
}
|
|
```
|