mirror of
https://github.com/microsoft/BCQuality.git
synced 2026-08-07 09:56:52 +01:00
Add new action skills for AL testing and documentation
- Introduced `al-test-writer` to generate AL test codeunits for production objects based on TDD principles. - Added `al-userguide-test-writer` to create test codeunits from user guide steps, mapping actions and assertions. - Implemented `bc-extension-test-guide` to generate a comprehensive TEST_GUIDE.md for Business Central extensions, covering various categories. - Created `bc-webclient-runner` to automate UI testing of the Business Central web client, capturing screenshots and asserting UI states. - Developed `page-scripting-e2e` to produce a recording plan for Page Scripting, ensuring a structured approach to browser-level testing.
This commit is contained in:
Tharanga Chandrasekara
parent
822cae1b27
commit
07140e2223
76 changed files with 4353 additions and 6 deletions
90
custom/skills/testing/ai-test-driven-development.md
Normal file
90
custom/skills/testing/ai-test-driven-development.md
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
---
|
||||
kind: action-skill
|
||||
id: ai-test-driven-development
|
||||
version: 1
|
||||
title: AI test-driven development
|
||||
description: TDD for Copilot features and custom agents covering Evaluation suites, JSONL/YAML datasets, AITest codeunits, agent turn loops, intervention validation, and credit tracking.
|
||||
inputs: [repository, file-path]
|
||||
outputs: [findings-report]
|
||||
bc-version: [all]
|
||||
technologies: [al]
|
||||
countries: [w1]
|
||||
application-area: [all]
|
||||
---
|
||||
|
||||
# AI test-driven development
|
||||
|
||||
Drives test-driven development for Business Central Copilot features and custom agents using the Evaluation suite (the data-driven tool where datasets describe inputs and expected outputs and the test codeunit drives the loop). It covers two flows: prompt-based AI tests for PromptDialog features (JSONL or YAML datasets) and multi-turn agent accuracy tests with intervention validation (YAML only). This is a generator-style skill: it generates AITest codeunits, dataset scaffolds, and suite XML as findings whose `suggested-code` carries the artifact, and it also reviews an existing AI test setup for misconfiguration. This is a leaf action skill: it invokes no sub-skills.
|
||||
|
||||
An orchestrator invokes this skill with a `repository` (the Copilot or agent extension and its test app) and a `file-path` (the capability, PromptDialog, or agent under test, or an existing AI test codeunit or dataset to review). It produces a single JSON document conforming to the DO output contract.
|
||||
|
||||
## Source
|
||||
|
||||
Read the BCQuality knowledge index once (the `knowledge-index.json` Entry's preparation step regenerates over the live, already-filtered clone). Take the index entries whose `domain` is `testing` as the citable candidate set across every enabled layer: AITest-codeunit attribute rules, isolation rules (`TestType = AITest`, `RequiredTestIsolation = Disabled` for agent tests), suite-setup discipline, intervention-contract rules, and credit-tracking guidance back the findings here. Do not open individual article files at this step; open an article's full body only once it enters the Worklist below. Generated artifacts and configuration observations with no curated backing are agent findings within this skill's domain (see Action).
|
||||
|
||||
## Relevance
|
||||
|
||||
Apply the frontmatter matching rules defined in READ against the task context:
|
||||
|
||||
- `bc-version`: the target BC version from the repository `app.json`, or `unknown` if unavailable.
|
||||
- `technologies`: `[al]`.
|
||||
- `countries`: the consuming app's declared countries, or `unknown` (the suite XML may enable multilingual evaluation, so countries can matter).
|
||||
- `application-area`: the application areas of the Copilot or agent feature, or `unknown`.
|
||||
|
||||
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when configuration permits; findings derived from them have `confidence` no higher than `medium`, and the finding `message` names the unknown dimensions.
|
||||
|
||||
## Worklist
|
||||
|
||||
Narrow to the AI test work the task requires:
|
||||
|
||||
- The flow: prompt-based AI test (PromptDialog plus an Azure OpenAI call, JSONL or YAML dataset) or agent accuracy test (multi-turn, YAML only).
|
||||
- The AITest codeunit: `Subtype = Test`, `TestType = AITest`, `TestPermissions = Disabled`, and for agent tests `RequiredTestIsolation = Disabled` (essential, because agent tasks run in a different session and span transactions).
|
||||
- The turn loop for agent tests: the `repeat ... until` delegating to `Library - Agent` (`RunTurnAndWait`, `FinalizeTurn`), with validators returning `false` and a populated `ErrorReason` rather than calling `Error()`.
|
||||
- The dataset: `test_setup` and `expected_data` keys for AI tests; `turns:` with `query`/`expected_data` for agent tests; the `intervention_request` sub-key the framework reads automatically (both directions: a declared intervention must pause with matching type and suggestions, an undeclared one must not pause); `$DateFormula-<...>$` placeholders, always quoted.
|
||||
- The suite XML: `TestRunnerId="130451"` (Isolation-Disabled runner, required for agent tests), `TestType="Agent"` versus `"AITest"`, `<Language>` children, and the install-time dataset load.
|
||||
- Suite-setup discipline: `AITTestContext.IsSuiteSetupDone()` is sticky; re-running setup needs the Reset Suite Setup action.
|
||||
- Credit and permission constraints: Evaluation runs consume Copilot credits (tracked per suite, per line, per entry; limited at environment and company level); users need the `AI TEST TOOLKIT` permission set.
|
||||
|
||||
A curated `testing` file enters the worklist when its `keywords` intersect these tokens. Read its full body only after it makes the worklist. Resolve layer-precedence conflicts per READ and record dropped files in `suppressed`.
|
||||
|
||||
## Action
|
||||
|
||||
When generating, emit one finding per generated artifact (the AITest codeunit, the dataset, the suite XML, the install codeunit), each carrying the artifact in `suggested-code` with a `message` stating what was generated and how to wire it. When reviewing an existing setup, emit a finding per defect: a missing `RequiredTestIsolation = Disabled` on an agent test, a wrong `TestRunnerId`, a validator calling `Error()` instead of returning `ErrorReason`, an unquoted date placeholder, an intervention contract the dataset does not exercise in both directions, or a sticky suite-setup that silently ignores edited setup YAML. Where a curated `testing` file states the rule, emit a knowledge-backed finding citing it: `id` equal to the file path, `severity` up to `major` (`blocker` only when the file states a platform-level guarantee, for example an isolation rule whose violation makes the agent runner unusable), `confidence` `high` for an unambiguous match. Where no curated file applies, emit an agent finding within this skill's domain: `references: []`, `id` slug prefixed `agent:` (for example `agent:generated-aitest-codeunit`, `agent:missing-disabled-isolation`, `agent:unquoted-date-placeholder`), `confidence` capped at `medium`, `severity` capped at `minor`, self-contained `message`. Put generated AL, YAML, or XML in `suggested-code`; for a mechanical fix to an existing file (adding the isolation attribute, quoting a placeholder) also set `suggested-code`. Where the fix is not local (restructuring a turn loop), set `suggested-code-omission-reason`. Hold every agent finding to the precision bar in `skills/do.md`.
|
||||
|
||||
Outcome selection: `completed` when the requested generation or review finished (including a clean review with empty `findings`); `not-applicable` when the repository has no Copilot capability, PromptDialog, or custom agent to test; `no-knowledge` when no curated knowledge survived and no agent finding was raised; `partial` or `failed` per the DO contract with `outcome-reason`.
|
||||
|
||||
## Output
|
||||
|
||||
Output conforms to the DO output contract. Generated artifacts and configuration findings with no curated backing are agent findings (`references: []`, `agent:` id, severity capped at `minor`); findings citing a `testing` file carry that file path as `id` and primary reference.
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "id": "ai-test-driven-development", "version": 1 },
|
||||
"outcome": "completed",
|
||||
"summary": {
|
||||
"counts": { "blocker": 0, "major": 1, "minor": 1, "info": 0 },
|
||||
"coverage": { "worklist-size": 3, "items-evaluated": 3 }
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "agent:missing-disabled-isolation",
|
||||
"severity": "minor",
|
||||
"message": "Agent accuracy codeunit 50202 sets TestType = AITest but not RequiredTestIsolation = Disabled. Impact is major: agent tasks run in a different session and span transactions, so the runner cannot enforce isolation and the suite fails to start. Add the attribute. Promote to a curated rule so it can gate.",
|
||||
"location": { "file": "test/MyAgentAccuracyTest.Codeunit.al", "line": 4 },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code": " RequiredTestIsolation = Disabled;"
|
||||
},
|
||||
{
|
||||
"id": "agent:generated-agent-dataset",
|
||||
"severity": "info",
|
||||
"message": "Generated a YAML agent dataset with a turns chain and an intervention_request the FinalizeTurn contract enforces in both directions. Date values use quoted $DateFormula placeholders so the dataset does not drift against WorkDate. Ship it under the test app .resources/ folder and load it in an Install codeunit.",
|
||||
"location": { "file": "test/.resources/datasets/MY-DATASET.yaml" },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code": "name: MY-DATASET\nsuite_setup: MY-AGENT\ntests:\n - turns:\n - query:\n message: \"Release all open sales orders for next week\"\n expected_data:\n orders_released: 2"
|
||||
}
|
||||
],
|
||||
"suppressed": []
|
||||
}
|
||||
```
|
||||
91
custom/skills/testing/al-test-coverage-enforcer.md
Normal file
91
custom/skills/testing/al-test-coverage-enforcer.md
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
---
|
||||
kind: action-skill
|
||||
id: al-test-coverage-enforcer
|
||||
version: 1
|
||||
title: AL test coverage enforcer
|
||||
description: Hard coverage gate, passing only when AL coverage meets the threshold and otherwise naming every uncovered path.
|
||||
inputs: [pr-diff, repository]
|
||||
outputs: [findings-report]
|
||||
bc-version: [all]
|
||||
technologies: [al]
|
||||
countries: [w1]
|
||||
application-area: [all]
|
||||
---
|
||||
|
||||
# AL test coverage enforcer
|
||||
|
||||
Decides whether a Business Central change has enough AL test coverage to ship. Unlike `al-test-coverage-validator`, which reports, this skill gates: it passes (an empty `findings` array) only when every new or behaviour-changed production surface has at least one identifiable covering test, and otherwise emits a finding per uncovered path at gating severity. When in doubt it fails, because the cost of one extra test is low and the cost of an uncovered regression is high. This is a leaf action skill: it invokes no sub-skills.
|
||||
|
||||
An orchestrator invokes this skill with a `pr-diff` (the production change to gate) and a `repository` (so the test index and any coverage report can be read). It produces a single JSON document conforming to the DO output contract.
|
||||
|
||||
## Source
|
||||
|
||||
Read the BCQuality knowledge index once (the `knowledge-index.json` Entry's preparation step regenerates over the live, already-filtered clone). Take the index entries whose `domain` is `testing` as the citable candidate set across every enabled layer: a curated rule about a coverage threshold or a mandatory regression test is the authoritative basis that lets this skill gate at `major` or `blocker`. Do not open individual article files at this step; open an article's full body only once it enters the Worklist below. Where the project threshold is a house default with no curated backing, see Action for how severity is handled.
|
||||
|
||||
## Relevance
|
||||
|
||||
Apply the frontmatter matching rules defined in READ against the task context:
|
||||
|
||||
- `bc-version`: the target BC version from the repository `app.json`, or `unknown` if unavailable.
|
||||
- `technologies`: `[al]`.
|
||||
- `countries`: the consuming app's declared countries, or `unknown`.
|
||||
- `application-area`: the application areas of the changed objects, or `unknown`.
|
||||
|
||||
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when configuration permits; findings derived from them have `confidence` no higher than `medium`, and the finding `message` names the unknown dimensions.
|
||||
|
||||
## Worklist
|
||||
|
||||
Narrow to the production surfaces the threshold applies to:
|
||||
|
||||
- New public procedures (default threshold: every one must have at least one direct or indirect covering test, no exceptions).
|
||||
- New event subscribers (must have a test that fires the publisher in a realistic context).
|
||||
- New table triggers (`OnInsert`, `OnModify`, `OnDelete`, field `OnValidate`): each must have a covering test.
|
||||
- Modified procedures with a behaviour change: an existing or new test must assert the new behaviour. A behaviour change whose existing tests still pass unchanged is itself a gap (the tests do not exercise the new behaviour).
|
||||
- Bug fixes: must add a regression test that names the bug and fails without the fix.
|
||||
- Pure refactors with no behaviour change: existing covering tests must still apply; no new test required.
|
||||
|
||||
Compute the covering set from the test index (procedure to referencing tests) and any supplied coverage report. A curated `testing` file enters the worklist when its `keywords` intersect a coverage-threshold rule. Read its full body only after it makes the worklist. Resolve layer-precedence conflicts per READ and record dropped files in `suppressed`.
|
||||
|
||||
## Action
|
||||
|
||||
For each worklisted surface, decide PASS or FAIL against the threshold. Emit a finding for every FAIL reason, each naming the specific procedure, subscriber, trigger, or bug fix. Where a curated `testing` knowledge file states the coverage requirement, emit a knowledge-backed finding citing it: `id` equal to the file path, `severity` `blocker` when the file states a platform-level guarantee, otherwise `major`; `confidence` `high` for an unambiguous gap. Where the requirement is the house default with no curated backing, the finding is an agent finding within this skill's domain (`references: []`, `id` slug prefixed `agent:` such as `agent:uncovered-new-public-procedure` or `agent:bug-fix-missing-regression-test`, `confidence` capped at `medium`). Per `skills/do.md`, an agent finding's `severity` is capped at `minor` even though this skill gates: keep the emitted severity at `minor`, state plainly in the `message` that the impact is gating (it blocks completion under the project threshold), and flag that the threshold should be promoted to a curated `testing` rule so the gate carries authoritative weight. The consuming orchestrator combines the threshold configuration with these findings to set the actual PASS/FAIL on the merge. The fix is a new test, so omit `suggested-code` and set `suggested-code-omission-reason` to `the gap is closed by adding a covering test`.
|
||||
|
||||
When no coverage report is supplied, fall back to static analysis of the test codeunits (which procedures each test references directly) and state the limitation in an `info` finding. When in doubt about whether a surface is covered, prefer to emit the finding.
|
||||
|
||||
Outcome selection: `completed` when every worklisted surface was decided (an empty `findings` array means PASS, the gate is satisfied); `not-applicable` when the diff has no new or behaviour-changed production surface (a pure refactor or doc-only change); `partial` or `failed` per the DO contract with `outcome-reason`.
|
||||
|
||||
## Output
|
||||
|
||||
Output conforms to the DO output contract. An empty `findings` array with `outcome: completed` is the PASS signal. Threshold findings with no curated backing are agent findings (`references: []`, `agent:` id, severity capped at `minor`, gating impact stated in the message); findings citing a `testing` file carry that file path as `id` and may gate at `major` or `blocker`.
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "id": "al-test-coverage-enforcer", "version": 1 },
|
||||
"outcome": "completed",
|
||||
"summary": {
|
||||
"counts": { "blocker": 0, "major": 0, "minor": 2, "info": 0 },
|
||||
"coverage": { "worklist-size": 3, "items-evaluated": 3 }
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "agent:uncovered-new-public-procedure",
|
||||
"severity": "minor",
|
||||
"message": "codeunit 50101 'Event Registration Mgt'.ReleaseRegistration is new in this diff and no test references it directly or indirectly. Impact is gating: under the project threshold this blocks completion. Promote the threshold to a curated testing rule so the gate carries authoritative weight.",
|
||||
"location": { "file": "src/Sales/EventRegistrationMgt.Codeunit.al" },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code-omission-reason": "the gap is closed by adding a covering test"
|
||||
},
|
||||
{
|
||||
"id": "agent:bug-fix-missing-regression-test",
|
||||
"severity": "minor",
|
||||
"message": "The commit references work item #1234 (a fix) but no new test names the item or asserts the prior failure mode. Impact is gating: a bug fix must ship with a regression test that fails without the fix.",
|
||||
"location": { "file": "src/Sales/EventRegistrationMgt.Codeunit.al" },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code-omission-reason": "the gap is closed by adding a covering test"
|
||||
}
|
||||
],
|
||||
"suppressed": []
|
||||
}
|
||||
```
|
||||
91
custom/skills/testing/al-test-coverage-validator.md
Normal file
91
custom/skills/testing/al-test-coverage-validator.md
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
---
|
||||
kind: action-skill
|
||||
id: al-test-coverage-validator
|
||||
version: 1
|
||||
title: AL test coverage validator
|
||||
description: Reports AL test coverage shape and identifies untested branches, error handlers, and edge cases without gating.
|
||||
inputs: [pr-diff, repository]
|
||||
outputs: [findings-report]
|
||||
bc-version: [all]
|
||||
technologies: [al]
|
||||
countries: [w1]
|
||||
application-area: [all]
|
||||
---
|
||||
|
||||
# AL test coverage validator
|
||||
|
||||
Reports what a production AL change is and is not covered by, and how meaningfully. For each production procedure or trigger the diff touches, it classifies coverage as covered, shallow, uncovered, or not-applicable, and surfaces uncovered branches, error handlers, and edge cases. It reports; it does not gate. The hard PASS/FAIL decision lives in `al-test-coverage-enforcer`. This is a leaf action skill: it invokes no sub-skills.
|
||||
|
||||
An orchestrator invokes this skill with a `pr-diff` (the production change to assess) and a `repository` (so the test codeunit index and any coverage report can be read). It produces a single JSON document conforming to the DO output contract.
|
||||
|
||||
## Source
|
||||
|
||||
Read the BCQuality knowledge index once (the `knowledge-index.json` Entry's preparation step regenerates over the live, already-filtered clone). Take the index entries whose `domain` is `testing` as the citable candidate set across every enabled layer; coverage is structural and rarely maps onto a curated rule, but a published `testing` rule about minimum coverage for a specific area would back a finding. Do not open individual article files at this step; open an article's full body only once it enters the Worklist below. Most coverage observations are agent findings within this skill's domain (see Action).
|
||||
|
||||
## Relevance
|
||||
|
||||
Apply the frontmatter matching rules defined in READ against the task context:
|
||||
|
||||
- `bc-version`: the target BC version from the repository `app.json`, or `unknown` if unavailable.
|
||||
- `technologies`: `[al]`.
|
||||
- `countries`: the consuming app's declared countries, or `unknown`.
|
||||
- `application-area`: the application areas of the changed objects, or `unknown`.
|
||||
|
||||
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when configuration permits; findings derived from them have `confidence` no higher than `medium`, and the finding `message` names the unknown dimensions.
|
||||
|
||||
## Worklist
|
||||
|
||||
Narrow to the production surface the diff touches and the coverage signal for each:
|
||||
|
||||
- Every production procedure and trigger (`OnInsert`, `OnValidate`, and so on) added or changed in the diff, excluding test files.
|
||||
- For each, the test index: which test codeunits reference it directly or via a clear chain, and whether any asserts an outcome that depends on its body.
|
||||
- Branch and error-handler coverage: uncovered `if`/`case` arms, uncovered `Error()` paths, uncovered `else` guards.
|
||||
- Edge-case coverage where the procedure can encounter them: nulls, empty sets, max values, permission failures, date boundaries.
|
||||
- Bug-fix regression coverage: when the diff message references a work item or fix, whether a test names it.
|
||||
- Mutation survivors, when supplied: mutants that survived because no test caught the logic change.
|
||||
|
||||
A curated `testing` file enters the worklist only when its `keywords` intersect a real coverage rule. Read its full body only after it makes the worklist. Resolve layer-precedence conflicts per READ and record dropped files in `suppressed`.
|
||||
|
||||
## Action
|
||||
|
||||
Classify each worklisted procedure or trigger: `covered` (a test calls it and asserts an outcome exercising its body), `shallow` (a test calls it but asserts nothing depending on its return or side effects), `uncovered` (no test references it), or `n/a` (deleted or a pure pass-through). Emit a finding for every gap. These are agent findings within this skill's domain: `references: []`, `id` slug prefixed `agent:` (for example `agent:uncovered-procedure`, `agent:shallow-coverage`, `agent:uncovered-branch`, `agent:missing-edge-case`), `confidence` capped at `medium`, `severity` capped at `minor` (this skill reports, it does not gate, so even a wholly uncovered new public procedure is `minor` here and the enforcer raises it). The `message` is self-contained: name the object and procedure, the coverage class, and the concrete gap (which branch, which edge case, which assertion is missing). Where a published `testing` rule genuinely backs a coverage requirement, upgrade that finding to knowledge-backed and cite the file. Hold every agent candidate to the precision bar in `skills/do.md`: a pass-through that genuinely needs no test is not a gap. The fix is a new or extended test rather than a local edit, so omit `suggested-code` and set `suggested-code-omission-reason` to `coverage gap is closed by adding a test, not a local code edit`.
|
||||
|
||||
When the test index is missing, report only what static analysis of the diff yields (new procedure count, new trigger count) and state the limitation in an `info` finding.
|
||||
|
||||
Outcome selection: `completed` when every touched procedure was classified (including an empty `findings` when coverage is complete); `not-applicable` when the diff has no production AL surface; `partial` or `failed` per the DO contract with `outcome-reason`.
|
||||
|
||||
## Output
|
||||
|
||||
Output conforms to the DO output contract. Coverage gaps are agent findings (`references: []`, `agent:` id, severity capped at `minor`); the gating decision lives in `al-test-coverage-enforcer`.
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "id": "al-test-coverage-validator", "version": 1 },
|
||||
"outcome": "completed",
|
||||
"summary": {
|
||||
"counts": { "blocker": 0, "major": 0, "minor": 2, "info": 0 },
|
||||
"coverage": { "worklist-size": 8, "items-evaluated": 8 }
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "agent:uncovered-procedure",
|
||||
"severity": "minor",
|
||||
"message": "codeunit 50101 'Event Registration Mgt'.ReleaseRegistration is touched by the diff but no test codeunit references it directly or indirectly. Add a covering test. The enforcer will gate on this; here it is reported only.",
|
||||
"location": { "file": "src/Sales/EventRegistrationMgt.Codeunit.al" },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code-omission-reason": "coverage gap is closed by adding a test, not a local code edit"
|
||||
},
|
||||
{
|
||||
"id": "agent:shallow-coverage",
|
||||
"severity": "minor",
|
||||
"message": "ValidateAttendeeCount is called by a test but the test asserts nothing depending on its outcome, so the resulting error path is not actually verified. Add an asserterror on the over-capacity case.",
|
||||
"location": { "file": "src/Sales/EventRegistrationMgt.Codeunit.al" },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code-omission-reason": "coverage gap is closed by adding a test, not a local code edit"
|
||||
}
|
||||
],
|
||||
"suppressed": []
|
||||
}
|
||||
```
|
||||
87
custom/skills/testing/al-test-runner.md
Normal file
87
custom/skills/testing/al-test-runner.md
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
---
|
||||
kind: action-skill
|
||||
id: al-test-runner
|
||||
version: 1
|
||||
title: AL test runner
|
||||
description: Executes AL test codeunits via the AL-Go runner or a local container and returns the run result as a findings report.
|
||||
inputs: [repository, object-list]
|
||||
outputs: [findings-report]
|
||||
bc-version: [all]
|
||||
technologies: [al]
|
||||
countries: [w1]
|
||||
application-area: [all]
|
||||
---
|
||||
|
||||
# AL test runner
|
||||
|
||||
Executes Business Central AL test codeunits and reports the run result in a shape the rest of the verifier chain can consume. It detects the project's runner (AL-Go pipeline, a Docker BC sandbox via BcContainerHelper, or a project-local build script), invokes it, parses the XUnit-style results, and maps each failure to a finding. It does not judge coverage (that is `al-test-coverage-validator` and `al-test-coverage-enforcer`), test quality (that is `al-test-validator`), or write tests (that is `al-test-writer`). This is a leaf action skill: it invokes no sub-skills.
|
||||
|
||||
An orchestrator invokes this skill with a `repository` (the project root to run in) and optionally an `object-list` (a filter narrowing the run to specific test codeunits). It produces a single JSON document conforming to the DO output contract.
|
||||
|
||||
## Source
|
||||
|
||||
Read the BCQuality knowledge index once (the `knowledge-index.json` Entry's preparation step regenerates over the live, already-filtered clone). Take the index entries whose `domain` is `testing` or `pipelines` as the citable candidate set across every enabled layer: runner-selection, isolation, and AL-Go pipeline rules can back a finding about how the run was configured. Do not open individual article files at this step; open an article's full body only once it enters the Worklist below. A reported test failure or a runner-startup failure rarely maps onto a curated rule, so it is emitted as an agent finding within this skill's domain (see Action).
|
||||
|
||||
## Relevance
|
||||
|
||||
Apply the frontmatter matching rules defined in READ against the task context:
|
||||
|
||||
- `bc-version`: the target BC version from the repository `app.json`, or `unknown` if unavailable.
|
||||
- `technologies`: `[al]`.
|
||||
- `countries`: the consuming app's declared countries, or `unknown`.
|
||||
- `application-area`: the application areas of the test objects, or `unknown`.
|
||||
|
||||
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when configuration permits; findings derived from them have `confidence` no higher than `medium`, and the finding `message` names the unknown dimensions.
|
||||
|
||||
## Worklist
|
||||
|
||||
Narrow to the run to perform and the artifacts it produces:
|
||||
|
||||
- Runner detection, in order: an AL-Go pipeline (`.AL-Go/settings.json` plus a `BuildALGoProject` script); a Docker BC sandbox via BcContainerHelper (a `BcContainerHelperVersion` setting or a `Run-TestsInBcContainer` call); a project-local `scripts/Build.ps1` or equivalent.
|
||||
- The test codeunits to run: every test codeunit in the repository, narrowed by the `object-list` filter when supplied.
|
||||
- The results file the run emits (`TestResults.xml` or equivalent) and the runner console output.
|
||||
|
||||
A curated `testing` or `pipelines` file enters the worklist when its `keywords` intersect these tokens. Read its full body only after it makes the worklist. Resolve layer-precedence conflicts per READ and record dropped files in `suppressed`.
|
||||
|
||||
## Action
|
||||
|
||||
Invoke the detected runner with the project's standard arguments, capture its output and the XUnit-style results file, and parse total, passed, failed, and skipped counts. Emit one finding per failed test: an agent finding (`references: []`, `id` slug prefixed `agent:` such as `agent:test-failed`, `confidence` capped at `medium`, `severity` capped at `minor`), with a self-contained `message` carrying the test codeunit, the test procedure, the assertion message, and the source location, and a `location` pointing at the failing line. Keep `severity` at `minor` even though a red test commonly blocks the chain, and say in the `message` that the run failed; the gating decision belongs to `al-test-coverage-enforcer` and the consuming orchestrator, not to this advisory channel. A run with many skipped tests emits an `info` finding naming the skip count. Where a curated `testing` or `pipelines` rule explains a misconfiguration the run surfaced (for example an isolation attribute that produced a spurious failure), upgrade that finding to knowledge-backed and cite the file. Mechanical fixes are rare here (the fix lives in the test or production code, not in the run), so omit `suggested-code` and set `suggested-code-omission-reason` to `fix lives in the test or production source under change`.
|
||||
|
||||
If the runner cannot start (Docker daemon down, BC image missing, AL-Go misconfigured) or exceeds the configured timeout, do not silently succeed: emit `outcome: "failed"` with `outcome-reason` carrying the exact command attempted and the error output, and an agent finding describing the startup failure.
|
||||
|
||||
Outcome selection: `completed` when the run finished and every failure was mapped to a finding (including a green run with empty `findings`); `not-applicable` when the repository has no AL test codeunit or no runner could be detected; `partial` when the run was cancelled on timeout after some tests ran (`summary.coverage` reflects the executed subset); `failed` when the runner could not start, with `outcome-reason` required.
|
||||
|
||||
## Output
|
||||
|
||||
Output conforms to the DO output contract. Test failures are agent findings (`references: []`, `agent:` id, severity capped at `minor`).
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "id": "al-test-runner", "version": 1 },
|
||||
"outcome": "completed",
|
||||
"outcome-reason": "al-go-pipeline runner, 27 tests, 1 failed",
|
||||
"summary": {
|
||||
"counts": { "blocker": 0, "major": 0, "minor": 1, "info": 1 },
|
||||
"coverage": { "worklist-size": 27, "items-evaluated": 27 }
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "agent:test-failed",
|
||||
"severity": "minor",
|
||||
"message": "Test ReleaseRegistrationShouldFailWhenOverCapacity in codeunit 50202 'Event Registration Tests' failed: expected error 'Capacity exceeded' but got 'Permission denied'. The run is red; fix the production code or the test before the chain can gate. Runner: al-go-pipeline.",
|
||||
"location": { "file": "test/EventRegistrationTests.al", "line": 88 },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code-omission-reason": "fix lives in the test or production source under change"
|
||||
},
|
||||
{
|
||||
"id": "agent:tests-skipped",
|
||||
"severity": "info",
|
||||
"message": "1 test was skipped via an explicit Skip() call (codeunit 50202, SmokeTest). Skipped tests do not fail the run; confirm the skip is intentional.",
|
||||
"references": [],
|
||||
"confidence": "medium"
|
||||
}
|
||||
],
|
||||
"suppressed": []
|
||||
}
|
||||
```
|
||||
87
custom/skills/testing/al-test-validator.md
Normal file
87
custom/skills/testing/al-test-validator.md
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
---
|
||||
kind: action-skill
|
||||
id: al-test-validator
|
||||
version: 1
|
||||
title: AL test validator
|
||||
description: Flags AL tests with no assertions, missing edge cases, weak names, poor isolation, and tests that exercise implementation not behaviour.
|
||||
inputs: [pr-diff, file-path]
|
||||
outputs: [findings-report]
|
||||
bc-version: [all]
|
||||
technologies: [al]
|
||||
countries: [w1]
|
||||
application-area: [all]
|
||||
---
|
||||
|
||||
# AL test validator
|
||||
|
||||
Reads AL test codeunits and reports whether each test is meaningful, well-named, and correctly configured. The failure modes are familiar: a `[Test]` with no `Assert.*`, a name that describes the call site instead of the expected behaviour, an isolation attribute set wrong for an AI test, a test that exercises implementation detail rather than behaviour, or a missing edge case the domain demands. This is a leaf action skill: it invokes no sub-skills.
|
||||
|
||||
An orchestrator invokes this skill with a `pr-diff` (the changed test files) or a `file-path` (a single test codeunit). It produces a single JSON document conforming to the DO output contract.
|
||||
|
||||
## Source
|
||||
|
||||
Read the BCQuality knowledge index once (the `knowledge-index.json` Entry's preparation step regenerates over the live, already-filtered clone). Take the index entries whose `domain` is `testing` as the citable candidate set across every enabled layer: test-attribute, isolation, transaction-model, and assertion-pattern rules are the authoritative basis for most findings here. Do not open individual article files at this step; open an article's full body only once it enters the Worklist below. Project-specific conventions (a naming pattern, a project helper codeunit) rarely map onto a curated file, so a finding about them is an agent finding within this skill's domain (see Action).
|
||||
|
||||
## Relevance
|
||||
|
||||
Apply the frontmatter matching rules defined in READ against the task context:
|
||||
|
||||
- `bc-version`: the target BC version from the branch `app.json`, or `unknown` if unavailable.
|
||||
- `technologies`: `[al]`.
|
||||
- `countries`: the consuming app's declared countries, or `unknown`.
|
||||
- `application-area`: the application areas of the tested objects, or `unknown`.
|
||||
|
||||
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when configuration permits; findings derived from them have `confidence` no higher than `medium`, and the finding `message` names the unknown dimensions.
|
||||
|
||||
## Worklist
|
||||
|
||||
Narrow to the test surface under review. Before raising findings, build a mental model of what the tests should assert: identify the business domain, the invariants and state transitions the system under test enforces, and the edge cases (empty sets, max values, null or blank fields, permission failures, date boundaries) the domain demands. Then compute overlap against:
|
||||
|
||||
- Every `[Test]` procedure in the changed or supplied test codeunits, and whether each has at least one `Assert.*` or `asserterror` expectation.
|
||||
- Codeunit configuration: `Subtype = Test`; for AI tests `TestType = AITest`, `TestPermissions = Disabled`, and for agent tests `RequiredTestIsolation = Disabled`; suite-setup guards via `AITTestContext.IsSuiteSetupDone()`.
|
||||
- Test names: behaviour-describing (`ReleaseRegistrationShouldFailWhenOverCapacity`) versus implementation-named (`TestReleaseRegistration`) versus opaque (`TestProc01`).
|
||||
- Anti-patterns: assertions inside an unguarded loop, a test calling `Commit()`, dependence on global state with no seed, deep mocking, a single test asserting multiple behaviours, undocumented `Sleep()`.
|
||||
- Mutation survivors, when supplied: which test should have caught each survivor and which assertion it lacks.
|
||||
|
||||
A curated `testing` file enters the worklist when its `keywords` intersect these tokens. Read its full `## Best Practice` / `## Anti Pattern` body only after it makes the worklist. Resolve layer-precedence conflicts per READ and record dropped files in `suppressed`.
|
||||
|
||||
## Action
|
||||
|
||||
For each worklisted test, evaluate it against the model and the worklisted knowledge. A missing assertion, a `Commit()` in a test, an assertion inside an unguarded loop, global-state dependence, or an isolation/transaction-model attribute wrong for the test type is a defect: when a curated `testing` file states the rule, emit a knowledge-backed finding citing it (`id` equal to the file path, `severity` up to `major`, `blocker` only when the file states a platform-level guarantee, `confidence` `high` for an unambiguous match). Implementation-named tests, single-test-multiple-behaviour, missing edge cases, and project-convention drift rarely map onto a curated file: emit them as agent findings within this skill's domain (`references: []`, `id` slug prefixed `agent:`, `confidence` capped at `medium`, `severity` capped at `minor`, self-contained `message`). When the underlying impact would otherwise be major (a missing assertion that lets a mutation survive), keep the emitted `severity` at `minor` but say so plainly in the `message` and note the concern should be promoted to a curated rule before it can gate. Hold every agent candidate to the precision bar in `skills/do.md`: steelman that the test is deliberate (a no-throw contract is sometimes the real contract) before emitting, and omit when in doubt. Set `suggested-code` when the fix is mechanical (rename a test, add a `[HandlerFunctions(...)]` attribute, add the missing isolation attribute); otherwise set `suggested-code-omission-reason`.
|
||||
|
||||
Outcome selection: `completed` when every worklisted test was evaluated (including an empty `findings`); `no-knowledge` when no curated knowledge survived and no agent finding was raised; `not-applicable` when the task carries no AL test to validate; `partial` or `failed` per the DO contract with `outcome-reason`.
|
||||
|
||||
## Output
|
||||
|
||||
Output conforms to the DO output contract. Findings without a knowledge file are agent findings (`references: []`, `agent:` id, severity capped at `minor`); findings citing a `testing` file carry that file path as `id` and primary reference.
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "id": "al-test-validator", "version": 1 },
|
||||
"outcome": "completed",
|
||||
"summary": {
|
||||
"counts": { "blocker": 0, "major": 1, "minor": 1, "info": 0 },
|
||||
"coverage": { "worklist-size": 4, "items-evaluated": 4 }
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "microsoft/knowledge/testing/transactionmodel-attribute-governs-test-transactions.md",
|
||||
"severity": "major",
|
||||
"message": "PostAndAssertNoChange posts a document but TransactionModel is unset, so posting rolls back at test end and the state-after-post assertion never observes a committed change.",
|
||||
"location": { "file": "test/EventRegistrationTests.al", "line": 88 },
|
||||
"references": [ { "path": "microsoft/knowledge/testing/transactionmodel-attribute-governs-test-transactions.md" } ],
|
||||
"confidence": "high"
|
||||
},
|
||||
{
|
||||
"id": "agent:implementation-named-test",
|
||||
"severity": "minor",
|
||||
"message": "Test procedure TestReleaseRegistration names the call site, not the expected behaviour. Rename to describe the outcome, for example ReleaseRegistrationShouldEmitTelemetry, so a failure reads as a broken contract.",
|
||||
"location": { "file": "test/EventRegistrationTests.al", "line": 40 },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code": " procedure ReleaseRegistrationShouldEmitTelemetry()"
|
||||
}
|
||||
],
|
||||
"suppressed": []
|
||||
}
|
||||
```
|
||||
79
custom/skills/testing/al-test-writer.md
Normal file
79
custom/skills/testing/al-test-writer.md
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
---
|
||||
kind: action-skill
|
||||
id: al-test-writer
|
||||
version: 1
|
||||
title: AL test writer
|
||||
description: Generates AL test codeunits for a target production object as findings carrying the test source, the TDD red step.
|
||||
inputs: [object-list, file-path]
|
||||
outputs: [findings-report]
|
||||
bc-version: [all]
|
||||
technologies: [al]
|
||||
countries: [w1]
|
||||
application-area: [all]
|
||||
---
|
||||
|
||||
# AL test writer
|
||||
|
||||
Generates Business Central AL test codeunits that exercise a target production object and assert specific behaviours. This is a generator-style skill: it expresses each generated test codeunit as a finding whose `suggested-code` carries ready-to-drop AL, with the `## Action` step explaining what is generated. The contract is the TDD red step: the generated test must fail against the current production code and pass once the intended behaviour is implemented. This is a leaf action skill: it invokes no sub-skills.
|
||||
|
||||
An orchestrator invokes this skill with an `object-list` (the target objects to cover) and a `file-path` (the production AL under test, plus the behaviour spec the caller supplies). It produces a single JSON document conforming to the DO output contract.
|
||||
|
||||
## Source
|
||||
|
||||
Read the BCQuality knowledge index once (the `knowledge-index.json` Entry's preparation step regenerates over the live, already-filtered clone). Take the index entries whose `domain` is `testing` as the citable candidate set across every enabled layer: test-attribute, isolation, transaction-model, and assertion-pattern rules govern how the generated codeunit must be shaped, so a finding that matches a curated rule cites that file. Do not open individual article files at this step; open an article's full body only once it enters the Worklist below. Where no curated rule covers a concrete generation choice, this skill emits an agent finding within its own domain (see Action).
|
||||
|
||||
## Relevance
|
||||
|
||||
Apply the frontmatter matching rules defined in READ against the task context:
|
||||
|
||||
- `bc-version`: the target BC version from the branch `app.json`, or `unknown` if unavailable.
|
||||
- `technologies`: `[al]`.
|
||||
- `countries`: the consuming app's declared countries, or `unknown`.
|
||||
- `application-area`: the application areas of the target objects, or `unknown`.
|
||||
|
||||
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when configuration permits; findings derived from them have `confidence` no higher than `medium`, and the finding `message` names the unknown dimensions.
|
||||
|
||||
## Worklist
|
||||
|
||||
Narrow to the generation work the task actually requires:
|
||||
|
||||
- One target per entry in `object-list`: a codeunit, table, page, or report whose behaviour the caller's spec describes.
|
||||
- The behaviour spec: which inputs should produce which outcome, side effect, or error. If the spec is missing for a target, that target produces no test and is reported as `info` rather than a guessed behaviour.
|
||||
- The project's existing test conventions (helper codeunits, fakes, naming pattern, assigned test object ID range) read from the supplied source.
|
||||
|
||||
A curated `testing` knowledge file enters the worklist when its `keywords` intersect the tokens of the target (`Subtype = Test`, `TestType = AITest`, `RequiredTestIsolation`, `TestPermissions`, `HandlerFunctions`, `Library Assert`, `Commit`, isolation, transaction-model). Read its full `## Best Practice` / `## Anti Pattern` body only after it makes the worklist. Resolve layer-precedence conflicts per READ and record dropped files in `suppressed`.
|
||||
|
||||
## Action
|
||||
|
||||
For each target in the worklist, generate an AL test codeunit. Set the codeunit attributes for the test type: `Subtype = Test` with `TestPermissions = Disabled` for a regular AL test; add `TestType = AITest` for a Copilot prompt test; add `RequiredTestIsolation = Disabled` as well for an agent accuracy test. Every generated `[Test]` procedure carries a behaviour-describing name (the `Given_When_Then` or `Behaviour_Should_Outcome` pattern), self-seeding setup with no production-data dependence, an invocation of the target procedure or trigger, at least one `Assert.*` call validating the outcome, and `[HandlerFunctions(...)]` where a modal or confirmation is expected. The generated assertions must fail against the current production code (the TDD red contract); when that cannot be guaranteed for a target, say so plainly in the finding `message`.
|
||||
|
||||
Emit one finding per generated test codeunit. Where a curated `testing` knowledge file backs the generation choice (for example an isolation or transaction-model rule the generated attributes satisfy), cite it: `id` equal to the file path, `references` carrying it, `confidence` `high` for an unambiguous match, `severity` `info` (a generator produces artifacts, not gating defects). Where no curated file applies, emit an agent finding: `references: []`, `id` slug prefixed `agent:` (for example `agent:generated-test-codeunit`), `confidence` capped at `medium`, `severity` capped at `minor`, and a self-contained `message` naming the target, the asserted behaviours, and the developer follow-up (implement the behaviour so the red test goes green). Put the generated AL in `suggested-code` since the artifact is mechanical; the `message` states what was generated and the TDD-red status. When a target lacks a behaviour spec, emit an `info` finding asking for the spec and omit `suggested-code` with `suggested-code-omission-reason` set to `behaviour spec missing for target`. Hold any agent finding to the precision bar in `skills/do.md`.
|
||||
|
||||
Outcome selection: `completed` when every worklist target was processed (including when no AL could be generated for lack of a spec); `no-knowledge` when no curated knowledge survived and no agent finding was raised; `not-applicable` when the task supplies no AL target to cover; `partial` or `failed` per the DO contract with `outcome-reason`.
|
||||
|
||||
## Output
|
||||
|
||||
Output conforms to the DO output contract. A populated example:
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "id": "al-test-writer", "version": 1 },
|
||||
"outcome": "completed",
|
||||
"summary": {
|
||||
"counts": { "blocker": 0, "major": 0, "minor": 1, "info": 0 },
|
||||
"coverage": { "worklist-size": 1, "items-evaluated": 1 }
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "agent:generated-test-codeunit",
|
||||
"severity": "minor",
|
||||
"message": "Generated test codeunit 50202 'Event Registration Tests' covering ReleaseRegistration. Asserts 'Capacity exceeded' is raised when attendee count exceeds capacity. Fails on current code (red); implement the capacity validation in Event Registration Mgt.ReleaseRegistration to make it pass.",
|
||||
"location": { "file": "test/EventRegistrationTests.al" },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code": "codeunit 50202 \"Event Registration Tests\"\n{\n Subtype = Test;\n TestPermissions = Disabled;\n\n [Test]\n procedure ReleaseRegistrationShouldFailWhenOverCapacity()\n begin\n // ... arrange, act, asserterror\n end;\n}"
|
||||
}
|
||||
],
|
||||
"suppressed": []
|
||||
}
|
||||
```
|
||||
88
custom/skills/testing/al-userguide-test-writer.md
Normal file
88
custom/skills/testing/al-userguide-test-writer.md
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
---
|
||||
kind: action-skill
|
||||
id: al-userguide-test-writer
|
||||
version: 1
|
||||
title: AL user-guide test writer
|
||||
description: Maps each USER_GUIDE.md step to a BC page, action, and assertion and emits Subtype=Test TestPage codeunits as findings.
|
||||
inputs: [file-path, repository]
|
||||
outputs: [findings-report]
|
||||
bc-version: [all]
|
||||
technologies: [al]
|
||||
countries: [w1]
|
||||
application-area: [all]
|
||||
---
|
||||
|
||||
# AL user-guide test writer
|
||||
|
||||
Reads a Business Central extension's end-user walkthrough (typically `USER_GUIDE.md`), maps each documented step to a page plus action plus assertion, and generates `Subtype = Test` codeunits that script the flow with AL's TestPage library so the suite can run in a container. This is a generator-style skill: each generated test codeunit is a finding whose `suggested-code` carries the AL, and the `## Action` step explains what is generated. It writes tests; it does not run them (that is `al-test-runner`). This is a leaf action skill: it invokes no sub-skills.
|
||||
|
||||
An orchestrator invokes this skill with a `file-path` (the user-guide markdown) and a `repository` (so page object names, action names, field names, and the test app `idRanges` can be resolved from the source). It produces a single JSON document conforming to the DO output contract.
|
||||
|
||||
## Source
|
||||
|
||||
Read the BCQuality knowledge index once (the `knowledge-index.json` Entry's preparation step regenerates over the live, already-filtered clone). Take the index entries whose `domain` is `testing` or `ux` as the citable candidate set across every enabled layer: TestPage patterns, isolation rules, handler-function conventions, and page-interaction guidance shape the generated codeunits, so a finding matching a curated rule cites it. Do not open individual article files at this step; open an article's full body only once it enters the Worklist below. Where no curated rule covers a generation choice or a step that cannot be mapped, this skill emits an agent finding within its own domain (see Action).
|
||||
|
||||
## Relevance
|
||||
|
||||
Apply the frontmatter matching rules defined in READ against the task context:
|
||||
|
||||
- `bc-version`: the target BC version from the repository `app.json`, or `unknown` if unavailable.
|
||||
- `technologies`: `[al]`.
|
||||
- `countries`: the consuming app's declared countries, or `unknown`.
|
||||
- `application-area`: the application areas of the pages the guide drives, or `unknown`.
|
||||
|
||||
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when configuration permits; findings derived from them have `confidence` no higher than `medium`, and the finding `message` names the unknown dimensions.
|
||||
|
||||
## Worklist
|
||||
|
||||
Narrow to the generation work the guide implies:
|
||||
|
||||
- One test codeunit per top-level guide section (`"UserGuide §N <Topic>_<SUFFIX>_TST"`, N the section number, suffix matching the extension's mandatory suffix), living in the test app's `idRanges`.
|
||||
- One `[Test]` procedure per documented step or substep, so a failure points at the specific step.
|
||||
- Per procedure: the page to open (`OpenNew()` / `OpenEdit()`), the field writes (display captions resolved to AL field names from the source), the action invocations (`Invoke()` on the AL action name, not the caption), and the assertions on documented outcomes (`Assert.AreEqual` / `Assert.IsTrue` via `Codeunit "Library Assert"`).
|
||||
- `[HandlerFunctions(...)]` where the guide implies a dialog or confirmation, with `asserterror` on negative paths.
|
||||
- Seed data via `LibrarySales` / `LibraryPurchase` / `LibraryInventory` / `LibraryWarehouse` or the extension's own seed library; an `IsInitialized` guard and an `Initialize()` procedure per codeunit.
|
||||
- A coverage map: which steps mapped to which procedures, and which sections could not be mapped (page does not exist, action unreachable from a TestPage, behaviour is server-side with no UI hook).
|
||||
|
||||
A curated `testing` or `ux` file enters the worklist when its `keywords` intersect these tokens. Read its full body only after it makes the worklist. Resolve layer-precedence conflicts per READ and record dropped files in `suppressed`.
|
||||
|
||||
## Action
|
||||
|
||||
For each mappable section, generate a TestPage codeunit as described in the worklist and emit one finding carrying it. Where a curated `testing` or `ux` file backs a generation choice (an isolation rule, a handler-function convention, a TestPage interaction pattern), cite it: `id` equal to the file path, `references` carrying it, `severity` `info`, `confidence` `high` for an unambiguous match. Where no curated file applies, emit an agent finding: `references: []`, `id` slug prefixed `agent:` (for example `agent:generated-userguide-test`), `confidence` capped at `medium`, `severity` capped at `minor`, and a self-contained `message` naming the section, the steps covered, and the file path the codeunit should be written to. Put the generated AL in `suggested-code`. For a section that cannot be cleanly mapped, emit a separate agent finding (`id` slug `agent:unmappable-userguide-step`) naming the section, the step, why it could not map, and the recommended workaround (for example a parent-subpage navigation pattern, or a note that a server-side behaviour needs a non-UI test); omit `suggested-code` and set `suggested-code-omission-reason` to `step cannot be expressed through a TestPage primitive`. When the extension has no seed library, emit an `info` finding recommending one. Hold every agent finding to the precision bar in `skills/do.md`.
|
||||
|
||||
Outcome selection: `completed` when every section was processed (mapped to a codeunit or reported as unmappable); `not-applicable` when the supplied path is not a user guide or no driven page exists in the repository; `no-knowledge` when no curated knowledge survived and no agent finding was raised; `partial` or `failed` per the DO contract with `outcome-reason`.
|
||||
|
||||
## Output
|
||||
|
||||
Output conforms to the DO output contract. Generated codeunits with no curated backing are agent findings (`references: []`, `agent:` id, severity capped at `minor`, AL in `suggested-code`); findings citing a `testing` or `ux` file carry that file path as `id` and primary reference.
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "id": "al-userguide-test-writer", "version": 1 },
|
||||
"outcome": "completed",
|
||||
"summary": {
|
||||
"counts": { "blocker": 0, "major": 0, "minor": 1, "info": 0 },
|
||||
"coverage": { "worklist-size": 2, "items-evaluated": 2 }
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "agent:generated-userguide-test",
|
||||
"severity": "minor",
|
||||
"message": "Generated TestPage codeunit for guide section '2. First-time setup', covering steps 2.1 Open Shipping Setup, 2.2 Review seeded reference data, 2.3 Set up Other Places. Write to src/UserGuide/SetupSection_TST.Codeunit.al in the test app id range.",
|
||||
"location": { "file": "src/UserGuide/SetupSection_TST.Codeunit.al" },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code": "codeunit 60001 \"UserGuide §2 Setup_SHP_EQL_TST\"\n{\n Subtype = Test;\n TestPermissions = Disabled;\n // [Test] procedures per documented step\n}"
|
||||
},
|
||||
{
|
||||
"id": "agent:unmappable-userguide-step",
|
||||
"severity": "minor",
|
||||
"message": "Section 5.1 'Click Add to Container on the line ribbon' is a subpage line-level action. A TestPage cannot invoke it directly; use the parent page's TestPage and a SubPage child reference (Page_PurchaseOrder.PurchLines.\"Add to Container_SHP_EQL\".Invoke()). Confirm the parent-subpage navigation matches AL conventions.",
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code-omission-reason": "step cannot be expressed through a TestPage primitive"
|
||||
}
|
||||
],
|
||||
"suppressed": []
|
||||
}
|
||||
```
|
||||
93
custom/skills/testing/bc-extension-test-guide.md
Normal file
93
custom/skills/testing/bc-extension-test-guide.md
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
---
|
||||
kind: action-skill
|
||||
id: bc-extension-test-guide
|
||||
version: 1
|
||||
title: BC extension test guide generator
|
||||
description: Inventories every page, field, relation, enum, action, state machine, permission set, and telemetry event and emits a category-driven release-audit TEST_GUIDE.md.
|
||||
inputs: [repository, object-list]
|
||||
outputs: [findings-report]
|
||||
bc-version: [all]
|
||||
technologies: [al]
|
||||
countries: [w1]
|
||||
application-area: [all]
|
||||
---
|
||||
|
||||
# BC extension test guide generator
|
||||
|
||||
Produces an exhaustive `DOCS/TEST_GUIDE.md` for a Business Central AL extension. The guide is exhaustive by construction: every field, relation, action, and reachable data state in the AL source appears in at least one of twelve category inventories, so the state pivots happy-path testing misses are caught. This is a generator-style skill: the generated guide markdown is carried as a finding's `suggested-code`, and the `## Action` step explains what is generated. It produces the artifact; it does not run tests. This is a leaf action skill: it invokes no sub-skills.
|
||||
|
||||
An orchestrator invokes this skill with a `repository` (the extension root with `app.json` and AL `src/`) and optionally an `object-list` (a scope narrowing the inventory). It produces a single JSON document conforming to the DO output contract.
|
||||
|
||||
## Source
|
||||
|
||||
Read the BCQuality knowledge index once (the `knowledge-index.json` Entry's preparation step regenerates over the live, already-filtered clone). Take the index entries whose `domain` is `testing`, `ux`, `security`, or `telemetry` as the citable candidate set across every enabled layer: the twelve categories pull on lookup, type-conditional relation, visibility-refresh, state-machine, permission-boundary, cross-company isolation, and telemetry-event rules, so a category whose contract matches a curated rule cites it. Do not open individual article files at this step; open an article's full body only once it enters the Worklist below. The generated guide itself, where no curated rule covers a category, is an agent finding within this skill's domain (see Action).
|
||||
|
||||
## Relevance
|
||||
|
||||
Apply the frontmatter matching rules defined in READ against the task context:
|
||||
|
||||
- `bc-version`: the target BC version from the repository `app.json`, or `unknown` if unavailable.
|
||||
- `technologies`: `[al]`.
|
||||
- `countries`: the consuming app's declared countries, or `unknown`.
|
||||
- `application-area`: the application areas of the inventoried objects, or `unknown`.
|
||||
|
||||
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when configuration permits; findings derived from them have `confidence` no higher than `medium`, and the finding `message` names the unknown dimensions.
|
||||
|
||||
## Worklist
|
||||
|
||||
Narrow to the full AL inventory the twelve categories require. Discover the repo (`app.json`, `.AL-Go/settings.json`, the primary app folder, the `src/**/*.al` file list), then build the inventory across these categories:
|
||||
|
||||
1. Lookup audit: every page field whose underlying table field has a `TableRelation`, both directions.
|
||||
2. Type-conditional `TableRelation`: every relation conditional on a sibling field, and each sibling enum or option value mapped to its target table.
|
||||
3. Eligibility filters: every lookup whose downstream `OnValidate` or OK handler rejects a subset of the target table, and whether the lookup pre-filters it.
|
||||
4. Visibility and Editable conditionals: every dynamic `Visible`/`Editable`, its driver, and whether the driver's `OnValidate` forces a `CurrPage.Update(false)` refresh.
|
||||
5. StandardDialog Mode pivots: every `PageType = StandardDialog` with a Mode selector, the visible-field set, OK side effect, and error per Mode.
|
||||
6. Subpage FK persistence: every `part(...)` with a `SubPageLink`, the FK propagated, and whether the explicit-push pattern or default-value behaviour alone is in use.
|
||||
7. State machine transitions: every status enum, its full from-by-to matrix marking allowed and disallowed transitions, and every code path mutating the status.
|
||||
8. Permission boundaries: every permission set with full RIMD per table and page or codeunit execute claims.
|
||||
9. Telemetry events: every custom-event log call site with event id, trigger, and payload keys.
|
||||
10. Mobile and tablet smoke: every top-level user-facing page.
|
||||
11. Cross-company isolation: every table with its `DataPerCompany` value, singletons called out.
|
||||
12. Upgrade paths: every upgrade codeunit and per-release schema delta with seed instructions.
|
||||
|
||||
Scope the inventory by `object-list` when supplied. A curated `testing`, `ux`, `security`, or `telemetry` file enters the worklist when its `keywords` intersect a category's tokens. Read its full body only after it makes the worklist. Resolve layer-precedence conflicts per READ and record dropped files in `suppressed`.
|
||||
|
||||
## Action
|
||||
|
||||
Generate `DOCS/TEST_GUIDE.md` with exactly the twelve categories in order, each carrying a definition, a procedure, and a populated inventory drawn from the actual AL (no placeholders). Run the self-audit pass: every `TableRelation` lands in category 1 or 2, every dynamic `Visible`/`Editable` in category 4, every `SubPageLink` in category 6, every status enum in category 7, every permission set in category 8, every telemetry call site in category 9; if an inventory has fewer rows than the AL warrants, the guide is incomplete and the missing rows are added. Emit one finding carrying the generated guide. Where a curated file backs a category contract (a state-machine rule, a cross-company isolation rule, a telemetry payload rule), cite it: `id` equal to the file path, `references` carrying it, `severity` `info`, `confidence` `high` for an unambiguous match. Where no curated file applies, emit an agent finding: `references: []`, `id` slug prefixed `agent:` (for example `agent:generated-test-guide`), `confidence` capped at `medium`, `severity` capped at `minor`, and a self-contained `message` stating the guide was generated, the categories that genuinely do not apply (stated explicitly, not stubbed), and the path to write it to. Put the generated markdown in `suggested-code`. Emit an `info` finding recommending the cross-link in `USER_GUIDE.md`. Hold every agent finding to the precision bar in `skills/do.md`.
|
||||
|
||||
Outcome selection: `completed` when the full inventory was built and the guide generated (including categories that do not apply, stated as such); `not-applicable` when the repository has no `app.json` and AL `src/`; `partial` when a token budget truncated the inventory (`summary.coverage` reflects the categories completed); `failed` per the DO contract with `outcome-reason`.
|
||||
|
||||
## Output
|
||||
|
||||
Output conforms to the DO output contract. The generated guide with no curated backing is an agent finding (`references: []`, `agent:` id, severity capped at `minor`, markdown in `suggested-code`); findings citing a `testing`, `ux`, `security`, or `telemetry` file carry that file path as `id` and primary reference.
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "id": "bc-extension-test-guide", "version": 1 },
|
||||
"outcome": "completed",
|
||||
"summary": {
|
||||
"counts": { "blocker": 0, "major": 0, "minor": 1, "info": 1 },
|
||||
"coverage": { "worklist-size": 12, "items-evaluated": 12 }
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "agent:generated-test-guide",
|
||||
"severity": "minor",
|
||||
"message": "Generated DOCS/TEST_GUIDE.md with all 12 categories populated from src/ (47 TableRelation rows in cat 1/2, 3 status enums in cat 7, 2 permission sets in cat 8). Category 12 (Upgrade paths) is empty: the extension ships no upgrade codeunit, stated explicitly in the guide rather than stubbed.",
|
||||
"location": { "file": "DOCS/TEST_GUIDE.md" },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code": "# <Extension Name>, Test Guide\n\n## 0. Category index\n... (12 categories, each with definition, procedure, and inventory table) ..."
|
||||
},
|
||||
{
|
||||
"id": "agent:userguide-cross-link",
|
||||
"severity": "info",
|
||||
"message": "Add a cross-link in the USER_GUIDE.md header pointing to the new TEST_GUIDE.md so QA can find the audit companion.",
|
||||
"references": [],
|
||||
"confidence": "medium"
|
||||
}
|
||||
],
|
||||
"suppressed": []
|
||||
}
|
||||
```
|
||||
93
custom/skills/testing/bc-webclient-runner.md
Normal file
93
custom/skills/testing/bc-webclient-runner.md
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
---
|
||||
kind: action-skill
|
||||
id: bc-webclient-runner
|
||||
version: 1
|
||||
title: BC web client runner
|
||||
description: Drives the rendered BC web client through a documented flow to catch UI residue AL TestPage cannot observe.
|
||||
inputs: [repository, file-path]
|
||||
outputs: [findings-report]
|
||||
bc-version: [all]
|
||||
technologies: [al]
|
||||
countries: [w1]
|
||||
application-area: [all]
|
||||
---
|
||||
|
||||
# BC web client runner
|
||||
|
||||
Drives a real Business Central web client through a documented user flow (typically `USER_GUIDE.md`), capturing screenshots and asserting on rendered UI state at every step. It catches the class of bug AL TestPage is structurally blind to: page layout, action enable and disable state, FactBox refresh timing, notification toasts, modal stacking, lookup usability, delayed-insert behaviour on subpages, and state-label drift between the guide and the enum. It executes scripted flows; it does not author AL. This is a leaf action skill: it invokes no sub-skills.
|
||||
|
||||
An orchestrator invokes this skill with a `repository` (the extension source, so page and action names and the documented sandbox URL and company can be resolved) and a `file-path` (the user-guide markdown to walk). It produces a single JSON document conforming to the DO output contract.
|
||||
|
||||
## Source
|
||||
|
||||
Read the BCQuality knowledge index once (the `knowledge-index.json` Entry's preparation step regenerates over the live, already-filtered clone). Take the index entries whose `domain` is `ux` or `testing` as the citable candidate set across every enabled layer: rendered-UI rules (delayed-insert, lookup usability, refresh-after-validate, state-label consistency) can back a finding the run surfaces. Do not open individual article files at this step; open an article's full body only once it enters the Worklist below. Where no curated rule covers an observed rendered-UI defect, this skill emits an agent finding within its own domain (see Action).
|
||||
|
||||
## Relevance
|
||||
|
||||
Apply the frontmatter matching rules defined in READ against the task context:
|
||||
|
||||
- `bc-version`: the target BC version from the repository `app.json`, or `unknown` if unavailable.
|
||||
- `technologies`: `[al]`.
|
||||
- `countries`: the consuming app's declared countries, or `unknown`.
|
||||
- `application-area`: the application areas of the pages walked, or `unknown`.
|
||||
|
||||
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when configuration permits; findings derived from them have `confidence` no higher than `medium`, and the finding `message` names the unknown dimensions.
|
||||
|
||||
## Worklist
|
||||
|
||||
Narrow to the flow to drive and the rendered checks per step. The environment must be non-production: refuse an on-prem host lacking `sandbox`, `dev`, `test`, or `staging`, and for a SaaS host on `businesscentral.dynamics.com` inspect the environment-name path segment and refuse if it matches `Production` or starts with `Prod`. Then build the worklist:
|
||||
|
||||
- Each top-level guide section (or the supplied subset), and within it each documented step.
|
||||
- Step-level state checks: read the documented outcome (status pill text, field value, subpage row count) after each action.
|
||||
- Action availability: confirm a button is enabled or disabled exactly as the guide states, reading `aria-disabled` from the accessibility tree.
|
||||
- Notification toasts: screenshot the toast region before auto-dismiss and read its content.
|
||||
- FactBox totals: read the numbers and compare to the documented arithmetic.
|
||||
- Lookup usability on every lookup-bearing field: open the lookup, confirm it lists records and a selection writes back.
|
||||
- Delayed-insert behaviour on every editable subpage: type into the first non-PK field, tab off, and watch for an out-of-filter banner, a blank PK column, or the row falling out of the parent filter.
|
||||
- Missing affordances the guide implies (a lookup drop-down a documented path needs).
|
||||
- State-label drift: compare the displayed status value and enum dropdown values against the names the guide uses.
|
||||
- Page-level errors: any `Error` notification, inline validation message, or console `ServerError`, captured even if the guide does not mention it.
|
||||
|
||||
A curated `ux` or `testing` file enters the worklist when its `keywords` intersect these tokens. Read its full body only after it makes the worklist. Resolve layer-precedence conflicts per READ and record dropped files in `suppressed`.
|
||||
|
||||
## Action
|
||||
|
||||
Drive the web client through each worklisted step, screenshot the result, and assert on the documented outcome. The skill requires a Chrome automation surface in the calling session; if it is unavailable, do not fall back to anything else: emit `outcome: "failed"` with `outcome-reason` stating the surface is missing.
|
||||
|
||||
Emit a finding for every rendered-UI defect. Where a curated `ux` or `testing` file states the rule (for example a delayed-insert rule, a refresh-after-validate rule, or a state-label-consistency rule), emit a knowledge-backed finding citing it: `id` equal to the file path, `severity` up to `major`, `blocker` only when the file states a platform-level guarantee, `confidence` `high` for an unambiguous match. Where no curated file covers the observed defect, emit an agent finding within this skill's domain: `references: []`, `id` slug prefixed `agent:` (for example `agent:userguide-action-disabled`, `agent:subpage-missing-delayed-insert`, `agent:factbox-stale`, `agent:state-label-drift`), `confidence` capped at `medium`, `severity` capped at `minor`, and a self-contained `message` carrying the section, the step, what was observed against what the guide promised, and the screenshot path. When the underlying impact would otherwise be major (a subpage missing `DelayedInsert = true` corrupting the parent FK, or state-label drift that breaks every downstream filter), keep the emitted `severity` at `minor` but say so plainly in the `message` and note the concern should be promoted to a curated rule before it can gate. Record the role the run used and the URL in the summary. Hold every agent finding to the precision bar in `skills/do.md`. The fix lives in AL, not in a renderable replacement, so omit `suggested-code` and set `suggested-code-omission-reason` to `fix is an AL change the developer applies after reading the report`.
|
||||
|
||||
Outcome selection: `completed` when every attempted step was driven and asserted (including a clean run with empty `findings`); `not-applicable` when the supplied path is not a user guide or the repository drives no rendered page; `partial` when a block stopped the run mid-flow and not every section was attempted (`summary.coverage` reflects the attempted subset); `failed` when the Chrome surface was unavailable or the run could not start, with `outcome-reason` required.
|
||||
|
||||
## Output
|
||||
|
||||
Output conforms to the DO output contract. Rendered-UI defects with no curated backing are agent findings (`references: []`, `agent:` id, severity capped at `minor`); findings citing a `ux` or `testing` file carry that file path as `id` and primary reference.
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "id": "bc-webclient-runner", "version": 1 },
|
||||
"outcome": "completed",
|
||||
"summary": {
|
||||
"counts": { "blocker": 0, "major": 0, "minor": 2, "info": 0 },
|
||||
"coverage": { "worklist-size": 9, "items-evaluated": 9 }
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "agent:userguide-action-disabled",
|
||||
"severity": "minor",
|
||||
"message": "Section 3.2: Release on the Freight Movement card is disabled although the guide says it should be enabled once the header is filled (aria-disabled=true on the command-bar item). Screenshot: screenshots/section-3-step-2-release-disabled.png. Verify the action's Enabled expression against the header-filled state.",
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code-omission-reason": "fix is an AL change the developer applies after reading the report"
|
||||
},
|
||||
{
|
||||
"id": "agent:subpage-missing-delayed-insert",
|
||||
"severity": "minor",
|
||||
"message": "Section 5.1: typing into the line subpage then tabbing off shows an out-of-filter banner and a blank No. column, indicating the subpage is missing DelayedInsert = true. Impact is major: OnInsert fires before the number series assigns the PK, so the row persists with a blank or wrong parent FK. Promote to a curated rule before it can gate.",
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code-omission-reason": "fix is an AL change the developer applies after reading the report"
|
||||
}
|
||||
],
|
||||
"suppressed": []
|
||||
}
|
||||
```
|
||||
79
custom/skills/testing/page-scripting-e2e.md
Normal file
79
custom/skills/testing/page-scripting-e2e.md
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
---
|
||||
kind: action-skill
|
||||
id: page-scripting-e2e
|
||||
version: 1
|
||||
title: Page Scripting e2e planner
|
||||
description: Decides what belongs in Page Scripting versus an AL test and produces a deterministic recording plan wired into the bc-replay harness.
|
||||
inputs: [repository, file-path]
|
||||
outputs: [findings-report]
|
||||
bc-version: [all]
|
||||
technologies: [al]
|
||||
countries: [w1]
|
||||
application-area: [all]
|
||||
---
|
||||
|
||||
# Page Scripting e2e planner
|
||||
|
||||
Builds a durable, repeatable browser-level test layer for a Business Central extension using BC's native Page Scripting (record and replay `.yml`) plus the `e2e-replay` (bc-replay) harness. It decides what belongs in Page Scripting versus an AL TestPage test, produces a deterministic recording plan a human follows, and wires the recordings so the whole set re-runs from one command. This is a generator-style skill: the recording plan markdown and the seed-factory scaffold are carried as findings' `suggested-code`, and the `## Action` step explains what is generated. This is a leaf action skill: it invokes no sub-skills.
|
||||
|
||||
An orchestrator invokes this skill with a `repository` (the extension source and any green AL suite) and a `file-path` (the `DOCS/TEST_GUIDE.md` or `USER_GUIDE.md` whose residue the plan covers). It produces a single JSON document conforming to the DO output contract.
|
||||
|
||||
## Source
|
||||
|
||||
Read the BCQuality knowledge index once (the `knowledge-index.json` Entry's preparation step regenerates over the live, already-filtered clone). Take the index entries whose `domain` is `testing` or `ux` as the citable candidate set across every enabled layer: rules about what a rendered client must verify (notification toasts, cue rendering, visibility refresh, factbox refresh) and about deterministic seeding can back a plan decision. Do not open individual article files at this step; open an article's full body only once it enters the Worklist below. The generated plan and the layer-allocation decisions, where no curated rule applies, are agent findings within this skill's domain (see Action).
|
||||
|
||||
## Relevance
|
||||
|
||||
Apply the frontmatter matching rules defined in READ against the task context:
|
||||
|
||||
- `bc-version`: the target BC version from the repository `app.json`, or `unknown` if unavailable.
|
||||
- `technologies`: `[al]`.
|
||||
- `countries`: the consuming app's declared countries, or `unknown`.
|
||||
- `application-area`: the application areas of the flows recorded, or `unknown`.
|
||||
|
||||
Discard files that are not applicable. Retain conditionally applicable files (any dimension `unknown`) only when configuration permits; findings derived from them have `confidence` no higher than `medium`, and the finding `message` names the unknown dimensions.
|
||||
|
||||
## Worklist
|
||||
|
||||
Narrow to the residue that needs a rendered client and the recordings that cover it, using the three-layer model: AL TestPage owns logic, state transitions, validation errors, action gates, proportional math, FlowField values, and permission RIMD; Page Scripting owns only what needs a rendered client; the manual checklist owns the irreducible (mobile, subjective look). Build the worklist:
|
||||
|
||||
- For each TEST_GUIDE category or USER_GUIDE flow, decide the layer. Anything verifiable by reading a record or asserting a field after invoking a codeunit stays layer 1 and is excluded here.
|
||||
- The residue items that need a rendered client: notification toasts firing, cue and tile rendering and Style, visibility and editability refresh after a field change, FactBox refresh on row change, dropdown and lookup population and filter-as-you-type narrowing, modal and dialog flow a user clicks through, real posting through standard codeunits driven from the UI.
|
||||
- One small single-purpose recording per residue item (`E2E-NN <flow>.yml`, the `NN` prefix sorting play order).
|
||||
- The deterministic precondition per recording: a Test Seed Factory codeunit gated behind an `Allow Test Data Seed` toggle that clears then seeds a fixed-prefix set, a reset No. Series so a recorded New yields a stable number, an `E2E-00 Clear and Seed.yml` head, and filter-as-you-type lookups that narrow to exactly one row.
|
||||
- The harness wiring: replay via `e2e-replay/run.ps1` or the VS Code task, accounting for the known constraints (bc-replay cannot run from a path with a space, credentials come from env vars not interactive login, the recordings globber rejects `..`).
|
||||
|
||||
A curated `testing` or `ux` file enters the worklist when its `keywords` intersect these tokens. Read its full body only after it makes the worklist. Resolve layer-precedence conflicts per READ and record dropped files in `suppressed`.
|
||||
|
||||
## Action
|
||||
|
||||
Generate `Page Scripting/E2E-PLAN.md`: a numbered chain of small recordings, each with the file name, the exact click-path the human follows while recording (Tell Me, page, field, value, action), the deterministic precondition, and the residue item it verifies. Generate the Test Seed Factory scaffold and the `E2E-00 Clear and Seed.yml` head as supporting artifacts. Emit one finding carrying the plan. Where a curated `testing` or `ux` file backs a residue allocation (a rule that a given behaviour needs a rendered client, or a determinism rule), cite it: `id` equal to the file path, `references` carrying it, `severity` `info`, `confidence` `high` for an unambiguous match. Where no curated file applies, emit an agent finding: `references: []`, `id` slug prefixed `agent:` (for example `agent:generated-e2e-plan`), `confidence` capped at `medium`, `severity` capped at `minor`, and a self-contained `message` listing the residue items, the recordings planned, and the path to write the plan to. Put the generated plan markdown in `suggested-code`. For any flow wrongly placed in layer 2 that an AL test could verify, emit a separate agent finding (`id` slug `agent:belongs-in-al-test`) recommending it stay in the TestPage suite; omit `suggested-code` and set `suggested-code-omission-reason` to `the recommendation is to keep the flow in the AL suite, not to generate a recording`. Hold every agent finding to the precision bar in `skills/do.md`.
|
||||
|
||||
Outcome selection: `completed` when the residue was identified and the plan generated (including when all flows are already covered by layer 1 and no recording is needed); `not-applicable` when the supplied path is not a test or user guide, or the repository has no rendered page to record; `no-knowledge` when no curated knowledge survived and no agent finding was raised; `partial` or `failed` per the DO contract with `outcome-reason`.
|
||||
|
||||
## Output
|
||||
|
||||
Output conforms to the DO output contract. The generated plan with no curated backing is an agent finding (`references: []`, `agent:` id, severity capped at `minor`, markdown in `suggested-code`); findings citing a `testing` or `ux` file carry that file path as `id` and primary reference.
|
||||
|
||||
```json
|
||||
{
|
||||
"skill": { "id": "page-scripting-e2e", "version": 1 },
|
||||
"outcome": "completed",
|
||||
"summary": {
|
||||
"counts": { "blocker": 0, "major": 0, "minor": 1, "info": 0 },
|
||||
"coverage": { "worklist-size": 4, "items-evaluated": 4 }
|
||||
},
|
||||
"findings": [
|
||||
{
|
||||
"id": "agent:generated-e2e-plan",
|
||||
"severity": "minor",
|
||||
"message": "Generated Page Scripting/E2E-PLAN.md with 4 recordings for the rendered-UI residue the green AL suite cannot reach: notification toast on release, Attention cue turning red at zero, factbox refresh on row change, and the real posting flow driven from the UI. Each recording is anchored to the FE2E- seed prefix with a reset No. Series so replays are deterministic. Write the plan and the Test Seed Factory scaffold into the repo.",
|
||||
"location": { "file": "Page Scripting/E2E-PLAN.md" },
|
||||
"references": [],
|
||||
"confidence": "medium",
|
||||
"suggested-code": "# E2E Page Scripting plan\n\n## E2E-00 Clear and Seed\n... numbered recording chain, each with click-path, precondition, and verified residue ..."
|
||||
}
|
||||
],
|
||||
"suppressed": []
|
||||
}
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue