Add Job Queue reliability and scheduling guidance (#148)

* knowledge(performance): add job queue reliability guidance

* Address Job Queue review feedback

* Address Job Queue routing review feedback

* Encode job queue conflict in fixtures

microsoft/knowledge/performance/httpclient-inside-write-transaction-holds-locks.md

@@ -17,7 +17,7 @@ The first database write opens an AL write transaction that the runtime holds un
 
 ## Best Practice
 
-Defer the HTTP call to a separate session. When the external operation must correspond to a committed database change, insert an outbox work item in the same transaction as that change and process committed outbox rows with a recurring job queue entry. The change and work item then commit or roll back together, and the worker performs HTTP before deleting the item so it holds no write lock during the call. Make the external operation idempotent because a failure after a successful HTTP response can cause the work item to be retried.
+Defer the HTTP call to a separate session. When the external operation must correspond to a committed database change, insert an outbox work item in the same transaction as that change and process committed outbox rows with a recurring job queue entry. The change and work item then commit or roll back together, and the worker performs HTTP before deleting the item so it holds no write lock during the call. The separate retry-safety requirement is covered by `job-queue-external-effects-must-be-idempotent.md`.
 
 A directly created scheduled task is suitable only when its work is independent of the caller's commit. An immediately ready task can run concurrently with the caller, so it must not assume that the caller's writes are already committed. Do **not** use `Commit()` as a general remedy: it irrevocably commits all prior writes in the current transaction, so any subsequent failure cannot roll them back. `Commit()` is appropriate only at top-level entry points where partial persistence is intentional and understood.
 

microsoft/knowledge/performance/job-queue-category-code-serializes-conflicting-jobs.bad.al

@@ -0,0 +1,11 @@
+codeunit 50113 "Job Queue Category Bad"
+{
+    procedure ConfigureJobsForSharedExclusiveResource(var SalesPostingJob: Record "Job Queue Entry"; var PurchasePostingJob: Record "Job Queue Entry"; ExclusiveResourceId: Text[250])
+    begin
+        // Both jobs update the same posting resources, but nothing prevents overlap.
+        SalesPostingJob.Validate("Parameter String", ExclusiveResourceId);
+        PurchasePostingJob.Validate("Parameter String", ExclusiveResourceId);
+        SalesPostingJob.Validate("Job Queue Category Code", '');
+        PurchasePostingJob.Validate("Job Queue Category Code", '');
+    end;
+}

microsoft/knowledge/performance/job-queue-category-code-serializes-conflicting-jobs.good.al

@@ -0,0 +1,18 @@
+codeunit 50113 "Job Queue Category Good"
+{
+    procedure ConfigureJobsForSharedExclusiveResource(var SalesPostingJob: Record "Job Queue Entry"; var PurchasePostingJob: Record "Job Queue Entry"; ExclusiveResourceId: Text[250])
+    var
+        JobQueueCategory: Record "Job Queue Category";
+    begin
+        if not JobQueueCategory.Get('POSTING') then begin
+            JobQueueCategory.Code := 'POSTING';
+            JobQueueCategory.Insert();
+        end;
+
+        // The shared category lets only one conflicting posting job run at a time.
+        SalesPostingJob.Validate("Parameter String", ExclusiveResourceId);
+        PurchasePostingJob.Validate("Parameter String", ExclusiveResourceId);
+        SalesPostingJob.Validate("Job Queue Category Code", 'POSTING');
+        PurchasePostingJob.Validate("Job Queue Category Code", 'POSTING');
+    end;
+}

microsoft/knowledge/performance/job-queue-category-code-serializes-conflicting-jobs.md

@@ -0,0 +1,28 @@
+---
+bc-version: [all]
+domain: performance
+keywords: [job-queue, category-code, concurrency, waiting, serialization, locking]
+technologies: [al]
+countries: [w1]
+application-area: [all]
+---
+
+# Use a job queue category to serialize conflicting jobs
+
+> Contributions welcome — open a PR to refine or extend this article.
+
+## Description
+
+Different job queue entries can run at the same time. When two jobs update the same exclusive resource, concurrent execution can cause lock contention, deadlocks, or conflicting results. Within one company, entries with the same Job Queue Category Code are serialized: while one runs, another entry in that category waits.
+
+## Best Practice
+
+Assign the same non-empty Job Queue Category Code to job queue entries in the same company that must not overlap, regardless of which codeunit they run. Define categories around the shared resource or exclusivity requirement, not merely around object names. Leave independent jobs in different categories so they can still run concurrently. A category does not serialize work across companies or environments, or coordinate workers outside the job queue dispatcher. Protect shared external or cross-company resources with a separate application-level locking mechanism.
+
+See sample: `job-queue-category-code-serializes-conflicting-jobs.good.al`.
+
+## Anti Pattern
+
+Creating or configuring multiple job queue entries that update the same exclusive resource while leaving their Job Queue Category Code empty or different. Do not flag jobs merely because they touch the same tables; the rule applies when their operation requires mutual exclusion.
+
+See sample: `job-queue-category-code-serializes-conflicting-jobs.bad.al`.

microsoft/knowledge/performance/job-queue-external-effects-must-be-idempotent.bad.al

@@ -0,0 +1,57 @@
+table 50112 "Queued Export Bad"
+{
+    DataClassification = CustomerContent;
+
+    fields
+    {
+        field(1; "Entry No."; Integer)
+        {
+            AutoIncrement = true;
+        }
+        field(2; Payload; Text[250])
+        {
+        }
+    }
+
+    keys
+    {
+        key(PK; "Entry No.")
+        {
+            Clustered = true;
+        }
+    }
+}
+
+codeunit 50112 "Queued Export Worker Bad"
+{
+    TableNo = "Job Queue Entry";
+
+    trigger OnRun()
+    var
+        QueuedExport: Record "Queued Export Bad";
+        Client: HttpClient;
+        Content: HttpContent;
+        Response: HttpResponseMessage;
+    begin
+        if not QueuedExport.FindFirst() then
+            exit;
+
+        Content.WriteFrom(QueuedExport.Payload);
+        Client.Post('https://example.local/exports', Content, Response);
+        if not Response.IsSuccessStatusCode() then
+            Error('Export failed with HTTP status %1.', Response.HttpStatusCode());
+
+        // If this local step fails, the external export exists but this row is retried.
+        UpdateLocalStatus();
+        FinalizeExport(QueuedExport);
+    end;
+
+    local procedure UpdateLocalStatus()
+    begin
+    end;
+
+    local procedure FinalizeExport(var QueuedExport: Record "Queued Export Bad")
+    begin
+        QueuedExport.Delete();
+    end;
+}

microsoft/knowledge/performance/job-queue-external-effects-must-be-idempotent.good.al

@@ -0,0 +1,65 @@
+table 50112 "Queued Export Good"
+{
+    DataClassification = CustomerContent;
+
+    fields
+    {
+        field(1; "Entry No."; Integer)
+        {
+            AutoIncrement = true;
+        }
+        field(2; Payload; Text[250])
+        {
+        }
+    }
+
+    keys
+    {
+        key(PK; "Entry No.")
+        {
+            Clustered = true;
+        }
+    }
+
+}
+
+codeunit 50112 "Queued Export Worker Good"
+{
+    TableNo = "Job Queue Entry";
+
+    trigger OnRun()
+    var
+        QueuedExport: Record "Queued Export Good";
+        Client: HttpClient;
+        Content: HttpContent;
+        ContentHeaders: HttpHeaders;
+        JsonPayload: JsonObject;
+        RequestBody: Text;
+        Response: HttpResponseMessage;
+    begin
+        if not QueuedExport.FindFirst() then
+            exit;
+
+        JsonPayload.Add('idempotencyKey', Format(QueuedExport.SystemId));
+        JsonPayload.Add('payload', QueuedExport.Payload);
+        JsonPayload.WriteTo(RequestBody);
+
+        Content.WriteFrom(RequestBody);
+        Content.GetHeaders(ContentHeaders);
+        ContentHeaders.Clear();
+        ContentHeaders.Add('Content-Type', 'application/json');
+        Client.Post('https://example.local/exports', Content, Response);
+        if not Response.IsSuccessStatusCode() then
+            Error('Export failed with HTTP status %1.', Response.HttpStatusCode());
+
+        // The external service must atomically create a record only when idempotencyKey
+        // does not exist. When the key already exists, it must return the existing record
+        // without repeating the side effect.
+        UpdateLocalStatus();
+        QueuedExport.Delete();
+    end;
+
+    local procedure UpdateLocalStatus()
+    begin
+    end;
+}

microsoft/knowledge/performance/job-queue-external-effects-must-be-idempotent.md

@@ -0,0 +1,30 @@
+---
+bc-version: [all]
+domain: performance
+keywords: [job-queue, idempotency, retry, outbox, httpclient, external-effect]
+technologies: [al]
+countries: [w1]
+application-area: [all]
+---
+
+# Job queue external effects must be idempotent
+
+> Contributions welcome — open a PR to refine or extend this article.
+
+## Description
+
+A job queue handler can successfully create something in an external system and then fail while updating Business Central. Business Central rolls back its database changes, but it cannot roll back the external request. The same work can later run again through configured retries, recurrence, rescheduling, or manual restart. Without a way for the external system to recognize the repeated request, a later run can create a duplicate shipment, payment, notification, or other side effect.
+
+## Best Practice
+
+Use a stable request ID that exists before the job queue processes the outbox row. For example, include the outbox record's `SystemId` as an `idempotencyKey` value in the JSON body of every POST attempt. The external service must enforce uniqueness on that value: when it receives the key again, it returns the existing record instead of creating another one. Delete the outbox row only after the external call and all required local updates succeed.
+
+A `Processed` flag set after the external call does not solve this failure window. If a later AL error rolls back that flag, the outbox row again looks unprocessed even though the external operation already happened.
+
+See sample: `job-queue-external-effects-must-be-idempotent.good.al`.
+
+## Anti Pattern
+
+Sending a state-changing request from a job queue handler with no stable request ID understood by the external API. Specifically, look for this sequence: read an outbox row, call `HttpClient.Post` or another side-effecting API, update or delete local data, and propagate an error after which the same outbox row can be processed again. The key may be part of the request body, URI, headers, or an existing business key; a naturally idempotent remote operation is already safe and should not be flagged.
+
+See sample: `job-queue-external-effects-must-be-idempotent.bad.al`.

microsoft/knowledge/performance/job-queue-handlers-must-not-require-ui.bad.al

@@ -0,0 +1,17 @@
+codeunit 50110 "Job Queue UI Bad"
+{
+    TableNo = "Job Queue Entry";
+
+    trigger OnRun()
+    begin
+        if not Confirm('Process the queued export now?') then
+            exit;
+
+        ProcessExport(Rec."Parameter String");
+        Message('The queued export completed.');
+    end;
+
+    local procedure ProcessExport(ParameterString: Text)
+    begin
+    end;
+}

microsoft/knowledge/performance/job-queue-handlers-must-not-require-ui.good.al

@@ -0,0 +1,14 @@
+codeunit 50110 "Job Queue UI Good"
+{
+    TableNo = "Job Queue Entry";
+
+    trigger OnRun()
+    begin
+        Rec.TestField("Parameter String");
+        ProcessExport(Rec."Parameter String");
+    end;
+
+    local procedure ProcessExport(ParameterString: Text)
+    begin
+    end;
+}

microsoft/knowledge/performance/job-queue-handlers-must-not-require-ui.md

@@ -0,0 +1,28 @@
+---
+bc-version: [all]
+domain: performance
+keywords: [job-queue, background-session, guiallowed, confirm, runmodal, client-callback]
+technologies: [al]
+countries: [w1]
+application-area: [all]
+---
+
+# Job queue handlers must not require user interaction
+
+> Contributions welcome — open a PR to refine or extend this article.
+
+## Description
+
+A job queue handler runs in a background session with no client UI. Calls that require a client callback, such as `Confirm`, `Page.RunModal`, `Report.RunModal`, upload, or download, can stop the job with a non-retriable callback error. `Message` is suppressed and logged by the server, so it cannot communicate a result to the user who scheduled the job.
+
+## Best Practice
+
+Make a dedicated job queue entry point non-interactive. Validate parameters and data in AL, persist business-visible status when needed, and let failures propagate to the job queue log. If one procedure genuinely serves both foreground and background callers, isolate optional UI-only behavior behind `GuiAllowed`; do not use the guard to silently skip a decision that the operation requires.
+
+See sample: `job-queue-handlers-must-not-require-ui.good.al`.
+
+## Anti Pattern
+
+Calling `Confirm`, `Page.Run`, `Page.RunModal`, `Report.Run`, `Report.RunModal`, `Hyperlink`, `File.Upload`, or `File.Download` from a codeunit run by the job queue. Another signal is using `Message` as the only success or failure notification: no user is attached to receive it.
+
+See sample: `job-queue-handlers-must-not-require-ui.bad.al`.

microsoft/knowledge/performance/job-queue-handlers-must-propagate-failures.bad.al

@@ -0,0 +1,23 @@
+codeunit 50111 "Job Queue Failure Bad"
+{
+    TableNo = "Job Queue Entry";
+
+    trigger OnRun()
+    begin
+        if not TryProcessCustomer(Rec."Parameter String") then
+            exit;
+    end;
+
+    [TryFunction]
+    local procedure TryProcessCustomer(CustomerNo: Code[20])
+    var
+        Customer: Record Customer;
+    begin
+        Customer.Get(CustomerNo);
+        ProcessCustomer(Customer);
+    end;
+
+    local procedure ProcessCustomer(Customer: Record Customer)
+    begin
+    end;
+}

microsoft/knowledge/performance/job-queue-handlers-must-propagate-failures.good.al

@@ -0,0 +1,16 @@
+codeunit 50111 "Job Queue Failure Good"
+{
+    TableNo = "Job Queue Entry";
+
+    trigger OnRun()
+    var
+        Customer: Record Customer;
+    begin
+        Customer.Get(Rec."Parameter String");
+        ProcessCustomer(Customer);
+    end;
+
+    local procedure ProcessCustomer(Customer: Record Customer)
+    begin
+    end;
+}

microsoft/knowledge/performance/job-queue-handlers-must-propagate-failures.md

@@ -0,0 +1,28 @@
+---
+bc-version: [all]
+domain: performance
+keywords: [job-queue, error-propagation, tryfunction, retry, dispatcher, job-queue-log]
+technologies: [al]
+countries: [w1]
+application-area: [all]
+---
+
+# Job queue handlers must propagate execution failures
+
+> Contributions welcome — open a PR to refine or extend this article.
+
+## Description
+
+The job queue dispatcher can mark an entry as failed, record the error, and apply its configured retry behavior only when the handler terminates with an error. A handler that catches a failed `TryFunction` or Boolean-returning operation and then returns normally reports success to the dispatcher, even though its work did not complete.
+
+## Best Practice
+
+Let an error that invalidates the whole run propagate out of the job queue entry point. Add context only when it helps an operator diagnose the failure and does not expose sensitive data. Per-item failures may be collected deliberately, but the batch must persist or emit an observable aggregate outcome instead of silently treating incomplete work as success.
+
+See sample: `job-queue-handlers-must-propagate-failures.good.al`.
+
+## Anti Pattern
+
+Calling a `TryFunction`, `Codeunit.Run`, or another Boolean-returning operation from a job queue handler and using `exit` or normal fall-through on failure without recording an intentional partial-success outcome. The dispatcher sees a successful return, so the entry's status and log do not represent the failed work and configured retries are not applied.
+
+See sample: `job-queue-handlers-must-propagate-failures.bad.al`.

microsoft/knowledge/performance/job-queue-on-hold-does-not-stop-running-work.bad.al

@@ -0,0 +1,18 @@
+codeunit 50114 "Job Queue On Hold Bad"
+{
+    TableNo = "Job Queue Entry";
+
+    trigger OnRun()
+    begin
+        repeat
+            if not ProcessNextBatch() then
+                exit;
+            Rec.Get(Rec.ID);
+        until Rec.Status = Rec.Status::"On Hold";
+    end;
+
+    local procedure ProcessNextBatch(): Boolean
+    begin
+        exit(false);
+    end;
+}

microsoft/knowledge/performance/job-queue-on-hold-does-not-stop-running-work.good.al

@@ -0,0 +1,49 @@
+table 50114 "Job Cancellation Control"
+{
+    DataClassification = SystemMetadata;
+
+    fields
+    {
+        field(1; "Job Queue Entry ID"; Guid)
+        {
+        }
+        field(2; "Stop Requested"; Boolean)
+        {
+        }
+    }
+
+    keys
+    {
+        key(PK; "Job Queue Entry ID")
+        {
+            Clustered = true;
+        }
+    }
+}
+
+codeunit 50114 "Job Queue On Hold Good"
+{
+    TableNo = "Job Queue Entry";
+
+    trigger OnRun()
+    begin
+        while not IsStopRequested(Rec.ID) do
+            if not ProcessNextBatch() then
+                exit;
+    end;
+
+    local procedure IsStopRequested(JobQueueEntryId: Guid): Boolean
+    var
+        JobCancellationControl: Record "Job Cancellation Control";
+    begin
+        if not JobCancellationControl.Get(JobQueueEntryId) then
+            exit(false);
+
+        exit(JobCancellationControl."Stop Requested");
+    end;
+
+    local procedure ProcessNextBatch(): Boolean
+    begin
+        exit(false);
+    end;
+}

microsoft/knowledge/performance/job-queue-on-hold-does-not-stop-running-work.md

@@ -0,0 +1,28 @@
+---
+bc-version: [all]
+domain: performance
+keywords: [job-queue, on-hold, cancellation, in-process, long-running, stop-request]
+technologies: [al]
+countries: [w1]
+application-area: [all]
+---
+
+# Putting a job queue entry on hold does not stop its current run
+
+> Contributions welcome — open a PR to refine or extend this article.
+
+## Description
+
+The On Hold status prevents a job queue entry from starting again, but it does not cancel a run that is already in process. A long-running handler continues until it completes, fails, reaches a cancellation point implemented by the application, or its session is stopped externally.
+
+## Best Practice
+
+Use On Hold to pause future scheduling. When a long-running operation must support graceful cancellation, store a separate application-owned stop request and check it before every bounded unit of work, including the first. Exit only at a point where completed work and the checkpoint are consistent. The code that resumes scheduling must clear the stop request before restarting the job. Use administrative session termination only when graceful cancellation is impossible.
+
+See sample: `job-queue-on-hold-does-not-stop-running-work.good.al`.
+
+## Anti Pattern
+
+Polling the job queue entry's Status field from inside its handler and expecting a change to On Hold to cancel the active run. The status controls scheduling, not cooperative cancellation, so the handler can continue processing despite the operator's action.
+
+See sample: `job-queue-on-hold-does-not-stop-running-work.bad.al`.

microsoft/knowledge/performance/oncompanyopen-subscribers-must-not-do-io.md

@@ -17,7 +17,7 @@ application-area: [all]
 
 ## Best Practice
 
-Keep company-open subscribers to cheap in-memory work: set a flag, enqueue a job-queue entry, or `TaskScheduler.CreateTask`. Perform HTTP and large SQL after the session is running, in that background work.
+Keep company-open subscribers to cheap in-memory work: set a flag, enqueue a job-queue entry, or `TaskScheduler.CreateTask`. Perform HTTP and large SQL after the session is running, in that background work. When the subscriber can run repeatedly, use `store-scheduled-task-id-to-avoid-duplicate-tasks.md` to avoid creating the same logical task more than once.
 
 See sample: [`oncompanyopen-subscribers-must-not-do-io.good.al`](oncompanyopen-subscribers-must-not-do-io.good.al).
 

microsoft/knowledge/performance/store-scheduled-task-id-to-avoid-duplicate-tasks.bad.al

@@ -0,0 +1,15 @@
+codeunit 50115 "Scheduled Task Duplicate Bad"
+{
+    procedure EnsureCleanupTask()
+    begin
+        // Every call creates another task for the same cleanup work.
+        TaskScheduler.CreateTask(Codeunit::"Scheduled Cleanup Work Bad", 0, true, CompanyName());
+    end;
+}
+
+codeunit 50116 "Scheduled Cleanup Work Bad"
+{
+    trigger OnRun()
+    begin
+    end;
+}

microsoft/knowledge/performance/store-scheduled-task-id-to-avoid-duplicate-tasks.good.al

@@ -0,0 +1,23 @@
+codeunit 50115 "Scheduled Task Duplicate Good"
+{
+    internal procedure EnsureCleanupTask()
+    var
+        TaskId: Guid;
+        StoredTaskId: Text;
+    begin
+        if IsolatedStorage.Get('CleanupTaskId', DataScope::Company, StoredTaskId) then
+            if Evaluate(TaskId, StoredTaskId) then
+                if TaskScheduler.TaskExists(TaskId) then
+                    exit;
+
+        TaskId := TaskScheduler.CreateTask(Codeunit::"Scheduled Cleanup Work Good", 0, true, CompanyName());
+        IsolatedStorage.Set('CleanupTaskId', Format(TaskId), DataScope::Company);
+    end;
+}
+
+codeunit 50116 "Scheduled Cleanup Work Good"
+{
+    trigger OnRun()
+    begin
+    end;
+}

microsoft/knowledge/performance/store-scheduled-task-id-to-avoid-duplicate-tasks.md

@@ -0,0 +1,28 @@
+---
+bc-version: [all]
+domain: performance
+keywords: [task-scheduler, scheduled-task, taskexists, duplicate-task, createtask, guid]
+technologies: [al]
+countries: [w1]
+application-area: [all]
+---
+
+# Store the scheduled task ID to avoid duplicate tasks
+
+> Contributions welcome — open a PR to refine or extend this article.
+
+## Description
+
+Every call to `TaskScheduler.CreateTask` creates a new scheduled task and returns its unique GUID. Repeating setup or lifecycle code without retaining that GUID can create multiple tasks for the same logical work, consuming scheduler capacity and running the work more than once.
+
+## Best Practice
+
+Persist the GUID returned by `CreateTask` at the same scope as the logical task. Before creating a replacement, parse the stored GUID and call `TaskScheduler.TaskExists`; create and store a new task only when the previous task no longer exists. `TaskExists` checks one GUID, not whether an equivalent codeunit is already scheduled, so callers that can schedule concurrently still need serialization around this check-and-create sequence.
+
+See sample: `store-scheduled-task-id-to-avoid-duplicate-tasks.good.al`.
+
+## Anti Pattern
+
+Calling `TaskScheduler.CreateTask` every time initialization, login, setup, or another repeatable path runs while ignoring its return value. Each invocation creates another independent task even when an equivalent task is already pending.
+
+See sample: `store-scheduled-task-id-to-avoid-duplicate-tasks.bad.al`.