Give AI the producer, queue, scheduler, worker, payload, retry, dead-letter, deployment, and observability evidence. Ask it to inventory stored work, describe the exact contract and side-effect delta, build a compatibility matrix for old and new producers and workers, and split the migration into accept, process, drain, and retire. Every phase needs delivery invariants, telemetry, a stop condition, and a recovery or replay action.
Queued Work Outlives the Deploy
A background job is not only a handler function. It is a durable agreement among the code that creates work, the serialized payload, the broker or scheduler that stores it, the worker that interprets it, and the external systems changed by its side effects. Delayed jobs, retries, scheduled tasks, dead-letter records, and paused queues may preserve older assumptions long after the producer that created them has been replaced.
Mixed versions are therefore normal during a migration. An old producer may publish while a new worker is deploying. A new payload may wait until an old worker restarts. A job first attempted before the release may be retried afterward. A dead-letter message may be replayed weeks later. Safe migration means deciding which combinations are valid, preventing the invalid ones, and proving that every accepted job reaches one intentional terminal state.
Use AI to assemble repository evidence and expose missing paths. Do not use it as the source of truth for queue depth, oldest-message age, live payload versions, broker delivery semantics, scheduled inventory, dead-letter contents, or whether an external side effect already happened. Confirm those facts with broker configuration, dashboards, logs, traces, deployment records, data queries, and responsible owners. Sanitize payload samples before adding them to a prompt.
The Five-Part Migration Plan
Locate every producer, queue or topic, scheduler, worker, retry queue, dead-letter path, replay tool, payload schema, idempotency store, and external side effect. Record routing keys, concurrency, acknowledgement point, visibility or lease behavior, delay and retention settings, retry limits, timeouts, and deployed versions from evidence. Measure current depth, oldest age, scheduled work, in-flight work, and dead letters instead of assuming the queue is nearly empty.
Write the current and target behavior side by side. Include payload fields, requiredness, defaults, version markers, routing, ordering key, deduplication identity, retry classification, timeout, acknowledgement point, and terminal states. Then describe the side effects: database writes, emails, payments, uploads, notifications, or calls to another service. A payload can remain structurally readable while changing business meaning, so schema compatibility alone is not enough.
Evaluate old and new producers against old and new workers, including delayed delivery, retry after deployment, and dead-letter replay. Mark each combination as supported, prevented by release order, transformed at a boundary, or quarantined for manual handling. Every supported combination needs a fixture and an expected terminal result. Every impossible combination needs a deployment or routing gate that makes it impossible in production.
Accept old and target payloads where the compatibility window requires both. Process target jobs in a controlled cohort or route while preserving the intended side-effect guarantees. Drain old work until measured depth, age, in-flight leases, scheduled jobs, and dead letters meet explicit thresholds. Retire old production first, then old consumption and replay support only after the final old-format path has cleared. These are separate gates, not one deployment.
Specify contract fixtures, mixed-version integration tests, duplicate-delivery tests, retry and timeout tests, side-effect invariants, and a controlled replay. Name the first queue or job class, monitoring window, owner, approval gate, and pause signal. Decide the recovery action for each phase: stop new production, route to the old worker, quarantine a payload version, pause consumption, repair state, or replay only jobs proven safe. “Roll back the code” is insufficient after messages or side effects have crossed the boundary.
A Minimal Compatibility Matrix
Separate production from consumption and first delivery from retry. That reveals the dangerous overlap a single happy-path test hides.
| Stored work or route | During migration | Required rule | Proof |
|---|---|---|---|
| Old payload → new worker | Usually expected | Preserve old semantics or transform through a tested, explicit boundary | Pre-migration payload fixture against the new worker |
| New payload → old worker | Prevent or support explicitly | Do not start target production until every reachable consumer can handle it | Deployment gate plus mixed-version integration test |
| Old job retried after cutover | Expected | Retain its stable job identity and classify the previous attempt correctly | Timeout, redelivery, and partial-side-effect test |
| In-flight worker during deploy | Expected | Define lease, shutdown, acknowledgement, and redelivery behavior | Controlled termination while the handler is running |
| Dead-letter replay → target worker | Policy decision | Inspect or transform deliberately; never bulk replay unknown failures blindly | Sampled dry run and bounded replay with stop conditions |
Ask what happens when a worker completes the external side effect but exits before acknowledgement, then the job is delivered to a different worker version. If the answer is “the queue prevents duplicates,” the plan is incomplete. Delivery deduplication and business-side-effect idempotency are different guarantees.
Retries Need a Stable Identity
Many queue systems can redeliver work after a timeout, crash, lost acknowledgement, or explicit retry. Make the logical job identity stable across those attempts and define what “already completed” means at the side-effect boundary. A newly generated identifier on every retry defeats deduplication. A check that is separate from the write can still race. The exact solution depends on the system, but the invariant must be explicit and tested where the irreversible effect occurs.
Also separate retryable infrastructure failures from permanent business failures. A malformed payload, missing account, revoked authorization, or invalid state transition should not loop forever merely because the worker has a generic catch-and-retry block. Record the terminal reason without leaking sensitive payload data, and give operators a deliberate repair or discard path.
Drain Is a Measured State, Not a Timer
Waiting an hour is not evidence that old work is gone. A drain gate should account for ready depth, delayed or scheduled messages, oldest age, in-flight leases, retry queues, dead letters, paused partitions, and producers that can still emit the old format. Record the last old-format production time and the longest relevant delay, retention, lease, and retry window. Then verify the absence of old work through the system's actual observability surfaces.
Keep the legacy reader and replay tooling until the drain gate clears and the rollback window closes. Give temporary compatibility code an owner and removal condition. Otherwise the migration quietly becomes permanent dual support, and a future replay can reactivate semantics nobody remembers.
Copy-Paste Prompt: Queue Migration Analysis
"Plan this background job or queue migration before editing files. Desired outcome: [outcome]. Producer, scheduler, payload, worker, retry, dead-letter, replay, idempotency, deployment, and observability paths: [paths]. Known queues, routes, deployed versions, delivery settings, queue depth, oldest age, scheduled work, and dead-letter evidence: [details or unknown]. External side effects and their current deduplication or transaction boundary: [details]. First inspect repository evidence and list runtime or broker facts that require owner confirmation. Then output: (1) runtime topology and stored-work inventory, (2) current-to-target payload, routing, retry, acknowledgement, and side-effect delta, (3) compatibility matrix for old/new producers and workers including delayed delivery, retry, and replay, (4) accept-process-drain-retire phases, (5) contract, mixed-version, duplicate-delivery, timeout, shutdown, and side-effect proof per phase, (6) queue and business telemetry, gates, and stop conditions, (7) recovery, quarantine, repair, and bounded replay action per phase, and (8) old-path removal criteria. Flag unstable job identities, unsafe retries, ordering assumptions, unbounded poison messages, PII exposure, and invented production facts. Do not implement yet."
Resolve the runtime unknowns before requesting a patch. Then implement the smallest reversible phase and review application code, broker configuration, deployment manifests, dashboards, alerts, replay tools, and runbooks together. Queue behavior often lives outside the repository, so a locally correct worker diff is not complete migration evidence.
Common Failure Modes
The first is changing the payload and worker in one release while ignoring messages already stored. The second is starting new production before every reachable worker is compatible. The third is treating retries as harmless without proving side-effect idempotency. The fourth is draining the ready queue while forgetting delayed jobs, in-flight leases, and dead letters. The fifth is bulk replaying failures after behavior has changed. The sixth is deleting the old reader because the dashboard looked quiet for an arbitrary period.
The correction is an evidence-backed sequence: map stored work, state the semantic delta, prove mixed versions and redelivery, separate production from consumption, drain every queue state, preserve a bounded repair path, and remove compatibility code only after its explicit gate clears.
Conclusion
AI is useful for tracing job paths, comparing payload handlers, generating fixtures, and challenging rollout order. The safety comes from verified runtime state and deliberate compatibility. Treat a background job change as a migration of durable work across independently deployed producers and workers—not as a rewrite of one handler.
Continue with The AI Database Migration Plan, The AI API Contract Change Plan, AI Observability, and AI-Assisted CI/CD.