A scheduled job can be perfectly healthy and still damage the data if two copies run at the same time. The usual cause is simple: one run takes longer than expected, the next timer fires, and both workers process the same queue, export window, or set of records.
The fix is not merely a faster server or a shorter schedule. Reliable automation needs an explicit answer to a basic question: what should happen when a second run starts before the first one finishes?
Why scheduled jobs overlap
Many workflows begin with a timer, cron entry, or platform scheduler. The scheduler starts a worker, but it often does not know whether the previous worker is still active. If the first run normally takes five minutes and the job runs every five minutes, a small delay can create overlap. A slow API, a locked database table, a large batch, or a transient retry can extend the first run into the next interval.
Overlap can also happen after deployment, when a job is accidentally registered twice, or when a manual recovery run is started while the regular schedule is still active. In distributed systems, multiple application instances may each believe they are responsible for the same scheduled task.
The visible symptom depends on the workflow. A CRM sync may create duplicate notes. An email process may enqueue the same recipient twice. A reporting export may write two files for one reporting period. A cleanup job may update records while another worker is still deciding which records qualify.
Start with a run policy
Before choosing a locking mechanism, define the intended behavior. There are three common policies:
- Skip: If a run is active, the new trigger records that it was skipped and exits. This is appropriate when the next scheduled run can safely cover the work.
- Queue: The new trigger becomes a pending run that waits for the active run to finish. This is useful when every trigger represents a distinct request or time window.
- Coalesce: Multiple triggers collapse into one pending run that processes the latest known state. This works well for refresh-style jobs where the goal is current data, not one execution per trigger.
Do not leave this decision hidden inside a timeout or an implementation detail. Record the policy in the runbook, because operators need to know whether a skipped run is expected behavior or an incident.
Use a lock that has an owner and an expiry
A practical run lock needs more than a boolean called is_running. At minimum, store:
- A unique lock name for the job.
- A run ID that identifies the worker holding the lock.
- The time the lock was acquired.
- An expiry or lease deadline.
- The worker or deployment instance that acquired it.
- The last heartbeat, if the job can run for a long time.
Acquisition must be atomic. Two workers should not be able to read an empty lock and then both write themselves as the owner. Depending on the environment, the lock can live in a database row with a uniqueness constraint, a distributed cache with an atomic set-if-not-exists operation, or a scheduler that provides a documented single-run guarantee.
Use a lease rather than a permanent lock. If a process crashes after acquiring the lock, the job must become eligible to run again. The lease should be longer than the expected work interval but short enough to recover from a failed worker. Long-running jobs should renew the lease with a heartbeat and stop if renewal fails.
Make the work safe if two workers still meet
A lock reduces overlap, but it should not be the only protection. Locks can expire, be misconfigured, or fail during an infrastructure incident. The work itself should be retry-safe.
Give each work item a stable identity, such as a source record ID plus the operation being performed. Store that identity where the side effect is recorded, and enforce uniqueness at the database boundary when possible. A retry can then discover that the item was already completed instead of sending, creating, or updating it again.
Separate the identity of a run from the identity of the work. A run ID answers, “Which execution attempted this?” A work ID answers, “Which business action is this?” If a recovery run processes the same source record, it should create a new run record but reuse the same work identity.
| Layer | Question to answer |
|---|---|
| Scheduler | Should a second trigger skip, queue, or coalesce? |
| Lock | Which worker owns the active run, and when does the lease expire? |
| Work item | What stable identity prevents the same action from being applied twice? |
| Destination | What constraint or lookup detects an existing result? |
| Reconciliation | How will uncertain outcomes be compared with the source of truth? |
Keep the batch boundary visible
Overlapping jobs are harder to diagnose when one run processes an unbounded query. Use explicit batch sizes and checkpoints. A run should be able to say which records it selected, which records it completed, which records failed, and which records remain.
Claim work before processing it when the queue or database supports that pattern. A claimed item should carry a worker ID, claim time, and lease expiry. Another worker can reclaim it only after the lease has expired. This turns an invisible race into an observable state transition.
Do not mark a batch complete just because the API returned a successful HTTP response. Record the destination identifier or confirmed state needed to prove that the intended side effect occurred. If the response is lost after the remote system accepts the request, reconciliation should check for the existing result before retrying.
Test the failure cases on purpose
A normal run is not enough to validate overlap protection. Test the cases that create the race:
- Start a second worker while the first worker is processing the same batch.
- Pause the first worker after it acquires the lock but before it writes a result.
- Let the lease expire and verify that recovery is possible without creating duplicates.
- Restart the application during a remote API call and then run reconciliation.
- Register two scheduler entries in a test environment and confirm that only one owner proceeds.
- Run a manual recovery while the normal schedule is active.
- Force a partial batch failure and confirm that a retry resumes from a clear checkpoint.
For every test, inspect both sides of the workflow. Check the run log, the work-item table, the remote destination, and any downstream analytics or reporting record. A green job status is not proof that the business outcome is correct.
Monitor overlap as an operational signal
Track more than success and failure. Useful measurements include lock-acquisition attempts, skipped triggers, queue depth, lease expirations, run duration, batch size, retry count, uncertain outcomes, and reconciliation findings.
An increase in run duration can predict future overlap before duplicates appear. A growing number of skipped triggers may mean the schedule is too aggressive, the batch is too large, or the destination API has slowed down. Alerts should point to the decision an operator needs to make, such as shortening a recovery window or investigating a stuck worker.
Build the recovery path before launch
Every scheduled workflow needs a documented recovery path. Define how to find an incomplete run, how to identify its unprocessed work, how to distinguish completed actions from uncertain ones, and how to replay only the safe remainder.
Keep recovery bounded. A recovery command should require a job name, a time or cursor range, and an explicit confirmation for large batches. It should write its own run record and produce a reconciliation summary. Avoid a “run everything again” button unless the underlying operation is genuinely idempotent and the consequences are understood.
DigitalWerks can review scheduled integrations, queue boundaries, identifiers, run locks, retry behavior, and reconciliation checks together. That review is useful when a workflow has grown from a simple cron task into a process that moves customer, donor, form, payment, or reporting data between systems.
Scheduled jobs do not become reliable because they run on a timer. They become reliable when overlap has a defined policy, ownership is explicit, work has stable identity, and recovery can prove what happened.