Laravel Job timeout vs retry_after: The Ordering Rule Nothing Enforces

Two numbers, in two different configuration files, decide whether a queued job runs once or twice. One is retry_after, in config/queue.php. The other is your worker's timeout, in config/horizon.php or on the job class itself. Laravel ships them 30 seconds apart, mentions the relationship twice in prose, and then never checks it again — not at boot, not at dispatch, not when a supervisor starts a worker.

Get the order wrong and a job that is still running is handed to a second worker. What you see next depends entirely on an unrelated setting: either two workers execute the same job concurrently and both report success, or a job you never saw run twice lands in your failed jobs list with MaxAttemptsExceededException — for work that actually completed fine. This article is about the mechanism underneath both symptoms, the exact defaults, and the ordering rule nothing enforces for you.

The two numbers, and what they actually default to#

Both settings are about time, both are measured in seconds, and they sound close enough to be confused for each other. They are not. retry_after belongs to the queue connection and describes how long a reservation is honoured. timeout belongs to the worker and describes how long a single job is allowed to occupy a process. Different owners, different clocks, no shared enforcement.

That second row of defaults is the one worth staring at. A stock Laravel skeleton pairs a 60-second timeout with a 90-second retry_after, which is a correct configuration with 30 seconds of slack. But the 90 is a value in your configuration file, not a framework default. The framework's own fallback, used whenever a connection array has no retry_after key, is 60:

// Illuminate\Queue\Connectors\RedisConnector
return new RedisQueue(
    $this->redis, $config['queue'],
    Arr::get($config, 'connection', $this->connection),
    Arr::get($config, 'retry_after', 60),   // not 90
    // ...
);

So a hand-rolled connection — a second Redis connection for a dedicated queue, a connection built in a test, anything copied from a blog post rather than from config/queue.php — silently lands on 60, exactly equal to the default worker timeout. That configuration is broken from the first job that runs long, and it looks perfectly reasonable in a pull request.

The safe defaults are in the skeleton's config file, not in the framework. Delete the line and you get an unsafe one.

What retry_after actually does#

The name suggests a delay before a retry. It is closer to a lease on a rental car: taking the job out puts a return deadline on it, and once that deadline passes anyone may drive it away — including while you are still in it.

On Redis, popping a job is a single Lua script. It lpops the payload off the ready list, increments the payload's attempts counter, and adds the result to a sorted set, scored with the moment the reservation expires:

-- Illuminate\Queue\LuaScripts::pop()
local job = redis.call('lpop', KEYS[1])
local reserved = false

if(job ~= false) then
    -- Increment the attempt count and place job on the reserved queue...
    reserved = cjson.decode(job)
    reserved['attempts'] = reserved['attempts'] + 1
    reserved = cjson.encode(reserved)
    redis.call('zadd', KEYS[2], ARGV[1], reserved)
    redis.call('lpop', KEYS[3])
end

ARGV[1] is now + retry_after. Nothing watches that score on a timer. Instead, every single pop() on that queue begins by sweeping it:

// Illuminate\Queue\RedisQueue
public function pop($queue = null, $index = 0)
{
    $this->migrate($prefixed = $this->getQueueRedisKey($queue));
    // ...
}

migrate() moves two things back onto the ready list: delayed jobs whose time has come, and reserved jobs whose lease has expired. It cannot tell the difference between a job whose worker was killed by a deploy and a job whose worker is happily still working on it. Both look like an expired score.

The database driver reaches the same outcome through SQL rather than a sorted set — isReservedButExpired() widens the "next available job" query to include any row whose reserved_at is older than now - retry_after. Same lease, same blindness to whether the original worker is alive.

Two connections behave differently and are worth knowing about. Beanstalkd has no retry_after default of its own and falls back to Pheanstalk's TTR. SQS has no retry_after at all — the connector never reads the key, because the lease lives on Amazon's side as the queue's Default Visibility Timeout. Its default is 30 seconds, which is half of Laravel's default worker timeout. An SQS queue created with the console defaults and consumed by a stock worker is misconfigured out of the box, and no amount of editing config/queue.php will fix it.

What timeout actually does#

The worker's timeout has nothing to do with the queue. It is a POSIX alarm, set per job, inside the worker process:

// Illuminate\Queue\Worker::registerTimeoutHandler()
pcntl_signal(SIGALRM, function () use ($job, $options) {
    // ...record the failure, dispatch JobTimedOut...
    $this->kill(static::$timedOutExitCode ?? static::EXIT_ERROR, $options, WorkerStopReason::TimedOut);
}, true);

pcntl_alarm(
    max($this->timeoutForJob($job, $options), 0)
);

Three consequences fall straight out of those seven lines, and each one is a way for the ordering rule to break without anyone touching config/queue.php:

• The job wins. timeoutForJob() returns the job's own $timeout property or #[Timeout] attribute if it has one, and only falls back to the worker's option. A single #[Timeout(300)] on a slow report job overrides a perfectly-configured 60-second supervisor and blows straight past a 90-second lease.
• --timeout=0 disables it entirely. pcntl_alarm(0) cancels the alarm rather than firing immediately, so a zero timeout means no timeout. Nothing bounds the job's runtime, while the lease keeps expiring on schedule.
• No pcntl, no timeout. The whole handler sits behind a supportsAsyncSignals() check. On a build without the extension the timeout silently does nothing, and long jobs are duplicated with no local symptom to reproduce.

Why the two collide, and why equal is already broken#

Line the clocks up. Assume a 90-second job, a 90-second retry_after, and a 90-second timeout — the "matching" configuration that looks tidy in a config file:

t=0.00   worker A pops the job; reservation scored to expire at t=90.00
t=0.02   JobReserved fires, the worker enters process()
t=0.03   pcntl_alarm(90) armed  ->  will fire at t=90.03
t=90.00  worker B pops the same queue; migrate() sees an expired score
         and moves the still-running job back onto the ready list
t=90.00  worker B pops it, attempts becomes 2
t=90.03  worker A's alarm finally fires — 30ms too late to matter

The lease starts at the pop. The alarm is armed several steps later — after the reservation is written, after the JobReserved event and its listeners, after the worker enters process(). With identical values the reservation is guaranteed to expire first, by that setup delta, every time. And even when the alarm does fire first, the handler still has to record the failure and exit before the process lets go of anything.

"Timeout must be less than retry_after" is not a style preference with a safety margin bolted on. Equality is a losing race, not a tie.

This is also why the bug resists reproduction. Migration is lazy: it only happens when somebody pops that queue. A supervisor with a single worker, fully occupied by the long job, never sweeps its own expired reservation — so the misconfiguration stays invisible until a second worker exists. Autoscale from one process to two, or move from a laptop to production, and the same code starts double-running.

The two symptoms, and which one you get#

Here is the part that is rarely spelled out. Once the job has been migrated back and popped by a second worker, what happens next is decided by tries — because the Lua script incremented attempts on that second pop, and Worker::process() checks the attempt count before it fires the job:

// Illuminate\Queue\Worker::process()
$this->raiseBeforeJobEvent($connectionName, $job);

$this->markJobAsFailedIfAlreadyExceedsMaxAttempts(
    $connectionName, $job, (int) $options->maxTries
);

// ...

$job->fire();

With tries => 1: a failure on work that succeeded#

One attempt is Horizon's default — if you do not set tries on a supervisor, and the job class defines no $tries, you get a single attempt. The migrated copy arrives at worker B carrying attempts = 2, exceeds the limit before fire() is reached, and is failed immediately with MaxAttemptsExceededException. Its body never runs.

That is the good news and the confusing news at once. There is no duplicate execution — but you now have a failed job in the dashboard for work that is still running and, a few seconds later, completes successfully. Worker A finishes, tries to delete its reservation, and removes nothing, because the entry was migrated away while it worked. One job id, one failure, one success, and a stack trace pointing at a timeout that never happened.

If you have ever chased "MaxAttemptsExceededException on a job that clearly worked", this is one of the two ways to get there. (The other is a lock-based middleware spending attempts on releases, which is its own set of failure modes.)

With tries => 2 or more: silent concurrent execution#

Raise tries — which everyone does eventually, because retries are the point of a queue, and because middleware like RateLimited and WithoutOverlapping consume attempts — and the attempt check passes. Worker B calls fire() while worker A is inside the same job's handle().

Both copies now run to completion. Both charge the card, both send the email, both write the export. Neither throws, neither is marked failed, and the dashboard shows one job that completed. Nothing in Laravel, Horizon or your error tracker registers an anomaly. The only trace is in your data: a duplicate row, a doubled counter, a customer with two receipts.

The failure mode with tries at 1 is loud and wrong. The failure mode with tries above 1 is quiet and correct-looking. The second one is worse.

Four realistic ways a correct config becomes an incorrect one#

Nobody sets timeout above retry_after deliberately. It happens as a second-order effect of a change that looked local:

• A job grew a timeout of its own. A nightly export starts taking four minutes, someone adds #[Timeout(300)] to the job class, and the supervisor's tidy 60 is now irrelevant for that job while the connection's 90-second lease is unchanged. The job class is the highest-precedence setting and the one furthest from the config file where the constraint lives.
• A supervisor timeout was raised to fix timeouts. Jobs are being killed at 60 seconds, so the supervisor timeout goes to 120. The symptom disappears and is replaced by a quieter one, because retry_after is in a different file and was not part of the change.
• A new connection was added without the key. The 60-second framework fallback, described above. Equal values, broken on day one, and nothing in the diff to notice.
• The queue moved to SQS. The lease is now a Default Visibility Timeout of 30 seconds set on the AWS side, and config/queue.php has no say in it.

There is a fifth, opposite mistake worth naming: setting 'retry_after' => null on a Redis connection. That does not mean "never expire" in a benign sense — it skips reserved-job migration altogether, so a job whose worker dies is never recovered. You trade duplicates for silent loss.

The ordering rule, with numbers#

The full chain has three links, not two, because Horizon's supervisor timeout is a separate value from the job's — and with balance => 'auto', Horizon force-kills workers it considers hung during scale-down:

job timeout < supervisor timeout < retry_after

Read left to right, each step needs slack rather than a single second of headroom. A worked example for a job whose realistic worst case is three minutes:

The cost of over-shooting retry_after is bounded and dull: a job whose worker really did die waits longer before being recovered. The cost of under-shooting it is duplicate execution. When in doubt, pad the lease.

A supervisor serves one connection, so the constraint is checkable — and since nothing checks it for you, a test is the cheapest place to put it:

// tests/Feature/QueueTimeoutOrderingTest.php
public function test_supervisor_timeouts_stay_below_their_connection_lease(): void
{
    foreach (config('horizon.defaults') as $name => $supervisor) {
        $connection = $supervisor['connection'];

        // The 60 mirrors the framework's own fallback when the key is absent.
        $retryAfter = config("queue.connections.{$connection}.retry_after", 60);

        $this->assertLessThan(
            $retryAfter,
            $supervisor['timeout'] ?? 60,
            "Supervisor [{$name}] can hand a running job to a second worker.",
        );
    }
}

That covers the supervisors. It cannot see a $timeout property or a #[Timeout] attribute on an individual job class, which is the most common way the chain breaks — for those you have to look at runtime, when the worker resolves the effective timeout for the job in front of it.

Catching it in the logs#

The reason this misconfiguration survives in production for months is that the queue's own instrumentation reports it as ordinary activity. A reservation expiring looks exactly like a recovered job. A second worker picking work up looks exactly like work being picked up.

Skyline's job lifecycle logging makes the sequence legible, because every line is tagged with the job id and the transitions Laravel passes through silently — reserved, migrated, released, timed out — each get one. The fingerprint is a single job id reserved twice, with a migration between and no completion in between:

[11:04:12] queue.DEBUG: [job:91827364] reserved from [exports] and started processing.
[11:05:42] queue.INFO:  [job:91827364] migrated to [exports] and is ready to run.
[11:05:45] queue.DEBUG: [job:91827364] reserved from [exports] and started processing.
[11:06:20] queue.DEBUG: [job:91827364] completed.

Read top to bottom: the job was picked up, and then ninety seconds later — while still running, because no completion line came first — it was migrated back onto the ready queue and immediately picked up again. One grep for a job id is enough to tell a genuine retry from a double reservation, which is a distinction the dashboard alone cannot make. The reserved lines are debug, so turn the channel down for the investigation and back up afterwards; the migration line is info and can stay on permanently.

The core takeaway#

retry_after is a lease on a job, and the worker's timeout is the only thing that guarantees the job hands that lease back before it expires. Laravel ships the two numbers 30 seconds apart, states the constraint in the documentation, and enforces it nowhere — so every change that touches either one, on a job class, on a supervisor, or on a connection, can quietly break it. Order the chain, leave real slack at each step, and put the assertion somewhere that runs. It is the rare production bug whose fix is a single integer, and whose cost of missing it is measured in duplicated side effects nobody reports.

If you are working through queue reliability more broadly, 12 best practices for Laravel background jobs covers the idempotency habits that make a double execution survivable rather than expensive, and rate-limited APIs and Laravel queues covers the middleware that pushes jobs toward their timeout in the first place. For the logging shown above, see job lifecycle logging — and Skyline vs Horizon for what else changes when you swap the package.