The Life of a Laravel Queued Job: Every State, Every Transition

A Laravel job spends almost none of its life as an object. It is an object for the microseconds between new SendInvoice($order) and dispatch(), and again for the milliseconds between a worker deserialising it and handle() returning. Everything in between — the waiting, the reserving, the retrying, the failing — happens to a JSON string sitting in one of three Redis keys.

This is a walk through all of it, in order: what dispatch() writes, how a worker takes exactly one copy of it, why the attempt is spent at that moment rather than when something goes wrong, what the reservation guarantees when a worker is killed mid-job, and how a delayed job, a retried job, an exhausted job, a rate-limited job and a deploy all move through the same small set of transitions. Every claim here is traced to the code that makes it true.

One queue, four Redis keys#

Before any of the transitions make sense, the storage has to. A single queue named emails on the Redis driver is not one key. It is four — three that hold jobs, plus one that exists so a worker can sleep instead of poll — and every state a job can be in is "which of these keys is it in":

Notice what is not in that list: there is no "running" key, no "processing" flag, and nothing anywhere that records which worker holds a job. The reserved set stores a payload and a deadline. That single design decision explains most of the behaviour in the rest of this article — including the good parts.

Dispatch: what actually gets written#

dispatch(new SendInvoice($order)) ends in RedisQueue::push(), which builds a payload array and hands it to a two-command Lua script. The payload is worth reading closely, because everything the worker later decides is decided from these fields — not from the job class:

// Illuminate\Queue\Queue::createObjectPayload()
[
    'uuid' => (string) Str::uuid(),
    'displayName' => 'App\Jobs\SendInvoice',
    'job' => 'Illuminate\Queue\CallQueuedHandler@call',
    'maxTries' => 3,          // $tries on the job, or #[Tries]
    'maxExceptions' => null,
    'failOnTimeout' => false,
    'backoff' => '30,120',    // $backoff, flattened to a string
    'timeout' => null,        // $timeout on the job, or #[Timeout]
    'retryUntil' => null,     // retryUntil() resolved to a timestamp, now
    'data' => ['commandName' => ..., 'command' => 'O:21:"App\Jobs\SendInvoice"...'],
    'createdAt' => 1756...,
    'id' => 'ZK3v...',        // 32 random chars, added by the Redis driver
    'attempts' => 0,          // added by the Redis driver
]

Two of those fields are frozen at dispatch and are a common source of surprise. retryUntil is resolved here, by calling your retryUntil() method at dispatch time and storing the resulting timestamp — so the window starts when the job is queued, not when it first runs. And command is the serialised job object, which is why renaming or restructuring a job class breaks everything already in the queue (covered in more depth here).

The write itself is two commands:

-- Illuminate\Queue\LuaScripts::push()
-- Push the job onto the queue...
redis.call('rpush', KEYS[1], ARGV[1])
-- Push a notification onto the "notify" queue...
redis.call('rpush', KEYS[2], 1)

rpush onto the tail, and a token onto the notify list. That second list is the whole mechanism behind block_for: a worker with nothing to do calls blpop on it and is woken by Redis the instant a job arrives, instead of sleeping for a fixed interval and discovering the job late.

Two things that can happen before the write#

The transaction gate. If the connection has after_commit set, or the job is ShouldQueueAfterCommit, or it was dispatched with ->afterCommit(), the push is not executed — it is registered as a callback on the current database transaction and runs on commit. If the transaction rolls back, the job is never queued at all. Without it, a worker can pop the job before the transaction commits and find no row.

The uniqueness gate. A ShouldBeUnique job acquires its lock before the push, and if the lock is held the dispatch is a silent no-op — nothing is written, nothing is logged, nothing appears in the dashboard. That is admission control at the door, and it is a completely different mechanism from the runtime locks discussed later (the failure modes of both).

Delayed jobs live somewhere else entirely#

dispatch(new SendInvoice($order))->delay(now()->addMinutes(10)) takes a different path. It never touches the ready list:

-- Illuminate\Queue\LuaScripts::later()
-- Push the job onto the delayed queue...
redis.call('zadd', KEYS[1], ARGV[1], ARGV[2])

One zadd into queues:emails:delayed, scored with the timestamp it becomes available. No token goes onto the notify list, because there is nothing to wake a worker for yet.

Nothing is watching that sorted set on a timer. There is no scheduler process, no zpop loop, no key-expiry listener. A delayed job becomes available only because some worker pops that queue and, as the first act of popping, sweeps the delayed set for anything whose score has passed:

// Illuminate\Queue\RedisQueue
protected function migrate($queue)
{
    $this->migrateExpiredJobs($queue.':delayed', $queue);

    if (! is_null($this->retryAfter)) {
        $this->migrateExpiredJobs($queue.':reserved', $queue);
    }
}

migrateExpiredJobs() is a Lua script that does a zrangebyscore up to "now", zremrangebyranks what it found, and rpushes it onto the ready list with a notify token per job. Two consequences follow immediately, and both are load-bearing for the rest of this article:

• A delayed job is late by design. Its availability is discovered on the next pop of that queue, so a queue with no workers running never migrates anything — the jobs are all still there, they are simply invisible until a worker looks.
• It rejoins at the back. rpush puts it at the tail of the ready list, behind everything already waiting. A delayed job is not a priority job. If you need one to go to the head of the line, that is front-of-queue dispatching, a different operation entirely.

The same script, on the same schedule, is what recovers reserved jobs whose lease has expired. Delayed and abandoned jobs are recovered by one mechanism — which is why they behave identically once you understand either.

The worker loop#

A worker is a while (true) loop that never touches the database of your application except through the job it is running. Stripped to the transitions that matter:

// Illuminate\Queue\Worker::daemon(), condensed
while (true) {
    if (! $this->daemonShouldRun($options, $connectionName, $queue)) {
        // maintenance mode, or paused by SIGUSR2
        [$status, $reason] = $this->pauseWorker($options, $lastRestart, $startTime);
        // ...
        continue;
    }

    $job = $this->getNextJob($this->manager->connection($connectionName), $queue);

    if ($supportsAsyncSignals) {
        $this->registerTimeoutHandler($job, $options);   // pcntl_alarm() armed here
    }

    if ($job) {
        $this->jobsProcessed++;
        $this->runJob($job, $connectionName, $options);
        // ...
    } else {
        $this->events->dispatch(new WorkerIdle($connectionName, $queue, $options));
        $this->sleep($options->sleep);
    }

    if ($supportsAsyncSignals) {
        $this->resetTimeoutHandler();                    // pcntl_alarm(0)
    }

    [$status, $reason] = $this->stopIfNecessary($options, $lastRestart, $startTime, $job);

    if (! is_null($status)) {
        return $this->stop($status, $options, $reason);
    }
}

Four things happen every iteration, in this order: decide whether to run at all, take a job, run it, then decide whether to keep living. That last check — stopIfNecessary() — is the one that makes deploys work, and we come back to it. Note where it sits: after the job. A worker never abandons a job in order to exit.

When multiple queues are listed, getNextJob() walks them in order and returns the first job it finds, which is why balance => false gives you strict priority and can starve the queues at the end of the list (the trade-off in full, and Skyline's weighted alternative).

The pop, and why exactly one worker wins#

Here is the single most important piece of code in the queue. Popping a job is not "read then write" — it is one Lua script, and Redis executes Lua scripts atomically on its single command thread:

-- Illuminate\Queue\LuaScripts::pop()
-- Pop the first job off of the queue...
local job = redis.call('lpop', KEYS[1])
local reserved = false

if(job ~= false) then
    -- Increment the attempt count and place job on the reserved queue...
    reserved = cjson.decode(job)
    reserved['attempts'] = reserved['attempts'] + 1
    reserved = cjson.encode(reserved)
    redis.call('zadd', KEYS[2], ARGV[1], reserved)
    redis.call('lpop', KEYS[3])
end

return {job, reserved}

That is the entire concurrency guarantee, and it does not need a lock. Twenty workers can call this script on the same queue in the same millisecond; Redis serialises them, and lpop is destructive, so the first script to run takes the payload and the other nineteen get false. There is no window in which two workers hold the same ready-list entry, because there is no window at all — no other command can run between the lpop and the zadd.

Uniqueness at pop is a property of Redis, not of Laravel. It is the one part of the lifecycle that needs no configuration and cannot be misconfigured.

ARGV[1] is now + retry_after: the score that will decide, later, whether this job is considered abandoned. And on the way out, the script lpops one token off the notify list, keeping the "how many jobs are waiting" signal in step with reality.

The script returns both payloads, and the difference between them matters: job is the original as it sat on the ready list, reserved is the re-encoded copy with the incremented attempt count. The worker keeps both, because deleting the reservation later requires the exact byte-for-byte string that was written into the sorted set.

The attempt is spent at pop, not at failure#

Read that Lua script again: reserved['attempts'] = reserved['attempts'] + 1 runs before the worker has seen the job, let alone executed it. tries does not count failures. It counts pops.

Everything that causes a job to be popped a second time therefore costs a try, whether or not any of your code ran:

This is why the default tries => 1 and a throttling middleware are a bad pair: the first time the limiter says "not yet", the job is released, popped again, and immediately fails with MaxAttemptsExceededException — a retry budget meant for errors, spent entirely on successful backpressure. The fix is to stop counting attempts and start counting time, with retryUntil().

One implementation detail closes the loop. RedisJob::attempts() reads the count off the payload it was given and adds one:

// Illuminate\Queue\Jobs\RedisJob
public function attempts()
{
    return ($this->decoded['attempts'] ?? null) + 1;
}

So on a job's first ever run, attempts() is 1, not 0. And when a job is released, the payload written back is the reserved copy — the one the Lua script already incremented — so the count survives every round trip.

Reserved: a lease with a deadline, not a lock#

Once the job is in the reserved set, one fact governs everything that follows: the entry is not owned by anybody. It carries a score, and the score is a deadline. Redis does not know which worker wrote it, whether that worker is alive, or whether it is still working.

Every pop on the queue sweeps that set (the second migrateExpiredJobs() call shown earlier) and moves anything past its deadline back onto the ready list. The sweep cannot distinguish between:

• a job whose worker was killed by a deploy thirty seconds ago, and
• a job whose worker is happily still running it.

Both are just an expired score. That is the mechanism that makes a crashed worker lose no work — and the same mechanism that double-runs a job when its timeout is not comfortably below retry_after. The ordering rule and the two symptoms of breaking it have a whole article to themselves; for the lifecycle, the thing to hold onto is that the reservation is a promise about time, and the worker's timeout is the only thing that makes the promise keepable.

Inside process(): four checks before your code runs#

The worker now has a job object. Before handle() is reached, it goes through Worker::process():

// Illuminate\Queue\Worker::process()
$this->raiseBeforeJobEvent($connectionName, $job);          // JobProcessing

$this->markJobAsFailedIfAlreadyExceedsMaxAttempts(
    $connectionName, $job, (int) $options->maxTries
);

if ($job->isDeleted()) {
    return $this->raiseAfterJobEvent($connectionName, $job);
}

$job->fire();

$this->raiseAfterJobEvent($connectionName, $job);           // JobProcessed

That second call is the gate that fails a job which has already been popped too many times. It is deliberately checked before execution, because the case it exists for is a job that has been coming back around without ever getting to report a failure — the classic timed-out-and-migrated job:

// Illuminate\Queue\Worker::markJobAsFailedIfAlreadyExceedsMaxAttempts()
$maxTries = ! is_null($job->maxTries()) ? $job->maxTries() : $maxTries;

$retryUntil = $job->retryUntil();

if ($retryUntil && Carbon::now()->getTimestamp() <= $retryUntil) {
    return;
}

if (! $retryUntil && ($maxTries === 0 || $job->attempts() <= $maxTries)) {
    return;
}

$this->failJob($job, $e = $this->maxAttemptsExceededException($job));

Two things stand out. retryUntil wins outright: if it is set and has not passed, the attempt count is not consulted at all — a job can be popped a hundred times inside its window. And maxTries === 0 means unlimited, not zero.

Past the gate, $job->fire() resolves CallQueuedHandler@call, which unserialises your job object out of the payload and sends it through the middleware pipeline into handle(). If unserialising throws ModelNotFoundException — the row was deleted while the job waited — the job is either quietly deleted (with deleteWhenMissingModels) or failed outright, and in both cases your code never runs.

Middleware: the ways a job leaves without running#

Job middleware sits between fire() and handle(), and the two that ship with Laravel both work by not calling $next. They are the reason a job can be reserved, counted, and returned to the queue having done nothing at all.

RateLimited#

// Illuminate\Queue\Middleware\RateLimited::handleJob()
foreach ($limits as $limit) {
    if ($this->limiter->tooManyAttempts($limit->key, $limit->maxAttempts)) {
        return $this->shouldRelease
            ? $job->release($this->releaseAfter ?: $this->getTimeUntilNextRetry($limit->key))
            : false;
    }

    $this->limiter->hit($limit->key, $limit->decaySeconds);
}

return $next($job);

Over the limit, the job is released with a delay of "however long until the window reopens, plus three seconds" — and, as established, the attempt is already gone. Under the limit, the limiter is hit and the job proceeds. Note what this does not do: it bounds how often jobs may start, never how many run at once. Sixty workers can all be inside the same API call and still be within "sixty per minute".

WithoutOverlapping#

// Illuminate\Queue\Middleware\WithoutOverlapping::handle()
$lock = Container::getInstance()->make(Cache::class)->lock(
    $this->getLockKey($job), $this->expiresAfter
);

if ($lock->get()) {
    try {
        $next($job);
    } finally {
        $lock->release();
    }
} elseif (! is_null($this->releaseAfter)) {
    $job->release($this->releaseAfter);
}

This one is a genuine lock, taken for the duration of handle() and released in a finally. Three details decide whether it behaves the way you expected:

• The default release delay is 0. A blocked job goes straight back and is very likely popped again immediately by the same worker, burning attempts in a tight loop. Give it a real releaseAfter.
• The key includes the job class unless you call ->shared(). Two different job classes with the same $key do not exclude each other by default.
• Without expireAfter, the lock has no TTL. A worker killed inside handle() never reaches the finally, and the lock outlives it — permanently, until someone clears it by hand.

The third bullet is the one that turns a nine-second outage into a nine-hour one, and it interacts with everything in the restart section below. Rate limiting and concurrency in Laravel queues covers the alternatives — funnels, and single-slot supervisors that need no locks at all.

Completion: the only thing that deletes a job#

handle() returns. CallQueuedHandler::call() then, in order: releases the ShouldBeUnique lock, dispatches the next job in the chain, records the batch success — and only then:

// Illuminate\Queue\CallQueuedHandler::call()
if (! $job->isDeletedOrReleased()) {
    $job->delete();
}

Which, on Redis, is a single zrem of the exact reserved payload:

// Illuminate\Queue\RedisQueue
public function deleteReserved($queue, $job)
{
    $this->getConnection()->zrem($this->getQueueRedisKey($queue).':reserved', $job->getReservedJob());
}

This is the whole reliability model in one line. The job is removed from Redis after your code succeeded, never before. Anything that stops the worker between the pop and this zrem leaves the reservation in place, which means the work is recoverable — and it also means the work may be done twice. There is no third option, and no queue driver has one.

Failure: release, backoff, and the delayed set again#

handle() throws. Worker::handleJobException() runs, and its structure is worth reading as two separate questions asked in order: should this job be failed now?, then if not, put it back.

// Illuminate\Queue\Worker::handleJobException(), condensed
if (! $job->hasFailed()) {
    $this->markJobAsFailedIfWillExceedMaxAttempts($connectionName, $job, (int) $options->maxTries, $e);
    $this->markJobAsFailedIfWillExceedMaxExceptions($connectionName, $job, $e);
    $this->markJobAsFailedIfItShouldntBeRetried($connectionName, $job, $e);
}

// ...

if (! $job->isDeleted() && ! $job->isReleased() && ! $job->hasFailed()) {
    $backoff = $this->calculateBackoff($job, $options);

    $job->release($backoff);

    $this->events->dispatch(new JobReleasedAfterException($connectionName, $job, $backoff, $e));
}

throw $e;

If the job survives all three failure checks, it is released with a backoff. The backoff is resolved per attempt, from the job's backoff() or the worker's --backoff, indexed by the attempt just used:

// Illuminate\Queue\Worker::calculateBackoff()
return (int) ($backoff[$job->attempts() - 1] ?? last($backoff));

So public $backoff = [30, 120, 600]; means: 30 seconds after the first failure, 120 after the second, 600 after the third and after every one beyond that, because last() is the fallback once the array runs out. A single integer is just an array of one — a flat delay forever.

The release itself is, once more, one Lua script — and it does not go where most people assume:

-- Illuminate\Queue\LuaScripts::release()
-- Remove the job from the current queue...
redis.call('zrem', KEYS[2], ARGV[1])

-- Add the job onto the "delayed" queue...
redis.call('zadd', KEYS[1], ARGV[2], ARGV[1])

A released job goes to the delayed set, not the ready list — even when the delay is zero. It becomes runnable again only when a subsequent pop migrates it, and it rejoins at the tail of the ready list, behind everything queued in the meantime. A retry is not a resumption; it is a fresh trip through the entire lifecycle, with the attempt count carried along in the payload.

When the retries run out#

There are two distinct routes to a failed job, and the difference explains a lot of confusing dashboards.

The normal route is markJobAsFailedIfWillExceedMaxAttempts(), and the word will is doing real work:

// Illuminate\Queue\Worker::markJobAsFailedIfWillExceedMaxAttempts()
if ($job->retryUntil() && $job->retryUntil() <= Carbon::now()->getTimestamp()) {
    $this->failJob($job, $e);
}

if (! $job->retryUntil() && $maxTries > 0 && $job->attempts() >= $maxTries) {
    $this->failJob($job, $e);
}

With tries => 3, the exception thrown on the third attempt fails the job immediately rather than releasing it for a fourth pop that would only be rejected. You get exactly three executions, and the failure is recorded at the moment of the third error — with the real exception, not a synthetic one.

The other route is the pre-execution gate from earlier, markJobAsFailedIfAlreadyExceedsMaxAttempts(), which fires when a job comes back around without anyone having been able to record a failure — a killed worker, or a lease that expired. That one fails with MaxAttemptsExceededException and a stack trace that points at the queue, not at your bug, because there is no exception to report: the code that would have thrown was killed mid-sentence.

Either way, failJob() calls $job->fail($e), which:

1. marks the job failed and deletes the reservation, so it can never be popped again;
2. calls CallQueuedHandler::failed(), which releases the ShouldBeUnique lock — important, since a leaked unique lock would block every future dispatch — records the failure against the job's batch and chain, and then calls your job's own failed() method if it has one;
3. fires JobFailed.

That last step is the one that persists it. The row in failed_jobs is not written by fail() at all — it is written by a listener the worker command registers:

// Illuminate\Queue\Console\WorkCommand::listenForEvents()
$this->laravel['events']->listen(JobFailed::class, function ($event) {
    $this->writeOutput($event->job, 'failed', $event->exception);

    $this->logFailedJob($event);
});

horizon:work extends that command, so it inherits the listener and behaves identically. But it is worth knowing where the boundary is: fail a job outside a worker process — in a test, in a Tinker session, from custom code calling $job->fail() — and the event fires with nobody listening, so no row is written and the failure leaves no trace.

Retrying from the dashboard or with queue:retry does not resurrect that job. It pushes a new job, with a new id, carrying the original payload with its attempt count reset. Skyline's lifecycle log records both ids on one line for exactly this reason — so the retry can be traced back to the failure it came from.

The timeout: what protects a job from a wedged worker#

A job that hangs — an HTTP call with no timeout, a lock wait, an infinite loop — would otherwise occupy a worker process forever. The guard is a POSIX alarm, armed per iteration of the worker loop:

// Illuminate\Queue\Worker::registerTimeoutHandler()
pcntl_signal(SIGALRM, function () use ($job, $options) {
    if ($job) {
        $this->markJobAsFailedIfWillExceedMaxAttempts(
            $job->getConnectionName(), $job, (int) $options->maxTries, $e = $this->timeoutExceededException($job)
        );

        $this->markJobAsFailedIfWillExceedMaxExceptions($job->getConnectionName(), $job, $e);
        $this->markJobAsFailedIfItShouldFailOnTimeout($job->getConnectionName(), $job, $e);

        $this->events->dispatch(new JobTimedOut($job->getConnectionName(), $job));
    }

    $this->kill(static::$timedOutExitCode ?? static::EXIT_ERROR, $options, WorkerStopReason::TimedOut);
}, true);

pcntl_alarm(
    max($this->timeoutForJob($job, $options), 0)
);

The important word is kill. A timeout does not release the job, does not roll anything back, and does not let the loop continue — it terminates the worker process from inside the signal handler. Which leaves the reserved entry exactly where it was, and hands recovery to the lease:

• Attempts left: the reservation sits in the sorted set until its score passes, then the next pop on that queue migrates it back and it runs again from the top.
• Attempts exhausted: the handler's own markJobAsFailedIfWillExceedMaxAttempts() call fails it first, which deletes the reservation — so there is nothing left to migrate, and the job stops for good.
• The worker: is gone. Horizon's supervisor notices the dead process on its next monitor pass and starts a replacement, which is why a timeout looks like nothing at all in an uninstrumented setup.

Three ways this guard silently does not exist: --timeout=0 cancels the alarm rather than firing it instantly; a $timeout property or #[Timeout] on the job class overrides the supervisor's value entirely; and with no pcntl extension the whole block is skipped, since it sits behind a supportsAsyncSignals() check.

Restarts, deploys, and why jobs survive them#

The reason jobs survive a restart is almost anticlimactic: workers hold no state worth losing. Queued jobs are in Redis, delayed jobs are in Redis, reserved jobs are in Redis. A worker process contains at most one job's worth of in-flight work. Everything about restart handling is therefore about that one job.

The graceful path#

php artisan horizon:terminate sends SIGTERM to the master, which terminates its supervisors, which signal their workers. Inside the worker, the signal does not interrupt anything:

// Illuminate\Queue\Worker::listenForSignals()
foreach ([SIGQUIT, SIGTERM, SIGINT] as $signal) {
    pcntl_signal($signal, function (int $signal) use ($connectionName, $queue, $options) {
        $this->shouldQuit = true;

        $this->events->dispatch(new WorkerInterrupted($signal, $connectionName, $queue, $options));

        $this->notifyJobOfSignal($signal);
    });
}

It sets a flag. The current job runs to completion, is deleted normally, and only then does stopIfNecessary() — at the bottom of the loop — see shouldQuit and exit with status 0. Nothing is released, nothing is retried, no attempt is wasted. A job dispatched a millisecond before the deploy is simply still sitting in Redis when the new workers come up.

horizon:terminate also writes the illuminate:queue:restart cache key, which plain queue:work workers check at the end of every loop iteration and use to exit at the same safe point. Same mechanism, same guarantee.

The ungraceful path, and the number that controls it#

Graceful only lasts so long. A supervisor that has told a worker to stop keeps it in a terminating list, and prunes it on a clock:

// Laravel\Horizon\ProcessPool::stopTerminatingProcessesThatAreHanging()
foreach ($this->terminatingProcesses as $process) {
    $timeout = $this->options->timeout;

    if ($process['terminatedAt']->addSeconds((int) $timeout)->lte(CarbonImmutable::now())) {
        $process['process']->stop();
    }
}

A job still running timeout seconds after the deploy started is killed outright. From the queue's point of view this is identical to a crash: the reserved entry survives, its lease expires retry_after seconds after the original pop, and the next worker to sweep the queue migrates it back and runs it again — with one attempt already consumed and any side effects from the first, partial run still committed.

That path also skips the finally block in WithoutOverlapping, so a lock created without expireAfter() is now orphaned. Deploys are the single most common way that particular bug is triggered.

Pausing is not stopping#

SIGUSR2 sets paused, which makes daemonShouldRun() return false; the worker sleeps in a loop, popping nothing, until SIGCONT clears it. Jobs accumulate on the ready list and nothing is reserved, released or failed. It is the safest lever there is when a downstream dependency is unhealthy, and Skyline exposes it per queue from the dashboard rather than for the whole fleet at once.

The whole lifecycle, on one page#

                          dispatch()
                              |
              +---------------+---------------+
              |                               | ->delay(...)
              v                               v
      queues:emails                  queues:emails:delayed
      LIST, ready, FIFO              ZSET, score = available at
              ^                               |
              |    migrate() — runs at the top of every pop()
              +-------------------------------+
              |
              |  ONE Lua script:  lpop  +  attempts++  +  zadd
              v
      queues:emails:reserved
      ZSET, score = now + retry_after
              |
    +---------+-----------+----------------------+
    |                     |                      |
 handle() ok         exception              worker killed
    |                     |                 (timeout, deploy,
   zrem            attempts >= tries ?       OOM, crash)
  (gone)             /            \               |
                   no             yes        lease expires
                    |               |             |
             zadd delayed      failed_jobs   migrate() back
             (backoff)         + failed()    to ready, retry
                    |
              back to ready on the next migrate()

Every transition in that diagram is a Redis command, most of them inside a Lua script, and none of them involves a worker telling another worker anything. There is no coordination, no leader, no heartbeat — just a list, two sorted sets, and a clock.

What the queue actually guarantees#

Put the pieces together and the contract is precise, and narrower than most people assume:

• A ready job is popped by exactly one worker. Guaranteed by Lua atomicity, unconditionally.
• A job is never lost. Guaranteed by deleting only on success, unconditionally.
• A job runs at least once. Guaranteed by the reservation lease and migration.
• A job runs at most once. Not guaranteed, ever. The lease has no way to know whether the original worker is alive, so any job outliving its lease can be running in two places.

That last line is not a Laravel limitation; it is the standard trade-off of every at-least-once queue. What Laravel adds is a set of dials — timeout, retry_after, tries, backoff — that decide how often you land on the wrong side of it. The engineering answer is to make the duplicate harmless: give each job a natural idempotency key and check it before doing the irreversible thing.

Design for at-least-once and the whole lifecycle becomes forgiving. Assume exactly-once and every one of these transitions is a bug waiting for a bad day.

Watching it happen#

Almost none of these transitions is logged by default. Laravel emits events for some of them, no event at all for others — a delayed job becoming ready, a dispatch discarded by a unique lock, a job dropped by WithoutOverlapping — and the dashboard shows states, not the moves between them.

Skyline's job lifecycle logging writes one line per transition, each tagged with the job id, so a single job's whole history is one grep:

[09:14:02] queue.DEBUG: [job:8f2ac1] queued onto [emails].
[09:14:02] queue.DEBUG: [job:8f2ac1] reserved from [emails] and started processing.
[09:14:03] queue.WARNING: [job:8f2ac1] released back to [emails] (reason=exception, delay=30s)
[09:14:33] queue.INFO:  [job:8f2ac1] migrated to [emails] and is ready to run.
[09:14:33] queue.DEBUG: [job:8f2ac1] reserved from [emails] and started processing.
[09:14:35] queue.DEBUG: [job:8f2ac1] completed on [emails].

Read against the diagram above, that is: pushed to the ready list, popped into reserved, released into the delayed set with a 30-second backoff, migrated back to ready when the backoff elapsed, popped again, and finally zremed on success. Two attempts, five state changes, one job id. The level of each line is chosen so the channel's log level is the volume dial — debug to trace a specific job, info or warning as a steady state.

The core takeaway#

The lifecycle is small enough to hold in your head, and worth holding there: a job is pushed to a list, moved atomically into a reserved set with its attempt count incremented, and removed only when it succeeds. Everything else — delays, backoffs, rate limits, overlap locks, timeouts, deploys — is a variation on moving that same JSON string between a list and two sorted sets.

Most production queue bugs come from assuming a step that does not exist. There is no scheduler making delayed jobs punctual, no lock proving a worker is still alive, no bookkeeping that separates "failed" from "never ran", and no counter that only increments when something goes wrong. Once you know which transitions are real, the odd behaviours stop being mysterious: a phantom MaxAttemptsExceededException, a job that ran twice, a retry that arrived later than its backoff, a rate-limited job that exhausted its tries without executing. They are all the same six transitions, seen from different angles.

For the individual failure modes in depth: timeout vs retry_after for the double-run, uniqueness controls for the locks, rate limiting and concurrency for the release churn, and 12 best practices for the habits that make at-least-once delivery survivable. For seeing it in your own queues, see job lifecycle logging and Skyline vs Horizon.