What to Monitor in a Laravel Queue, from Dispatch to Done

Queue monitoring usually grows one outage at a time. A backlog goes unnoticed for an afternoon, so someone adds an alert on queue length. A job fails quietly for a week, so failed jobs start posting to Slack. Each alert answers the last incident and says nothing about the next one.

This post goes the other way. It follows a job from dispatch to done and lists what can go wrong at each step and which signal sees it. Most of the space goes to the part that looks redundant. An alert on the age of the oldest waiting job will eventually fire for a stalled queue and for an overloaded one, so why keep separate alerts for those? The answer is in how long "eventually" takes, and in what each alert tells you to do when it arrives.

A job's path from dispatch to done#

A queued job passes through five places where it can go wrong, and each one needs a different signal:

The sections below start with a queue falling behind, which is the outage people notice first, and work outward from there.

Is work getting through in time?#

Measure the oldest job, not the length#

Queue length is the number most teams alert on first, and Laravel ships a command for it. php artisan queue:monitor redis:default --max=1000 fires a QueueBusy event once the queue holds a thousand jobs. Length is the wrong measure, though. Fifty jobs whose oldest has waited twenty minutes is a worse queue than five thousand that will be gone in thirty seconds, and a threshold on the count can't tell them apart.

What users feel is how long a job waits before it starts. The cheapest measure of that is the age of the job at the head of the queue. It is zero on an empty queue, small on a fast one however deep it gets, and it climbs for as long as nothing takes the job. On Redis, Laravel reads it with creationTimeOfOldestPendingJob(), and the Prometheus exporter post publishes it as a gauge. That method reads the timestamp written at dispatch, so a delayed job reads as old the moment it becomes due. Where you can, measure from when the job became ready to run.

Horizon's long wait alert is a forecast#

Horizon's LongWaitDetected looks like this alert, but it measures something else. Its "wait" is the number of ready jobs multiplied by the queue's average runtime, divided by the processes serving it. Nothing in that formula depends on time. Take a queue that ran two-second jobs on four processes and then stopped finishing anything: 40 waiting jobs read as a 20-second wait. Thirty minutes later they still read as 20 seconds, and the oldest job is thirty minutes old.

The next snapshot makes it worse. Horizon keeps the average runtime in a hash that horizon:snapshot deletes every time it runs, and only a finished job writes it back. Push 200 jobs onto that stalled queue and the forecast reads 100 seconds, over the 60-second default, so the alert fires. After the next snapshot it reads 0 and stays there. Two hours later the oldest job is two hours old, 200 jobs are waiting, and Horizon reports no wait at all.

The one alert Horizon ships can stay silent through a stall. If it is all you have, add an alert on the oldest job's real age before anything else in this post.

The oldest job catches a stall and an overload, eventually#

With a real oldest-job alert in place, it is tempting to stop. A stalled queue's head job keeps getting older, and so does an overloaded queue's, so both cross the threshold sooner or later. How much later depends on how fast the wait grows.

On a stalled queue nothing is taken, so the head job's wait grows by one second every second. On a queue receiving more than it can process, the wait grows more slowly, because the job at the head arrived a while ago, when the backlog was shorter. With jobs arriving at rate a and workers able to finish them at rate c, the head job's wait after t seconds of overload is t × (1 − c/a). At 25% over capacity, that is 0.2 seconds of wait per second. At 10% over, it is 0.09.

The table shows the time from the start of the problem until the oldest job crosses its threshold, before any hold window the alert adds. On a queue with a one-minute threshold, the oldest-job alert is the first to fire in every case, and that is fine. The other two signals earn their place on the queues further down the table.

The threshold is set by the worst normal day#

Nobody sets a two-hour threshold to be slow. They set it because the queue regularly waits that long while working exactly as intended:

• A scheduled command drops twenty thousand jobs on a bulk queue at 2am, and the last of them starts an hour and a half later.
• A newsletter fills the mail queue for forty minutes every Tuesday.
• Workers given high,default without balancing take from high first, so default waits out every burst on high.
• A customer starts an import from the UI and queues four thousand jobs at once.

Each of these pushes the threshold up, and every minute added is another minute before anyone notices the queue has died. On those queues, one number would have to be loose enough for the busiest normal day and tight enough to catch a failure, and it can't be both. The way out is to measure things a busy, healthy queue never does.

Nothing is finishing#

A stalled queue has jobs waiting and workers assigned to it, and none of its jobs finish. A worker stuck on an HTTP call with no timeout looks like this, and so does a worker that lost its Redis connection without noticing. The dashboard shows the configured number of processes the whole time.

The question to ask is whether any job finished in the last few minutes. Count failures as finished, since a failing job still means a worker is taking work. A batch draining at 2am finishes jobs constantly, so this check stays quiet however long the batch takes. It catches a stall in the same few minutes on a queue whose oldest-job threshold is two hours.

Two details keep it from firing on healthy queues. Ask the question of the pool of workers rather than of each queue: during a burst on high, nothing on default finishes, and that is the oldest-job alert's to report. And scale the silence to the job. A queue of ten-minute jobs finishes nothing for most of ten minutes, so silence should only count once it has lasted well past one job's runtime.

The common shortcut is to compare the queue's length with itself a few minutes ago. It misses in both directions: a busy queue can land on the same number, and a stalled queue that keeps receiving dispatches changes length every minute. The PromQL version of "did anything finish" is in Laravel Horizon alerting with Prometheus.

A backlog that keeps growing#

An overloaded queue still finishes jobs, so the stall check says nothing about it. What gives it away is direction. A queue holding steady at 900 jobs is fine. A queue at 400 and climbing by eighty a minute is an incident that hasn't happened yet, and the cheapest point to fix it.

Watch the backlog across a window of ten minutes or so. It is falling behind if it ended the window higher than it started and never dropped back below that starting point in the second half. Its estimated time to clear should be rising too, and over a threshold. A batch queued in one go only shrinks, so this alert can stay at fifteen minutes on a queue whose oldest-job threshold has to allow hours. On a bulk queue 25% over capacity, the time to clear crosses fifteen minutes about an hour in. A two-hour oldest-job threshold on the same queue takes ten hours.

Time to clear runs ahead of the oldest job's age by a factor of a/c, because it describes everything queued now rather than the job that arrived first. The alert can also give the numbers that matter for the fix, jobs arriving per minute against jobs finishing per minute, which is what tells you how many workers to add.

Each alert calls for a different first move#

A stall is the one to wake someone for. With only an oldest-job alert, you choose between being paged for every slow batch and leaving a dead queue until morning. With all three, the severity can follow the cause. Page on the stall, and send the other two somewhere they will be read during the day, unless the queue is one where a minute's delay is itself the outage.

When nothing is consuming at all#

Every alert so far is computed by something, and reads the queues that something knows about. Both assumptions can fail.

Horizon isn't running#

LongWaitDetected is evaluated by a running supervisor, and the gauges a Prometheus exporter publishes come from the supervisor list. When Horizon is down, the first has nothing to run it and the second has nothing to report. The outage that costs the most is the one with no alert. Check it from outside instead: php artisan horizon:status exits 0 when Horizon is running, 1 when it is paused and 2 when it is inactive, so a cron entry or an uptime monitor can alert on any non-zero exit. With Prometheus, add an absent() rule so that a series that disappears also fires.

A heartbeat job#

A check from outside can go one step further and test the whole path. Schedule a small job onto a queue every minute, and have it ping a dead man's switch, such as Healthchecks.io or Cronitor, from inside handle(). The switch alerts when the pings stop. That happens when the scheduler stops, when Redis is unreachable, when Horizon is down, or when the queue the job is on has stalled:

// app/Jobs/QueueHeartbeat.php
class QueueHeartbeat implements ShouldQueue
{
    use Queueable;

    public function handle(): void
    {
        // One check URL per queue, e.g. services.heartbeats.default
        Http::timeout(5)->get(config("services.heartbeats.{$this->queue}"));
    }
}

// routes/console.php
Schedule::job(new QueueHeartbeat, 'default')->everyMinute();
Schedule::job(new QueueHeartbeat, 'webhooks')->everyMinute();

Ping from the job, not with the scheduler's pingOnSuccess(). That runs once the scheduler has pushed the job, which says nothing about whether a worker ran it. The grace period has the same problem as the oldest-job threshold: a heartbeat on the bulk queue waits behind the 2am batch. Put heartbeats on the queues you keep fast.

Paused and forgotten#

php artisan queue:pause redis:default in Laravel, and horizon:pause or horizon:pause-supervisor in Horizon, stop work on purpose. The risk is the resume that never happens after a maintenance window. Leave paused queues out of the alerts above, since a paused queue grows by design, and give pauses their own alert: paused for longer than fifteen minutes, sent somewhere it will be read during the day. horizon:status exiting with 1 is the outside view of the same thing.

A queue no worker reads#

A typo in onQueue(), or a queue that a deploy dropped from config/horizon.php, leaves jobs waiting that will never run. No metric built from the workers can see them. Horizon's wait calculation only covers the queues a supervisor works, so 500 jobs on a misspelled defualt queue, the oldest an hour old, don't appear in it at all.

The fix is to compare the queues jobs are pushed to with the queues your workers read. Record each queue name as jobs are queued:

// app/Providers/AppServiceProvider.php, in boot()
Event::listen(function (JobQueued $event) {
    $queue = $event->queue ?? config("queue.connections.{$event->connectionName}.queue");

    Redis::sadd('monitoring:pushed-queues', $event->connectionName.':'.$queue);
});

JobQueued carries a null queue when the job goes to the connection's default queue, so the listener reads the default from the connection config. It costs one SADD per dispatch. Then compare the two lists on a schedule:

// routes/console.php
Schedule::call(function () {
    $served = collect(config('horizon.environments.'.app()->environment(), []))
        ->map(fn ($supervisor, $name) => array_merge(config("horizon.defaults.{$name}", []), $supervisor))
        ->flatMap(fn ($supervisor) => collect(explode(',', implode(',', (array) $supervisor['queue'])))
            ->map(fn ($queue) => ($supervisor['connection'] ?? 'redis').':'.trim($queue)));

    foreach (Redis::smembers('monitoring:pushed-queues') as $key) {
        [$connection, $queue] = explode(':', $key, 2);

        if (! $served->contains($key) && Queue::connection($connection)->size($queue) > 0) {
            Log::critical("Jobs are waiting on {$key}, which no Horizon supervisor processes.");
        }
    }
})->everyFiveMinutes();

Send the message wherever your other alerts go. Without Horizon, the list of queues your workers read lives in your Supervisor or systemd config, so keep a copy of it in a config file and check against that.

A queue that should be busy and isn't#

Everything above fires when there is too much work, and nothing fires when there is too little. If the scheduler stops on one server, or a webhook route starts returning 404 after a deploy, the queues look healthier than ever.

This alert needs knowledge only you have: which jobs should run, and when. For a job that runs on a schedule, use the heartbeat pattern again, pinging a dead man's switch from the job's own handle() with a grace period that matches its schedule. For jobs driven by traffic, alert on a floor, such as no SendInvoice completed in an hour on a weekday afternoon. That needs a count of completed jobs per class, which the Prometheus exporter above provides, and a time-of-day condition so the rule stays quiet overnight.

Failures, and the jobs that never count as one#

Take fifty jobs that throw on their first line, with three tries and a 60-second backoff. After one pass of a worker, all fifty have left the ready list and sit in the delayed set waiting for their retry. The oldest-job age reads nothing, because the queue is empty, and the failed_jobs table is empty too, because every job has tries left. Every signal in the first half of this post reads that queue as healthy. Whether a job succeeded is a separate question, with its own alerts.

Failure rate, per job class#

Queue::failing(), or a package such as spatie/laravel-failed-job-monitor, sends a message for every failed job. That is the right place to start at a few hundred jobs a day. Above that it breaks in a predictable way. One bad payload or one vendor outage fails four hundred jobs in ten minutes, and the channel gets muted. The next real failure lands in a channel nobody reads.

Alert on a rate instead: failures in a rolling window, sent once and resolved when the rate drops. Count per job class as well as per queue. One class failing every run on a queue shared with twenty others is a small share of that queue, and a per-queue ratio can hide it indefinitely. If you alert on a percentage, require a minimum number of jobs in the window, or a queue that ran one job and failed it reads as 100%.

Retries delay the failure#

A job only counts as failed once its tries run out. With $tries = 5 and $backoff = [10, 60, 300, 900], that happens about 21 minutes after the first exception, so a failure alert hears about an outage 21 minutes late. Laravel fires JobReleasedAfterException each time a job is put back after throwing. Count that per class and you see the outage on the first attempt. A class that is released more often than it completes is leaning on its tries to get through at all. Some releases are expected, such as an API asking you to wait, and the post on 429s in queued jobs covers releasing those without spending tries.

Timeouts#

A job that runs past its timeout is killed along with its worker. It is only marked failed if that was its last try or it sets $failOnTimeout. Otherwise it stays reserved, comes back once retry_after passes, and runs again. Laravel fires JobTimedOut just before the worker exits, so count that per class. Five timeouts in ten minutes for one class is usually an API that stopped answering, and the failure count won't show it until the tries run out.

Jobs dropped and counted as done#

Some job middleware removes a job without failing it. RateLimited and WithoutOverlapping with dontRelease() delete the job when the limit is hit or the lock is held, and ThrottlesExceptions with deleteWhen() deletes a job that threw a matching exception. None of these writes a failed job or calls failed(), and the job counts as processed, so a failure-rate alert can't see it. If you use them, count the drops yourself and alert when the count jumps. Rate-limited APIs and Laravel queues goes through which middleware drops a job and which releases it.

Jobs that run twice, or never#

The checks above count a job that ran twice as two successes, and a job that never ran as nothing at all.

Twice: a timeout at or above retry_after#

A Redis or database queue gives a reserved job to another worker once the connection's retry_after passes. If the first worker died, that is the recovery. If it is still running the job, the job now runs twice and both runs count as processed. The usual cause is a worker timeout at or above retry_after. That is a fact about configuration, so a test catches it before an alert would:

public function test_worker_timeouts_stay_below_retry_after(): void
{
    foreach (config('horizon.environments', []) as $environment => $supervisors) {
        foreach ($supervisors as $name => $supervisor) {
            $supervisor = array_merge(config("horizon.defaults.{$name}", []), $supervisor);
            $connection = $supervisor['connection'] ?? 'redis';

            $this->assertLessThan(
                config("queue.connections.{$connection}.retry_after"),
                $supervisor['timeout'] ?? 60,
                "{$environment}.{$name} runs long jobs twice",
            );
        }
    }
}

A job's own $timeout property overrides the worker's, so add an assertion for any class that sets one. Laravel job timeout vs retry_after covers how the two collide and how to find the double runs in your logs.

Never: a unique lock nobody released#

A ShouldBeUnique job takes a cache lock at dispatch and releases it once the job finishes. When the job leaves by a path that skips that release, such as a worker killed with SIGKILL or a queue:clear, and the lock has no expiry, every later dispatch of that job is skipped. Nothing is queued, so there is no backlog. Nothing runs, so there is no failure.

Laravel 13.25 added a UniqueJobSkipped event for each skipped dispatch. Skips on their own are normal, since preventing duplicates is the lock's job. Skips for a class that completed nothing over the same window point to a stranded lock. What breaks with ShouldBeUnique under real traffic covers how locks get stranded, and draining a Redis queue backlog has a script that finds the ones already stuck.

Worker health#

Worker problems show up in the alerts above as a stall or a falling-behind queue. Their own alerts explain the cause before you log in. A worker started with --memory exits with code 12 when it passes the limit, and the process manager starts a new one. That is housekeeping until it happens many times an hour, when one job class is usually holding on to memory. Horizon fires events for the larger versions of the problem, such as SupervisorOutOfMemory and UnableToLaunchProcess, with no notification attached.

The one to page on is a crash loop: workers dying faster than they can finish work, with no exception, because the kernel's out-of-memory killer or a segfault ended them. Frequent memory restarts can wait for the morning. The orphan workers postmortem is an example of the opposite fault: workers that kept running with nothing supervising them, and raised no alert at all.

What pages someone, and what waits for the morning#

A useful rule: page when work isn't getting done, and send a message when it is slow or at risk.

Every alert in that table should also say when it clears. A channel that has gone quiet could mean the queue recovered or that someone muted it, and only a recovery message tells the two apart.

The same set in Skyline#

Skyline is our fork of Horizon, and its alert checks cover most of this post without a metrics stack: queue_wait for the oldest job, queue_stalled, queue_not_draining, horizon_down from a scheduled horizon:check, queue_paused, unserved_queue, failure_rate per queue or per job class, job_timeout, limiter_drop for the middleware imported from Skyline, stranded_lock, reservation_expired, misconfiguration for a timeout at or above retry_after, and the worker checks. Two things here have no built-in check: the too-little-work floor, which you write as a custom check, and retry churn.

For an application with a fast webhooks queue, a default queue, a mail queue that fills on Tuesdays and a nightly bulk queue, the thresholds from this post look like this:

// config/horizon.php
'waits' => [
    'redis:webhooks' => 30,
    'redis:default' => 120,
    'redis:mail' => 3600,    // the Tuesday newsletter
    'redis:bulk' => 0,       // the 2am batch: left to the two checks below
],

'alerts' => [
    // queue_stalled is critical by default, queue_wait and queue_not_draining are warnings
    'routes' => [
        'critical' => ['slack', 'sms', 'webhook'],
        'warning' => ['slack'],
    ],

    'checks' => [
        'queue_not_draining' => [
            'threshold' => 900,          // fifteen minutes to clear
            'queues' => [
                'redis:mail' => 3600,    // the newsletter grows the queue for forty minutes
            ],
        ],
    ],
],

queue_wait reads the waits thresholds Horizon already uses, and a zero turns it off for bulk, whose normal wait is the length of the batch. queue_stalled and queue_not_draining still cover that queue, because a batch queued in one go and draining steadily trips neither of them. queue_wait takes one severity for every queue, warning by default. If a slow webhooks queue should wake someone, set it to critical and keep the other queues' thresholds at a wait that would be worth a page too. Setting HORIZON_ALERTS=true and one channel turns it all on.