Laravel Horizon in Prometheus: Why Exported Throughput Reads Low

Horizon's dashboard is good at one question: what is happening on the queues right now. It is less good at the questions that come after an incident. What did the backlog look like at 03:10? Did failures on webhooks start before or after the deploy? Should somebody have been paged? Those answers live in Prometheus, next to your database and host metrics, and Horizon has no exporter of its own.

The obvious fix is to export the figures the dashboard already shows, and the graphs that produces look right and aren't. This post explains why: where Horizon keeps its numbers, why they don't survive rate(), what the existing exporter packages read, and a small counter-based exporter you can drop into any Laravel app today. The PromQL and alert rules at the end work with it as written.

Where Horizon keeps its numbers#

Every successful job updates two Redis hashes, one for its queue (queue:emails) and one for its class (job:App\Jobs\SendInvoice). Each hash holds two fields: throughput, a count, and runtime, a running average in milliseconds. The update is a Lua script fired from Horizon's UpdateJobMetrics listener on JobDeleted, and the first thing that listener does is return if the job has failed. Failed jobs appear in the dashboard's failed list, never in its metrics.

horizon:snapshot is what turns those hashes into the graphs. For every measured queue and class it reads both fields and deletes the hash in one transaction, then appends the values to a sorted set trimmed to the last 24 entries. The hashes start again from zero. At the five-minute schedule the Horizon docs recommend, the dashboard's graphs cover about two hours.

Scheduling it more often doesn't change that. The command takes a lock for horizon.metrics.snapshot_lock minus 30 seconds, which is 270 seconds at the default of 300, and never releases it. An everyMinute() schedule still produces roughly one snapshot every five minutes. The other four runs find the lock held and do nothing.

What rate() does with a number that resets every five minutes#

Prometheus counters are allowed to reset. A process restarts, its counter drops to zero, and rate() sees the drop, assumes the series restarted from zero, and carries on. So exporting the throughput field as a counter looks like it should work, because the snapshot is just a reset.

The problem is what happens between the last scrape before a snapshot and the snapshot itself. Jobs processed in that gap were counted into the hash, and the hash was deleted before any scrape saw them. rate() can't recover increments it never observed. With a 30-second scrape interval the gap is anywhere from zero to thirty seconds, fifteen on average, out of every five-minute window.

The loss is systematic, so the graph is smooth and consistently low. Nobody notices a throughput panel that reads 8% under. The runtime field has a milder version of the same issue: it is the average of jobs since the last snapshot, so each window's first minute is an average of a handful of jobs and jumps around before settling.

What the Horizon exporter packages read#

The two packages most people find are spatie/laravel-prometheus, a general Laravel exporter with a set of Horizon collectors, and the older lkaemmerling/laravel-horizon-prometheus-exporter. Their Horizon metrics are near-identical: master supervisor count and status, workload and processes per queue, a jobs-per-minute figure, a failed-jobs figure and recent jobs. All of them are gauges read from Horizon's repositories at scrape time.

The gauges that read live state are fine. Queue length, process counts and Horizon status are current-value questions, and that is what a gauge is for. The two that describe a rate are where it goes wrong.

• Jobs per minute calls MetricsRepository::jobsProcessedPerMinute(), which divides the throughput since the last snapshot by the minutes since the last snapshot, with a floor of one minute. For the first minute after every snapshot the divisor is pinned at 1 while the count starts from zero, so the figure drops to near zero and climbs back. Graphed, that is a notch every five minutes. And if horizon:snapshot isn't scheduled, reading the figure writes last_snapshot_at on the first scrape and the hashes are never reset again, so the gauge becomes a lifetime average that gets flatter every day.
• Failed jobs (named horizon_failed_jobs_per_hour in the Spatie collector) calls countRecentlyFailed(), which counts the recent_failed_jobs set. Horizon keeps that set for trim.recent_failed minutes, and the default is 10080, so the figure counts the past week.

Two smaller things are worth checking in either package. Workload rows are labelled with the supervisor's queue string, so queues balanced together arrive as one series named high,default. And both packages' IP allowlists default to empty, which both treat as "allow everyone". Set it before the route is deployed.

A counter exporter in two files#

The fix is to stop reading Horizon's windowed figures and count outcomes yourself, into a store that never resets. The PHP-specific wrinkle is where the counter lives. A Go service keeps its counters in memory and serves them from the same process. A Laravel app has dozens of short-lived worker processes and a separate pool of web processes, none of which can see another's memory. So workers write to Redis, and the scrape endpoint is an ordinary web request that reads it.

First, the counting. Register two listeners in a service provider's boot():

use App\Support\QueueMetrics;
use Illuminate\Queue\Events\JobAttempted;
use Illuminate\Queue\Events\JobFailed;
use Illuminate\Support\Facades\Event;

Event::listen(JobAttempted::class, function (JobAttempted $event) {
    $outcome = match (true) {
        $event->job->isReleased() => 'retried',
        $event->successful() => 'processed',
        default => null, // failures are counted from JobFailed, below
    };

    if ($outcome !== null) {
        QueueMetrics::record($outcome, $event->job);
    }
});

Event::listen(JobFailed::class, function (JobFailed $event) {
    QueueMetrics::record('failed', $event->job);
});

namespace App\Support;

use Illuminate\Contracts\Queue\Job;
use Illuminate\Support\Facades\Redis;

class QueueMetrics
{
    public const KEY = 'queue-metrics:counters';

    public static function record(string $outcome, Job $job): void
    {
        Redis::pipeline(function ($pipe) use ($outcome, $job) {
            $pipe->hincrby(self::KEY, "queue|{$outcome}|{$job->getQueue()}", 1);
            $pipe->hincrby(self::KEY, "job|{$outcome}|{$job->resolveName()}", 1);
        });
    }
}

The choice of events is the part that is easy to get wrong. Queue::after(), the listener most examples reach for, listens to JobProcessed, and the worker raises that whenever fire() returns without throwing. A job released by RateLimited or WithoutOverlapping returns without throwing, so it counts as processed. So does a job the worker failed for exceeding its attempts before it ran, which also raises JobFailed and is counted twice. JobAttempted fires once per attempt, in a finally block, and isReleased() separates the two cases.

Failures come from JobFailed rather than JobAttempted because of timeouts. A job that hits its timeout is failed from the alarm handler, which then kills the worker process. JobFailed fires before that; the finally block never runs. The same kill means a timed-out job with attempts left isn't released at all. It stays reserved until retry_after expires and is picked up again, and the timed-out attempt lands in none of the three counters. The timeout vs retry_after post covers that path in detail.

Then the endpoint. It renders the counters, adds two gauges per queue, and emits the text exposition format by hand, which is short enough that a client library isn't worth the dependency:

use App\Support\QueueMetrics;
use Illuminate\Support\Facades\Queue;
use Illuminate\Support\Facades\Redis;
use Laravel\Horizon\Contracts\MasterSupervisorRepository;
use Laravel\Horizon\Contracts\SupervisorRepository;
use Symfony\Component\HttpFoundation\IpUtils;

Route::get('/queue-metrics', function () {
    abort_unless(IpUtils::checkIp(request()->ip(), ['127.0.0.1', '::1', '10.0.0.0/8']), 403);

    $series = [];
    $labels = ['queue' => 'queue', 'job' => 'job_class'];

    // Counters: written by every worker, never reset.
    foreach (Redis::hgetall(QueueMetrics::KEY) as $field => $count) {
        [$scope, $outcome, $name] = explode('|', $field, 3);
        $series["horizon_{$scope}_{$outcome}_total"][] = [[$labels[$scope] => $name], (int) $count];
    }

    // Gauges: every queue a running supervisor serves. Keyed by connection and
    // queue, because each server runs its own supervisor for the same queues.
    foreach (app(SupervisorRepository::class)->all() as $supervisor) {
        foreach (array_keys((array) $supervisor->processes) as $pool) {
            [$connection, $names] = explode(':', $pool, 2);

            foreach (explode(',', $names) as $name) {
                $queue = Queue::connection($connection);
                $oldest = $queue->creationTimeOfOldestPendingJob($name);
                $id = "{$connection}:{$name}";
                $key = ['queue' => $name, 'connection' => $connection];

                $series['horizon_queue_length'][$id] = [$key, $queue->pendingSize($name)];
                $series['horizon_queue_oldest_pending_seconds'][$id] = [$key, $oldest ? now()->timestamp - $oldest : 0];
            }
        }
    }

    $masters = app(MasterSupervisorRepository::class)->all();
    $series['horizon_up'][] = [[], (int) ! empty($masters)];
    $series['horizon_master_supervisors'][] = [[], count($masters)];

    $escape = fn ($value) => str_replace(['\\', '"', "\n"], ['\\\\', '\\"', '\n'], (string) $value);
    $body = '';

    foreach ($series as $metric => $samples) {
        $body .= "# TYPE {$metric} ".(str_ends_with($metric, '_total') ? 'counter' : 'gauge')."\n";

        foreach ($samples as [$set, $value]) {
            $pairs = collect($set)->map(fn ($v, $k) => $k.'="'.$escape($v).'"')->implode(',');
            $body .= $metric.($pairs === '' ? '' : '{'.$pairs.'}').' '.$value."\n";
        }
    }

    return response($body, 200, ['Content-Type' => 'text/plain; version=0.0.4']);
});

Three details in there matter. The label is job_class rather than job because Prometheus reserves job for the scrape config's job_name and renames any exposed one to exported_job. The escaping is required, not defensive: every job class contains backslashes, and an unescaped App\Jobs\SendInvoice makes the whole scrape fail to parse. And the IP check is only as good as request()->ip(). Behind a load balancer, or nginx in front of php-fpm, every request comes from the proxy until trusted proxies are configured.

What this exporter doesn't have is anything about how long jobs take or how many workers are on them. Runtime is the one to add next if you need it, and building it yourself means you can do better than Horizon's average. Record the start time in a JobProcessing listener (same process, so a static array keyed by job id is enough) and increment histogram buckets in the JobAttempted listener. That gives you a p95, which an average can't.

Scrape one target, not every node#

Every piece of state the exporter reads is in Redis, so any web node answers with the same fleet-wide numbers. If Prometheus discovers all your web nodes and scrapes each one, every series arrives once per node, distinguished only by the instance label. sum(rate(...)) then reports three times your real throughput on three nodes. Point Prometheus at a single address: one node, or the load balancer.

scrape_configs:
  - job_name: horizon
    metrics_path: /queue-metrics
    scrape_interval: 30s
    static_configs:
      - targets: ['app-internal.example.com']

The PromQL behind the useful panels#

Jobs per minute, by queue, computed from the counter rather than read from Horizon:

sum by (queue) (rate(horizon_queue_processed_total[5m])) * 60

Failure ratio per queue. Note that the denominator counts successful jobs only, so this is failures relative to successes; add the failed rate to the denominator if you want a share of all outcomes:

sum by (queue) (rate(horizon_queue_failed_total[5m]))
  / sum by (queue) (rate(horizon_queue_processed_total[5m]))

Queues with a backlog that nobody is draining. unless rather than and, so a queue that has never processed a job still matches:

max by (queue) (horizon_queue_length) > 0
  unless sum by (queue) (rate(horizon_queue_processed_total[10m])) > 0

Release churn, which usually means a rate limiter or an overlap lock is doing more work than the jobs are:

sum by (job_class) (rate(horizon_job_retried_total[15m]))
  / sum by (job_class) (rate(horizon_job_processed_total[15m]))

Alert rules#

groups:
  - name: horizon
    rules:
      - alert: HorizonDown
        expr: max(horizon_up) == 0
        for: 2m

      - alert: HorizonDuplicateMasters
        expr: max(horizon_master_supervisors) > 2   # your server count
        for: 10m

      - alert: QueueBacklogAgeing
        expr: max by (queue) (horizon_queue_oldest_pending_seconds) > 900
        for: 5m

      - alert: QueueFailureRate
        expr: |
          sum by (queue) (rate(horizon_queue_failed_total[5m]))
            / sum by (queue) (rate(horizon_queue_processed_total[5m])) > 0.05
        for: 10m

QueueBacklogAgeing is the one we would keep if we could only have one. A queue of ten thousand fast jobs is healthy, and a queue of one job that nobody has picked up in fifteen minutes is not. Queue length can't tell those apart and oldest-job age can. It has one blind spot: the gauges come from the supervisor list, so when Horizon is fully down there are no queue series left to alert on. HorizonDown covers that case.

HorizonDuplicateMasters catches the failure from our supervisor upgrade postmortem, where a second complete Horizon tree registered itself and kept running. Set the threshold to the number of servers that should be running Horizon. More masters than servers means one of them is running two.

Putting it in Grafana#

Every rate panel should use $__rate_interval rather than a fixed [5m], so it stays correct when someone zooms to a seven-day range or you change the scrape interval. Put queue and job_class in dashboard variables and filter every panel with =~"$queue", and one dashboard covers every queue instead of needing a copy each.

You don't have to build it. The metric names in the exporter above match the ones Skyline exports, so Skyline's Grafana dashboard imports straight onto it. The throughput, failure and release panels for queues and job classes populate, along with queue length, oldest pending job and Horizon status. The runtime, process and paused-state panels stay empty, because stock Horizon has nothing correct to feed them.

What Skyline exports instead#

We built Skyline's Prometheus endpoint to feed the panels that stay empty on stock Horizon. It is off until HORIZON_PROMETHEUS_ENABLED=true, and while off the route isn't registered at all. When on, it serves /horizon/prometheus to loopback only until you widen the allowlist.

The main difference is the counters. Skyline's horizon:snapshot folds each window it closes into a running total before resetting it, and a scrape reads that total plus the window still open. Nothing is lost between scrape and snapshot, and failures and releases are counted per queue and per job class alongside throughput. The oldest-pending age is measured from Horizon's own job record, which is updated when a job is released or leaves the delayed set, so a job that has just become due reads as new. Queues balanced as one pool are split into one series each, with a group label. The dashboard linked above is the same file the package publishes, and all 27 of its panels populate.

Skyline doesn't export runtime histograms either; runtime and wait are moving averages, exported as gauges. If percentiles matter more to you than the rest of this, the DIY route with buckets is the better one.

Most of the events in the listener above come from the life of a Laravel queued job, which walks through when each one fires and why a release puts a job at the back of the queue. For the heavier release churn that the retried counter tends to expose, rate-limited APIs and Laravel queues is the next one to read.