# Prometheus metrics & Grafana

> Scrape the measurements behind the dashboard, and import the bundled Grafana dashboard.

Source: https://boring-observability.dev/skyline/docs/prometheus-metrics
Section: Observability — Laravel Skyline documentation
Updated: 2026-08-05

---

Laravel Skyline can export the measurements behind the dashboard's metric graphs in the **Prometheus text exposition format**, so the numbers you watch during an incident are also the numbers your alerts fire on. Throughput, failures, retries, runtime, wait time, queue length, worker processes and job states — labelled by queue and by job class.

A Grafana dashboard covering all of it ships with the package. There is nothing to build.

## Enabling the endpoint

The endpoint is **off by default**, and off means off: no route is registered at all until you turn it on, so an application that never opts in cannot leak its queue topology to anyone who guesses the URL.

```bash
HORIZON_PROMETHEUS_ENABLED=true
```

That is the whole setup. Metrics are then served at `/horizon/prometheus` — or `{HORIZON_PATH}/prometheus` if you moved the dashboard, because the scrape endpoint follows it. Point Prometheus at it:

```yaml
scrape_configs:
  - job_name: horizon
    metrics_path: /horizon/prometheus
    scrape_interval: 30s
    static_configs:
      - targets: ['127.0.0.1:8000']
```

> **Keep horizon:snapshot scheduled**
>
> Counters are folded forward by `horizon:snapshot`, so keep it on its usual every-minute schedule and prefer a scrape interval at or below that. See [Metrics & trends](https://boring-observability.dev/skyline/docs/metrics-and-trends) for what the snapshot command does and why the metrics page is empty without it.

## Access control

A Prometheus server has no session to authenticate with, so this endpoint cannot sit behind the [`viewHorizon`](https://boring-observability.dev/skyline/docs/security-and-access) gate the way the rest of the dashboard does. It is deliberately registered outside the dashboard's middleware group and guarded by an **IP allowlist** instead — which, out of the box, admits nothing but loopback:

```php
'prometheus' => [
    'enabled' => env('HORIZON_PROMETHEUS_ENABLED', false),
    // Defaults to "{horizon.path}/prometheus" when left empty.
    'path' => env('HORIZON_PROMETHEUS_PATH'),
    'domain' => env('HORIZON_PROMETHEUS_DOMAIN'),
    'allowed_ips' => array_filter(array_map('trim', explode(
        ',', (string) env('HORIZON_PROMETHEUS_ALLOWED_IPS', '127.0.0.1,::1')
    ))),
    'middleware' => [],
    'prefix' => env('HORIZON_PROMETHEUS_PREFIX', 'horizon'),
],
```

Entries may be exact addresses or CIDR ranges, in either IP family, and the single entry `*` allows any address:

```bash
HORIZON_PROMETHEUS_ALLOWED_IPS="127.0.0.1,::1,10.0.4.0/24"
```

> **The allowlist checks the address your application sees**
>
> Read it as *"which addresses this application sees"*, not *"which machines may scrape"*. The address checked is the one Laravel resolves for the request, so behind a load balancer or reverse proxy — including nginx in front of php-fpm, or Octane, **on the same host** — every request arrives from the proxy, and the loopback default then admits anyone who can reach that proxy.
>
> Configure the application's trusted proxies so the real client address is what gets checked, and treat the allowlist as one layer rather than the whole of your access control. Anything listed under `middleware` runs after it, which is where a token check or a rate limiter belongs.

Setting `enabled` back to `false` removes the route entirely — there is no disabled-but-present state to get wrong.

## Exported metrics

Throughput, failures and retries are **counters**; everything else is a **gauge**. Runtime and wait time are reported in seconds here, where the dashboard's own API reports milliseconds.

| Metric | Type | Labels | Meaning |
| --- | --- | --- | --- |
| `horizon_job_processed_total` | counter | job | Jobs of this class processed |
| `horizon_job_failed_total` | counter | job | Jobs of this class that failed |
| `horizon_job_retried_total` | counter | job | Jobs of this class released back onto a queue |
| `horizon_job_runtime_seconds` | gauge | job | Moving average runtime |
| `horizon_job_wait_seconds` | gauge | job | Moving average wait before processing |
| `horizon_queue_processed_total` | counter | queue | Jobs processed on this queue |
| `horizon_queue_failed_total` | counter | queue | Jobs that failed on this queue |
| `horizon_queue_retried_total` | counter | queue | Jobs released back onto this queue |
| `horizon_queue_runtime_seconds` | gauge | queue | Moving average runtime |
| `horizon_queue_wait_seconds` | gauge | queue | Moving average wait before processing |
| `horizon_queue_length` | gauge | queue, connection | Jobs waiting on the queue |
| `horizon_queue_oldest_pending_seconds` | gauge | queue, connection | Age of the oldest waiting job |
| `horizon_queue_processes` | gauge | queue, connection | Worker processes assigned |
| `horizon_queue_paused` | gauge | queue, connection | 1 while the queue is paused |
| `horizon_queue_time_to_clear_seconds` | gauge | queue, connection | Estimated time to drain the queue |
| `horizon_jobs` | gauge | status | Jobs retained per status — pending, reserved, delayed, completed, failed, silenced |
| `horizon_recent_jobs` | gauge | — | Jobs inside the recent retention window |
| `horizon_recent_failed_jobs` | gauge | — | Failed jobs inside the recent retention window |
| `horizon_supervisor_processes` | gauge | supervisor, connection, queue | Processes per supervisor pool |
| `horizon_processes` | gauge | — | Total worker processes |
| `horizon_master_supervisor_paused` | gauge | master | 1 while a master supervisor is paused |
| `horizon_up` | gauge | — | 1 while a master supervisor is reporting in |
| `horizon_info` | gauge | name | Always 1; carries the Horizon name |
| `horizon_scrape_duration_seconds` | gauge | — | Time spent collecting the scrape |

Change the `horizon_` prefix with `HORIZON_PROMETHEUS_PREFIX` if it collides with something else you scrape — and remember to change it in the Grafana queries too.

## Why throughput is a counter

Horizon's metric graphs are *windowed*: `horizon:snapshot` resets the counters every time it runs. That is the opposite of what Prometheus wants, where a counter must only ever go up so that `rate()` can tell the difference between a quiet minute and a restart.

So throughput, failures and retries are exported as true counters. Each snapshot folds the window it closes into a running total, and a scrape reads that total plus the window still open. `rate()` and `increase()` behave correctly across snapshots, and nothing is double counted — the total is read before the open window on purpose, so a snapshot landing mid-scrape makes the figure briefly *under*-count, which self-corrects, rather than over-count, which Prometheus would read as a counter reset.

Runtime and wait time stay moving averages, so they remain gauges.

## There is no jobs-per-minute gauge

Deliberately. Horizon derives that figure from the time since the last snapshot, and *reading* it **writes** — so a scrape would move the window the dashboard's own figure is measured against. Scraping this endpoint never writes to Redis. Compute the same thing from the counters instead:

```promql
sum(rate(horizon_queue_processed_total[5m])) * 60
```

## Alerts worth having

The two failure modes that page you at 3am are a queue nobody is draining and a worker fleet that has quietly died:

```yaml
groups:
  - name: horizon
    rules:
      - alert: HorizonDown
        expr: max(horizon_up) == 0
        for: 2m

      - alert: QueueBacklogAgeing
        expr: horizon_queue_oldest_pending_seconds > 300
        for: 5m

      - alert: QueueFailureRate
        expr: |
          sum by (queue) (rate(horizon_queue_failed_total[5m]))
            / sum by (queue) (rate(horizon_queue_processed_total[5m])) > 0.05
        for: 10m

      - alert: QueueLeftPaused
        expr: horizon_queue_paused == 1
        for: 30m
```

`horizon_queue_oldest_pending_seconds` is the one to alert on rather than queue length: a queue of ten thousand fast jobs is fine, and a queue of one job nobody has picked up in an hour is not. `QueueLeftPaused` exists because pausing a queue is a deliberate act that somebody eventually forgets to undo.

## The Grafana dashboard

Publish the bundled dashboard into your project:

```bash
php artisan vendor:publish --tag=horizon-grafana
```

That writes `grafana/horizon-dashboard.json`, plus a short README with a sample scrape config. In Grafana choose **Dashboards → New → Import**, upload the file, and pick your Prometheus data source.

Or take it straight from here — this is the same file, byte for byte:

[**Download horizon-dashboard.json ↓** 30 panels across Overview, Queues, Job classes and Supervisors. Import it into Grafana and pick your Prometheus data source.](https://boring-observability.dev/skyline/grafana/horizon-dashboard.json)

The **Queue** and **Job class** variables at the top filter every panel, so one dashboard serves a fleet of queues rather than needing a copy per queue. Every rate panel uses `$__rate_interval`, which means the graphs stay correct when you change the time range or the scrape interval.

| Row | Panels |
| --- | --- |
| Overview | Horizon up, jobs/min, failures/min, worker processes, pending jobs, recent failures, jobs by state, throughput vs. failures |
| Queues | Throughput, failures, releases, average runtime, average wait, queue length, oldest pending job, estimated time to clear, worker processes, paused queues |
| Job classes | Throughput, failures, releases, average runtime, average wait, and a table of the most failing job classes |
| Supervisors | Processes by supervisor, paused master supervisors |

> **If you changed the metric prefix**
>
> The dashboard's queries are written against the default `horizon_` prefix. If you set `HORIZON_PROMETHEUS_PREFIX` to something else, find and replace `horizon_` in the JSON before importing.


## Common questions

### How do I export Laravel Horizon metrics to Prometheus?

Set HORIZON_PROMETHEUS_ENABLED=true. Skyline then serves the Prometheus text exposition format at /horizon/prometheus — or {HORIZON_PATH}/prometheus if you moved the dashboard — covering throughput, failures, retries, runtime, wait time, queue length, worker processes and job states, labelled by queue and by job class. The endpoint is off by default and no route exists until you enable it.

### How is the Prometheus endpoint secured if the viewHorizon gate does not apply?

A Prometheus server has no session to authenticate with, so the endpoint sits outside the dashboard middleware group and is guarded by an IP allowlist instead — horizon.prometheus.allowed_ips, which defaults to loopback only. Entries may be exact addresses or CIDR ranges, and * allows any. Behind a proxy or load balancer every request appears to come from the proxy, so configure trusted proxies and treat the allowlist as one layer rather than the whole of your access control.

### Why is there no jobs-per-minute metric?

Because reading Horizon's own figure writes: it is derived from the time since the last snapshot, and asking for it moves the window the dashboard measures against. Scraping must never mutate state, so the exporter omits it. Derive the same number in PromQL with sum(rate(horizon_queue_processed_total[5m])) * 60.

### Do Prometheus counters survive horizon:snapshot resetting the metrics window?

Yes. Horizon's metric graphs are windowed and horizon:snapshot resets them, which is the opposite of what Prometheus wants, so throughput, failures and retries are exported as true counters: each snapshot folds the window it closes into a running total. rate() and increase() behave correctly across snapshots and nothing is double counted. Runtime and wait time stay moving averages and are exported as gauges, in seconds.