# Metrics & trends

> Aggregate charts, workload and failure trends, and how wait time is measured.

Source: https://boring-observability.dev/skyline/docs/metrics-and-trends
Section: Observability — Skyline for Laravel documentation
Updated: 2026-07-13

---

Skyline reworks Horizon's metrics page around **aggregate** charts — one line across all queues, rather than a sparkline per queue and per job class — and adds three rolling time-series: workload, wait time and failures. Underneath sits a sortable per-queue and per-job breakdown, for finding the culprit behind a spike.

The reason for the rework: Horizon renders a chart per job class and a chart per queue, so at ten queues and forty job classes you get fifty sparklines and no answer.

## The metrics dashboard

Each chart plots a single line across all queues, so adding a queue does not add a chart. Underneath, a sortable table breaks throughput, runtime, retries, failures and wait time down per queue and per job — sort by wait time to find the starving queue, by runtime to find the job that got slow.

## Trend charts

The dashboard shows three time-series over a rolling window: **workload** (how much work is waiting), **wait** (how long it is waiting), and **failures**. Workload and wait are gauges, sampled periodically; failures are counted as they happen.

```php
'trends' => [
    'interval' => 15,   // minutes per bucket, and the sampling cadence
    'retention' => 24,  // hours of history to keep and display
],
```

The defaults give 96 buckets across 24 hours. A shorter interval yields a finer chart at the cost of more Redis keys and a heavier render; a longer retention keeps more history. Buckets older than the retention window are pruned automatically.

Where a supervisor serves a comma-joined queue group, samples are recorded against the individual sub-queues, so the workload line for a queue lines up with its failure line.

> **Trends are sampled by the master process**
>
> Workload and wait samples are taken from the `php artisan horizon` master supervisor loop — there is no separate command to schedule. If the master is not running, no samples land for that period; the charts show a gap and nothing else is affected.

## Scheduling horizon:snapshot

The per-job and per-queue metrics tables are still driven by Horizon's snapshot command, exactly as upstream. If you have not scheduled it, the metrics page stays empty:

```php
use Illuminate\Support\Facades\Schedule;

Schedule::command('horizon:snapshot')->everyFiveMinutes();
```

`metrics.trim_snapshots` controls how many snapshots are kept, which — combined with the schedule interval — determines how far back the metrics tables reach. Snapshots take a short lock so two concurrent runs cannot double up; `metrics.snapshot_lock` (default `300` seconds) sizes it to the five-minute cadence above.

To clear metrics and trends and start over:

```bash
php artisan horizon:clear-metrics
```

## Expected vs. actual wait time

Skyline reports two wait numbers, because there are two questions and Horizon answers only one of them.

|  | Actual wait time | Expected wait time |
| --- | --- | --- |
| **Answers** | How bad is the backlog *right now*? | How long will a job I dispatch now wait? |
| **Measured from** | The age of the oldest job still waiting in the queue. | A moving average of how long recently-processed jobs waited. |
| **Behaviour** | Point-in-time. Jumps immediately when a queue stalls. | Smoothed. Lags a sudden change, ignores outliers. |

They agree on a healthy queue and diverge exactly when you need to know. A queue whose workers have died has an actual wait time climbing by a second per second, while its expected wait time — computed from jobs processed before the workers died — sits happily near zero. The reverse also happens: a queue that just drained a large backlog shows a near-zero actual wait and an elevated expected wait for a while afterwards.

Read **actual** when you are diagnosing an incident, and **expected** when you are making a promise about latency.

> **Front-of-queue dispatch skews the actual figure**
>
> Actual wait time assumes the job at the head of the list is the oldest one. A job prepended with [`onFront()`](https://boring-observability.dev/skyline/docs/front-of-queue-dispatching) sits at the head while being brand new, so a queue that receives front-pushed jobs will under-report its actual wait time and its long-wait notifications.

## Tuning the smoothing factor

Expected wait time and the runtime estimates behind the workload panel's ETA are both exponential moving averages, sharing one smoothing factor:

```php
'metrics' => [
    'ema_alpha' => 0.05,
],
```

It must lie between 0 and 1. Lower values weight history more heavily: the estimate is stable and a single pathological job barely moves it, but it is slow to notice that a job class has genuinely got slower. Higher values do the opposite. The default of `0.05` favours stability, which is usually right — an estimate that reacts to every outlier is not an estimate.

Raise it if your job durations shift legitimately and often, and you would rather the dashboard track the change than average it away.


## Common questions

### What is the difference between expected and actual queue wait time?

Actual wait time is measured from the age of the oldest job still waiting in the queue, and answers "how bad is the backlog right now?". Expected wait time is a moving average of how long recently-processed jobs waited, and answers "how long will a job I dispatch now wait?". They agree on a healthy queue and diverge exactly when you need to know: a queue whose workers have died has an actual wait climbing by a second per second, while its expected wait — computed from jobs processed before the workers died — sits happily near zero.

### Why is my Skyline metrics page empty?

The per-job and per-queue metrics tables are driven by Horizon's snapshot command, exactly as upstream. If horizon:snapshot is not scheduled, the page stays empty. Add Schedule::command('horizon:snapshot')->everyFiveMinutes() to routes/console.php.

### Do the trend charts need a scheduled command?

No. Workload and wait samples are taken from the php artisan horizon master supervisor loop, and failures are counted as they happen. If the master process is not running, no samples land for that period and the charts show a gap — nothing else is affected.