Observability

Metrics & trends

Aggregate charts, workload and failure trends, and how wait time is measured.

Horizon's metrics page renders a chart per job class and a chart per queue. At ten queues and forty job classes that is fifty sparklines and no answer. Skyline reworks the page around aggregate charts you can actually read, plus a sortable breakdown for when you need to find the culprit behind a spike.

The metrics dashboard

Each chart plots a single line across all queues, so adding a queue does not add a chart. Underneath, a sortable table breaks throughput, runtime, retries, failures and wait time down per queue and per job — sort by wait time to find the starving queue, by runtime to find the job that got slow.

Trend charts

The dashboard shows three time-series over a rolling window: workload (how much work is waiting), wait (how long it is waiting), and failures. Workload and wait are gauges, sampled periodically; failures are counted as they happen.

'trends' => [
    'interval' => 15,   // minutes per bucket, and the sampling cadence
    'retention' => 24,  // hours of history to keep and display
],

The defaults give 96 buckets across 24 hours. A shorter interval yields a finer chart at the cost of more Redis keys and a heavier render; a longer retention keeps more history. Buckets older than the retention window are pruned automatically.

Where a supervisor serves a comma-joined queue group, samples are recorded against the individual sub-queues, so the workload line for a queue lines up with its failure line.

Trends are sampled by the master process

Workload and wait samples are taken from the php artisan horizon master supervisor loop — there is no separate command to schedule. If the master is not running, no samples land for that period; the charts show a gap and nothing else is affected.

Scheduling horizon:snapshot

The per-job and per-queue metrics tables are still driven by Horizon's snapshot command, exactly as upstream. If you have not scheduled it, the metrics page stays empty:

use Illuminate\Support\Facades\Schedule;

Schedule::command('horizon:snapshot')->everyFiveMinutes();

metrics.trim_snapshots controls how many snapshots are kept, which — combined with the schedule interval — determines how far back the metrics tables reach. Snapshots take a short lock so two concurrent runs cannot double up; metrics.snapshot_lock (default 300 seconds) sizes it to the five-minute cadence above.

To clear metrics and trends and start over:

php artisan horizon:clear-metrics

Expected vs. actual wait time

Skyline reports two wait numbers, because there are two questions and Horizon answers only one of them.

	Actual wait time	Expected wait time
Answers	How bad is the backlog right now?	How long will a job I dispatch now wait?
Measured from	The age of the oldest job still waiting in the queue.	A moving average of how long recently-processed jobs waited.
Behaviour	Point-in-time. Jumps immediately when a queue stalls.	Smoothed. Lags a sudden change, ignores outliers.

They agree on a healthy queue and diverge exactly when you need to know. A queue whose workers have died has an actual wait time climbing by a second per second, while its expected wait time — computed from jobs processed before the workers died — sits happily near zero. The reverse also happens: a queue that just drained a large backlog shows a near-zero actual wait and an elevated expected wait for a while afterwards.

Read actual when you are diagnosing an incident, and expected when you are making a promise about latency.

Front-of-queue dispatch skews the actual figure

Actual wait time assumes the job at the head of the list is the oldest one. A job prepended with onFront() sits at the head while being brand new, so a queue that receives front-pushed jobs will under-report its actual wait time and its long-wait notifications.

Tuning the smoothing factor

Expected wait time and the runtime estimates behind the workload panel's ETA are both exponential moving averages, sharing one smoothing factor:

'metrics' => [
    'ema_alpha' => 0.05,
],

It must lie between 0 and 1. Lower values weight history more heavily: the estimate is stable and a single pathological job barely moves it, but it is slow to notice that a job class has genuinely got slower. Higher values do the opposite. The default of 0.05 favours stability, which is usually right — an estimate that reacts to every outlier is not an estimate.

Raise it if your job durations shift legitimately and often, and you would rather the dashboard track the change than average it away.