Insights: job cost, worker resources and restarts

What each job class costs in memory and CPU, what the fleet is using, and why workers are being replaced.

Horizon tells you how many jobs ran and how fast. Insights answer the next question — what they cost. Skyline records the memory and CPU each job class needs, how much memory and CPU the whole worker fleet is using, and how often workers are dying and being replaced.

All of it is off until you turn it on. One switch covers all three:

HORIZON_INSIGHTS=true

Workers read the flag when they boot, so run php artisan horizon:terminate for a change to take effect.

What each job class costs#

A worker runs one job at a time, and Horizon already brackets every run with events inside that worker process. So memory and CPU are measured as a delta around a single job — no process scraping, no /proc, no shelling out. The cost is roughly two microseconds per job and no extra Redis round trips: the measurements ride along in the script call that already records the runtime.

Three figures are kept, per job class and per queue, and they appear as columns on the metrics screen:

Peak memory is the high-water mark of the worker's memory while a job ran, not the job's own allocation, so it carries the process baseline with it. That is deliberate: it is measured on the same scale the supervisor's memory option is enforced on, so the two numbers can be compared directly. The peak is reset per job, so a heavy class does not inflate whatever runs after it.

The three deliberately do not share an aggregation. Averaging peaks would smooth away the one heavy run that actually causes a restart, so peak keeps the maximum. Growth is summed rather than averaged so it stays comparable between a rare fat job and a frequent small one — a class leaking 1KB across 10,000 runs outranks one leaking 100KB across ten, which is the right way round. CPU averages, which is what keeps cpu / runtime meaningful: a class whose CPU sits far below its runtime is blocked on a database or an API, and giving it more workers will help.

What the numbers will not tell you#

• Failed jobs are not measured. Runtime already works this way — a job that throws never reaches the point where its metrics are recorded.
• The first job each worker runs is excluded, by default. It pays for framework boot and for autoloading everything it touches, which belongs to whichever class happened to come off the queue first rather than to the class itself. Set skip_first_job to false to include it.
• Peak memory needs PHP 8.2, where memory_reset_peak_usage() arrived. On older versions the field is omitted entirely rather than reported inaccurately, and the chart reads "Not Enough Data" instead of a confident zero. Growth and CPU are unaffected.
• Attribution is comparative, not absolute. Autoloading and framework singletons are charged to whichever job touched them first. Read the numbers as a ranking between classes, not as a budget for one.

Worker memory and CPU#

The dashboard charts the memory and CPU used by all workers together, across every supervisor on every machine, over the same 24 hours as the workload chart.

Job cost measures each job from inside the worker. This is measured from the outside instead: every ten seconds each supervisor reads the resident memory and CPU time of its own worker processes, from /proc on Linux or ps elsewhere. It therefore includes what the per-job figures leave out — idle workers, framework boot, and workers still finishing a job after being scaled down.

Each point is an average over its bucket. Memory is the average total held. CPU is the average number of cores in use, so 2.5 means two and a half cores' worth of work across the fleet. A bucket nothing was sampled in is drawn as a gap, not as a fleet using nothing.

Worker restarts#

Every worker that dies and is replaced is recorded against its supervisor, its queue group, and the reason it stopped. The reason comes from the worker itself — it is the only thing that knows why it is exiting — and a worker that never got the chance to say, because of a PHP fatal or a kill from outside, is recorded as crashed.

These are unplanned restarts. A worker retired by scaling down, by horizon:terminate or by a supervisor restart is stopped by its pool and never passes through this path, so a deploy does not show up here.

A rising memory line next to a job class with heavy heap growth is the pair worth looking for: the restart chart says workers are being replaced, and the growth column says which class is doing it.

Configuration#

'insights' => [
    'enabled' => env('HORIZON_INSIGHTS', false),
    'skip_first_job' => true,
],

skip_first_job applies to job cost only; see what the numbers will not tell you above.

Restart and worker resource history share the trends window (interval and retention), so those charts line up with the workload and wait charts beside them.

Turning enabled off hides the dashboard's restart, worker memory and worker CPU charts and its memory, CPU and heap growth columns, stops the matching Prometheus series, stops supervisors sampling their workers, and returns 404 from the restart and worker resource endpoints. Anything already recorded stays in Redis until php artisan horizon:clear-metrics, which clears it either way.