[ field notes ]
What we've learned running Laravel queues in production on Horizon and Redis, checked against the framework source.
· 15 min read
Does interrupted() work with horizon:terminate out of the box, or do you check for the stop signal in your own loop? Both. What SIGALRM lets a job do before the kill, why a curl call holds signals back, and how Horizon's autoscaler interrupts jobs with no deploy running.
Read more· 16 min read
An alert on the oldest waiting job eventually fires for a stalled queue and an overloaded one, so why keep separate alerts? How long "eventually" takes, why Horizon's long wait alert goes quiet in a stall, and the failures, drops and dead queues no backlog alert sees.
Read more· 9 min read
performance_schema hands back one running total per counter, so an hour of full scans yesterday is invisible in it. Sailfish reads it every five minutes from your Laravel app and charts it: each index's reads, each table's scans, the server and finished transactions.
Read more· 13 min read
A vendor API answers 429 and says how long to wait, and Laravel ignores it. Reading Retry-After in every form vendors send, releasing a job without spending its tries, and a job middleware that makes one 429 pause every job for that API.
Read more· 22 min read
Laravel runs middleware around incoming requests, queued jobs and outgoing HTTP calls. How each kind works, every middleware the framework ships, the packages people install, and why a job middleware that skips a job still lets the rest of its chain run.
Read more· 10 min read
Dispatch 300 five-job chains on 10 workers and the first one finishes after 40 minutes, not two. Where the framework queues each next link, two ways to keep chains in order without the wait, and how a ShouldBeUnique link silently ends other chains.
Read more· 14 min read
Horizon ships one notification, and it cannot fire when Horizon is down. Which alert rules Spatie's Prometheus gauges support, which ones need a counter exporter, how to carry them into Grafana, and the failures that never show up in a metric at all.
Read more· 9 min read
A full-page cache stores every rendered page whole, and ours reached 25MB in Redis. How predis and phpredis differ with values that size, sizes and timings for lzf, lz4 and zstd, the counter gotcha on a compressed connection, and a serializer that compresses on predis.
Read more· 9 min read
Pages behind a one-hour response cache kept rendering from scratch. The causes: click ids like srsltid and gbraid splitting the cache per visitor, a list cache whose keys never matched, a clear() that flushed the whole store, and an ad visitor's noindex page served to Googlebot.
Read more· 9 min read
A retried job is released, not dispatched again, so each lock behaves differently while it waits out its backoff. What ShouldBeUnique, ShouldBeUniqueUntilProcessing and WithoutOverlapping do in that gap, and how overlap releases and retryUntil() bend the backoff itself.
Read more· 11 min read
A Guzzle middleware on Laravel's HTTP client can log every outgoing call, but only if it records from on_stats: timeouts and DNS failures never produce a response. What the client events miss, SDKs with their own Guzzle client, and when to use metrics instead of logs.
Read more· 12 min read
A job dispatched inside a transaction can reach a worker before its rows commit. Where to turn on afterCommit, how it behaves with nested transactions, deadlock retries and ShouldBeUnique, and why Queue::fake() won't catch a rollback.
Read more· 13 min read
Horizon deletes its throughput hash on every snapshot, so a Prometheus exporter that reads it loses the jobs processed between the last scrape and the reset. Why the existing exporters undercount, and a two-file counter exporter with the PromQL and alert rules to go with it.
Read more· 13 min read
Clearing a Laravel queue deletes the payloads but not the ShouldBeUnique locks they hold, and with the default uniqueFor those locks never expire. How to see what is in a backlog, what queue:clear deletes, and how to drain it without leaving a job that can't be dispatched again.
Read more· 19 min read
Horizon ships its dashboard as a compiled Vue bundle with no plugin API, so extending it usually means forking. knobik/laravel-horizon-job-output adds panels, a page and a sidebar link without forking: it patches the rendered layout and mounts into Vue's static nodes.
Read more· 24 min read
For most of its life a queued job is a JSON string moving between a Redis list and two sorted sets. The lifecycle in order: what dispatch writes, why exactly one worker gets it, why the attempt is spent at pop, and how a job survives a deploy.
Read more· 13 min read
Two numbers in two config files decide whether a job runs once or twice, and nothing in Laravel checks their order. What retry_after leases and why equal values already fail. Sometimes the symptom is a failure recorded for work that succeeded.
Read more· 20 min read
An unattended package upgrade restarted supervisord, and a full set of Horizon workers kept draining the queues with no supervisor above them. Nothing errored or alerted. The two defaults behind it, and the systemd drop-in that fixes it without turning off fast_termination.
Read more· 16 min read
"60 requests a minute" and "no parallel requests" are different limits, handled in different places in Laravel. RateLimited, WithoutOverlapping, Redis funnels and single-worker supervisors compared, and how to let urgent jobs go first without breaking the limiter.
Read more· 11 min read
Many job bugs come from assuming a job runs once, on time, on the code that dispatched it. In development that holds; under production load it doesn't. Twelve practices for Laravel queued jobs, ending with what happens to queued payloads when you deploy a change to the job class.
Read more· 10 min read
Every Horizon supervisor makes you choose: turn balancing on and pay for a floor of idle workers, or turn it off and let a busy queue starve the ones listed after it. How both costs grow with the number of queues, and how Skyline's weighted queues avoid them.
Read more· 15 min read
ShouldBeUnique and WithoutOverlapping solve different problems and fail in different ways. What each one locks and when, and the failures that show up only at volume: dispatches dropped without an error, chains and retries that skip the lock, and locks that never expire.
Read more