[ field notes ]
What we've learned running Laravel queues in production on Horizon and Redis, checked against the framework source.
· 11 min read
A Guzzle middleware on Laravel's HTTP client can log every outgoing call, but only if it records from on_stats: timeouts and DNS failures never produce a response. What the client events miss, SDKs with their own Guzzle client, and when to use metrics instead of logs.
Read more· 10 min read
A job dispatched inside a transaction can reach a worker before its rows commit. Where to turn on afterCommit, how it behaves with nested transactions, deadlock retries and ShouldBeUnique, and why Queue::fake() won't catch a rollback.
Read more· 13 min read
Horizon deletes its throughput hash on every snapshot, so a Prometheus exporter that reads it loses the jobs processed between the last scrape and the reset. Why the existing exporters undercount, and a two-file counter exporter with the PromQL and alert rules to go with it.
Read more· 13 min read
Clearing a Laravel queue deletes the payloads but not the ShouldBeUnique locks they hold, and with the default uniqueFor those locks never expire. How to see what is in a backlog, what queue:clear deletes, and how to drain it without leaving a job that can't be dispatched again.
Read more· 19 min read
Horizon ships its dashboard as a compiled Vue bundle with no plugin API, so extending it usually means forking. knobik/laravel-horizon-job-output adds panels, a page and a sidebar link without forking: it patches the rendered layout and mounts into Vue's static nodes.
Read more· 24 min read
For most of its life a queued job is a JSON string moving between a Redis list and two sorted sets. The lifecycle in order: what dispatch writes, why exactly one worker gets it, why the attempt is spent at pop, and how a job survives a deploy.
Read more· 13 min read
Two numbers in two config files decide whether a job runs once or twice, and nothing in Laravel checks their order. What retry_after leases and why equal values already fail. Sometimes the symptom is a failure recorded for work that succeeded.
Read more· 20 min read
An unattended package upgrade restarted supervisord, and a full set of Horizon workers kept draining the queues with no supervisor above them. Nothing errored or alerted. The two defaults behind it, and the systemd drop-in that fixes it without turning off fast_termination.
Read more· 16 min read
"60 requests a minute" and "no parallel requests" are different limits, handled in different places in Laravel. RateLimited, WithoutOverlapping, Redis funnels and single-worker supervisors compared, and how to let urgent jobs go first without breaking the limiter.
Read more· 11 min read
Many job bugs come from assuming a job runs once, on time, on the code that dispatched it. In development that holds; under production load it doesn't. Twelve practices for Laravel queued jobs, ending with what happens to queued payloads when you deploy a change to the job class.
Read more· 10 min read
Every Horizon supervisor makes you choose: turn balancing on and pay for a floor of idle workers, or turn it off and let a busy queue starve the ones listed after it. How both costs grow with the number of queues, and how Skyline's weighted queues avoid them.
Read more· 15 min read
ShouldBeUnique and WithoutOverlapping solve different problems and fail in different ways. What each one locks and when, and the failures that show up only at volume: dispatches dropped without an error, chains and retries that skip the lock, and locks that never expire.
Read more