Laravel Horizon config checker

What it checks#

Each check is a behaviour one of our posts or docs pages works through in detail, and each finding links to that explanation. A setting with no published behaviour behind it, such as memory sizing or nice, is not checked.

Supervisor timeout at or above the connection's retry_after#

Problem. retry_after is a lease, not a delay. Once it expires, the next worker to pop the queue takes the job even while the first is still running it. Equal values fail too, because the lease starts at pop() and the timeout alarm a few steps later. With tries => 1 the second copy is failed with MaxAttemptsExceededException; with more tries, both copies run.

Source: Laravel queue timeout vs retry_after

A Redis or database connection with no retry_after key#

Warning. The published config/queue.php says 90. A connection written by hand without the key gets 60 with no warning, which equals the default worker timeout.

Source: The two numbers and their defaults

retry_after set to null on a Redis connection#

Problem. Null skips reserved-job migration altogether. Duplicates stop, and a job whose worker dies is never recovered, with no error to show for it.

Source: Four ways a correct config becomes an incorrect one

retry_after on an SQS connection#

Note. The SQS connector never reads the key. The lease is the queue's Default Visibility Timeout, set on the AWS side and 30 seconds by default, which is half of Laravel's default worker timeout.

Source: What retry_after does

A supervisor timeout of 0#

Warning. pcntl_alarm(0) cancels the alarm rather than firing it, so zero means no timeout. Nothing bounds the job's runtime while the lease keeps expiring on schedule.

Source: What timeout does

A supervisor with no tries key#

Note. The published config sets tries => 1. Drop the key and horizon:supervisor runs with --tries=0, which means unlimited attempts for every job class that does not set its own.

Source: With tries => 1: a failure on work that succeeded

Several queues on one unbalanced pool#

Warning. Without balancing (balance => false, or no balance key at all) the queue list is a strict left-to-right priority. A worker looks at the second queue only when the first is empty, so a flood on a higher queue starves every queue below it until it drains.

Source: Option B: balance => false and starvation by list order

An auto-balanced floor that eats the process ceiling#

Warning. The auto balancer never takes a queue below minProcesses. One worker per queue across thirty queues is a resident floor of thirty, and against maxProcesses 40 it leaves ten processes for the queue that is actually busy.

Source: Option A: balance => 'auto' and the idle-worker floor

queueWeights on a balanced supervisor, or a weight of zero#

Warning. Weights order the queues one shared pool polls, so they mean nothing once each queue has its own pool. Skyline refuses to provision the combination; stock Horizon does not know the key. A zero or negative weight falls back to 1.

Source: Weighted queues: only with balance => false

after_commit off on a connection Horizon works#

Warning. Every connection in the published config/queue.php ships with after_commit => false, so a job dispatched inside DB::transaction() can reach a worker before the row it needs is committed. With no transaction open the job is pushed at once, as before.

Source: Laravel dispatch after commit

fast_termination on#

Note. With the flag on, Horizon exits within about two seconds of being signalled while its workers are still finishing jobs. When supervisord itself is restarted under systemd, as a package upgrade did in our postmortem, those workers keep draining the queues with nothing above them. The fix is a systemd drop-in and a scheduled horizon:purge. Turning the flag off is only safe once the unit is fixed.

Source: Postmortem: why fast_termination stays on

A waits threshold of 0#

Note. Zero turns LongWaitDetected off for that queue, which is sometimes what you want for a housekeeping queue and worth knowing either way.

Source: What Horizon sends on its own

A waits key that matches no supervisor#

Warning. Horizon keys a wait by connection and queue the way a supervisor works them. A pool working high,default is the single entry redis:high,default, so a threshold written as redis:high never applies and the pool gets the 60-second default.

Source: What Horizon sends on its own

trim.recent_failed below a week#

Note. The dashboard's failed-jobs figure counts failures newer than this many minutes. Lowering it, for example to make a Prometheus gauge mean "the past hour", changes the dashboard number too.

Source: The rule that does not port

Questions#

Does my config leave the browser?

No. The page reads both files with a small parser that runs in your browser, and nothing you paste is sent anywhere. It does not evaluate PHP either: anything other than arrays, literals, casts and env() calls is treated as unknown, and a rule that needs an unknown value says it skipped rather than guessing.

What does it do with env() calls?

It uses the default written in the call, because it cannot see your .env. Every env() it met is listed with the value it assumed, and you can type the real value next to any of them to re-run the checks with it.

What can it not see?

Anything set on a job class. A $timeout property or #[Timeout] attribute takes precedence over the supervisor's timeout, so a job can break the timeout and retry_after ordering on its own. It also cannot see an SQS queue's visibility timeout, which lives on the AWS side, or your systemd or supervisord unit.

Which supervisors does it check?

The ones the chosen environment starts, resolved the way Horizon's provisioning plan does it: the first environments key that matches (keys can be patterns), merged recursively over every supervisor in defaults. A supervisor that appears only in defaults still runs, and one with maxProcesses 0 does not.