Alerts and notifications

What opens an alert, how it stays open and clears, and how alerts reach mail, Slack or a webhook.

requizon:detect runs every five minutes and checks every API for the ways it can go wrong. An API that meets a condition opens an alert, and the alert stays open while the condition holds. It resolves on the first run where it does not, so an API that is down for a day is one alert, not three hundred. Open alerts are listed on the dashboard's Alerts page, counted in its header, and shown above the overview and each API's paths page.

Detection needs nothing but the scheduler. Notifications are off until you name a channel.

What opens an alert#

Every failure counts towards the failure rate and the surge: connection errors, HTTP errors and the application errors your detector flags. Authentication failing usually means a key expired or was revoked, which is the alert most worth sending to a person straight away.

Changing the thresholds#

Every threshold is under alerts.detectors in config/requizon.php, and each detector takes 'enabled' => false. Switching one off resolves the alerts it has open.

'alerts' => [
    'detectors' => [
        'api_unreachable' => ['enabled' => true, 'window_minutes' => 10, 'min_requests' => 5, 'min_share' => 0.5],
        'failure_rate_high' => ['enabled' => true, 'window_minutes' => 15, 'min_requests' => 20, 'min_share' => 0.2],
        'failure_surge' => ['enabled' => true, 'history_days' => 7, 'ratio' => 3, 'min_per_hour' => 25],
        'server_errors' => ['enabled' => true, 'window_minutes' => 60, 'min_matching' => 10, 'min_share' => 0.1],
        'auth_failing' => ['enabled' => true, 'window_minutes' => 60, 'min_matching' => 5, 'min_share' => 0.5],
        'rate_limited' => ['enabled' => true, 'window_minutes' => 60, 'min_matching' => 10],
        'timeouts' => ['enabled' => true, 'window_minutes' => 60, 'min_matching' => 5],
        'latency_surge' => ['enabled' => true, 'history_days' => 7, 'ratio' => 2, 'min_increase_ms' => 300, 'min_requests' => 30],
        'latency_high' => [
            'enabled' => true,
            'window_minutes' => 15,
            'min_requests' => 10,
            'targets' => [],
            'default_ms' => null,
        ],
    ],
    // ...
],

min_requests is the least traffic an API needs before its share is judged at all, and min_matching the least number of calls that have to end that way. The floors keep an API called twice in an hour from opening an alert because one of the two failed.

Latency targets#

Over latency target is the one alert that does nothing until you configure it. targets maps API names, as Str::is() patterns, to the average in milliseconds that is too slow for them, and default_ms covers every API the map leaves out:

'latency_high' => [
    'enabled' => true,
    'window_minutes' => 15,
    'min_requests' => 10,
    'targets' => ['stripe' => 800, 'nausys*' => 2000],
    'default_ms' => null,
],

Alerts that compare with previous days#

Failure surge and Slowdown compare the last hour with the same hour on the previous history_days days. Detectors read the detail rows, not the hourly rollup, so that history can reach back no further than retention.detail_days allows: history_days is capped at one day less. Both wait until the detail rows go back far enough, and the alerts page says while they are waiting. With detail_days at 1 they never run.

How an alert stays open and clears#

• An open alert holds on its share alone. An API failing every call it gets stays open when its traffic drops under min_requests, rather than resolving because fewer calls were made.
• An API that gets no calls at all keeps its alert as it was, for up to 24 hours after its last call (idle_hours on the detectors that judge a share).
• An open Failure surge, Slowdown or Over latency target alert holds while its figures stay above 80% of their thresholds, so one hovering around a threshold does not open and resolve on every run.

requizon_alerts keeps one row for every stretch of time an alert was open, with the figures that opened it. It is not pruned, as it grows with incidents rather than with traffic.

The alerts page#

/requizon/alerts lists open alerts first, the critical ones at the top, and filters by state (open, resolved or all), type and API. Each alert links to its API's paths and requests, and every notification links back to the alert. Two actions sit on every open alert:

• Acknowledge says someone is on it. Its reminders stop, and its recovery is still sent, since whoever is on it wants to know when it clears.
• Mute for 1, 7 or 30 days, until a date, or for good. A muted alert sends neither reminders nor its recovery, and the mute carries over to that API's next alert of the same type. Unmuting clears it for both. An immediate alert still open when its mute ends is sent then.

The page sits behind the same gate as the rest of the dashboard, so anyone who can see the traffic can acknowledge and mute.

Notifications#

Alerts can go out by mail, to Slack and to a webhook. Nothing is sent until at least one channel is named:

REQUIZON_NOTIFY_CHANNELS=mail,slack
REQUIZON_NOTIFY_MAIL=ops@example.com,oncall@example.com
REQUIZON_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
REQUIZON_WEBHOOK_URL=https://example.com/hooks/requizon

To set destinations in code instead, call Requizon::routeMailNotificationsTo(), routeSlackNotificationsTo() or routeWebhookNotificationsTo() in your RequizonServiceProvider. A destination in the config wins over one set in code.

The webhook payload#

{
    "application": "Acme",
    "environment": "production",
    "state": "firing",
    "repeat": false,
    "seconds": 300,
    "alert": {
        "id": 412,
        "type": "server_errors",
        "api": "stripe",
        "label": "Server errors",
        "severity": "critical",
        "title": "stripe is answering with server errors",
        "summary": "37 of 212 calls in the last hour were answered with a 5xx status: 17.5%.",
        "url": "https://acme.test/requizon/alerts?alert=412#alert-412",
        "subject_url": "https://acme.test/requizon/stripe/paths",
        "payload": {
            "api": "stripe", "minutes": 60, "requests": 212, "matching": 37,
            "share": 0.1745, "peak_share": 0.1745, "called_at": "2026-11-10T09:20:00+00:00"
        },
        "detected_at": "2026-11-10T09:15:00+00:00",
        "resolved_at": null
    }
}

state is firing or resolved, repeat is true for a reminder, and seconds is how long the alert has been open. payload holds the figures the detector saw, and its keys differ by type.

When an alert is sent#

• On opening. An alert set to go out immediately is sent on the run that opens it.
• Reminders. It is sent again every repeat_minutes (360, so every six hours) while it stays open, until someone acknowledges it. 0 sends no reminders. REQUIZON_NOTIFY_REPEAT_MINUTES sets it from the environment.
• Recovery. Once it clears, a recovery is sent, unless resolved is false or the alert was muted.
• Cooldown. An alert that clears and comes back within cooldown_minutes (60) of the latest notification about it stays quiet, so an API that keeps flapping notifies about once an hour at most.

Each channel is sent on its own, so one that fails does not stop the others. A notification counts as sent when any channel took it, and when every channel failed it is tried again on the next run.

Severities, routes and types#

routes sends each severity to its own channels, and the daily digest to its own as well. A severity with no route goes to every channel. types sets each alert's severity, and whether it goes out immediately, in the digest, or not at all (none). A type can also carry its own repeat_minutes and resolved:

'notifications' => [
    // ...
    'routes' => [
        'critical' => ['mail', 'slack'],
        'warning' => ['slack'],
        'digest' => ['mail'],
    ],
    'types' => [
        'auth_failing' => ['severity' => 'critical', 'send' => 'immediate', 'repeat_minutes' => 60],
        'latency_surge' => ['severity' => 'warning', 'send' => 'digest'],
        'rate_limited' => ['severity' => 'warning', 'send' => 'none'],
        // ...
    ],
],

types is a nested array, so write out every type you want to keep. One you leave out of your list is not sent.

The daily digest#

requizon:daily-digest runs once a day at schedule.daily_digest_at (09:00, or REQUIZON_DAILY_DIGEST_AT) and sends the alerts whose type is set to digest as one message: every one no digest has listed yet, from up to a week back, leaving out the acknowledged and muted ones. No digest is sent on a day with nothing in it.

Checking where alerts go#

php artisan requizon:notifications                      # where each severity goes, and what is wrong with it
php artisan requizon:notify:test --severity=critical    # send a test over every channel critical is routed to

requizon:notifications prints the channels, routes and destinations in force, flags a channel named without a destination or a Slack channel without its package installed, and lists the open alerts it is sending. The test takes critical, warning or info, and is not stored as an alert.

More than one server#

Run requizon:detect and requizon:daily-digest on one server only, or every alert is opened and sent once per server. The built-in schedule already marks both onOneServer(), which needs a default cache store that supports locks and is shared by every server. If you schedule the commands yourself, keep onOneServer() on them.