# Alerts and notifications > What opens an alert, how it stays open and clears, and how alerts reach mail, Slack or a webhook. Source: https://boring-observability.dev/requizon/docs/alerts Section: Configuration — Requizon documentation (version 1.1) Updated: 2026-11-10 --- `requizon:detect` runs every five minutes and checks every API for the ways it can go wrong. An API that meets a condition opens an alert, and the alert stays open while the condition holds. It resolves on the first run where it does not, so an API that is down for a day is one alert, not three hundred. Open alerts are listed on the dashboard's **Alerts** page, counted in its header, and shown above the overview and each API's paths page. Detection needs nothing but the scheduler. Notifications are off until you [name a channel](#notifications). ## What opens an alert | Alert | Opens when (defaults) | Severity | Key | | --- | --- | --- | --- | | API unreachable | Half the calls in the last 10 minutes got no response (refused, DNS, TLS, timeout), over at least 5 calls | critical | `api_unreachable` | | High failure rate | A fifth of the calls in the last 15 minutes failed in any way, over at least 20 calls | critical | `failure_rate_high` | | Server errors | 10% of the last hour's calls were answered 5xx, and at least 10 of them | critical | `server_errors` | | Authentication failing | Half the last hour's calls were answered 401 or 403, and at least 5 of them | critical | `auth_failing` | | Failure surge | At least 25 failures in the last hour, and 3× the average for that hour over the previous 7 days | warning | `failure_surge` | | Rate limited | 10 answers of 429 in the last hour | warning | `rate_limited` | | Timeouts | 5 calls in the last hour timed out | warning | `timeouts` | | Slowdown | Successful calls in the last hour took twice as long on average as at that hour over the previous 7 days, and 300 ms longer, over at least 30 calls both now and then | warning | `latency_surge` | | Over latency target | Successful calls in the last 15 minutes averaged more than the target you set for that API, over at least 10 calls | warning | `latency_high` | Every failure counts towards the failure rate and the surge: connection errors, HTTP errors and the [application errors](https://boring-observability.dev/requizon/docs/failure-detection#application-errors) your detector flags. Authentication failing usually means a key expired or was revoked, which is the alert most worth sending to a person straight away. ## Changing the thresholds Every threshold is under `alerts.detectors` in `config/requizon.php`, and each detector takes `'enabled' => false`. Switching one off resolves the alerts it has open. ```php 'alerts' => [ 'detectors' => [ 'api_unreachable' => ['enabled' => true, 'window_minutes' => 10, 'min_requests' => 5, 'min_share' => 0.5], 'failure_rate_high' => ['enabled' => true, 'window_minutes' => 15, 'min_requests' => 20, 'min_share' => 0.2], 'failure_surge' => ['enabled' => true, 'history_days' => 7, 'ratio' => 3, 'min_per_hour' => 25], 'server_errors' => ['enabled' => true, 'window_minutes' => 60, 'min_matching' => 10, 'min_share' => 0.1], 'auth_failing' => ['enabled' => true, 'window_minutes' => 60, 'min_matching' => 5, 'min_share' => 0.5], 'rate_limited' => ['enabled' => true, 'window_minutes' => 60, 'min_matching' => 10], 'timeouts' => ['enabled' => true, 'window_minutes' => 60, 'min_matching' => 5], 'latency_surge' => ['enabled' => true, 'history_days' => 7, 'ratio' => 2, 'min_increase_ms' => 300, 'min_requests' => 30], 'latency_high' => [ 'enabled' => true, 'window_minutes' => 15, 'min_requests' => 10, 'targets' => [], 'default_ms' => null, ], ], // ... ], ``` `min_requests` is the least traffic an API needs before its share is judged at all, and `min_matching` the least number of calls that have to end that way. The floors keep an API called twice in an hour from opening an alert because one of the two failed. ### Latency targets Over latency target is the one alert that does nothing until you configure it. `targets` maps API names, as `Str::is()` patterns, to the average in milliseconds that is too slow for them, and `default_ms` covers every API the map leaves out: ```php 'latency_high' => [ 'enabled' => true, 'window_minutes' => 15, 'min_requests' => 10, 'targets' => ['stripe' => 800, 'nausys*' => 2000], 'default_ms' => null, ], ``` ### Alerts that compare with previous days Failure surge and Slowdown compare the last hour with the same hour on the previous `history_days` days. Detectors read the detail rows, not the hourly rollup, so that history can reach back no further than `retention.detail_days` allows: `history_days` is capped at one day less. Both wait until the detail rows go back far enough, and the alerts page says while they are waiting. With `detail_days` at 1 they never run. ## How an alert stays open and clears - An open alert holds on its share alone. An API failing every call it gets stays open when its traffic drops under `min_requests`, rather than resolving because fewer calls were made. - An API that gets no calls at all keeps its alert as it was, for up to 24 hours after its last call (`idle_hours` on the detectors that judge a share). - An open Failure surge, Slowdown or Over latency target alert holds while its figures stay above 80% of their thresholds, so one hovering around a threshold does not open and resolve on every run. `requizon_alerts` keeps one row for every stretch of time an alert was open, with the figures that opened it. It is not pruned, as it grows with incidents rather than with traffic. ## The alerts page `/requizon/alerts` lists open alerts first, the critical ones at the top, and filters by state (open, resolved or all), type and API. Each alert links to its API's paths and requests, and every notification links back to the alert. Two actions sit on every open alert: - **Acknowledge** says someone is on it. Its reminders stop, and its recovery is still sent, since whoever is on it wants to know when it clears. - **Mute** for 1, 7 or 30 days, until a date, or for good. A muted alert sends neither reminders nor its recovery, and the mute carries over to that API's next alert of the same type. Unmuting clears it for both. An immediate alert still open when its mute ends is sent then. The page sits behind the same [gate](https://boring-observability.dev/requizon/docs/dashboard#access) as the rest of the dashboard, so anyone who can see the traffic can acknowledge and mute. ## Notifications Alerts can go out by mail, to Slack and to a webhook. Nothing is sent until at least one channel is named: ```bash REQUIZON_NOTIFY_CHANNELS=mail,slack REQUIZON_NOTIFY_MAIL=ops@example.com,oncall@example.com REQUIZON_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... REQUIZON_WEBHOOK_URL=https://example.com/hooks/requizon ``` | Channel | Needs | | --- | --- | | `mail` | A working mailer, and one or more addresses in `REQUIZON_NOTIFY_MAIL`, comma separated | | `slack` | `laravel/slack-notification-channel`. `REQUIZON_SLACK_WEBHOOK_URL` takes an incoming webhook URL, or a `#channel` or channel id to post to through the bot token in `services.slack.notifications` | | `webhook` | A URL to POST JSON to. `alerts.notifications.webhook.headers` adds headers, such as a token your endpoint checks, and `webhook.timeout` is 5 seconds | To set destinations in code instead, call `Requizon::routeMailNotificationsTo()`, `routeSlackNotificationsTo()` or `routeWebhookNotificationsTo()` in your `RequizonServiceProvider`. A destination in the config wins over one set in code. ### The webhook payload ```json { "application": "Acme", "environment": "production", "state": "firing", "repeat": false, "seconds": 300, "alert": { "id": 412, "type": "server_errors", "api": "stripe", "label": "Server errors", "severity": "critical", "title": "stripe is answering with server errors", "summary": "37 of 212 calls in the last hour were answered with a 5xx status: 17.5%.", "url": "https://acme.test/requizon/alerts?alert=412#alert-412", "subject_url": "https://acme.test/requizon/stripe/paths", "payload": { "api": "stripe", "minutes": 60, "requests": 212, "matching": 37, "share": 0.1745, "peak_share": 0.1745, "called_at": "2026-11-10T09:20:00+00:00" }, "detected_at": "2026-11-10T09:15:00+00:00", "resolved_at": null } } ``` `state` is `firing` or `resolved`, `repeat` is true for a reminder, and `seconds` is how long the alert has been open. `payload` holds the figures the detector saw, and its keys differ by type. > **Notifications and your own traffic** > > Slack notifications go out through Laravel's HTTP client, so they are recorded like any other call. Webhook notifications are not, so a webhook endpoint that is down does not raise alerts about itself and send them to itself. ### When an alert is sent - **On opening.** An alert set to go out immediately is sent on the run that opens it. - **Reminders.** It is sent again every `repeat_minutes` (360, so every six hours) while it stays open, until someone acknowledges it. `0` sends no reminders. `REQUIZON_NOTIFY_REPEAT_MINUTES` sets it from the environment. - **Recovery.** Once it clears, a recovery is sent, unless `resolved` is `false` or the alert was muted. - **Cooldown.** An alert that clears and comes back within `cooldown_minutes` (60) of the latest notification about it stays quiet, so an API that keeps flapping notifies about once an hour at most. Each channel is sent on its own, so one that fails does not stop the others. A notification counts as sent when any channel took it, and when every channel failed it is tried again on the next run. ### Severities, routes and types `routes` sends each severity to its own channels, and the daily digest to its own as well. A severity with no route goes to every channel. `types` sets each alert's severity, and whether it goes out `immediate`ly, in the `digest`, or not at all (`none`). A type can also carry its own `repeat_minutes` and `resolved`: ```php 'notifications' => [ // ... 'routes' => [ 'critical' => ['mail', 'slack'], 'warning' => ['slack'], 'digest' => ['mail'], ], 'types' => [ 'auth_failing' => ['severity' => 'critical', 'send' => 'immediate', 'repeat_minutes' => 60], 'latency_surge' => ['severity' => 'warning', 'send' => 'digest'], 'rate_limited' => ['severity' => 'warning', 'send' => 'none'], // ... ], ], ``` `types` is a nested array, so write out every type you want to keep. One you leave out of your list is not sent. ### The daily digest `requizon:daily-digest` runs once a day at `schedule.daily_digest_at` (09:00, or `REQUIZON_DAILY_DIGEST_AT`) and sends the alerts whose type is set to `digest` as one message: every one no digest has listed yet, from up to a week back, leaving out the acknowledged and muted ones. No digest is sent on a day with nothing in it. ## Checking where alerts go ```bash php artisan requizon:notifications # where each severity goes, and what is wrong with it php artisan requizon:notify:test --severity=critical # send a test over every channel critical is routed to ``` `requizon:notifications` prints the channels, routes and destinations in force, flags a channel named without a destination or a Slack channel without its package installed, and lists the open alerts it is sending. The test takes `critical`, `warning` or `info`, and is not stored as an alert. ## More than one server Run `requizon:detect` and `requizon:daily-digest` on one server only, or every alert is opened and sent once per server. The built-in schedule already marks both `onOneServer()`, which needs a default cache store that supports locks and is shared by every server. If you [schedule the commands yourself](https://boring-observability.dev/requizon/docs/retention#own-schedule), keep `onOneServer()` on them. ## Common questions ### Does Requizon alert me when an API starts failing? Yes. requizon:detect runs every five minutes on the scheduler and opens an alert when an API is unreachable, failing a fifth of its calls, answering 5xx, rejecting your credentials, rate limiting you, timing out, failing three times more than usual, or running slower than usual or than a target you set. Alerts show on the dashboard straight away, and are sent by mail, Slack or webhook once REQUIZON_NOTIFY_CHANNELS names a channel. ### Why did Requizon not send an alert? Run php artisan requizon:notifications, which prints where alerts go and what is wrong with that. Nothing is sent until REQUIZON_NOTIFY_CHANNELS names a channel with a destination. Acknowledged alerts send no reminders, muted ones send nothing, a type set to digest waits for the daily digest, and an alert that comes back within cooldown_minutes of its last notification stays quiet. Without the scheduler, requizon:detect never runs and no alert opens at all. ### How do I stop Requizon alerting about one API? Mute its alert on the alerts page, for a day, a week, a month, until a date or for good. The mute carries over to that API's next alert of the same type. To stop one kind of alert for every API, set enabled to false on its detector under alerts.detectors in config/requizon.php.