# Notifications

> Send events by mail, Slack or webhook, route them by severity, and get reminders and recoveries.

Source: https://boring-observability.dev/sailfish/docs/notifications
Section: Events and notifications — Sailfish documentation
Updated: 2026-10-02

---

Sailfish sends nothing until you give it a channel. Without one the Events page on the dashboard is the only record, which is fine for a first week while the detectors gather history.

## Channels

Set `SAILFISH_NOTIFY_CHANNELS` to any of `mail`, `slack` and `webhook`, and give each a destination:

```bash
SAILFISH_NOTIFY_CHANNELS=mail,slack,webhook
SAILFISH_NOTIFY_MAIL=db-team@example.com,ops@example.com
SAILFISH_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T000/B000/XXXX
SAILFISH_WEBHOOK_URL=https://example.com/hooks/sailfish
```

Slack needs `composer require laravel/slack-notification-channel`. An incoming webhook URL posts an attachment. A `#channel` or channel id instead posts Block Kit through the bot token in `services.slack.notifications`.

Destinations can also be set from code, in a service provider. The config wins when it has one:

```php
Sailfish::routeMailNotificationsTo(['ops@example.com']);
Sailfish::routeSlackNotificationsTo('#database');
Sailfish::routeWebhookNotificationsTo('https://example.com/hooks/sailfish');
```

## What is sent, and when

Each event type has a severity and a `send` mode: `immediate` sends it as it opens, `digest` puts it in the daily digest, and `none` records it on the dashboard only. The defaults are in the table on [events](https://boring-observability.dev/sailfish/docs/events). To change one:

```php
// config/sailfish.php
'notifications' => [
    'types' => [
        'scan_surge' => ['severity' => 'critical', 'send' => 'immediate'],
        'schema_change' => ['severity' => 'info', 'send' => 'none'],
        // ...
    ],
],
```

### The daily digest

Events whose type is sent by `digest` are gathered into one message a day, at `SAILFISH_DAILY_DIGEST_AT` (09:00 by default). Index went cold is a digest event on purpose: a deploy that changes queries can turn many indexes cold at once, and that should be one message with a line each.

## Routing by severity

`notifications.routes` sends each severity, and the digest, over some of the channels only. A severity it does not list goes to all of them:

```php
'routes' => [
    'critical' => ['slack', 'webhook'],
    'warning' => ['slack'],
    'digest' => ['mail'],
],
```

## Reminders and recoveries

An immediate event is sent as it opens. While it stays open it is sent again every `repeat_minutes` (six hours by default, `SAILFISH_NOTIFY_REPEAT_MINUTES`) until someone acknowledges it. Once it clears, a recovery is sent, saying why it ended: the condition cleared, or the table or index it was about was dropped. Each type can set its own `repeat_minutes` and `resolved`.

|  | First message | Reminders | Recovery |
| --- | --- | --- | --- |
| Muted | Held until the mute ends | Held until the mute ends | Not sent |
| Acknowledged | Not sent | Stop | Sent |

An event that flaps notifies at most once per `cooldown_minutes` (six hours by default, counted from its latest notification). One that never notified, because of the cooldown, ends without a recovery. Once a recovery has gone out, the cooldown no longer holds back the event coming back: whoever was told it cleared hears that it is back. With recoveries on, the cooldown only damps types that send none.

## When a channel fails

Each channel is sent on its own. One failing is logged as `sailfish.notification_failed` and does not stop the others, and a notification counts as sent when any channel took it. When every channel failed, it is tried again by the next `sailfish:detect`, and the digest is left for the next day.

A channel that fails three times in a row is skipped for the rest of the run, so one that is down costs one timeout per run rather than one per event. A route naming a channel that has no destination counts as that channel failing, and is logged as `sailfish.notification_unroutable`: the event waits until it can be sent rather than being marked as sent.

## The webhook payload

The webhook channel POSTs JSON, with any headers in `notifications.webhook.headers`:

```json
{
  "application": "Laravel",
  "environment": "production",
  "state": "firing",
  "repeat": false,
  "seconds": 0,
  "event": {
    "id": 42, "type": "scan_surge", "label": "Table-scan surge", "severity": "critical",
    "title": "Full-table scans on app.orders surged", "summary": "…",
    "url": "https://example.com/sailfish/events?type=scan_surge&state=all#event-42",
    "subject_url": "https://example.com/sailfish/tables/7",
    "payload": {}, "detected_at": "…", "resolved_at": null, "resolution": null
  }
}
```

`state` is `firing` or `resolved`, and `repeat` is true on a reminder. Once resolved, `resolution` is `cleared`, or `gone` when its table or index was dropped. The digest sends `"state": "digest"` with a list of `events`.

## Testing the setup

```bash
php artisan sailfish:notify:test --severity=warning
php artisan sailfish:notifications
```

`sailfish:notify:test` sends a test over the channels that severity is routed to, and fails if any channel does. `sailfish:notifications` warns about channels without a destination and routes that go nowhere, shows the routes, and lists the open events with when each was last sent.

## Customising the message

To change what is sent, bind your own notification class. It is constructed with `event`, `repeat` and `resolved`:

```php
$this->app->bind(
    \BoringO11y\MySailfish\Contracts\EventNotification::class,
    MyEventNotification::class,
);
```

Sailfish also dispatches Laravel events you can listen for: `Events\EventOpened` and `Events\EventResolved` as events open and clear, and `Events\EventNotified` after each notification, with the channels it went over and any that failed.


## Common questions

### What happens when a notification channel is down?

Each channel is sent on its own, so one failing is logged as sailfish.notification_failed and does not stop the others. When every channel failed, the notification is tried again by the next sailfish:detect. A channel that fails three times in a row is skipped for the rest of that run.

### How do I check that notifications reach us?

Run php artisan sailfish:notify:test --severity=warning. It sends a test over the channels that severity is routed to and fails if any of them does. php artisan sailfish:notifications lists channels without a destination and routes that go nowhere.