Notifications

Send events by mail, Slack or webhook, route them by severity, and get reminders and recoveries.

Sailfish sends nothing until you give it a channel. Without one the Events page on the dashboard is the only record, which is fine for a first week while the detectors gather history.

Channels#

Set SAILFISH_NOTIFY_CHANNELS to any of mail, slack and webhook, and give each a destination:

SAILFISH_NOTIFY_CHANNELS=mail,slack,webhook
SAILFISH_NOTIFY_MAIL=db-team@example.com,ops@example.com
SAILFISH_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T000/B000/XXXX
SAILFISH_WEBHOOK_URL=https://example.com/hooks/sailfish

Slack needs composer require laravel/slack-notification-channel. An incoming webhook URL posts an attachment. A #channel or channel id instead posts Block Kit through the bot token in services.slack.notifications.

Destinations can also be set from code, in a service provider. The config wins when it has one:

Sailfish::routeMailNotificationsTo(['ops@example.com']);
Sailfish::routeSlackNotificationsTo('#database');
Sailfish::routeWebhookNotificationsTo('https://example.com/hooks/sailfish');

What is sent, and when#

Each event type has a severity and a send mode: immediate sends it as it opens, digest puts it in the daily digest, and none records it on the dashboard only. The defaults are in the table on events. To change one:

// config/sailfish.php
'notifications' => [
    'types' => [
        'scan_surge' => ['severity' => 'critical', 'send' => 'immediate'],
        'schema_change' => ['severity' => 'info', 'send' => 'none'],
        // ...
    ],
],

The daily digest#

Events whose type is sent by digest are gathered into one message a day, at SAILFISH_DAILY_DIGEST_AT (09:00 by default). Index went cold is a digest event on purpose: a deploy that changes queries can turn many indexes cold at once, and that should be one message with a line each.

Routing by severity#

notifications.routes sends each severity, and the digest, over some of the channels only. A severity it does not list goes to all of them:

'routes' => [
    'critical' => ['slack', 'webhook'],
    'warning' => ['slack'],
    'digest' => ['mail'],
],

Reminders and recoveries#

An immediate event is sent as it opens. While it stays open it is sent again every repeat_minutes (six hours by default, SAILFISH_NOTIFY_REPEAT_MINUTES) until someone acknowledges it. Once it clears, a recovery is sent, saying why it ended: the condition cleared, or the table or index it was about was dropped. Each type can set its own repeat_minutes and resolved.

An event that flaps notifies at most once per cooldown_minutes (six hours by default, counted from its latest notification). One that never notified, because of the cooldown, ends without a recovery. Once a recovery has gone out, the cooldown no longer holds back the event coming back: whoever was told it cleared hears that it is back. With recoveries on, the cooldown only damps types that send none.

When a channel fails#

Each channel is sent on its own. One failing is logged as sailfish.notification_failed and does not stop the others, and a notification counts as sent when any channel took it. When every channel failed, it is tried again by the next sailfish:detect, and the digest is left for the next day.

A channel that fails three times in a row is skipped for the rest of the run, so one that is down costs one timeout per run rather than one per event. A route naming a channel that has no destination counts as that channel failing, and is logged as sailfish.notification_unroutable: the event waits until it can be sent rather than being marked as sent.

The webhook payload#

The webhook channel POSTs JSON, with any headers in notifications.webhook.headers:

{
  "application": "Laravel",
  "environment": "production",
  "state": "firing",
  "repeat": false,
  "seconds": 0,
  "event": {
    "id": 42, "type": "scan_surge", "label": "Table-scan surge", "severity": "critical",
    "title": "Full-table scans on app.orders surged", "summary": "…",
    "url": "https://example.com/sailfish/events?type=scan_surge&state=all#event-42",
    "subject_url": "https://example.com/sailfish/tables/7",
    "payload": {}, "detected_at": "…", "resolved_at": null, "resolution": null
  }
}

state is firing or resolved, and repeat is true on a reminder. Once resolved, resolution is cleared, or gone when its table or index was dropped. The digest sends "state": "digest" with a list of events.

Testing the setup#

php artisan sailfish:notify:test --severity=warning
php artisan sailfish:notifications

sailfish:notify:test sends a test over the channels that severity is routed to, and fails if any channel does. sailfish:notifications warns about channels without a destination and routes that go nowhere, shows the routes, and lists the open events with when each was last sent.

Customising the message#

To change what is sent, bind your own notification class. It is constructed with event, repeat and resolved:

$this->app->bind(
    \BoringO11y\MySailfish\Contracts\EventNotification::class,
    MyEventNotification::class,
);

Sailfish also dispatches Laravel events you can listen for: Events\EventOpened and Events\EventResolved as events open and clear, and Events\EventNotified after each notification, with the channels it went over and any that failed.