Sailfish

The dashboard

What each page of /sailfish shows, and the two gates that decide who sees it.

The dashboard lives at /sailfish. It is self-contained (Blade, Alpine and uPlot, served by the package itself), so there is nothing to build or publish for it, and it runs under a Content-Security-Policy without 'unsafe-eval'. Who can open it is up to the gates described in installation.

Every rate on it is what moved over the seconds the snapshots cover, never a raw delta. After a gap, one snapshot covers hours, and its rows are spread over them rather than drawn as one spike.

Overview#

Open events, rows read through an index against rows read without one, the five most scanned tables, and markers on the charts for MySQL restarts, schema changes and setting changes. This is the page to open after a deploy. It warns when most statements reach MySQL as native prepared statements, which get no digest (see statement digests).

Server#

Connections against max_connections, statements, time waiting for row locks, buffer pool misses, temporary tables and sorts spilling to disk, and InnoDB's history list. It also shows the buffer pool's size against the data being watched, which answers whether the working set still fits in memory.

Below the charts are two tables of settings:

  • Settings changed, the latest 100 in the window: each global variable whose value moved, from what to what, and how. A SET names the account that ran it and says whether a restart undoes it. A restart says MySQL came back with another value, from an option file, mysqld-auto.cnf or its default. A change with no SET of its own is one MySQL made itself, as super_read_only turns on read_only.
  • Set at runtime: the settings changed with SET since MySQL last started, who set each and when, and what a restart does to it. Only SET PERSIST keeps a runtime change, unless an option file sets the same value, so each is marked kept, undone, or changed to the value that was persisted. MySQL records a SET SESSION here too, so a variable your application sets on every connection shows as set by its own user moments ago. Laravel's MySQL connection does that for sql_mode in strict mode, its default.

The server-wide charts, here and on the overview, mark the same changes. How they are read, and how values that may be secrets are hidden, is in what gets recorded.

Activity#

Read live when you open it, and never stored. It shows who is blocking whom, as trees from the session everyone waits for, over row locks and metadata locks. The classic case is an ALTER TABLE stuck behind an idle transaction, with every query on that table queued behind the ALTER. Locks wider than a table count too: on a schema, the commit lock, the backup lock, and the global read lock, so a FLUSH TABLES WITH READ LOCK left holding it shows with every write queued behind it. It also lists open transactions oldest first, flagging those idle in a transaction, and the statements running now.

Detection takes the same reading after every collection. A session that holds others up for 30 seconds, or a transaction left idle, opens an event even while nobody has this page open, and the event keeps who was holding up whom.

A transaction is listed as open for so long only while MySQL times it, which takes the events_transactions_current consumer and the transaction instrument, both on by default. One that began while they were off is listed as not timed. If they are off now, the page says an open transaction may be missing from the list, rather than that none is open.

Under each open transaction, in the list and in the blocking trees, are the statements it has run, with their time, rows affected, sent and examined, and any error. What an idle transaction locked was usually taken by an earlier statement than its last, and the statement that took row locks on a table someone is waiting for is marked. They come from events_statements_history, on by default, which keeps the last 10 statements of each session. A native prepared statement takes three of them and has no text, so a transaction that ran more says its list may be incomplete.

Two more columns appear when their consumers are on:

  • What each running statement is waiting on right now: disk I/O, the redo log, a row, table or metadata lock, a mutex or latch, the network. It needs the events_waits_current consumer and most of the wait/% instruments, which MySQL leaves off. Timing every mutex costs a busy server measurably, so turn those instruments on for an investigation and off again; the Instrumentation page has the SQL for both. Keep the consumer itself on: the sessions blocked event reads it, with only the waits MySQL times by default.
  • The stage each running statement is in, and for an ALTER TABLE a progress bar and, while it reads the table, about how long the rest should take. It needs events_stages_current. MySQL leaves that consumer off but times only the ALTER TABLE stages and a few other long operations, so it costs next to nothing. Keep it on: a stage that began while it was off is never seen.

The page needs nothing beyond SELECT on performance_schema. Statements are shown with each literal replaced by ?, and error messages left out, unless the viewer passes the viewSailfishQueryText gate.

Transactions#

The Activity page sees a transaction only while it is open and someone is looking. One that held its locks for 40 seconds and committed between two collections leaves no trace there, and neither does a rising share of rollbacks. The Transactions page shows them after they finished, read with every collection.

It charts the share of transactions rolled back over time, and lists the long ones, those that ran at least SAILFISH_LONG_TRANSACTION_SECONDS (5 by default), slowest first: how long each ran, whether it committed or rolled back, how many statements it ran, its access mode and isolation level, and the account and program that ran it. Never its text. The rollback surge and long transaction events are shown beside them.

The page says which history it read. With events_transactions_history_long on, it sees every transaction. Without it, it reads events_transactions_history, which keeps only the last 10 of each session and drops them when the session disconnects, so the counts are a sample and most long transactions go unseen. See what gets recorded.

Redis#

Shown when SAILFISH_REDIS_CONNECTION is set. Commands and the time each one takes, hits against misses, memory against maxmemory with when it gets there at the current pace, keys evicted and expired, keys for each database, clients, connections and network, with persistence and replication as of the latest read. A Commands table shows what each command costs, with KEYS and FLUSHDB marked. The charts mark Redis's restarts and failovers. See Redis.

Cache#

Shown when SAILFISH_CACHE_ENABLED is set. Each group of cache keys with its lookups, hit ratio, generations after a miss and how long they took, the time spent generating in all, generations that ran in two processes at once, the share of entries that ended without being read, the size of an entry and what the group holds now, sorted by time spent generating. Each group has a page charting the same over time. See Cache.

Indexes#

Every index with its status, its last use, its size, its columns and 30 days of reads and writes. The status is one of:

StatusMeaning
hotLast fetched within the last 7 days
coldFetched before, but not in the last 7 days
never usedNo fetch in the history Sailfish has
droppedGone from the schema, kept with its history

PRIMARY and unique keys are hidden until you ask for them: they enforce a constraint, so "unused" means little. Each index has a page with its full history and the indexes that cover it, and a redundant one shows how to drop it.

Tables#

Rows fetched through an index against rows read without one, over time and broken down by index, with the time spent waiting for table locks. Rows read without an index are a full scan's, and the rows an index merge fetches by position after reading its indexes. Size and row count are read once a day and charted beside them.

A dropped index or a deploy that ships a query MySQL cannot use an index for shows here as the scanned line climbing while an index line falls. Eloquent calls that skip an index covers the common ones, such as whereDate().

Errors#

Errors the server raised over time, the most frequent, and when each was first seen. Deadlocks (1213), lock wait timeouts (1205) and duplicate keys (1062) are pointed out. Errors are counted server-wide, not per schema.

Instrumentation#

Read live when you open it. Each part of Sailfish, with the performance_schema consumers and instruments it needs and whether they are on. Every consumer is listed with what it records and what Sailfish reads it for, including one set to YES but switched off by the consumer above it. It also shows the share of statements sent as native prepared statements, which get no digest, counted since the server started across every client.

For anything off it gives the UPDATE to run now and the my.cnf line that survives a restart. What MySQL leaves off by default is never counted as missing. What a running statement waits on is marked costly, with the UPDATEs to turn the wait/% instruments on and to put MySQL's defaults back. ALTER TABLE progress, events_transactions_history_long and the events_waits_current consumer the sessions blocked event needs are cheap, so they get the UPDATE and the my.cnf line to keep them on.

The page changes nothing itself. That takes UPDATE on performance_schema, more than the dashboard's connection should hold.

Events#

The full record of every event, open and resolved, filterable by type. This is where an event is acknowledged, which stops its reminders, or muted until a date or for good. Detectors still waiting for enough history say here how many more days each one needs.

Install Sailfish today.

Checkout ends with your license key, and the installation guide takes it from there.