The dashboard
What each page of /sailfish shows, and the two gates that decide who sees it.
The dashboard lives at /sailfish. It is self-contained (Blade, Alpine and uPlot, served by the package
itself), so there is nothing to build or publish for it, and it runs under a Content-Security-Policy without
'unsafe-eval'. Who can open it is up to the gates described in
installation.
Every rate on it is what moved over the seconds the snapshots cover, never a raw delta. After a gap, one snapshot covers hours, and its rows are spread over them rather than drawn as one spike.
Overview#
Open events, rows read through an index against rows read without one, the five most scanned tables, and markers on the charts for MySQL restarts, schema changes and setting changes. This is the page to open after a deploy. It warns when most statements reach MySQL as native prepared statements, which get no digest (see statement digests).
Server#
Connections against max_connections, statements, time waiting for row locks, buffer pool misses,
temporary tables and sorts spilling to disk, and InnoDB's history list. It also shows the buffer pool's size against
the data being watched, which answers whether the working set still fits in memory.
Below the charts are two tables of settings:
-
Settings changed, the latest 100 in the window: each global variable whose value moved, from what
to what, and how. A
SETnames the account that ran it and says whether a restart undoes it. A restart says MySQL came back with another value, from an option file,mysqld-auto.cnfor its default. A change with noSETof its own is one MySQL made itself, assuper_read_onlyturns onread_only. -
Set at runtime: the settings changed with
SETsince MySQL last started, who set each and when, and what a restart does to it. OnlySET PERSISTkeeps a runtime change, unless an option file sets the same value, so each is marked kept, undone, or changed to the value that was persisted. MySQL records aSET SESSIONhere too, so a variable your application sets on every connection shows as set by its own user moments ago. Laravel's MySQL connection does that forsql_modein strict mode, its default.
The server-wide charts, here and on the overview, mark the same changes. How they are read, and how values that may be secrets are hidden, is in what gets recorded.
Activity#
Read live when you open it, and never stored. It shows who is blocking whom, as trees from the session everyone
waits for, over row locks and metadata locks. The classic case is an ALTER TABLE stuck behind an idle
transaction, with every query on that table queued behind the ALTER. Locks wider than a table count
too: on a schema, the commit lock, the backup lock, and the global read lock, so a
FLUSH TABLES WITH READ LOCK left holding it shows with every write queued behind it. It also lists
open transactions oldest first, flagging those idle in a transaction, and the statements running now.
Detection takes the same reading after every collection. A session that holds others up for 30 seconds, or a transaction left idle, opens an event even while nobody has this page open, and the event keeps who was holding up whom.
A transaction is listed as open for so long only while MySQL times it, which takes the
events_transactions_current consumer and the transaction instrument, both on by default.
One that began while they were off is listed as not timed. If they are off now, the page says an open
transaction may be missing from the list, rather than that none is open.
Under each open transaction, in the list and in the blocking trees, are the statements it has run, with their time,
rows affected, sent and examined, and any error. What an idle transaction locked was usually taken by an earlier
statement than its last, and the statement that took row locks on a table someone is waiting for is marked. They
come from events_statements_history, on by default, which keeps the last 10 statements of each session.
A native prepared statement takes three of them and has no text, so a transaction that ran more says its list may
be incomplete.
Two more columns appear when their consumers are on:
-
What each running statement is waiting on right now: disk I/O, the redo log, a row, table or
metadata lock, a mutex or latch, the network. It needs the
events_waits_currentconsumer and most of thewait/%instruments, which MySQL leaves off. Timing every mutex costs a busy server measurably, so turn those instruments on for an investigation and off again; the Instrumentation page has the SQL for both. Keep the consumer itself on: the sessions blocked event reads it, with only the waits MySQL times by default. -
The stage each running statement is in, and for an
ALTER TABLEa progress bar and, while it reads the table, about how long the rest should take. It needsevents_stages_current. MySQL leaves that consumer off but times only theALTER TABLEstages and a few other long operations, so it costs next to nothing. Keep it on: a stage that began while it was off is never seen.
The page needs nothing beyond SELECT on performance_schema. Statements are shown with each
literal replaced by ?, and error messages left out, unless the viewer passes the
viewSailfishQueryText gate.
Transactions#
The Activity page sees a transaction only while it is open and someone is looking. One that held its locks for 40 seconds and committed between two collections leaves no trace there, and neither does a rising share of rollbacks. The Transactions page shows them after they finished, read with every collection.
It charts the share of transactions rolled back over time, and lists the long ones, those that ran at least
SAILFISH_LONG_TRANSACTION_SECONDS (5 by default), slowest first: how long each ran, whether it committed
or rolled back, how many statements it ran, its access mode and isolation level, and the account and program that
ran it. Never its text. The rollback surge and
long transaction events are shown beside them.
The page says which history it read. With events_transactions_history_long on, it sees every
transaction. Without it, it reads events_transactions_history, which keeps only the last 10 of each
session and drops them when the session disconnects, so the counts are a sample and most long transactions go
unseen. See what gets recorded.
Redis#
Shown when SAILFISH_REDIS_CONNECTION is set. Commands and the time each one takes, hits against misses,
memory against maxmemory with when it gets there at the current pace, keys evicted and expired, keys for
each database, clients, connections and network, with persistence and replication as of the latest read. A Commands
table shows what each command costs, with KEYS and FLUSHDB marked. The charts mark Redis's
restarts and failovers. See Redis.
Cache#
Shown when SAILFISH_CACHE_ENABLED is set. Each group of cache keys with its lookups, hit ratio,
generations after a miss and how long they took, the time spent generating in all, generations that ran in two
processes at once, the share of entries that ended without being read, the size of an entry and what the group holds
now, sorted by time spent generating. Each group has a page charting the same over time. See
Cache.
Indexes#
Every index with its status, its last use, its size, its columns and 30 days of reads and writes. The status is one of:
| Status | Meaning |
|---|---|
| hot | Last fetched within the last 7 days |
| cold | Fetched before, but not in the last 7 days |
| never used | No fetch in the history Sailfish has |
| dropped | Gone from the schema, kept with its history |
PRIMARY and unique keys are hidden until you ask for them: they enforce a constraint, so "unused" means little. Each index has a page with its full history and the indexes that cover it, and a redundant one shows how to drop it.
Tables#
Rows fetched through an index against rows read without one, over time and broken down by index, with the time spent waiting for table locks. Rows read without an index are a full scan's, and the rows an index merge fetches by position after reading its indexes. Size and row count are read once a day and charted beside them.
A dropped index or a deploy that ships a query MySQL cannot use an index for shows here as the scanned line
climbing while an index line falls. Eloquent
calls that skip an index covers the common ones, such as whereDate().
Errors#
Errors the server raised over time, the most frequent, and when each was first seen. Deadlocks (1213), lock wait timeouts (1205) and duplicate keys (1062) are pointed out. Errors are counted server-wide, not per schema.
Instrumentation#
Read live when you open it. Each part of Sailfish, with the performance_schema consumers and instruments it needs
and whether they are on. Every consumer is listed with what it records and what Sailfish reads it for, including
one set to YES but switched off by the consumer above it. It also shows the share of statements sent
as native prepared statements, which get no digest, counted since the server started across every client.
For anything off it gives the UPDATE to run now and the my.cnf line that survives a
restart. What MySQL leaves off by default is never counted as missing. What a running statement waits on is marked
costly, with the UPDATEs to turn the wait/% instruments on and to put MySQL's defaults
back. ALTER TABLE progress, events_transactions_history_long and the
events_waits_current consumer the sessions blocked event needs are cheap, so they get the
UPDATE and the my.cnf line to keep them on.
The page changes nothing itself. That takes UPDATE on performance_schema, more than the
dashboard's connection should hold.
Events#
The full record of every event, open and resolved, filterable by type. This is where an event is acknowledged, which stops its reminders, or muted until a date or for good. Detectors still waiting for enough history say here how many more days each one needs.