Paths and table size

How identifiers become :id, rewrite rules for the ones it misses, the allow-list that is the only hard bound, merging paths already stored, and the path each call was made to.

The hourly rollup, requizon_http_request_stats, has one row per API, host, path and hour. APIs and hosts are few. Paths are the column that can run away: a REST API puts an identifier in almost every URL, and stored raw, every order number would be its own row in every hour it was fetched. So paths are normalised before anything is stored.

What the normaliser does#

Each segment of the path is checked in turn. A segment becomes :id when it is:

• all digits (8134)
• a UUID (2f1c3a9e-7b0d-4c1e-9f55-0c8e6f1d2a47)
• a ULID (01J8Z3K9Q4M2V6X7B1C5D8E0FG)
• a 32, 40 or 64 character hex string, the shape of an MD5, SHA-1 or SHA-256
• longer than paths.max_segment_length (40), which catches signed tokens and encoded filenames nobody names by hand

Only the first paths.max_segments segments (four) are kept. When there were more, the stored path ends in /*, so nobody mistakes the prefix for the endpoint that was called. The query string is never part of the path.

The first and fourth rows are the heuristic missing. cus_Q2x8fL0aZkP3m is an identifier, but it is under 40 characters and not in any of the recognised shapes. A UUID with a file extension is no longer a bare UUID. Both would add a path per customer or per file. The rest of this page is about what to do when your API looks like that.

Paths are compared case-sensitively, on every database, because /files/abc and /files/AbC are two resources.

Choosing max_segments#

Four segments is enough for most REST APIs, and it bounds the damage when an identifier the normaliser missed sits deep in a path. Raise it when the part of the path that tells your endpoints apart comes later, as with APIs that carry a long fixed prefix:

// /CBMS-external/rest/yachtReservation/v6/freeYachts
// with max_segments 4: /CBMS-external/rest/yachtReservation/v6/*
// with max_segments 6: /CBMS-external/rest/yachtReservation/v6/freeYachts
'paths' => [
    'max_segments' => 6,
    'max_segment_length' => 40,
    'max_length' => 255,
    'patterns' => [],
    'other_label' => 'other',
],

Changing it starts new rows on the paths page from the deploy onwards. Existing rows keep the paths they were stored with until you merge them.

Rewriting paths the heuristic misses#

When an identifier does not look like one, declare the path to record instead. paths.rewrite maps Str::is() patterns, matched against the raw path, to the path stored in its place:

'paths' => [
    // ...
    'rewrite' => [
        '/files/*' => '/files/:id',                 // /files/4313261196505276_DSCF0122.jpg
        '/v1/customers/cus_*/invoices' => '/v1/customers/:id/invoices',
    ],
],

Rewrites run before the heuristic, and the first pattern that matches wins, so put the narrower ones first. A rewritten path skips max_segments, since you wrote it, but it still goes through patterns below. The list applies to every API at once. For a rule that has to look at the host or the request, use a resolver instead.

The allow-list#

Everything above is a heuristic, and an API with an open-ended URL space will eventually leak through it. The one hard bound is paths.patterns. When the list is not empty, a normalised path must match one of its Str::is() patterns, or it is stored as other:

'paths' => [
    // ...
    'patterns' => [
        '/v1/customers*',
        '/v1/payment_intents*',
        '/v1/refunds*',
        '/',
    ],
    'other_label' => 'other',
],

Patterns are matched against the normalised path, so write /v1/users/:id rather than a pattern for the digits. They apply to every API at once, including the root path, which is why / is in the list above. It is empty by default because a default list would quietly relabel every path of every application that never configured one.

other is a real bucket: its calls, failures and response times are all counted, just not told apart. If it grows, the requests list for that API still has every call's full path to show you what to add.

Replacing the normaliser#

For an API whose identifiers the heuristic cannot recognise, register a resolver in your RequizonServiceProvider. Return the path to store, or null to hand the call to the built-in normaliser:

use BoringO11y\Requizon\Requizon;

Requizon::resolvePathUsing(function ($uri, $request) {
    if ($uri->getHost() !== 'api.stripe.com') {
        return null;
    }

    // cus_Q2x8fL0aZkP3m, pi_3NcQ... → :id, while /v1/payment_intents stays
    return preg_replace('#/[a-z]{2,5}_[A-Za-z0-9]{10,}(?=/|$)#', '/:id', $uri->getPath());
});

A resolved path is stored as returned, cut to paths.max_length. It does not go through max_segments or patterns, so a resolver is responsible for keeping its own output bounded.

Merging paths already stored#

Everything on this page changes how calls are recorded from the deploy onwards. The rows already stored keep the paths they were recorded with, so the paths page shows the old, unbounded paths next to the new bucket until they age out. requizon:merge-paths folds them in:

php artisan requizon:merge-paths --dry-run        # every path that would move, and where to
php artisan requizon:merge-paths                  # merge
php artisan requizon:merge-paths --api=nausys     # one API only

It puts every stored path through the current paths config, renames the detail rows and adds the old hourly buckets into the new ones, so the charts keep their history under the new path. A path that new calls still land in is left where it is. A rewrite is matched against the stored path, which is the raw path unless the normaliser had already changed it.

Deploy the config change and restart long-running workers (queue:restart, octane:reload) before merging. A worker still on the old config keeps recording the old paths. Running the command again afterwards folds those in too.

It is safe to run while the scheduler is running: the merge and requizon:aggregate take turns through a cache lock, one batch of paths at a time. With more than one server, the default cache store has to be one they all share, as it does for withoutOverlapping().

A path resolver cannot be applied to stored rows, because it needs the request and a stored row only has its path. Merging would re-normalise the paths it produced by the config alone, while new calls keep landing where the resolver puts them. So with a resolver registered, the command refuses to merge unless you pass --ignore-path-resolver. Check --dry-run first, and use --api to limit the merge to the APIs your resolver leaves alone.

The path as it was called#

Normalising is what keeps the rollup small, and it is also what stops /cbm/documents/:id telling you which document a failed call asked for. So each row also keeps its raw path, /cbm/documents/123, for every call, successful or not. Nothing groups by it, so it costs the rollup nothing. The requests list shows it in place of the normalised path, and a row's details show the URL it adds up to: method, host, path and the stored query, redacted.

What the raw path masks#

A raw path can hold a secret as easily as a filename, so some segments are stored as ***:

In a path, a substring rule has to match a whole word of the segment, so client_secret names one and authors, passengers and tokens do not. A call that a path resolver names stores no raw path at all, since the resolver may be what keeps a segment like that out of the database.

For a rule about values rather than names, register a callback. It gets the segments after the leading slash and returns the ones to store:

use BoringO11y\Requizon\Requizon;
use Psr\Http\Message\RequestInterface;

Requizon::redactPathUsing(function (array $segments, string $api, RequestInterface $request): array {
    return array_map(fn (string $segment) => preg_match('/^\d{13,19}$/', $segment) ? '***' : $segment, $segments);
});

Three settings move the line either way:

• paths.raw_unmasked lists Str::is() patterns, matched against the raw path on every host, for paths you know hold no secret. Neither a rewrite's * nor the length rule masks them, so with '/files/*' listed, /files/report.pdf is stored as itself and still counted under /files/:id. The redaction rules and the callback still run.
• paths.raw_long_segments keeps long segments, for APIs that put no tokens in their paths.
• paths.store_raw (REQUIZON_STORE_RAW_PATHS=false) stores no raw path at all, for an API that puts a credential in a segment short enough to be kept that nothing names, as a webhook URL does.

A call recorded before a setting changed keeps what it was stored with. A masked value was never kept, so there is nothing to unmask.

Searching by path or URL#

The Path filter on the requests list takes either form. A normalised path lists every call counted under it. A raw path, or a whole URL pasted from a log, lists the calls made to exactly that path, and with a query string, to exactly that URL. The overview's search box takes a URL too, and its host picks the API.

Masking does not get in the way. Every call also stores a hash of its path and of its query string as they were sent, before anything was masked, redacted or cut to fit, and a search by URL matches those. So /files/report.pdf finds the call stored as /files/***, a resolver's call is found by the URL it was made to, and so is a path longer than the column. Both sides are spelt one way first, so /My Doc.pdf copied from a log finds /My%20Doc.pdf.

The hashes are keyed with your APP_KEY, so they give a short secret away to nobody who does not also hold the key. Rotating the key leaves calls recorded before it unfindable by URL until they age out.

max_length#

The path column is VARCHAR(255), and paths.max_length cuts the stored value to fit. Leave it at 255 unless you have altered the column. A longer path could be rejected by the database, and because recording never lets an error reach your call, that row would be silently lost.