BitSpeed Inspect Can Now Fail a CI Build When a Request Gets Slower

A comparison you read once is useful. A comparison that runs on every pull request and stops the merge when something got slower is more useful. Inspect 0.2.0 adds that: compare can now act as a CI gate.

You name the intervals that mustn’t get slower, optionally with a minimum change worth caring about:

bitspeed-inspect compare --fail-if-slower=first_byte_wait,connection_setup --min-change=20ms before.json after.json

The comparison prints exactly as before, byte for byte. The gate adds one line per interval on stderr and sets the exit status your pipeline can act on.

What it looks like

Here’s the released 0.2.0 binary on the saved demo recordings, with the slow run as the candidate:

gate: connection_setup hop 0: pass (inconclusive, +0.029 ms, +7.7%) observed ranges overlap, so these samples show no slowdown
gate: first_byte_wait hop 0: regressed (higher_in_samples, +180.162 ms, +877.3%) candidate median is higher and observed ranges do not overlap
gate failed: a gated interval got slower in these samples

Exit status 3. Swap the files round and both lines pass, exit 0.

Run against our own live-site recordings, two runs of the bitspeed.org homepage a minute apart show first-byte wait 8 percent higher in the second. That’s the kind of wobble that makes naive speed checks flaky. Inspect sees the ranges overlap and passes it. A gate that fires on noise gets switched off within a week.

The rules

The gate doesn’t invent its own statistics. It fails only when compare itself says higher_in_samples: the candidate median is higher and the observed ranges don’t overlap. The change also has to clear --min-change and --min-percent if you set them.

Anything compare can’t stand behind is undecided, exit 4. That covers a comparison blocked because the targets or responses differ, and runs with fewer than three complete samples, failures or a cancelled batch. That last case came up in an independent review before release. The first version of the gate let a 180 ms slowdown through as a pass when the run had been cut short, because compare marks withheld verdicts “inconclusive”. The review caught it, the gate now treats them as undecided, and a second review confirmed the fix. If you’d rather not block on undecided results, --allow-undecided lets them through.

An interval that never happened on either side, like TLS on a plain HTTP redirect hop, passes as not applicable. Nothing there can have got slower.

In a pipeline

Record a baseline from your main branch, record the candidate from the pull request on the same runner type, and compare:

bitspeed-inspect record --samples=10 https://staging.example.com/ > after.json
bitspeed-inspect compare --fail-if-slower=first_byte_wait --min-change=20ms baseline.json after.json

Ten samples is the default and a sensible minimum. Shared runners are noisy, so expect overlapping ranges, and passes, more often than you’d like. That’s the honest answer. Nothing else changed in 0.2.0, and recordings made with 0.1.1 work as they are. Get it from the download page.