How we cut p95 latency by 38% without touching the model
Three changes did most of the work, and none of them involved the model itself. Continuous batching, speculative decoding and region-aware routing between them moved p95 from 664ms to 412ms.
Where the time actually went
Before any of this, a third of the tail was queueing — requests waiting on a batch that had already closed. Once we could admit work into a running batch, the shape of the histogram changed more than any kernel optimisation had.
The fastest request is one that never waited in line for a batch that had already left.
Continuous batching
Instead of forming a batch and running it to completion, the scheduler admits new sequences on every forward pass. Set MERIDIAN_BATCH_MODE=continuous and the rest follows.
const region = pickRegion({
from: request.geo,
healthy: await health.snapshot(),
// saturation beats distance past ~85%
penalise: (r) => r.saturation > 0.85 ? 120 : 0,
});The routing change is the least glamorous and the most valuable — see the Next.js routing docs for the request-level hooks we used. Outbound links in prose get their arrow automatically.
What we would do differently
- Measure the queue before the kernel.
- Ship the routing change first — it was a day of work.
- Keep a per-region histogram, not a global one.