Your LLM Server Monitoring Says Fine. Your Users Say Otherwise.
I ran a test against an LLM server where every single user request got slower — the median by 8.5× — and standard LLM server monitoring barely noticed: the metric most teams put on their dashboard moved by 9%.
I ran another where a server rejected 11 of 16 requests, and every one of those rejections arrived as HTTP 200 OK.
Both of these came out of a few weeks spent tearing apart two popular LLM inference servers, vLLM and SGLang — reading their schedulers line by line, then running controlled experiments against them on pinned commits. I expected to write about throughput. Instead I found that the two most important failures these servers produce are ones your existing monitoring is structurally unable to detect.
This post is about those two failures, why each happens at the source level, and what to change. If you run an LLM endpoint where chat traffic and bulk jobs share a URL, at least one of these is probably happening to you right now.
First, where an LLM server actually spends its time
You need four ideas to follow the rest. If you already serve LLMs, skip ahead.
When a request arrives, the server does two completely different jobs.
Prefill is reading your prompt. The model ingests the entire prompt to build up its internal state. This is one big, parallel, compute-heavy pass, and its cost scales with prompt length — roughly linearly at the lengths that matter here, with an additional quadratic attention term that grows as prompts get long. But the reason one long prompt dominates a scheduling step isn't the quadratic term: it's that the scheduler works against a fixed per-step token budget, and a single large prefill can consume all of it. Hold onto that — it's what makes the chunked-prefill defaults later in this post so consequential.
Decode is writing the answer, one token at a time. Each step is individually cheap but you need one per output token, and they're sequential — token 50 can't start until token 49 exists.
While a request is alive — through prefill and every decode step — the server holds its intermediate attention state in accelerator memory. This is the KV cache — the same structure that makes prompt caching worth paying for — and it is the genuinely scarce resource. It's proportional to the number of tokens in flight, it's pinned for the entire lifetime of the request, and when it runs out the server must queue, evict, or fail. Not "gets slower" — one of those three things.
The number users actually feel is TTFT, time to first token: the gap between hitting enter and seeing the first word appear. A slow TTFT reads as "it's broken." A slow token rate after that reads as "it's thinking."
So the scheduler's real job, every single step, is deciding who gets compute and who gets memory. Everything below follows from that one decision.
Failure one: the latency p95 cannot see
Here's an experiment with exactly one variable.
Six short requests (64 input tokens, 16 output — think chat messages) plus one long request (2,048 input tokens, 64 output — think a document summarization job). Same seven requests every time. The only thing I changed was the order they were submitted: the long one first, or the long one last.
I was strict about the controls, because I got this wrong the first time (more on that at the end). The server was restarted before every arm and every repetition, prompt content was varied per repetition so nothing could be served from cache, and I alternated which arm ran first. All six server logs confirm Prefix cache hit rate: 0.0% — every run genuinely did the work.
Here's what the six short requests experienced, across three repetitions (n=18 per arm; see the note below the next table on why pooling matters for the p95 column). p50 is the median — the experience of a typical request; p95 is the 95th percentile — roughly the worst request in the sample:
| Submission order | TTFT p50 | TTFT p95 | Requests delayed > 1s |
|---|---|---|---|
| Long request first | 6.2720 s | 6.4321 s | 18 of 18 |
| Long request last | 0.7377 s | 5.8770 s | 3 of 18 |
Read those last two columns again. Putting one large request at the front of the queue delayed every single short request past a second, and made the median 8.50× worse.
And p95 — the 95th percentile, the number in most latency SLOs and on most dashboards — moved 1.09×. Nine percent. Essentially noise.
That headline is not an artifact of pooling. Here is every repetition on its own, six short requests per arm per run:
| Repetition | p50 long-first | p50 long-last | p50 ratio | Delayed > 1 s |
|---|---|---|---|---|
| 1 | 6.1442 s | 0.7423 s | 8.28× | 6/6 vs 1/6 |
| 2 | 6.3640 s | 0.7336 s | 8.68× | 6/6 vs 1/6 |
| 3 | 6.3065 s | 0.7410 s | 8.51× | 6/6 vs 1/6 |
One caveat I owe you on the p95 figure. Per repetition it reads 6.2265 / 6.4542 / 6.4056 s long-first against 4.6013 / 4.5998 / 4.6447 s long-last — a ratio nearer 1.38× than the pooled 1.09×. That gap is an estimator artifact rather than a disagreement: a 95th percentile of six samples interpolates between the fifth and sixth values, so it lands below the one genuinely stuck request, which is why the pooled n=18 figure is the better estimate. Slice it either way and the conclusion holds — p95 moves by tens of percent while p50 moves 8.5×.
Why the percentile hides it
This isn't a quirk of my data; it's arithmetic that will reproduce anywhere.
In the good arm, the long request goes last, so the short requests mostly clear in well under a second — the fastest near 0.2 s, the rest queued behind each other's prefills, for a median of 0.74 s. One per repetition still takes ~5.9 s, and it's worth being precise about why, because the submission order rules out the obvious explanation. Arrival order was correct — the long request really did arrive last. But the scheduler had not yet picked up every short request by the time it did, so one straggler was still sitting in the WAITING queue when the long prefill claimed the entire step budget, and waited out the whole thing. Perfect ordering is not enough; what matters is whether the queue has drained before the large prefill arrives. With 18 samples, the 95th percentile picks out roughly the worst one — which is exactly one of those stuck requests. So p95 reports ~5.9 s.
In the bad arm, everyone is stuck at ~6.3 s, so p95 reports ~6.4 s.
Both arms put a stuck request at p95. The percentile is faithfully reporting the tail in both cases — and the tail barely changed. What changed is how much of the population is in the tail, and a tail metric cannot express that by construction.
Head-of-line blocking doesn't make your worst case worse. It makes your typical case as bad as your worst case. p95 is blind to that specific transformation, and that is the transformation mixed traffic produces.
"But chunked prefill solves this"
This is the part I want to push back on hardest, because it's the standard answer and it did not hold.
Chunked prefill is supposed to fix exactly this problem: instead of processing a 2,048-token prompt in one indivisible gulp, the scheduler slices it across several steps and interleaves other work between the slices. vLLM has it, and in my runs enable_chunked_prefill was true.
It still happened. Two defaults are why.
The per-request clamp is long_prefill_token_threshold, and it defaults to 0 — which means disabled, because of the guard it sits behind (scheduler.py:586-590):
if 0 < self.scheduler_config.long_prefill_token_threshold < num_new_tokens:
num_new_tokens = self.scheduler_config.long_prefill_token_thresholdWith the threshold at 0, that condition is never true and the clamp never fires. Meanwhile the step-level budget, max_num_batched_tokens, resolves to 2,048 for this configuration — precisely the size of my prefill. A 2,048-token prompt fits in one 2,048-token budget, so there was nothing to chunk. (In fairness to the evidence: that 2,048 is derived by resolving vLLM's own config code at this commit, not scraped from a log line — the server doesn't print it in the logs I captured.)
The feature was on. Both parameters that would have made it do something were at values that made it inert.
There's a second reason the scheduler doesn't protect you here, and it's visible right in the source. vLLM schedules in a deliberate order — # First, schedule the RUNNING requests. and then, 200 lines later, # Next, schedule the WAITING requests.. That ordering protects requests that have already started from being starved by new arrivals. It does precisely nothing for the six requests still sitting in the queue behind a large prompt that got there first.
Failure two: the rejection that looks like a success
Second experiment, and a single burst per runtime rather than a repeated measurement — every figure in this section is one run. Deliberately shrink each server's capacity — SGLang via --max-queued-requests 4 and --max-running-requests 2, vLLM via VLLM_CPU_KVCACHE_SPACE=1 and --max-model-len 4096, which is as close an equivalent as vLLM exposes, since it has no queue-bound argument to set — then fire 16 concurrent 1,024-token requests at each and watch what the caller sees.
The two servers did completely different things — and I want to be clear that this is a difference in policy, not a performance comparison. (These ran on different backends; comparing their speeds would be meaningless, and I don't.)
vLLM never says no. All 16 requests were served successfully. TTFT stretched from roughly 2.7 s to 49 s — about a 19× spread across requests that arrived together. Server-side queue wait summed to ~328 seconds. Preemptions: zero. There is no queue bound by default, so vLLM absorbs everything, degrades smoothly, and waits for your clients to give up. Unbounded queueing is a design choice, and 328 seconds of accumulated waiting serves nobody — those users left.
SGLang does say no. It admitted 5 of the 16 and rejected the other 11 — the exact split depends on how many requests had already been promoted from the queue to running when the burst landed. Correct behavior — better than the alternative.
It just delivered the rejection like this:
data: {"error": {"object": "error", "message": "The request queue is full.", "type": "SERVICE_UNAVAILABLE", "param": null, "code": 503}}
data: {"id":"698ac1e4...","object":"text_completion","created":1787543038,"model":"default","choices":[],"usage":{"prompt_tokens":0,"total_tokens":0,"completion_tokens":0,"reasoning_tokens":0}}
data: [DONE]That is the body of an HTTP 200 OK response. All 16 requests returned 200. The rejected 11 carried zero output tokens, no finish reason, and completed in ~0.196 s.
To any client that branches on HTTP status — which is most clients, most retry middleware, and essentially every default metrics exporter — those 11 rejections are successful requests that happened to return an empty answer. Your success-rate panel stays green through a hard overload. Your p95 latency actually improves, because 11 requests just completed in 196 milliseconds.
Why it's structural, not a bug
Once you see the sequence, it's hard to see how it could work otherwise:
To stream a response you must send headers before you have any content. That header commits the status code. Whatever the scheduler decides afterwards — queue full, out of memory, client vanished — can only reach the caller through the body. In SGLang's case the rejection is raised inside the scheduler process and travels the ordinary output path as an AbortReq carrying status_code=HTTPStatus.SERVICE_UNAVAILABLE, by which point 200 has been on the wire for milliseconds.
Every streaming API you own over SSE or chunked HTTP has this shape. If you stream — over SSE or any of the alternatives — your status code describes admission and nothing else. HTTP trailers can carry a status after the body, and gRPC is the one widely deployed protocol that requires clients to read them — which is why it escapes this problem in practice, while on ordinary SSE and chunked HTTP trailers are optional and routinely ignored or stripped by intermediaries. If you are reading this about an OpenAI-compatible endpoint, you are not the exception.
Two more outcomes worth knowing, each from a single run rather than a repeated measurement. When a client disconnected mid-stream after receiving 5 tokens, SGLang kept generating for roughly 14 more seconds with no abort recorded anywhere in its log — billable compute and pinned KV cache for a user who had already closed the tab. And when the accelerator's memory allocator was exhausted beneath the scheduler, the caller got a bare TCP disconnect with no HTTP status at all.
Why LLM server monitoring misses both failures
These look like two unrelated bugs. They're one problem: we're observing in the wrong place.
A status code was designed for unary request/response, where the server knows the outcome before it replies. Streaming breaks that assumption — the status now describes whether you were let in, while the outcome lives in the body, minutes later.
A percentile was designed to catch outliers in a stable population. Contention doesn't create outliers; it shifts the whole distribution. A tail metric is the wrong instrument for a problem that moves the middle.
Both of our defaults are inherited from an era of short, unary, uniform requests. LLM traffic is none of those things: requests differ in cost by orders of magnitude, run for tens of seconds, and stream. The monitoring conventions didn't follow.
What to change in your LLM server monitoring — and your serving config
Roughly in order of payoff per hour spent:
-
Parse stream bodies. Never trust the status code alone. Treat "HTTP 200, zero tokens, no finish reason" as a failure, because it is one. Export an empty-completion rate and alert on it; if you already run trace-level observability, that is the natural place to put it. If you wrap these servers in your own API, emit a real terminal error event so your callers don't inherit this problem.
-
Alert on p50 and "fraction of requests above target," not p95 alone. Keep p95 as your SLO if you like — it's a reasonable contract. But it is a terrible detector. In my data, p50 moved 8.50× and the count of affected requests went from 3 to 18, while p95 shrugged. Graph all three.
-
Check whether your chunking is actually on. Look up
long_prefill_token_thresholdandmax_num_batched_tokens(orchunked_prefill_sizeon SGLang) and compare them against your real p99 prompt length. "The feature exists" and "the feature is doing something" are different claims, and the defaults are not on your side. -
Separate interactive and batch traffic, and defer large prefills rather than interleaving them. This is the only fix that addresses the cause — and the only one with a real cost: two pools means two sets of idle accelerators. One shared queue means one 30-second job can own the head of the line, and no amount of tuning inside the runtime changes who arrived first. But if your prompt-length distribution is genuinely uniform, a single queue is the better trade and none of this applies to you. It's the spread in request cost, not the load, that makes separation pay.
-
Bound your queue and reject early — loudly. Rejecting at admission with a real status code is kinder than 328 seconds of silent waiting. Just make sure the rejection is legible to a machine.
-
Set server-side deadlines that don't depend on client disconnect, and cap
max_tokens. At the commit I tested, vLLM ships no server-side request deadline at all — if you don't own that timeout, nobody does. -
Add idempotency keys before you add retries. A naive retry on an empty response amplifies load at the exact moment the queue is already full.
Method, and one thing I got wrong
Both repositories were pinned to full commit SHAs (vLLM 185cada36, SGLang d1af3c89), and the model — Qwen/Qwen3-0.6B — to revision c1899de289a04d12100db370d81485cdf75e47ca, which pins the tokenizer with it, since those files ship in the same repository. Generation was greedy — necessary but not sufficient for bitwise reproducibility — and every condition in the ordering experiment ran three repetitions with per-request records kept rather than just summary statistics. The overload, cancellation and allocator-exhaustion results are single observations, labelled as such where they appear.
Two honest caveats. This ran on a laptop — an M4 Pro with a 0.6B-parameter model, the sort of local setup anyone can reproduce — so the absolute numbers do not transfer to production accelerators. What transfers is the mechanisms, because those were read out of the source, not inferred from the benchmark. And the two servers ran on different backends — vLLM on its native macOS CPU path (VLLM_CPU_KVCACHE_SPACE=1, --dtype float16, --max-model-len 4096, --enforce-eager), SGLang on its in-tree MLX/Metal path (SGLANG_USE_MLX=1, --dtype float16, --disable-cuda-graph) — so I make no speed comparison between them anywhere; each result is a within-runtime observation. Both ran at fp16 deliberately: vLLM's macOS CPU backend documents FP32 and FP16 only, so fp16 is the one precision the two runtimes share, which removes weight precision as a confound. One detail of full disclosure: vLLM's effective max_num_batched_tokens=2048 is derived by resolving its own config code at that commit, not scraped from a log line, because that line isn't emitted in the logs I captured.
And the correction. My first version of the ordering experiment was invalid. Both arms ran against a single server instance with identical prompts, so the cache quietly served the second arm's big prefill — I was comparing a real prefill against a cache hit and calling it an ordering effect. An independent review caught it. I quarantined the data, rebuilt the experiment with restarts and varied prompts, and re-ran it; the numbers in this post are from that second version. The effect turned out to be real, but my original magnitudes were wrong and are retracted.
Earlier still, my measurement client read the response stream one byte at a time. Under concurrency, Python's GIL made the client the bottleneck and produced a beautiful, entirely fictional server saturation cliff — p95 of 16.09 s, 50 tokens/sec. Buffering the reads gave 0.12 s and 542 tokens/sec against the same server, unchanged.
Which is the meta-lesson, and the reason I trust the two findings above: validate the instrument before you believe the result. Most of what I initially "discovered" about these servers was something I had accidentally built myself.
