GLM-5.3 on Workers AI Returned Other Sessions' KV Cache Pages

TL;DR

• Workers AI is Cloudflare's hosted inference service, and GLM-5.3 is the Z.ai text-generation model it served to customers. A KV cache is the working memory an inference engine keeps for one request: the keys and values already computed for the prompt, stored in fixed-size GPU memory pages that later requests reuse.
• Cloudflare's incident report, published on 26 October 2026, records 155 responses between 27 August and 9 September 2026 that carried data from other customers' sessions, every one of them to a prompt longer than 681,280 tokens.
• Two numbers came from different places: the configured context window was 1,048,576 tokens, and the prefill KV pool on the serving GPUs held 681,280, because SGLang sizes that pool at startup from whatever GPU memory is left after the weights load.
• In disaggregated prefill/decode serving, a request the scheduler had marked as rejected still reached the decode node, which read the stale pages left by earlier requests and answered with HTTP 200.
• Cloudflare set the model private 40 minutes after declaring the incident, raised the pool to about 2.04 million tokens and patched the rejection path; SGLang v0.5.20 retired aborted requests when their final prefill result was processed, and v0.5.21 moved rejection to admission before bootstrap, allocation or transfer.
• The same coordinated disclosure covers a second bug class, writes still in flight when a slot is freed, fixed by a deferred release that became the default in v0.5.21; the operator checks that follow are a build past pull request 41128, a router that pairs aborts, and the release metric.

When a provider serves a model on your behalf, the isolation boundary sits inside software you never see. Cloudflare published its account of a boundary that failed on its own inference stack, alongside a matching write-up from the maintainers of the engine involved. The two documents describe the same nine days from different seats, and together they say something more useful than either alone: the leak needed a capacity mismatch and an admission check to line up, and each of those has a check an operator can put in place.

A KV cache is a pool of pages that has to be handed back in order

An inference engine does not recompute a prompt from scratch for every token it generates. It stores, for each request, the attention keys and values it has already derived, in a pool of fixed-size pages inside GPU memory. Cloudflare's report is unusually direct about how that pool behaves in production: pages are assigned when a request is admitted and released when it finishes, retaining their existing contents until the next request that receives a page overwrites them. That design is common across inference engines and it is what makes continuous batching fast, because allocation happens without a memory wipe in the hot path.

Which puts the ownership bookkeeping squarely on the engine. The report names the difficulty precisely: a page's ownership is shared within a node between the offload path and token generation, and it is also shared with the remote prefill node, which holds a simultaneous mutable reference to the same memory. Get that accounting wrong and stale text becomes live state for a different customer.

The context window and the KV pool come from different places

The configured context length of GLM 5.3 was 1,048,576 tokens. The pool on Cloudflare's prefill GPUs held 681,280. The pool capacity could move when changes to the deployment altered the GPU memory remaining after the model weights loaded.

Cloudflare states that it knew about the mismatch before launch and expected the engine to reject any prompt it could not process. Its long-context tests returned successful responses, with healthy status codes and a streaming body carrying no error frames, and those results became the reason to ship. The company's own post-mortem names the hole in that test: it validated HTTP response codes and engine performance at long contexts, and it did not validate model output or coherence. A status code is a claim about the request, and here the claim was false.

The pool size deserves its own line, because it is the part that quietly changes. SGLang derives that number at startup from the GPU memory remaining after the model weights load. It sizes the pool at startup from the GPU memory remaining after the model weights load, and it prints the result in the log and exposes it as a metric. Quantization, tensor parallelism, CUDA graph sizes and model versions all move it. Nothing in the stack forces the pool and the context length to agree; Cloudflare's report says plainly that the job of checking belongs to the deployer.

How a rejected request reached the decode node with stale pages

In a disaggregated deployment the work is split. One pool of GPUs prefills the prompt and produces its KV cache, then hands that cache over RDMA, through Mooncake in this deployment, to a second pool that generates the response. Cloudflare runs GLM 5.3 this way.

The rejection path was written for a single node. When a prompt exceeds what the pool can hold, the scheduler records an error for the client and swaps the prompt for a one-token placeholder that carries the request through the normal batch machinery. In a single-node deployment, the placeholder runs, the batch finishes, and the client gets an error.

The disaggregated path admitted the rejected request. Both admission queues asked whether a request was finished, and a rejected request reported itself as live, because the rejection sat in a different field. Decode reserved eight pages for the request; prefill admitted it, ran the placeholder, transferred one page and signalled completion; decode accepted the handoff and pointed attention at all eight. Seven of those pages belonged to earlier prompts. The client received HTTP 200, the token count matched its long prompt, the answer was coherent, and the content came from other users' cache.

SGLang's write-up adds the part that matters for anyone who assumes a vendor's tests cover this: the bug had been sitting in the project's tracker since 6 July 2026, and Cloudflare found it while investigating its own incident.

The shutdown, the patch, and the upstream release

The remediation ran on one day. The report received the HackerOne submission at 23:11 UTC on 8 September 2026, began verifying it at 08:00 the next morning, declared an incident at 13:22 and reproduced the leak with two independent accounts within 25 minutes. GLM 5.3 went private at 14:00, requests returned HTTP 400, the root cause was confirmed at 14:35 and every prefill node was rolled to the corrected configuration by 14:52.

Two changes closed it, each addressing a separate condition. The configuration change made the pool hold the full context: matching the GLM 5.2 settings freed enough GPU memory for about 2.04 million tokens, roughly twice the advertised window, which means a valid prompt always fits and the rejection path never runs. The engine work proceeded in two stages. PR 36988 retired an aborted request when its final prefill result was processed. PR 38935 then moved rejection to the disaggregation admission queue, before bootstrap, allocation or transfer. Cloudflare proposed the second half upstream and notes that a decode-side length check was added with it.

The release trail is worth pinning. PR 36988 shipped in SGLang v0.5.20 and retired aborted requests after their final prefill result was processed. PR 38935 shipped in v0.5.21 and rejected them at admission before handoff. Cloudflare recommends that operators run v0.5.22 or later. It also states that GLM 5.3 was the only model affected on Workers AI, because every other disaggregated deployment there has pools larger than its context length on both sides of the handoff, and that its deployment checks now require the engine fix for every new model.

The second bug class: writes still in flight when a slot is freed

The SGLang write-up covers a family of failures that the same investigation surfaced. In prefill-decode serving the cache transfer is asynchronous: the transfer engine issues one-sided RDMA writes that stream chunks into slots decode has already reserved, while prefill computation is still running. If a request dies mid-transfer, decode historically returned its slots to the allocator right away. A write already posted to the NIC cannot be cancelled, so the late write could land in a slot that now belonged to a different request, and that request would decode on foreign keys and values.

Three properties make this hard to reason about. The allocator can mark a slot free while writes to it are in flight, so "free" carries a hidden second condition: safe to reuse once no remote writer can still touch it. Launched RDMA work cannot be recalled, in either push or pull designs, because the API has no call that cancels an individual work request already posted to a queue pair. And complete handling starts at the router, which can cancel both halves of a pair, while safety ends in the engine, which decides when memory may be released.

A deferred release keeps a slot reserved until the writes drain

The fix keeps the aborted request's slots reserved for longer than its writes can keep arriving. Decode notifies every prefill rank, each rank drops the chunks still queued and acknowledges once its outstanding count reaches zero, and decode releases on a full set of acknowledgements or on a timeout that defaults to 30 seconds. The timeout is a bounded fallback, and its expiry does not prove that outstanding writes have drained.

The evidence behind that design is worth reading as a method. The maintainers built a harness that triggers the race on demand, then reproduced it end to end on two H200 GPUs with Mooncake over RDMA: the victim counted correctly to 80, then its output turned into the donor's prompt text. A community audit published on 11 September 2026 found two further ways the race could reopen, and both were closed on main between 22 and 28 September 2026. The 30-second timeout is a bounded fallback, and the write-up says so.

What an operator can check before the next serving change

Three checks come straight out of the two documents and each one is mechanical.

Read the pool size the engine prints at startup and compare it with the context length you advertise, on both prefill and decode. That comparison is the entire difference between a rejection path nobody reaches and one that runs in production under load, and the number is already in the logs and in the metrics endpoint.

Cut your build past the pull request that carries the deferred-release metrics, then watch sglang:decode_deferred_kv_release_total{outcome} after rollout. A deployment whose releases land through the drained outcome is behaving; one whose releases cluster on the timeout is waiting out the backstop on every abort, and the write-up treats that ratio as the signal to investigate. Operators on v0.5.19 or v0.5.20 can set SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1 on both sides in the meantime, which covers decode-initiated aborts.

Run a router that pairs aborts, so a client disconnect reaches both halves of a prefill/decode pair. The engine can hold memory only after it learns a request died. The same discipline belongs in the review that runs before a serving change ships, which is what a security audit and hardening pass looks at when it compares what an engine reports with what the product promises; the earlier Netics article on SGLang's own benchmark exchange covered the performance side of the same engine.

One honest limit closes the piece. The SGLang write-up states that a deployer has no reliable way to tell whether exposure already occurred, because the corruption leaves no error and no log line. For teams running GLM 5.3 on Workers AI between 27 August and 9 September 2026, Cloudflare's report is the record to check; for everyone else running disaggregated serving, the configuration comparison and the release metric are the two instruments that catch this class of fault before a customer does.

Sources

Source: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang — blog.cloudflare.com, 2026-10-26. Companion: SGLang security update: improving request isolation — lmsys.org, 2026-10-26. Primary artifact: SGLang pull request 36988 — github.com, merged 2026-09-06.