A GKE API Timeout Traced to Three Bugs in Three Layers
Google traced weeks of kube-apiserver timeouts across five GKE clusters to a dropped challenge ACK, an Envoy half-close and a kernel 6.1 ACK path, and shipped the fix in GKE 1.33.5.
TL;DR
- The Kubernetes API server is the component of a cluster that answers every request from kubectl, controllers, operators and the pods themselves.
- An API timeout means a client opened a TCP connection toward that server and never completed the handshake, so the failure sits on the path before any Kubernetes logic runs.
- Google Cloud published the case study on 30 October 2026: from January 2026, pods in five GKE clusters in europe-west intermittently failed to reach the kube-apiserver, and at its peak the incident blocked around 800 developers from deploying to production.
- The escalation timeline ran from intermittent timeouts on day 1 to simultaneous packet captures on day 22, after the custom kopf operator, External Secrets, security agents, conntrack exhaustion and control plane sizing had all been cleared as suspects.
- The captures showed a fresh SYN answered by an ACK that belonged to an earlier connection, and Private Service Connect was reusing source ports through NAT.
- A challenge ACK is the RFC 5961 reply that tells a client its remembered connection is stale, and a change in Google's user space packet switch started dropping those replies as invalid ACKs.
- Packet reordering at the PIGv moved the client FIN behind the ACKs, Envoy held the socket open with shutdown(SHUT_WR) so tcp_fin_timeout never started, and Linux 6.1 discarded the FIN after an old ACK acceptable under RFC 5961.
- The kernel fix, commit 795a7dfbc3d9 by Eric Dumazet, landed in Linux 6.9.7 with backports to 6.6.y and 6.1.y and shipped in GKE 1.33.5.
The ticket that opened this case read as one sentence: pods are intermittently failing to reach the Kubernetes API server. The investigation that followed ended with three separate bugs in three different layers, and the last of them lived in the Linux kernel's TCP state machine.
The case is useful because its diagnosis spans three layers. Each layer explained one part of the same timeout, and the full diagnosis required evidence from all three. Separating them took captures at three points in the path, correlated against a metric that had been flat for months.
Pods dial a virtual IP and Private Service Connect brings the connection to the control plane
The customer ran a large multi-cluster GKE environment in europe-west, with Private Service Connect exposing the control plane privately and a full set of system workloads: Twistlock scanners, the External Secrets Operator, Cilium for network policy and several internal operators. Pods logged TimeoutError: Connect call failed ('1.2.3.4', 443) and dial tcp 1.2.3.4:443: i/o timeout, where 1.2.3.4 is the cluster's internal VIP for the kube-apiserver.
That VIP is a virtual endpoint. From there, a pod connection crosses Google's networking fabric, Private Service Connect NAT, an internal passthrough Network Load Balancer and Envoy before reaching kube-apiserver. Private Service Connect also adds a proxy protocol header to each new connection so the API server can see the real client address.
The path is long for something a pod experiences as a local connection, and every hop on it can drop or delay a packet. Google's post states the consequence directly: this is a case where the application layer had nothing left to say, so the investigation went to the packets.
The first timeouts looked like a customer workload and the timeline cleared every suspect
The customer's first instinct was their own code. They had deployed a custom operator built with the kopf framework, and the logs showed the errors had started eight days before that deployment, and continued after it was scaled to zero.
From there the timeline worked through the obvious candidates. The External Secrets Operator was removed entirely on the customer's request and timeouts continued in workloads that only needed control plane connectivity, including cluster system pods. Security agents were uninstalled with the same result. One theory blamed conntrack table exhaustion, and the table sat at roughly 50% of a default maximum of 1.3M entries. Another blamed an overloaded control plane, so the control plane VM was resized while kube-apiserver latencies stayed healthy and the timeouts did not move.
Scope settled the question of where to look. Five clusters were affected, across DaemonSets, Deployments and system pods, and external clients such as developer laptops and CI runners hit the same wall. Google's scope evidence placed the fault on the shared path to the API server.
Simultaneous packet captures showed a fresh SYN answered by an ACK from an earlier connection
Captures taken at the control plane VM's NIC and from inside the kube-apiserver container at the same moment produced the first real signal. SYN packets from the pods arrived at the NIC, and the reply was an out-of-place ACK that belonged to a different, earlier connection. The capture isolated the stale ACK as the response. Wireshark flagged the streams with the label "TCP Port numbers reused".
The same source port was reused for a new connection, and the VM answered the new SYN with an ACK carrying the previous stream's sequence numbers. Google's own description of the pattern is that the VM responded to a fresh SYN with an ACK belonging to a connection that had ended minutes before, and that the VM never fully let go of it.
A challenge ACK is the RFC 5961 reply that clears stale TCP state
A challenge ACK is the acknowledgement RFC 5961 requires when a SYN arrives for a five-tuple the stack still remembers: the server replies with the last known sequence number, the client recognises stale state and sends a RST, and the next SYN is accepted normally. The mechanism exists to stop an attacker injecting data into a session the kernel believes is open.
That mechanism works when the challenge ACK reaches the client. Google found a counter called "Proxy protocol drop: invalid ACK packet" in the Andromeda metrics, flat for months, which spiked in February 2026 to several drops per second across the affected europe-west regions. The date matched a user space packet switch (USPS) version rollout, a change written to fix a different security issue in proxy protocol handling that also dropped ACKs arriving before an initial SYN-ACK, which is exactly the shape of a challenge ACK.
A user space packet switch change started dropping challenge ACKs
Private Service Connect uses NAT to share one endpoint across many client pods, so pods on different nodes and in different namespaces can land on the same source port after translation, and colliding five-tuples happen far more often than on a direct connection. The challenge ACK fired regularly, and the new USPS behaviour dropped it silently.
Google rolled the change back in February 2026 and the counter returned to baseline almost immediately. The customer confirmed by early March that all affected clusters had recovered, and Google is explicit in the post: the rollback restored challenge ACK forwarding, and the root cause remained unresolved. Why had the control plane VM held connections in FIN_WAIT_2 for more than two minutes?
Bug one: packet reordering at the PIGv put the FIN behind the ACKs
The answer arrived by comparing the host NIC capture with a capture at the PIGv, the programmable virtual NIC as the VM sees it. Packets were in order at the host NIC and subtly reordered by the time they reached the PIGv, a consequence of Private Service Connect's miss-thread processing: the first packet of an unseen flow takes a slower path that adds the proxy protocol header and sets up NAT state, while later packets take a fast path. A FIN or RST belonging to a flow in mid-close can therefore arrive slightly out of order relative to the ACKs around it.
Bug two: Envoy half-closes the socket and tcp_fin_timeout stays idle
Linux has a parameter for exactly this situation. tcp_fin_timeout on the control plane VM was set to 60 seconds, and it applies to orphaned sockets, which are sockets no process holds open. Envoy performs a half-close with shutdown(SHUT_WR), closing the write side while keeping the socket open to receive, which is valid TCP behaviour and the normal way to terminate half-duplex traffic gracefully.
The side effect is that the kernel never treated the socket as orphaned, so the timeout never started. Google's netstat capture shows a control plane socket sitting in FIN_WAIT_2 for more than 120 seconds while tcp_fin_timeout stayed at 60.
Bug three: kernel 6.1 discards a valid FIN after an old acceptable ACK
Even with a reordered FIN, a healthy stack should process the FIN and move the socket to TIME_WAIT. The missing piece sat in tcp_ack() in kernel 6.1, the version underpinning the Container-Optimized OS images on the control plane VMs. RFC 5961 allows an ACK that is old and inside an acceptable sequence window, and in that case the 6.1 code path returned zero, meaning the packet was not fully processed. The caller in the FIN processing path checked the return value, saw zero, and did nothing with the FIN.
Google publishes the upstream commit that fixed it, 795a7dfbc3d9 by Eric Dumazet, with the message "net: tcp: accept old ack during closing" and the explanation that tcp_ack() returns 0 when an old ACK remains acceptable under RFC 5961, which left connections stuck in FIN_WAIT_2. The bug was introduced in 6.1, fixed in 6.9.7, backported to the 6.6.y and 6.1.y branches.
The fix arrived at three levels and shipped in GKE 1.33.5
Google lists three separate actions. The immediate mitigation was the February rollback of the USPS change, which restored forwarding of challenge ACKs and let the clusters recover. The correctness fix handles the challenge ACK case in the proxy protocol layer, with a PacketDrill test added to the suite to prevent a regression. The root cause fix is the kernel patch, which ships in GKE 1.33.5 and later.
Three bugs formed one outage chain
Google attributes the outage's magnitude to the interaction of all three issues. Reordering is mildly disruptive and the kernel normally processes the FIN eventually. Envoy's half-close is correct behaviour. The kernel's handling of old ACKs acceptable under RFC 5961 would have left some connections lingering with limited impact.
Together, the three issues formed the outage chain: reordering delivered the FIN out of sequence, the kernel bug discarded it, Envoy's SHUT_WR kept the socket out of the orphan timeout, Private Service Connect reused the port, the challenge ACK carried the stale sequence number, and the USPS change dropped that reply. The client saw an i/o timeout at 443 with nothing in the logs to suggest any of it.
What this case teaches about capture points and rollouts
The published causal chain has an evidentiary limitation; the anonymized case and unpublished raw captures limit independent verification. Google needed three simultaneous captures and one metric correlation to explain the failure. Teams that operate a managed control plane can copy the habit cheaply: capture at the node and inside the container when a generic timeout resists application-layer reasoning, and keep a note of the exact window when a platform change landed in the region.
The second lesson concerns rollouts. A change that fixes one security bug in a shared packet-processing path can alter behaviour for a different class of traffic, and the metric that reveals it may be a counter nobody watches until it spikes. A counter that sits flat for months and then moves is the cheapest alarm an operator can build from.
A private, reachable control plane is an infrastructure concern in managed Kubernetes design, and the case is a useful reminder that the p99 of an API call mixes application latency with the whole path underneath. Related Netics coverage examines how GKE handles inference routing across clusters.
Sources
- When the Kubernetes API went silent: A three-layer networking mystery, Akash Thakur, cloud.google.com, 2026-10-30
- Private Service Connect overview, cloud.google.com/vpc/docs/private-service-connect
Source: the Google Cloud case study of 2026-10-30 and the Private Service Connect overview page, read on 2026-10-30. The five affected clusters, the kopf and External Secrets and security agent eliminations, the conntrack figure of about 50% of a 1.3M maximum, the control plane resize, the day 1 to day 22 timeline, the USPS invalid ACK counter, the RFC 5961 challenge ACK mechanism, the PIGv reordering on the miss path, the Envoy SHUT_WR half-close, the FIN_WAIT_2 socket, the tcp_fin_timeout value of 60 seconds, the kernel 6.1 tcp_ack() return value, commit 795a7dfbc3d9, the 6.9.7 and 6.6.y and 6.1.y backports, GKE 1.33.5 and the PacketDrill test all come from those pages. The advice on capture points and rollout counters is Netics analysis.