The SageMaker AI Inference Router Moves Requests Across Regions After Errors or Timeouts
Amazon SageMaker AI made cross-Region inference routing generally available on 18 November 2026. Salesforce's failover test shows how the router shifts traffic and what it leaves to you.
TL;DR
- The SageMaker AI inference router is a regional Amazon Web Services resource that receives an application's inference calls and forwards each request to one of the endpoints registered for that model, which can sit in other Regions and other AWS accounts.
- Cross-Region inference routing became generally available on 18 November 2026 in every commercial Region where SageMaker AI serves real-time inference, while Salesforce serves models across more than thirty Regions and hundreds of accounts.
- A routing target names one logical model and lists its destinations by full ARN; a resource-based endpoint policy on the owning account lets one named router invoke a remote endpoint.
- The router moves traffic when recent requests fail with platform faults, 5xx responses, throttles or timeouts, typically within about a minute, and it keeps traffic in place when the only symptom is slow model responses.
- Salesforce's failover test pushed probe p50 latency from 0.2 seconds to 45 and 58 seconds during five minutes of saturation, then saw the first requests arrive in us-east-2 about two minutes after errors began and roughly 78 percent of probe traffic served there within another minute.
- Every response carries the invoked resource ARN and a routing reason, and the router publishes per-destination CloudWatch metrics, which turns a traffic shift into something a team can explain after the fact.
- Model deployment and capacity remain the platform team's responsibility, the CrossRegionTrafficConfig cap limits inbound remote traffic per endpoint, and each Region still needs the same artifacts and its own autoscaling headroom.
Salesforce's Agentforce Foundation team serves models from the Region a request originates in, across more than thirty global Regions and hundreds of AWS accounts. Each customer has its own peak window and those windows overlap, so provisioning for the combined peak costs more than the traffic justifies, and provisioning per Region leaves idle instances in one place while another runs out. That gap between fragmented capacity and availability is the problem AWS is selling the router against.

Why thirty Regions split one model's capacity
Before the router, Salesforce routed overflow through its own Internal Gateway, with health checks, retries and Region-switching logic that the team built and operated. Two gaps made that path expensive. The gateway had no capacity signal from the local or the remote endpoint, so it could not tell when to move traffic or whether the destination had room. Cross-Region traffic also had to leave the VPC through a NAT gateway and an internet gateway and travel over public networking. In design reviews, Salesforce's team raised a third problem that any failover design inherits: shifting load during an incident can overload the Region that receives it, so the receiving side needs a ceiling.
What a routing target registers
The router is a regional resource in the customer's account. Applications invoke the router with a routing target name and the same body and content type they already send to InvokeEndpoint. The routing target is a name for one logical model, such as a classifier, and it lists the endpoints and inference components that may serve it by full Amazon Resource Name. One router holds one routing target per model, so a problem with one model's destinations leaves the others alone, and a target can mix single-model endpoints with inference components.

Because a destination is identified by ARN, it can live in another account. The owning account attaches a resource-based policy that allows the router account to invoke its endpoint, with a condition that limits access to one named router, so the endpoint owner keeps the revoke switch. This is the piece that makes the pattern usable inside a platform team: the model owner authorises one consumer, and the consumer does not need credentials across the whole account.
How the router decides a destination is struggling
The router determines health from recent request outcomes. Platform faults, 5xx responses from the model container, throttles and timeouts lower a destination's standing, and as they persist the router sends a growing share of new requests elsewhere, typically starting within about a minute and taking several minutes for a destination with little traffic. After recovery, traffic returns more slowly than it left, which is what keeps the system from oscillating.
Two exclusions carry most of the design intent. Client errors such as invalid input move nothing, because the same request would fail identically in any Region. Model latency moves nothing either, because a slow answer often reflects a long prompt and says little about endpoint health. The documentation limits the router to one retry on another destination when the destination did not run the model, for example when it throttled the request.
What the Salesforce failover test measured
The team ran two configurations. First, same-Region routing inside us-west-2, where the gateway called the local router and the router sent requests to the local endpoint. Then a cross-account, cross-Region setup in which the router in us-west-2 also listed an endpoint in us-east-2 owned by a separate infrastructure account. Before sending production traffic they called DescribeInferenceRouter and confirmed both destinations reported Routable.
The failover test flooded the local endpoint and watched with a separate probe. Probe p50 latency rose from 0.2 seconds to between 45 and 58 seconds across about five minutes of saturation, and traffic stayed in us-west-2 because successful slow responses count as healthy for routing. Once requests exceeded the model container's 60-second handler timeout and started returning errors, the router began to shift. The first requests reached us-east-2 about two minutes after the first errors, and within another minute roughly 78 percent of probe traffic was served there. The remaining 20 to 25 percent stayed local so the router could detect recovery.

The slice that stayed local matters as much as the share that moved. That 20 to 25 percent of probe traffic kept touching the impaired endpoint and supplied the request outcomes the router uses to notice a recovery. Keeping 20 to 25 percent of probe traffic local preserved the request-outcome signal used to detect recovery. The local share follows the router's assessment of destination health.
With traffic moved, p50 latency returned to 0.2 seconds and the cross-Region path added under 100 milliseconds at p50. After the local endpoint recovered, traffic returned to us-west-2 gradually over about ten minutes, and the probe recorded no errors during failover or failback. Salesforce plans to repeat the test with a longer saturation window, measure cross-Region latency at millisecond precision, and inspect cascading behaviour at the traffic cap before widening the rollout.

What every response and every metric reports
Each response carries InvokedRoutedResourceArn and RoutingReason, mirrored in the X-Amzn-SageMaker-Invoked-Routed-Resource-Arn and X-Amzn-SageMaker-Invoked-Routing-Reason headers. The reason takes one of four values: local-preferred for a healthy local destination, failover when a local destination exists and is not serving reliably, capacity-available when no local destination could take the request, and only-healthy when a single eligible destination was the only healthy one. Because the ARN carries Region and account, one log line answers whether a request left its boundary and why.
The aggregate view comes from OpenTelemetry metrics published to CloudWatch in the router's own account and Region: invocations per destination with an error.type attribute splitting 4xx, 5xx including timeouts, and model errors; destination.duration; overhead.duration for the time the router itself adds; and destination.status reporting healthy, overloaded or degraded. Because those metrics sit next to each endpoint's existing SageMaker metrics, teams can query them together in a cross-account dashboard.

Where data residency and the traffic cap sit
Salesforce kept its residency check in the gateway and pinned tagged requests to an in-Region routing target whose only destination is the local endpoint. Those requests never cross a boundary, which is the arrangement a European or public-sector customer will want examined before adoption. Everything else goes to the router with the body, content type and accept type the gateway already sent.
The cap is the second control. CrossRegionTrafficConfig on the receiving endpoint sets a per-instance request rate and the largest share of that endpoint's capacity remote traffic may take, and local traffic never counts against it. When remote traffic reaches the cap, the endpoint stops accepting requests from other Regions and the router looks elsewhere, which is how the design avoids the cascading failure Salesforce's reviewers predicted.
The work the router leaves to the platform team
Three obligations survive the managed layer. The same model, container image and artifacts have to exist in every registered Region, because each destination uses the artifacts deployed in its own Region and a destination running a different version answers the same request differently. Each Region has to be sized for its own traffic plus the overflow it expects to absorb, with autoscaling headroom, and its endpoint autoscaling policy supplies additional instances as traffic grows. And the cap has to match a real capacity calculation, since a receiving Region that cannot scale fast enough simply returns its own errors.
Richard Chang, a software engineering architect at Salesforce, frames the outcome in terms of what the team stopped maintaining: "we can add capacity securely and keep our data residency commitments". The engineering reading is narrower and more useful. A routing layer relocates the queue to capacity sized in another Region, and the cap is the part of the design that keeps a bad minute in one Region from becoming a bad hour in two.
Teams that already plan regional capacity for sovereign infrastructure and cloud exit work gain one thing from the router: a model's placement becomes a routing policy with an audit trail, and the questions to answer are which Regions may serve which requests, how much remote traffic each may absorb, and who reads the routing reason when a shift happens.

Sources
- How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI — AWS Machine Learning Blog, 18 November 2026
- Route requests across AWS Regions with cross-Region inference routing — Amazon SageMaker AI Developer Guide
- CreateInferenceRouter — Amazon SageMaker API Reference
Source: "How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI" — aws.amazon.com/blogs/machine-learning, 2026-11-18, https://aws.amazon.com/blogs/machine-learning/how-salesforce-is-advancing-global-scale-model-serving-with-cross-region-inference-routing-in-amazon-sagemaker-ai, retrieved 2026-11-18, with the SageMaker AI Developer Guide page at https://docs.aws.amazon.com/sagemaker/latest/dg/inference-router.html; both saved under sources/2026-11-19/sagemaker-routing/.