Amazon Web Services introduced SageMaker HyperPod Global Inference Gateway on November 20, 2026, giving organizations one managed endpoint for inference requests across registered HyperPod clusters in one or multiple AWS Regions. The service handles cluster selection and automatic failover, while the existing gateway inside each cluster continues routing requests to model pods.
AWS says the gateway supports OpenAI-compatible requests, authenticates callers through AWS Signature Version 4 (SigV4) and IAM, and forwards requests to clusters over AWS PrivateLink. It is available in 13 Regions across the Americas, Europe and Asia Pacific, according to the AWS announcement.
The consequential change is where routing logic lives. Applications no longer need to choose among cluster endpoints or maintain their own cross-Region switching logic. Operators still decide which capacity is available and how routing priorities should work.
One Endpoint, Two Routing Decisions
AWS illustrates the endpoint problem with four models deployed across five HyperPod clusters in two Regions. Without an inference gateway, clients would track 20 endpoints: one per model per cluster. HyperPod’s existing per-cluster Inference Gateway reduces that to five, but clients must still select a cluster.

The global gateway adds a routing layer above those cluster gateways. A request follows this path:
- The client signs the request with SigV4 and sends it to the gateway’s regional endpoint.
- The global gateway authenticates the caller and selects a cluster, using health and load signals.
- AWS PrivateLink carries the request to that cluster, within the same Region or across Regions.
- The cluster’s existing Inference Gateway selects a model pod.
This preserves the separation between choosing a cluster and choosing a pod. AWS operates the global routing fleet, including TLS termination, authentication, cluster scoring and request forwarding, as described in its architecture walkthrough.
OpenAI compatibility applies to the request interface, not the authentication system. Applications still need AWS signing and appropriate IAM permissions; an OpenAI-style request body alone is insufficient.
One naming detail also matters: the “global” gateway receives requests through a regional endpoint. Cross-Region destination routing should not be read as a guarantee that the entry endpoint itself survives every possible regional disruption.
GPU-Aware Routing With Adjustable Weights
The gateway considers more than whether a cluster responds to health checks. A cluster-level metrics publisher supplies load information so the service can account for available inference capacity.
AI.info’s report on the launch lists four scoring signals and their default weights:
| Routing signal | Default weight | Practical purpose |
|---|---|---|
| KV-cache and GPU memory headroom | 0.20 | Accounts for memory available to serve requests |
| Queue depth | 0.25 | Accounts for requests already waiting |
| Regional proximity | 0.40 | Gives locality a role in cluster selection |
| Recent error rate | 0.15 | Accounts for serving reliability |
KV cache is the memory used to retain attention keys and values during language-model generation. That cached state avoids recomputing earlier work, but consumes memory as requests progress. Available cache space therefore tells a router something useful that a simple healthy/unhealthy check cannot: whether a running cluster has room for more inference work.
Customers can override the weights. According to AI.info, every value must fall between 0.0 and 1.0, all four must be supplied when customizing them, and they must add up to 1.0. Setting a weight to zero disables that signal.
These are scoring weights, not traffic percentages. A proximity weight of 0.40 does not mean 40% of requests stay local. It is the largest individual default weight, while the other three collectively total 0.60; the values alone cannot predict the destination of a particular request.
I’d treat this configuration as a capacity-versus-locality control, not a data-residency policy. A gateway allowed to use clusters in multiple Regions can send work across those boundaries. For a workload that must remain within one Region, restricting the registered cluster set is the clearer architectural boundary.
Publishing Clusters Creates Resources in Your Account
Setup has two stages: publish each eligible cluster, then create the gateway that groups them.
Operators first enable the spec.globalRouting block in the cluster’s existing InferenceGatewayConfig. The HyperPod Inference controller then creates or publishes the resources needed to make that cluster privately reachable:
- A Network Load Balancer in front of the cluster’s Inference Gateway.
- A VPC endpoint service for private connectivity.
- The listener’s certificate, uploaded to the customer’s TLS bucket.
AWS says these resources reside in the customer’s account. Operators then call CreateGlobalInferenceGateway with the clusters they want to register, and SageMaker returns the managed endpoint. Listing an unpublished cluster causes creation to fail with a message identifying what needs fixing, according to the setup instructions.
The operational distinction is worth spelling out: this removes the need to design and maintain a custom cross-Region load-balancing system, but it does not mean there are no load balancers or networking resources. The controller provisions supporting infrastructure in the customer account, while AWS operates the managed routing layer.
That division should simplify application code. Infrastructure teams still need to account for the resulting resources in their security, inventory and cost-management processes.
Automatic Failover Still Needs Spare Capacity
The gateway automatically moves traffic away from a failed cluster to another suitable registered cluster. Clients continue using the same endpoint.
Operators can also change routing weights after creation, add a prepared cluster or detach an existing one. Newly added clusters are validated and begin receiving traffic once healthy, AI.info reports.
AWS includes a beta account from Bagelcode’s VP of AI Platform, Jinhyung Park, who said the gateway routed requests across two Regions and:
“kept serving them when one cluster could no longer serve the model.”
That is an attributed customer experience in the AWS announcement, not an independent reliability measurement. The supplied sources provide no failover-time or latency benchmarks, and do not explain recovery of an already-running response stream.
Frequently Asked Questions
3 questions
1Can the Gateway Serve Clusters in One Region?
Yes. AWS says a gateway can route across registered clusters within one Region or across multiple Regions.