AWS published an implementation guide on October 29, 2026 explaining how to share SageMaker Training GPU capacity through AWS Batch quota management. The guide covers project-level allocations, automatic lending and borrowing, and preemption for reclaiming capacity or advancing higher-priority jobs.
This is implementation guidance for an existing capability. AWS announced quota management and preemption for SageMaker Training on March 25, 2026. The October article supplies configuration patterns using scheduling policies, service environments, job queues and the SageMaker Python SDK v3.
The practical distinction is between owning an allocation and temporarily using spare capacity. A team can let another project use its idle allocation without permanently giving it up. But borrowed capacity comes with an interruption risk, and job priority operates alongside those allocation rules.
For ML-platform teams, that distinction matters more than a general promise of higher GPU utilization. It determines which workloads can safely fill idle capacity and what happens when the owner returns.
How Quota Shares Divide Training Capacity
AWS Batch quota shares are virtual queues nested inside a job queue. Each share has capacity limits and a resource-sharing strategy, allowing administrators to organize allocations by team, project or workload.
The AWS Batch documentation specifies two useful limits:
- A job queue can have up to 20 quota shares.
- Each quota share must have at least one capacity limit and can have up to five, expressed as instance limits for supported SageMaker Training instance types.
These are instance-specific allocations. In AWS’s example, each project receives a limit of one ml.g5.xlarge instance. Administrators therefore need to describe their capacity in supported instance types, treating the configuration as something other than an undifferentiated pool of GPU memory.
Each share then gets one of three strategies:
| Strategy | What happens to idle capacity | Borrowing behavior |
|---|---|---|
RESERVE | Capacity stays available only to that share’s jobs | Does not enable borrowing |
LEND | Other shares can use the idle capacity | Enables lending, without enabling borrowing |
LEND_AND_BORROW | Other shares can use idle capacity, and this share can use theirs | Requires a configured borrow limit |
Here, RESERVE describes scheduling behavior. It should not be confused with purchasing or provisioning reserved GPU capacity.
AWS Batch prioritizes shares below their capacity limits as it schedules work across the queue. A share at or above its limit can use available capacity from another share when its borrowing configuration permits it. That separates a project’s allocation from the additional capacity it might temporarily consume.
I’d treat the borrow limit as an operational control, not a minor setting. It defines how far a project can expand beyond its allocation and, consequently, how much of its running workload may depend on capacity another team can reclaim.
Two Different Reasons a Job Can Be Preempted
AWS’s guide describes two preemption mechanisms. They solve different scheduling problems and should be configured with different expectations.
Cross-Share Preemption Restores Lent Capacity
Cross-share preemption happens when jobs arrive in a share whose capacity has been lent elsewhere. AWS Batch reclaims resources from borrowing workloads so the lending share can use its allocation.
The quota-share documentation says AWS considers the number and type of instances being borrowed, relative job priorities and job duration when selecting work to preempt. AWS calls this a custom heuristic; the published material does not disclose its full algorithm.
That limits how precisely administrators can predict which borrowing job will be interrupted. The documented policy is automatic restoration of lent capacity, not a simple rule such as always stopping the newest or lowest-priority borrower first.
In-Share Preemption Advances Urgent Work
In-share preemption is an optional setting that lets higher-priority work displace lower-priority work within the same quota share. Eligible lower-priority jobs can be in the SCHEDULED, STARTING or RUNNING state.
AWS’s quota-management documentation also says job priorities can be set at submission and updated later. Those updated priorities are considered in preemption decisions.
The operational distinction is straightforward:
- Cross-share preemption restores an allocation to its owner.
- In-share preemption changes which jobs get to use a share’s available resources.
Disabling in-share preemption therefore does not make borrowed capacity interruption-proof. Conversely, assigning a job a high priority should not be treated as a documented way to override another project’s allocation.
One less obvious detail deserves attention during configuration changes: lowering a share’s capacity limits can make its SCHEDULED, STARTING or RUNNING jobs eligible for preemption if its consumption exceeds the new limits. AWS documents this behavior even for shares using LEND or RESERVE. A quota reduction can affect active work, not merely future submissions.
The Resources Teams Need to Configure
Quota management is supported only for job queues connected to a SAGEMAKER_TRAINING service environment. This guide should not be read as a generic quota-sharing recipe for every AWS Batch workload.
AWS’s setup requires four connected resource types:
- A scheduling policy containing
quotaSharePolicy. This establishes quota-based scheduling for the queue. - A service environment with instance-specific capacity limits. This describes the training capacity available to the connected queue.
- A SageMaker Training job queue connected to both. The queue associates the scheduling policy with the service environment.
- Quota shares attached to that queue. Each share receives its own capacity limits, sharing strategy and applicable preemption settings.
The guide requires an AWS account with permission to create and manage the relevant AWS Batch and SageMaker resources. Its examples target the SageMaker Python SDK v3, using the aws_batch integration.
The following excerpt shows the initial client and resource-manager setup used in AWS’s walkthrough:
from sagemaker.train.aws_batch.boto_client import get_batch_boto_client
from utils.aws_batch_resource_management import AwsBatchResourceManager
INSTANCE_TYPE = "ml.g5.xlarge"
resource_manager = AwsBatchResourceManager(get_batch_boto_client())
This selects the example instance type and initializes the helper used to manage resources. It does not, by itself, create the scheduling policy, service environment, queue or quota shares.
The utils.aws_batch_resource_management import is also important: it comes from the sample notebook’s helper code, not a standalone import supplied merely by installing the SageMaker SDK. Teams adapting the example need those helpers or equivalent resource-creation code. AWS’s walkthrough also uses QuotaShareConfig and create_quota_management_resources from that helper module.
I’d keep that dependency explicit in an internal runbook. Otherwise, a short example can look ready to paste into a fresh environment while silently depending on files from the larger sample.
What the Three-Project Example Demonstrates
Frequently Asked Questions
3 questions
1Is This a New SageMaker Product Launch?
No. AWS announced quota management and preemption on March 25, 2026. The October 29 guide explains how to configure and use that capability.
