SageMaker’s MIG support lets you slice GPUs, but you still need a good layout
According to Artificial Intelligence, SageMaker Inference now supports NVIDIA Multi‑Instance GPU (MIG) partitioning, letting you carve a single A100 or H100 into up to seven isolated slices. The change promises higher utilization and cheaper inference, but the benefit hinges on picking the right partition layout.
What MIG actually does on a SageMaker endpoint
MIG is a hardware feature that splits a physical GPU into independent partitions, each with its own compute cores, memory, and bandwidth. In SageMaker the feature is exposed through two optional fields in the existing CreateEndpointConfig and CreateInferenceComponent APIs – no new operations or IAM permissions are required. When you create an endpoint, SageMaker carves the GPUs according to the AcceleratorPartitionLayouts you specify; each model (inference component) then declares the partition type it wants via AcceleratorPartitionType. The endpoint provisioning time grows by about 17 seconds for the carving step, after which the partitions sit idle until components are placed on them.
Existing ways to squeeze more models onto a GPU and why they fall short
| Approach | How it works | Main drawback |
|---|---|---|
| Larger single‑model endpoints with batch processing | Packs many requests into one GPU pass | Only works when models are interchangeable; fine‑tuned or per‑tenant variants still need separate GPUs |
| Multi‑model endpoints | Loads and unloads models on demand, sharing the whole GPU | No hardware isolation; a heavy request on one model can slow all others, and first‑use incurs a cold‑start delay |
| Self‑managed Kubernetes with MIG | Runs the official NVIDIA driver and device plugin to create partitions | You must maintain driver versions, device‑plugin configs, and scheduler extensions across the fleet |
All three either waste capacity or demand extra ops work. MIG on SageMaker removes the ops burden while delivering hardware isolation, but the trade‑off is you must decide a static layout when the endpoint is created.
How to pick a partition profile – the practical checklist
- Identify your GPU family –
ml.p4d.24xlargeuses an A100,ml.p5.24xlargean H100, etc. The table below lists the profiles each family supports. - Match compute vs. memory needs – Prompt pre‑fill is compute‑heavy (TFLOPs), while token decode and model state need memory. A 3‑compute‑unit slice on an A100 (mig‑3g.20gb) gives 3/7 of the cores but 4/8 of the memory, leaving no room for another slice.
- Count partitions – The maximum per GPU is seven for A100/H100, two for G7, four for G7e. Your total layout must not exceed these limits.
- Validate driver support – Some profiles (e.g., mig‑1g.10gb on P4d) need NVIDIA driver R525+. Set
InferenceAmiVersiontoal2023-ami-sagemaker-inference-gpu-4-1to get the required driver; otherwise the carve silently fails. - Run a quick load test – Deploy a single component with the chosen profile, send a burst of prompts that match your typical length, and measure p99 latency. If latency spikes, you likely need a larger compute slice or fewer concurrent copies.
What actually changes – the hidden trade‑offs
You gain isolation without managing Kubernetes, but you lose flexibility. The layout is fixed at creation time; changing it requires a new endpoint deployment, which means a brief outage while the new partitions are carved. Also, a model that needs more than one GPU still must be assigned whole devices, so MIG does not help with the largest LLMs. The cost benefit appears only when you have many small‑to‑mid‑size models that would otherwise sit idle on a full GPU. In practice, teams that already run multi‑model endpoints see a 30‑40 % reduction in GPU spend after moving to MIG, but only after they trim their partition map to match real traffic patterns. Larger enterprises may also appreciate the clean charge‑back: each slice maps directly to a cost line item, simplifying budgeting.
Putting MIG in the broader SageMaker ecosystem
If you already use auto‑scaling for endpoint traffic, you can combine it with MIG partitions to keep the instance count low while still serving spikes. Our earlier guide on Getting SageMaker Endpoint Auto‑Scaling to Deploy Without Manual Steps shows how to configure target utilization; with MIG you can set a lower target because each partition can handle its own load without affecting its neighbors. Likewise, the Amazon SageMaker Inference 2026 article highlighted new features such as serverless inference and larger instance families – MIG adds a complementary knob for those who cannot afford a full GPU per model but still need the predictability of dedicated hardware.
Try it today: a step‑by‑step starter kit
- Open the SageMaker console and create a new endpoint configuration for a
ml.p4d.24xlargeinstance. - Add an
AcceleratorPartitionLayoutsentry like:
"AcceleratorPartitionLayouts": [
{"AcceleratorPartitionTypes": ["mig-4g.20gb", "mig-2g.10gb", "mig-1g.5gb"]}
]
This gives you three slice types that together fill the A100.
3. Create an inference component for a test model and set its AcceleratorPartitionType to mig-2g.10gb.
4. Deploy the endpoint, wait the extra ~17 seconds, then send a few requests with the InvokeEndpoint API.
5. Observe the latency and GPU utilization in CloudWatch; if the GPU memory is under‑used, try a smaller slice; if the compute side shows high p99, move to a larger compute slice.
By the end of the day you’ll have a functioning MIG‑enabled endpoint and a concrete sense of whether the extra granularity pays off for your workload.
