Getting SageMaker Endpoint Auto‑Scaling to Deploy Without Manual Steps

Updating a SageMaker real‑time endpoint while auto‑scaling is enabled has been a source of deployment headaches. The new CDK construct swaps the native CloudFormation scaling resources for three custom resources that deregister, update, and re‑register the scalable target in a single stack operation, eliminating the need for hand‑rolled CLI steps.

Why native CloudFormation scaling breaks on updates

When a stack creates a SageMaker endpoint together with an AWS::ApplicationAutoScaling::ScalableTarget and a matching ScalingPolicy, the resources appear in the template in the order CloudFormation chooses. If you later change the endpoint configuration (instance type, count, or environment variables) and redeploy, CloudFormation still thinks the scaling target exists and tries to update the endpoint first. SageMaker rejects the request with an error like:

The variant(s) "[AllTraffic]" must be deregistered as scalable targets with Application Auto Scaling before they can be removed or have their instance type updated.

Manually deregistering the target via the CLI lets the endpoint update succeed, but CloudFormation then skips the scaling resources because it sees no change in the scaling section. The endpoint ends up running without any auto‑scaling and the stack reports success. If you also modify scaling parameters after manual deregistration, CloudFormation throws a resource not found error and the stack can get stuck in UPDATE_ROLLBACK_FAILED, requiring console‑level rescue.

How the CDK construct solves the orchestration

The construct replaces the two native scaling resources with three custom resources:

1. Deregister – runs before the endpoint update, calls deregister_scalable_target. It catches ObjectNotFoundException so the first deployment (when no target exists) proceeds silently.
2. Register Scalable Target – fires after the endpoint reaches InService. It uses the CDK provider’s asynchronous completion pattern: the Lambda on_event returns immediately, while an is_complete handler polls every 30 seconds until the endpoint is ready. The total timeout is capped at one hour.
3. Attach Scaling Policy – runs after the target is queryable, again with an asynchronous waiter that times out after 15 minutes.

A single ConfigHash property is calculated from the combined endpoint and scaling settings. Any change to either section forces CloudFormation to treat the custom resources as updated, guaranteeing the full deregister‑update‑register cycle runs automatically.

What actually changes – the hidden trade‑off

The trade‑off is clear: you pay for a larger CloudFormation footprint and a handful of Lambda executions to gain deterministic, observable deployments. In practice the extra cost is tiny—each Lambda runs for a few seconds and the Step Functions state machines exist only as metadata. What you lose is the simplicity of a two‑resource template, but you gain safety and repeatability.

Quick start you can try today

1. Clone the sample repo

   git clone https://github.com/aws-samples/sample-sagemaker-endpoint-autoscaling.git
   cd sample-sagemaker-endpoint-autoscaling
   

2. Install dependencies (requires uv and Python 3.12+)

   uv sync
   

3. Edit config/config.yaml with your model details, instance type, and scaling limits. The file drives the ConfigHash, so any change will trigger the full orchestration.
4. Bootstrap your account (if you haven’t already)

   uv run cdk bootstrap
   

5. Deploy

   uv run cdk deploy InferenceStack
   

   CloudFormation will create the model, endpoint, and the three custom resources. When the stack finishes, verify scaling is active:

   aws application-autoscaling describe-scalable-targets \
     --service-namespace sagemaker \
     --resource-ids endpoint/<your-endpoint>/variant/AllTraffic
   

6. Test an update – change instance_type in config.yaml (e.g., ml.m5.large → ml.m5.xlarge) and redeploy with the same command. The custom resources will automatically deregister the old target, apply the new configuration, and re‑register scaling with the updated instance type.

What to watch next

The construct relies on the CDK provider framework’s asynchronous waiter. If you push the timeout limits (e.g., a model that takes longer than an hour to become InService), the deployment will fail and you’ll need to adjust the total_timeout values in the construct’s code. Also keep an eye on Lambda concurrency limits in large accounts—although each deployment only spins up a handful of functions, a burst of parallel stack updates could hit account‑wide caps. Finally, the pattern of wrapping native CloudFormation resources with custom resources is reusable; you can adopt it for other services that require pre‑ or post‑deployment actions (e.g., updating a Lambda alias after a new version rolls out).

Sources

• Orchestrating SageMaker AI endpoint auto scaling with AWS CDK