SOCI‑indexed containers shave minutes off SageMaker jobs

Every SageMaker training or processing job starts by pulling a container image from Amazon ECR. When the image is 10 GB or larger, that pull can take several minutes and is billed as job time. AWS announced Seekable OCI (SOCI) indexing for SageMaker containers, a lazy‑loading mechanism that moves most of the download work off the critical path, letting jobs start in seconds.

How SOCI indexing works

SOCI is an open‑source format that adds a small index file to a container image. The index lists every file inside the image and tells the runtime which files are needed to launch the container. When a SageMaker job starts, the service checks whether the specified image has a SOCI index. If it does, SageMaker fetches only the handful of files required for the container’s entry point, starts the container, and streams the remaining layers in the background as the code accesses them. If the index is missing or unreadable, SageMaker falls back to a normal Docker pull, so existing workflows never break.

What the benchmark numbers actually mean

AWS measured eight images ranging from 1.8 GB to 35.4 GB compressed on a variety of GPU and CPU instance types. The median download time for a standard pull grew with image size, while the lazy‑loaded version stayed between 3 and 11 seconds regardless of size. The table below reproduces the key figures:

Across 13 instance‑type/size combinations, the 13.2 GB image consistently dropped from 3 min 16 s–4 min 20 s (standard) to 6‑8 s (SOCI). The benchmark also showed that after the first training epoch, per‑epoch time was identical between the two approaches. The only extra cost was a 3‑16 second “first‑epoch penalty” while the lazy loader fetched the files the framework accessed for the first time.

Trade‑off: first‑epoch delay vs overall savings

In practice the trade‑off is simple: you pay a one‑time extra few seconds at the start of the job to avoid minutes of idle time later. The net saving is the download time minus the first‑epoch penalty. For a 13 GB image on a ml.p5.48xlarge instance, the job finished 1‑6 minutes sooner, depending on how much of the image the code touched before training began.

If your entry point eagerly scans the whole image—e.g., checksum verification, pre‑loading weights, or custom code that reads most files—SOCI’s benefit shrinks because the lazy loader ends up fetching the same bytes anyway. In the AWS tests, even a 6 GB model baked into the image added 2‑3.5 minutes of first‑epoch latency, yet the overall job still finished about 2‑2.5 minutes faster. The rule of thumb is: the less you touch the container at launch, the larger the win.

How to enable SOCI on your own jobs

There are two paths:

1. Use a pre‑indexed AWS Deep Learning Container – Amazon publishes SOCI‑enabled variants of popular DL containers. The tag ends with -soci (e.g., pytorch-training:2.9-gpu-py312-cu130-ubuntu22.04-sagemaker-v1-soci). Switching to the new tag is the only change required in your job definition.
2. Create a SOCI index for a custom image – On a Linux host with containerd, the SOCI snapshotter, and either finch or nerdctl installed, run nerdctl image convert --soci to produce a new tag (e.g., my-image:v1-soci) and push it to ECR. The original image digest stays the same; the index is stored as a separate artifact.

Both methods need no code changes. For CI/CD pipelines, AWS offers a CloudFormation template that watches ECR push events and automatically builds a SOCI index, so you can keep the workflow fully automated.

What to watch next

The lazy‑loading model depends on the underlying ECR and network performance. As container sizes keep growing—especially with large language models bundled inside the image—the relative gain of SOCI will increase. However, any future changes to the OCI spec or to SageMaker’s runtime could affect compatibility. Keep an eye on:

• Updates to the SOCI snapshotter and its supported platforms (arm64 is still limited).
• Changes in SageMaker’s fallback logic; a regression that skips the fallback could cause job failures.
• Vendor‑provided pre‑indexed images for emerging frameworks (e.g., vLLM, SGLang) that might incorporate additional optimisations beyond SOCI.

Try it today

Pick one of your regular training jobs, note the container image tag, and replace it with the -soci variant if it exists (check the AWS Deep Learning Containers Gallery). Run a quick experiment on a small dataset and compare the total job duration with the original tag. If you build custom images, follow the step‑by‑step nerdctl commands in the blog post to produce a SOCI‑indexed tag, push it, and rerun the same job. The difference should be visible within a single run, and you’ll instantly see whether the first‑epoch penalty matters for your code.

Sources

• Announcing SOCI‑indexed images for faster SageMaker AI jobs