How to Share a GPU Cluster Across Teams with SageMaker HyperPod and Keep Costs Fair

According to the AWS Machine Learning blog post, Amazon SageMaker HyperPod can be turned into a multi‑tenant GPU farm that lets data‑science, computer‑vision, and research groups work side‑by‑side without stepping on each other’s toes.

In this test, two fictitious teams – Team A and Team B – each got their own Kubernetes namespace, their own SageMaker Studio domain, and a separate cost bucket, all on a single HyperPod EKS cluster. The result was a shared cluster that stayed fast, stayed private, and produced per‑team spend reports that could be fed directly into internal chargeback dashboards.

---

Architecture Overview

The reference setup stacks three layers on top of a single HyperPod‑managed EKS cluster:

1. Authentication – AWS IAM Identity Center (formerly AWS SSO) federates with the corporate IdP (e.g., Microsoft Entra ID). Users log in either through the CLI (aws sso login) or the SageMaker Studio web UI.
2. Authorization – Per‑team IAM roles feed into two downstream controls: service‑level permissions (S3, CloudWatch, SageMaker) and Kubernetes RBAC policies that bind those roles to a dedicated namespace.
3. Cluster Services – HyperPod adds two platform layers: Observability (metrics, dashboards) and Task Governance (quota caps, priority queues). Below them sit the isolated namespaces where each team runs Spaces, distributed PyTorch jobs, and inference endpoints.

The diagram in the source shows the flow from the user, through Identity Center, into a team‑specific SageMaker AI domain, then into the EKS cluster. The same path works for both GUI and CLI traffic, which means teams can pick the tool they prefer without breaking the isolation model.

---

Setting Up Identity and Access

The first practical hurdle is wiring corporate identities to AWS. The blog walks through three steps that work with any SAML 2.0 IdP:

To keep things tidy, create a group per team in the corporate IdP (TeamA, TeamB, Admin). Enable SCIM provisioning so that any change in the IdP automatically creates or removes the matching group in Identity Center. Then attach a permission set to each group that grants the CLI role the eks:AccessKubernetesApi and eks:MutateViaKubernetesApi actions. The same group also gets a SageMaker domain that points to a team‑specific execution role.

---

Enforcing Namespace Isolation and Fair Scheduling

Kubernetes namespaces are the workhorse for isolation. After the IAM roles are in place, map each role to a namespace‑scoped RBAC policy. A typical policy looks like:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-a‑binding
  namespace: team-a
subjects:
- kind: User
  name: "arn:aws:sts::123456789012:assumed-role/TeamA-permissionset-role"
roleRef:
  kind: Role
  name: team-a‑admin
  apiGroup: rbac.authorization.k8s.io

Because the binding is limited to namespace: team-a, any kubectl or Studio request that tries to touch team-b resources is rejected outright. HyperPod Task Governance sits on top of the scheduler and enforces per‑namespace quotas (e.g., 4 GPUs max per team) and priority classes (training jobs get higher priority than ad‑hoc inference). The trade‑off is that you must pre‑define quota limits; if a team needs a burst, an admin must raise the limit manually or approve a temporary override.

---

Cost Allocation Made Visible

One of the biggest complaints about shared GPU farms is the lack of transparent billing. The reference architecture solves this by tagging every Kubernetes resource with the team name and enabling namespace‑level cost allocation in the AWS Cost Explorer. The steps are:

1. Turn on Cost Allocation Tags for kubernetes.io/namespace and a custom Team tag that you add to each pod spec.
2. In the SageMaker console, enable AI Domain cost reporting so that Studio‑generated resources also inherit the tags.
3. Build a simple weekly report that groups spend by tag, then feed the numbers into your internal chargeback model.

In practice this means you can answer the question “How many GPU‑hours did Team A consume last month?” with a click, and you can trace any unexpected spike back to a specific pod or notebook.

---

What Actually Changes and What to Watch Next

The headline claim – “share one GPU cluster across teams without losing isolation or cost visibility” – holds up, but the shift in day‑to‑day operations is subtle. Teams no longer need dedicated on‑prem GPUs or separate managed clusters; instead they inherit a shared pool that is governed centrally. The upside is lower capital expense and faster provisioning of new domains. The downside is added dependency on the central IAM/Identity Center configuration: a mis‑configured permission set can lock an entire team out of the cluster.

Key trade‑offs:

• Operational overhead vs. agility – Setting up the IAM groups and RBAC policies takes a few hours, but once in place, onboarding a new team is a matter of adding a group in the IdP.
• Quota rigidity vs. burst flexibility – Fixed namespace quotas protect against a runaway training job, yet they can frustrate researchers who need a sudden surge of GPUs. Consider implementing a request‑for‑quota‑increase workflow (e.g., a Slack bot that calls an AWS Lambda to adjust HyperPod Task Governance limits).
• Visibility vs. privacy – Detailed cost tags give finance the data they crave, but they also expose usage patterns. If you have strict confidentiality requirements, you may need to mask the tags for certain teams.

Who should care: Cloud architects and ML platform engineers will find the most value, because they are the ones balancing cost, security, and usability. Individual data scientists can ignore the IAM plumbing as long as their domain works; they just need to be aware of the quota limits.

What to watch: Upcoming updates to SageMaker HyperPod include dynamic quota scaling based on real‑time utilization, and tighter integration with AWS CloudWatch Metrics for per‑GPU‑hour billing. Keep an eye on the HyperPod release notes and test any new feature in a staging namespace before rolling it out cluster‑wide.

---

Quick Start Checklist for Today

1. Create groups in your corporate IdP for each AI team.
2. Enable SCIM sync to AWS IAM Identity Center so the groups appear automatically.
3. Define a permission set for CLI access that includes eks:AccessKubernetesApi and eks:MutateViaKubernetesApi.
4. Spin up a SageMaker AI domain per team and attach the matching execution role.
5. Add a namespace in the HyperPod EKS cluster for each team and bind the IAM role to that namespace via a RoleBinding.
6. Turn on cost allocation tags for kubernetes.io/namespace and add a Team tag to pod specs.
7. Run a test job (e.g., a short PyTorch training run) from both the CLI and Studio to verify that:
   • the job lands in the correct namespace,
   • the GPU usage appears in Cost Explorer under the right tag, and
   • a job from Team A cannot list pods in Team B.

If all six steps succeed, you now have a shared, isolated, and billable GPU cluster that any team can use without stepping on each other’s toes.

---

Sources

• Share GPU clusters across teams with isolation and fairness using Amazon SageMaker HyperPod