How to Choose the Right AI Customization Level on AWS without Wasting Money
According to the AWS Machine Learning blog post “The generative AI customization spectrum: From prompt engineering to custom models on AWS”, Amazon offers a ladder of ways to adapt large language models (LLMs) – from using them unchanged to building a model from scratch. The ladder is meant to keep teams from over‑engineering a solution or under‑investing and ending up with a model that either costs too much or fails to meet domain needs.
The staircase of effort and control
AWS splits the eight steps into three buckets:
- USE – keep the model weights frozen and only change how you talk to it (steps 1‑2).
- ENHANCE – wrap the model with external data or optimisations while the weights stay frozen (steps 3‑5).
- TRAIN – actually change the weights, either by fine‑tuning, continued pre‑training, or building a model from the ground up (steps 6‑8).
Each higher step adds more engineering work, more data, higher compute cost, but also more domain specificity and lower latency variance. The core rule is to start at step 1 and only climb when the current level can’t meet accuracy, latency, or format requirements.
When a prompt is enough (steps 1‑2)
Step 1 – Use the model as‑is
A plain API call to a foundation model (Claude, Nova, Llama, Mistral, etc.) is enough for generic summarisation, translation or brainstorming. No data, no setup, just a request‑response cycle.
Step 2 – Prompt engineering and in‑context learning
If the raw output is too generic or the format is wrong, the next move is to shape the request. System prompts, few‑shot examples, and chain‑of‑thought reasoning let you steer the model without touching its weights. AWS suggests keeping prompts under roughly 2,000 tokens; beyond that you start hitting context‑window limits and the cost of each call rises.
A real‑world example is DoorDash, which built a voice‑operated contact‑center prototype on Anthropic Claude in two months by focusing on prompt design rather than any model training.
Adding knowledge without training (steps 3‑5)
When the model needs private or rapidly changing data, you move to the ENHANCE bucket. The three most common techniques are:
| Step | Technique | Typical effort | Data needed | Cost impact | When to move on |
|---|---|---|---|---|---|
| 3 | Retrieval‑augmented generation (RAG) | Set up a Bedrock Knowledge Base, index documents | Structured docs, policies, catalogs | Moderate – you pay for storage and retrieval queries | Retrieval latency exceeds SLA or model still hallucinates domain facts |
| 4 | Prompt caching & context optimisation | Cache common system prompts or few‑shot blocks | None beyond what step 2 already uses | Low – saves compute by avoiding repeated tokenisation | Need a smaller, cheaper model because caching alone can’t cut cost |
| 5 | Model distillation | Train a small “student” model to mimic a large “teacher” | Output logs from the teacher on representative queries | High upfront compute, then lower inference cost | Student fails tone, format or reasoning style required |
RAG in action – Fractal Analytics attached a Bedrock Knowledge Base to a call‑center assistant and saw a 10–15 % reduction in handling time, a 30 % lift in call deflection and over 200 k queries per month. EXL reported an 80 % cut in underwriting costs after deploying a RAG‑powered virtual assistant.
Distillation payoff – Amazon Bedrock Model Distillation (announced May 2025) claims student models can be up to 500 % faster and up to 75 % cheaper to run, with less than 2 % loss in accuracy. Goodnotes switched from a self‑hosted model on EKS to an Anthropic Claude instance after confirming the cost‑performance trade‑off.
Fine‑tuning and beyond (steps 6‑8)
If prompting, retrieval, caching or distillation still can’t meet domain‑specific terminology, you need to TRAIN the model.
- Step 6 – Fine‑tuning – Provide thousands of labeled input‑output pairs. AWS offers Parameter‑Efficient Fine‑Tuning (PEFT/LoRA) for small datasets and Full Fine‑Tuning for larger changes. Trellix saved 40 hours per integration and cut time‑to‑market by 90 % after fine‑tuning Claude for cyber‑security tasks.
- Step 7 – Continued pre‑training – Feed massive unlabeled corpora to extend the model’s knowledge base. This step is rarely needed for most business use cases because step 3’s RAG already gives fresh facts at inference time.
- Step 8 – Build a custom model (Amazon Nova Forge) – Start from scratch, which can take months and large GPU budgets. The chef analogy in the source likens this to sending a head chef to culinary school: high cost, high risk, but complete control over the menu.
The hidden trade‑offs teams usually miss
Cost vs. latency is not linear – Moving from step 3 to step 5 may look like a pure cost win because a distilled model runs cheaper, but the upfront distillation job can consume the same GPU hours as a modest fine‑tune. Teams often underestimate the engineering effort to monitor and re‑train a student model when the underlying data distribution drifts.
Data quality matters more than quantity – Fine‑tuning with a few thousand high‑quality examples typically beats a larger, noisy dataset. The source notes that many teams jump straight to fine‑tuning “when a well‑structured prompt would have solved the problem in an afternoon.” The real expense is the time spent cleaning and labeling data, not the compute.
Operational overhead – Prompt caching (step 4) is free in terms of model cost but adds a layer of infrastructure (e.g., Redis or Lambda edge) that must be maintained. For a high‑volume chatbot, the latency savings of caching can be significant, yet the added ops burden can offset the monetary gain.
Vendor lock‑in – All steps map to specific AWS services (Bedrock, SageMaker, Nova Forge). Switching providers later means re‑implementing pipelines, especially for steps that embed AWS‑specific features like Bedrock Knowledge Bases.
Quick checklist to apply today
- Run a baseline – Call a foundation model with a minimal prompt (step 1) on a representative sample of your real queries.
- Measure – Record accuracy (does the output match a known correct answer?), latency, and cost per 1 k tokens.
- Iterate prompt – Add system instructions, a few examples, or chain‑of‑thought reasoning (step 2). Re‑measure.
- Add retrieval – If the model still hallucinates domain facts, index your internal docs in a Bedrock Knowledge Base and enable RAG (step 3).
- Cache common prefixes – For high‑throughput endpoints, cache the prompt prefix (step 4) and watch latency drop.
- Consider distillation – If inference cost is >$0.02 per 1 k tokens and latency is >800 ms, spin up a distillation job and compare performance.
- Fine‑tune only if needed – When step 5 fails to meet tone or format, gather a few thousand labeled examples and launch a PEFT job.
By following this ladder, you avoid the two common pitfalls the AWS blog warns about: spending weeks on prompt engineering when fine‑tuning would have been cheaper, or paying for a fine‑tune when a simple RAG layer would have solved the problem.
