Choosing the right OpenAI model on Bedrock: cost per outcome, not per token
Beyond the headline price‑per‑token, the real question for most teams is how much does a correct outcome cost? A recent benchmark from Amazon’s Machine Learning blog measured exactly that across five OpenAI‑derived models – three on Bedrock (gpt‑5.6‑luna, gpt‑5.6‑terra, gpt‑5.6‑sol) and two on the public OpenAI API (gpt‑5.4‑mini and gpt‑5.4‑nano). The results expose trade‑offs that spreadsheets of token rates simply hide.
How the test was run
The authors built a single‑code harness that calls the OpenAI Responses API, swapping only the backend endpoint and model ID. All other logic – prompts, evaluation scripts, and even the LLM judge (gpt‑5.5) – stayed constant. Three workloads were covered:
- Single‑call accuracy on academic benchmarks (AIME math, GPQA Diamond science, MMLU‑Pro).
- Multi‑turn agent tasks using the DeepSearchQA web‑research suite.
- Professional document production with the GDPval rubric set.
For Bedrock models reasoning was disabled; the OpenAI baselines used their default settings. The authors stress that this reflects deployment configurations, not a pure “intrinsic capability” test.
Cost per correct answer on straight‑call tasks
The benchmark divides total spend (right + wrong attempts) by the number of correct answers. After Amazon’s July 30 2026 price cut (luna ‑80 %, terra ‑20 %), the observed costs were:
| Model | AIME accuracy | Cost per correct answer |
|---|---|---|
| gpt‑5.6‑luna | 75 % | $0.0021 |
| gpt‑5.6‑terra | 68 % (GPQA) | $0.0139 (mini reference) |
| gpt‑5.6‑sol | 75 % (AIME) | – |
| gpt‑5.4‑mini | 37 % | $0.0139 |
| gpt‑5.4‑nano | – | – |
Two takeaways emerge. First, capability tiers are evident: Luna and Sol solve roughly twice as many AIME problems as mini. Second, token efficiency mattered even before the price cut – Luna used fewer billed tokens than mini because reasoning was disabled. After the cut, Luna’s $0.0021 per correct answer made it the cheapest outcome across the board, despite nano’s nominally lower token price.
Turn‑count matters for agents
Agentic workloads resend the entire conversation each turn, so input tokens grow roughly quadratically with turn count. In the DeepSearchQA test (50 questions), turn averages were:
| Model | Avg. turns per question | Cost per passing answer |
|---|---|---|
| gpt‑5.4‑mini | 7.6 | $0.40 |
| gpt‑5.6‑terra | fewer than mini | $0.31 |
| gpt‑5.6‑luna | fewer than mini | $0.05 |
| gpt‑5.4‑nano | – | $0.07 |
Luna’s dramatically lower cost ($0.05) stems from both fewer turns and a cheaper token rate. Nano’s lower token price cannot compensate for its 18 % pass rate, which inflated its per‑pass cost. The lesson is clear: turn efficiency is a hidden pricing variable that can dominate the bill for tool‑chaining agents.
Professional document quality vs. price
The GDPval suite (48 real‑world deliverables) grades outputs against human‑authored rubrics. Pass thresholds were set at 70 % of weighted points. Results:
| Model | Pass rate | Cost per passing deliverable |
|---|---|---|
| gpt‑5.6‑luna | 56 % (27/48) | $0.010 |
| gpt‑5.4‑mini | 42 % (20/48) | $0.030 |
| gpt‑5.4‑nano | 35 % (≈17/48) | $0.012 |
| gpt‑5.6‑terra | higher F1, higher cost | – |
| gpt‑5.6‑sol | highest capability tier, premium cost | – |
Luna not only passed more documents than mini or nano but did so at the lowest observed cost per pass. The only caveat: output length was capped at 8,192 tokens, truncating a handful of higher‑tier model outputs. Lifting the cap could shift the balance.
The hidden trade‑off: turn‑count versus token price
What the numbers don’t say outright is where the savings actually come from. In the agent test, Luna’s advantage came from a combination of three factors:
- Fewer turns – each turn adds the full system prompt, tool results, and prior conversation, inflating input tokens.
- Reasoning disabled – removes extra internal steps that would otherwise consume tokens.
- Lower per‑token price – after the July price cut, Luna’s $0.22/$1.32 per M tokens (input/output) is an order of magnitude cheaper than mini’s $2.20/$13.20.
If your workload involves chained tool calls, the turn count alone can eclipse raw token price differences. Conversely, a single‑call document generator may benefit more from raw capability (accuracy) than from turn‑efficiency. The benchmark therefore suggests a two‑dimensional decision matrix: outcome complexity versus interaction depth.
What to watch next
Amazon’s price reductions are recent, and the reported figures rely on a specific region (us‑west‑2) and a snapshot of latency. Future price updates, regional pricing tiers, or changes to Bedrock’s reasoning defaults could shift the balance. Also, the benchmark used a frozen gpt‑5.5 judge; any bias in that judge propagates to the pass/fail counts. Teams should re‑run the harness on their own data, especially if they operate in regulated domains where rubric nuances matter.
Try it on your own workload today
- Clone the public repository
openai-on-aws/benchmarks-openai. - Replace the supplied test prompts with a small slice of your own tickets, research queries, or document drafts (5‑10 examples).
- Run the harness against gpt‑5.4‑mini and gpt‑5.6‑luna on Bedrock, keeping reasoning disabled for both.
- Compare cost per successful outcome and turn count for each model.
If Luna’s per‑pass cost is lower and its quality meets your internal gate, you have a concrete data point to justify migrating from the OpenAI API to Bedrock.
