Why does GPT-4 cost $10 per million output tokens but $2.50 per million input?

OpenAI charges 4× more for output tokens than input tokens. Anthropic charges 5×. Google, 8×. DeepSeek, 4×. Meta-via-Together is the lone exception, and only because they bill by GPU-hour rather than per-token. The pattern holds across every other commercial provider.

A reasonable first reaction is suspicion: providers raised the output price because they could. The data here only shows that they all did the same thing, which is what you’d expect from a cartel — or from a physical constraint. The rest of the page argues for the second reading.

Across providers and model families, output tokens cost 4× to 8× more than input tokens. The ratio is too consistent to be a marketing decision.

Here’s the asymmetry in one diagram. When you send 500 input tokens, the model processes them all together — the matrix multiplications are wide enough to cover the whole prompt in essentially one parallel pass. When the model produces 500 output tokens, it runs the model 500 times, one pass per token, because each token has to be sampled before the next pass can start.

This is the architectural truth behind the price. Same model, same GPU, same prompt. The number of forward passes through the network differs by hundreds. Output costs more because output is more work.

500 output tokens require 500 sequential forward passes; 500 input tokens require roughly one parallel pass. The work isn’t the same.

If the work were 500× as much you’d expect output tokens to be 500× more expensive, not 4–8×. They aren’t, because each forward pass during decode is much cheaper than the prefill pass. To see why, look at what a single decode step actually does on a GPU.

A 70B model in FP16 weighs about 140 GB. To produce one token the GPU has to stream those weights from HBM through the matrix multipliers and back. On an H100 SXM that takes ~42 ms per step — about 24 tok/s for a single user. Switch GPUs in the widget; the step time tracks the bandwidth column on each spec sheet, almost exactly. Compute (TFLOPs) barely moves the number. Bandwidth does.

This is the key idea you’ll spend the next stage exploring. For now: decode tokens are cheap-per- step but slow-per-user. Many short waits, one after another.

Decode tok/s is bounded by how fast the GPU can stream the weights once per token — decode_tps ≈ HBM bandwidth ÷ model bytes.

Cost is rented GPU time divided by tokens produced. If you’re paying a few dollars per hour for a GPU and decoding at tens of tokens per second, every output token costs you that hourly rate divided by tokens-per-hour. Rescale to a million tokens and you get $/M output. The same arithmetic on prefill TPS gives $/M input. Two numbers, one formula.

The formula is the whole pricing story. A single GPU dollar-per-hour figure plus two throughput numbers (prefill TPS and decode TPS) determine the entire price card. The $/hr is set by the cloud; the throughputs are set by hardware specs and model architecture. Nothing else enters.

Cost per million tokens equals dollars per GPU-hour divided by tokens-per-second, rescaled by 3,600 seconds per hour and a million-tokens-per-million.

The ratio of output cost to input cost equals the ratio of prefill TPS to decode TPS. That ratio is a property of the hardware-and-model combination, not the provider. For a 70B model on an H100 it sits around 100×–500× depending on prompt length and engine; commercial providers price the output at only 4–8× the input. They’re already absorbing most of the asymmetry into the input price.

Pick different model + GPU combinations in the widget and watch the ratio. It moves with hardware choice but stays in the same general band. The provider’s job is to find a price card customers will accept; the shape of that card — output more expensive than input — is fixed before the provider gets a vote.

Provider pricing reflects the prefill-versus-decode asymmetry, with the input price subsidising the output price by a factor that varies but rarely shrinks below 2×.

The takeaway isn’t that providers are charity. It’s that the compute asymmetry forces the shape of the price card before any business decision happens. Output tokens are physically more expensive to produce, and every provider compresses that fact into the same general-purpose 4–8× ratio because the market won’t accept 100×.

The “output is marked up” framing has the sign backwards: relative to true compute cost, output is underpriced. The asymmetry survives because customers already think 4× is steep.