Qwen 2.5 72B needs 72.7 GB of VRAM for its weights at FP8 (E4M3) — 74.0 GB once a KV cache is loaded. Below is the full breakdown: which GPUs fit, how many you need, what it serves, and what it costs per million tokens. Every figure comes from tokcalc's open-source capacity model.
| Quantization | Weights | With KV cache | Smallest single GPU |
|---|---|---|---|
| FP16 | 145.4 GB | 146.7 GB | B200 SXM 192GB (192 GB) |
| FP8 (E4M3)recommended | 72.7 GB | 74.0 GB | H100 SXM5 80GB (80 GB) |
| AWQ 4-bit | 40.0 GB | 41.3 GB | L40S 48GB (48 GB) |
| GPU | GPUs | Total VRAM | Decode tok/s | Aggregate tok/s | Cost / 1M tokens |
|---|---|---|---|---|---|
| RTX 4080 16GB | 8× | 128 GB | 65.39 | 6277 | $0.1062 |
| RTX 5090 32GB | 4× | 128 GB | 81.71 | 3922 | $0.1416 |
| RTX 3090 24GB | 4× | 96 GB | 42.68 | 1959 | $0.1418 |
| RTX A4000 16GB | 8× | 128 GB | 40.86 | 3922 | $0.1700 |
| RTX 4090 24GB | 4× | 96 GB | 45.96 | 2206 | $0.2014 |
| T4 16GB | 8× | 128 GB | 29.18 | 1783 | $0.2493 |
| V100 SXM2 32GB | 4× | 128 GB | 41.04 | 1970 | $0.2820 |
| A100 40GB SXM4 | 4× | 160 GB | 70.68 | 3393 | $0.2948 |
| A10G 24GB | 4× | 96 GB | 27.36 | 1313 | $0.3046 |
| TPU v6e Trillium (32GB) | 4× | 128 GB | 37.35 | 1793 | $0.4959 |
| A100 80GB SXM4 | 2× | 160 GB | 45.60 | 1094 | $0.6092 |
| L4 24GB | 4× | 96 GB | 13.68 | 656.6 | $0.6769 |
RTX 4080 16GB has no NVLink, so 8-way tensor parallelism runs over PCIe. tokcalc applies a flat 0.85× multi-GPU efficiency factor for this, which is realistic for NVLink-equipped datacenter cards but optimistic for PCIe scale-out — treat the throughput and cost figures for this configuration as an upper bound, and measure before committing.
| Context | Concurrent users | KV per request | Fits? |
|---|---|---|---|
| 4K | 41 | 1.34 GB | ✓ fits |
| 8K | 20 | 2.68 GB | ✓ fits |
| 33K | 5 | 10.74 GB | ✓ fits |
| 66K | 2 | 21.47 GB | ✓ fits |
| 131K | 1 | 42.95 GB | ✓ fits |
Yes. At FP8 (E4M3) the model needs 74.0 GB of VRAM, so it fits on a single TPU v5p (per-chip, 95GB) or anything with more VRAM. At full FP16 it needs 146.7 GB, which fits on one B200 SXM 192GB or larger.
At FP8 (E4M3) with 8,192-token context and 8 concurrent users, the cheapest configuration that fits is 8× RTX 4080 16GB (128 GB total VRAM). $0.1062 per 1M output tokens.
On 4× Cerebras CS-3 (WSE-3) at FP8 (E4M3), the aggregate throughput is 43774 tokens/sec with continuous batching. That hardware has no published rental price, so its cost per million tokens is not comparable.
Weights alone: 145.4 GB at FP16, 72.7 GB at FP8 (E4M3). Add KV cache for your context length and concurrency — every concurrent request at 8,192 tokens costs extra VRAM, so budget headroom above the weight figure.
FP8 halves the weight footprint (145 GB → 73 GB) and is faster than FP16 on Hopper and Blackwell, where it has native kernels. FP16 remains the reference quality bar; FP8 loses a small amount of accuracy in exchange for roughly doubling the batch size you can serve per GPU.
At 8× RTX 4080 16GB the marginal cost is $0.1062 per 1M output tokens, before you add idle capacity, ops staff, or egress. That is typically cheaper than frontier API pricing at sustained volume, but API calls win when your load is bursty or too small to keep a GPU busy. The Build vs Buy tab on tokcalc prices this against a specific provider for you.
The calculator uses the exact same formulas as this page — change the workload and watch every number update.
| Total parameters | 72.70B |
| Active parameters per token | 72.70B |
| Architecture | Dense |
| Attention layout | Full attention (GQA) |
| Transformer layers | 80 |
| Hidden dimension | 8192 |
| Attention heads | 64 query / 8 KV (GQA) |
| Head dimension | 128 |
| Vocabulary size | 152,064 |
| Max context (catalog) | 131K |
| KV cache per token @ 8K | 320.0 KB |
Every layer caches keys and values for the whole context, so the KV cache grows linearly with conversation length.