TextDense405B params131K max context

Llama 3.1 405B VRAM Requirements: How Many GPUs & What It Costs

Llama 3.1 405B needs 222.8 GB of VRAM for its weights at AWQ 4-bit — 224.9 GB once a KV cache is loaded. Below is the full breakdown: which GPUs fit, how many you need, what it serves, and what it costs per million tokens. Every figure comes from tokcalc's open-source capacity model.

Weights (AWQ 4-bit)
222.8 GB
224.9 GB with KV cache
Fits on 1 GPU?
Yes
B300 SXM 288GB or larger
Cheapest config
$0.7697
8× A100 40GB SXM4 / 1M tokens
Fastest config
33527
tok/s on 8× Cerebras CS-3 (WSE-3)
VRAM requirement by quantization
Weight footprint = params × bytes per param. The “with KV cache” column adds one request at 4K context.
QuantizationWeightsWith KV cacheSmallest single GPU
FP16810.0 GB812.1 GB— needs multi-GPU
FP8 (E4M3)405.0 GB407.1 GB— needs multi-GPU
AWQ 4-bitrecommended222.8 GB224.9 GBB300 SXM 288GB (288 GB)
Which GPU should you use?
Reference workload: 8K context, 8 concurrent users, continuous batching, AWQ 4-bit. Ranked by fitting first, then by tokens per dollar-hour.
GPUGPUsTotal VRAMDecode tok/sAggregate tok/sCost / 1M tokens
A100 40GB SXM48×320 GB27.072598$0.7697
L40S 48GB8×384 GB15.091448$1.07
RTX 6000 Ada 48GB8×384 GB16.761609$1.10
A100 80GB SXM44×320 GB17.46838.2$1.59
Intel Gaudi 3 128GB4×512 GB28.811383$1.61
H100 SXM5 80GB4×320 GB29.251404$1.98
RTX PRO 6000 Blackwell 96GB4×384 GB15.65751.0$2.22
AMD MI300X 192GB2×384 GB23.14555.3$2.50
H100 PCIe 80GB4×320 GB17.46838.2$2.65
B200 SXM 192GB2×384 GB34.92838.2$3.98
H200 SXM5 141GB2×282 GB20.95502.9$4.42
AMD MI325X 288GB1×288 GB15.41184.9$5.26

Cerebras CS-3 (WSE-3) has no NVLink, so 8-way tensor parallelism runs over PCIe. tokcalc applies a flat 0.85× multi-GPU efficiency factor for this, which is realistic for NVLink-equipped datacenter cards but optimistic for PCIe scale-out — treat the throughput and cost figures for this configuration as an upper bound, and measure before committing.

How concurrency changes with context length
On 8× A100 40GB SXM4 at AWQ 4-bit. KV cache scales linearly with context, so long contexts cut the number of users you can serve from the same GPUs.
ContextConcurrent usersKV per requestFits?
4K462.11 GB✓ fits
8K234.23 GB✓ fits
33K516.91 GB✓ fits
66K233.82 GB✓ fits
131K167.65 GB✓ fits
Frequently asked questions

Can Llama 3.1 405B run on a single GPU?

Yes. At AWQ 4-bit the model needs 224.9 GB of VRAM, so it fits on a single B300 SXM 288GB or anything with more VRAM.

How many GPUs does Llama 3.1 405B need?

At AWQ 4-bit with 8,192-token context and 8 concurrent users, the cheapest configuration that fits is 8× A100 40GB SXM4 (320 GB total VRAM). $0.7697 per 1M output tokens.

How fast can Llama 3.1 405B be served?

On 8× Cerebras CS-3 (WSE-3) at AWQ 4-bit, the aggregate throughput is 33527 tokens/sec with continuous batching. That hardware has no published rental price, so its cost per million tokens is not comparable.

How much VRAM does Llama 3.1 405B need?

Weights alone: 810.0 GB at FP16, 222.8 GB at AWQ 4-bit. Add KV cache for your context length and concurrency — every concurrent request at 8,192 tokens costs extra VRAM, so budget headroom above the weight figure.

Should I run Llama 3.1 405B in FP8 or FP16?

FP8 halves the weight footprint (810 GB → 405 GB) and is faster than FP16 on Hopper and Blackwell, where it has native kernels. FP16 remains the reference quality bar; FP8 loses a small amount of accuracy in exchange for roughly doubling the batch size you can serve per GPU.

Is it cheaper to self-host Llama 3.1 405B than to call an API?

At 8× A100 40GB SXM4 the marginal cost is $0.7697 per 1M output tokens, before you add idle capacity, ops staff, or egress. That is typically cheaper than frontier API pricing at sustained volume, but API calls win when your load is bursty or too small to keep a GPU busy. The Build vs Buy tab on tokcalc prices this against a specific provider for you.

Try Llama 3.1 405B with your own context and concurrency

The calculator uses the exact same formulas as this page — change the workload and watch every number update.

Open in calculator

Llama 3.1 405B on specific GPUs

Llama 3.1 405B on A100 80GB SXM4Llama 3.1 405B on H100 SXM5 80GBLlama 3.1 405B on B200 SXM 192GBLlama 3.1 405B on H200 SXM5 141GBLlama 3.1 405B on AMD MI325X 288GBLlama 3.1 405B on B300 SXM 288GB
Model architecture
The numbers behind the KV-cache and throughput math.
Total parameters405.00B
Active parameters per token405.00B
ArchitectureDense
Attention layoutFull attention (GQA)
Transformer layers126
Hidden dimension16384
Attention heads128 query / 8 KV (GQA)
Head dimension128
Vocabulary size128,256
Max context (catalog)131K
KV cache per token @ 8K504.0 KB

Every layer caches keys and values for the whole context, so the KV cache grows linearly with conversation length.