TextDense70.60B params8K max context

Llama 3 70B VRAM Requirements: How Many GPUs & What It Costs

Llama 3 70B needs 70.6 GB of VRAM for its weights at FP8 (E4M3) — 71.9 GB once a KV cache is loaded. Below is the full breakdown: which GPUs fit, how many you need, what it serves, and what it costs per million tokens. Every figure comes from tokcalc's open-source capacity model.

Weights (FP8 (E4M3))
70.6 GB
71.9 GB with KV cache
Fits on 1 GPU?
Yes
TPU v5p (per-chip, 95GB) or larger
Cheapest config
$0.1031
8× RTX 4080 16GB / 1M tokens
Fastest config
45076
tok/s on 4× Cerebras CS-3 (WSE-3)
VRAM requirement by quantization
Weight footprint = params × bytes per param. The “with KV cache” column adds one request at 4K context.
QuantizationWeightsWith KV cacheSmallest single GPU
FP16141.2 GB142.5 GBB200 SXM 192GB (192 GB)
FP8 (E4M3)recommended70.6 GB71.9 GBH100 SXM5 80GB (80 GB)
AWQ 4-bit38.8 GB40.2 GBL40S 48GB (48 GB)
Which GPU should you use?
Reference workload: 8K context, 8 concurrent users, continuous batching, FP8 (E4M3). Ranked by fitting first, then by tokens per dollar-hour.
GPUGPUsTotal VRAMDecode tok/sAggregate tok/sCost / 1M tokens
RTX 4080 16GB8×128 GB67.336464$0.1031
RTX 5090 32GB4×128 GB84.144039$0.1376
RTX 3090 24GB4×96 GB43.952017$0.1377
RTX A4000 16GB8×128 GB42.074039$0.1651
RTX 4090 24GB4×96 GB47.332272$0.1956
T4 16GB8×128 GB30.051836$0.2421
V100 SXM2 32GB4×128 GB42.262028$0.2739
A100 40GB SXM44×160 GB72.783493$0.2863
A10G 24GB4×96 GB28.171352$0.2958
TPU v6e Trillium (32GB)4×128 GB38.461846$0.4816
A100 80GB SXM42×160 GB46.951127$0.5916
L4 24GB4×96 GB14.09676.1$0.6573

RTX 4080 16GB has no NVLink, so 8-way tensor parallelism runs over PCIe. tokcalc applies a flat 0.85× multi-GPU efficiency factor for this, which is realistic for NVLink-equipped datacenter cards but optimistic for PCIe scale-out — treat the throughput and cost figures for this configuration as an upper bound, and measure before committing.

How concurrency changes with context length
On 8× RTX 4080 16GB at FP8 (E4M3). KV cache scales linearly with context, so long contexts cut the number of users you can serve from the same GPUs.
ContextConcurrent usersKV per requestFits?
4K421.34 GB✓ fits
8K212.68 GB✓ fits
33K510.74 GB✓ fits
66K221.47 GB✓ fits
131K142.95 GB✓ fits
Frequently asked questions

Can Llama 3 70B run on a single GPU?

Yes. At FP8 (E4M3) the model needs 71.9 GB of VRAM, so it fits on a single TPU v5p (per-chip, 95GB) or anything with more VRAM. At full FP16 it needs 142.5 GB, which fits on one B200 SXM 192GB or larger.

How many GPUs does Llama 3 70B need?

At FP8 (E4M3) with 8,192-token context and 8 concurrent users, the cheapest configuration that fits is 8× RTX 4080 16GB (128 GB total VRAM). $0.1031 per 1M output tokens.

How fast can Llama 3 70B be served?

On 4× Cerebras CS-3 (WSE-3) at FP8 (E4M3), the aggregate throughput is 45076 tokens/sec with continuous batching. That hardware has no published rental price, so its cost per million tokens is not comparable.

How much VRAM does Llama 3 70B need?

Weights alone: 141.2 GB at FP16, 70.6 GB at FP8 (E4M3). Add KV cache for your context length and concurrency — every concurrent request at 8,192 tokens costs extra VRAM, so budget headroom above the weight figure.

Should I run Llama 3 70B in FP8 or FP16?

FP8 halves the weight footprint (141 GB → 71 GB) and is faster than FP16 on Hopper and Blackwell, where it has native kernels. FP16 remains the reference quality bar; FP8 loses a small amount of accuracy in exchange for roughly doubling the batch size you can serve per GPU.

Is it cheaper to self-host Llama 3 70B than to call an API?

At 8× RTX 4080 16GB the marginal cost is $0.1031 per 1M output tokens, before you add idle capacity, ops staff, or egress. That is typically cheaper than frontier API pricing at sustained volume, but API calls win when your load is bursty or too small to keep a GPU busy. The Build vs Buy tab on tokcalc prices this against a specific provider for you.

Try Llama 3 70B with your own context and concurrency

The calculator uses the exact same formulas as this page — change the workload and watch every number update.

Open in calculator

Llama 3 70B on specific GPUs

Llama 3 70B on RTX 5090 32GBLlama 3 70B on A100 80GB SXM4Llama 3 70B on H100 SXM5 80GBLlama 3 70B on AMD MI300X 192GBLlama 3 70B on L40S 48GBLlama 3 70B on AMD MI325X 288GBLlama 3 70B on Intel Gaudi 3 128GBLlama 3 70B on B200 SXM 192GB
Model architecture
The numbers behind the KV-cache and throughput math.
Total parameters70.60B
Active parameters per token70.60B
ArchitectureDense
Attention layoutFull attention (GQA)
Transformer layers80
Hidden dimension8192
Attention heads64 query / 8 KV (GQA)
Head dimension128
Vocabulary size128,256
Max context (catalog)8K
KV cache per token @ 8K320.0 KB

Every layer caches keys and values for the whole context, so the KV cache grows linearly with conversation length.