TextMixture-of-Experts141B params39B active66K max context

Mixtral 8x22B (MoE) VRAM Requirements: How Many GPUs & What It Costs

Mixtral 8x22B (MoE) needs 77.6 GB of VRAM for its weights at AWQ 4-bit — 78.5 GB once a KV cache is loaded. Below is the full breakdown: which GPUs fit, how many you need, what it serves, and what it costs per million tokens. Every figure comes from tokcalc's open-source capacity model.

Weights (AWQ 4-bit)
77.6 GB
78.5 GB with KV cache
Fits on 1 GPU?
Yes
TPU v5p (per-chip, 95GB) or larger
Cheapest config
$0.0534
8× RTX 4080 16GB / 1M tokens
Fastest config
87040
tok/s on 4× Cerebras CS-3 (WSE-3)
VRAM requirement by quantization
Weight footprint = params × bytes per param. The “with KV cache” column adds one request at 4K context.
QuantizationWeightsWith KV cacheSmallest single GPU
FP16282.0 GB282.9 GBB300 SXM 288GB (288 GB)
FP8 (E4M3)141.0 GB141.9 GBB200 SXM 192GB (192 GB)
AWQ 4-bitrecommended77.6 GB78.5 GBH100 SXM5 80GB (80 GB)
Which GPU should you use?
Reference workload: 8K context, 8 concurrent users, continuous batching, AWQ 4-bit. Ranked by fitting first, then by tokens per dollar-hour.
GPUGPUsTotal VRAMDecode tok/sAggregate tok/sCost / 1M tokens
RTX 4080 16GB8×128 GB130.012482$0.0534
RTX 5090 32GB4×128 GB162.57799$0.0712
RTX 3090 24GB4×96 GB84.863651$0.0761
RTX A4000 16GB8×128 GB81.247799$0.0855
RTX 4090 24GB4×96 GB91.394387$0.1013
T4 16GB8×128 GB58.033323$0.1337
V100 SXM2 32GB4×128 GB81.603917$0.1418
A100 40GB SXM44×160 GB140.56746$0.1482
A10G 24GB4×96 GB54.402611$0.1532
TPU v6e Trillium (32GB)4×128 GB74.263564$0.2494
A100 80GB SXM42×160 GB90.672176$0.3064
L4 24GB4×96 GB27.201306$0.3404

RTX 4080 16GB has no NVLink, so 8-way tensor parallelism runs over PCIe. tokcalc applies a flat 0.85× multi-GPU efficiency factor for this, which is realistic for NVLink-equipped datacenter cards but optimistic for PCIe scale-out — treat the throughput and cost figures for this configuration as an upper bound, and measure before committing.

How concurrency changes with context length
On 8× RTX 4080 16GB at AWQ 4-bit. KV cache scales linearly with context, so long contexts cut the number of users you can serve from the same GPUs.
ContextConcurrent usersKV per requestFits?
4K53962.1 MB✓ fits
8K261.88 GB✓ fits
33K67.52 GB✓ fits
66K315.03 GB✓ fits
131K130.06 GB✓ fits
Frequently asked questions

Can Mixtral 8x22B (MoE) run on a single GPU?

Yes. At AWQ 4-bit the model needs 78.5 GB of VRAM, so it fits on a single TPU v5p (per-chip, 95GB) or anything with more VRAM. At full FP16 it needs 282.9 GB, which fits on one B300 SXM 288GB or larger.

How many GPUs does Mixtral 8x22B (MoE) need?

At AWQ 4-bit with 8,192-token context and 8 concurrent users, the cheapest configuration that fits is 8× RTX 4080 16GB (128 GB total VRAM). $0.0534 per 1M output tokens.

How fast can Mixtral 8x22B (MoE) be served?

On 4× Cerebras CS-3 (WSE-3) at AWQ 4-bit, the aggregate throughput is 87040 tokens/sec with continuous batching. That hardware has no published rental price, so its cost per million tokens is not comparable.

How much VRAM does Mixtral 8x22B (MoE) need?

Weights alone: 282.0 GB at FP16, 77.6 GB at AWQ 4-bit. Add KV cache for your context length and concurrency — every concurrent request at 8,192 tokens costs extra VRAM, so budget headroom above the weight figure.

Should I run Mixtral 8x22B (MoE) in FP8 or FP16?

FP8 halves the weight footprint (282 GB → 141 GB) and is faster than FP16 on Hopper and Blackwell, where it has native kernels. FP16 remains the reference quality bar; FP8 loses a small amount of accuracy in exchange for roughly doubling the batch size you can serve per GPU.

Why does Mixtral 8x22B (MoE) need more VRAM than its active parameter count suggests?

Mixtral 8x22B (MoE) is a mixture-of-experts model: 39B parameters are active per token, but all 141B experts must stay resident in VRAM. Capacity is bounded by the resident set, while decode throughput is bounded by the active set — so it serves fast per token but needs the full footprint.

Is it cheaper to self-host Mixtral 8x22B (MoE) than to call an API?

At 8× RTX 4080 16GB the marginal cost is $0.0534 per 1M output tokens, before you add idle capacity, ops staff, or egress. That is typically cheaper than frontier API pricing at sustained volume, but API calls win when your load is bursty or too small to keep a GPU busy. The Build vs Buy tab on tokcalc prices this against a specific provider for you.

Try Mixtral 8x22B (MoE) with your own context and concurrency

The calculator uses the exact same formulas as this page — change the workload and watch every number update.

Open in calculator

Mixtral 8x22B (MoE) on specific GPUs

Mixtral 8x22B (MoE) on RTX 5090 32GBMixtral 8x22B (MoE) on A100 80GB SXM4Mixtral 8x22B (MoE) on H100 SXM5 80GBMixtral 8x22B (MoE) on AMD MI300X 192GBMixtral 8x22B (MoE) on L40S 48GBMixtral 8x22B (MoE) on AMD MI325X 288GBMixtral 8x22B (MoE) on Intel Gaudi 3 128GBMixtral 8x22B (MoE) on B200 SXM 192GB
Model architecture
The numbers behind the KV-cache and throughput math.
Total parameters141.00B
Active parameters per token39.00B
ArchitectureMixture-of-Experts
Attention layoutFull attention (GQA)
Transformer layers56
Hidden dimension6144
Attention heads48 query / 8 KV (GQA)
Head dimension128
Vocabulary size32,000
Max context (catalog)66K
KV cache per token @ 8K224.0 KB

Every layer caches keys and values for the whole context, so the KV cache grows linearly with conversation length.