TextMixture-of-Experts46.70B params13B active33K max context

Mixtral 8x7B (MoE) VRAM Requirements: How Many GPUs & What It Costs

Mixtral 8x7B (MoE) needs 46.7 GB of VRAM for its weights at FP8 (E4M3) — 47.2 GB once a KV cache is loaded. Below is the full breakdown: which GPUs fit, how many you need, what it serves, and what it costs per million tokens. Every figure comes from tokcalc's open-source capacity model.

Weights (FP8 (E4M3))
46.7 GB
47.2 GB with KV cache
Fits on 1 GPU?
Yes
Mac M3 Max (64GB) or larger
Cheapest config
$0.0119
8× RTX 3080 10GB / 1M tokens
Fastest config
61674
tok/s on 2× Cerebras CS-3 (WSE-3)
VRAM requirement by quantization
Weight footprint = params × bytes per param. The “with KV cache” column adds one request at 4K context.
QuantizationWeightsWith KV cacheSmallest single GPU
FP1693.4 GB93.9 GBTPU v5p (per-chip, 95GB) (95 GB)
FP8 (E4M3)recommended46.7 GB47.2 GBL40S 48GB (48 GB)
AWQ 4-bit25.7 GB26.2 GBV100 SXM2 32GB (32 GB)
Which GPU should you use?
Reference workload: 8K context, 8 concurrent users, continuous batching, FP8 (E4M3). Ranked by fitting first, then by tokens per dollar-hour.
GPUGPUsTotal VRAMDecode tok/sAggregate tok/sCost / 1M tokens
RTX 3080 10GB8×80 GB390.637498$0.0119
RTX 3090 24GB4×96 GB240.511039$0.0252
RTX 4090 24GB4×96 GB259.012434$0.0357
RTX 4080 16GB4×64 GB184.38844$0.0377
RTX 5090 32GB2×64 GB230.35526$0.0503
A10G 24GB4×96 GB154.27401$0.0540
RTX A4000 16GB4×64 GB115.15526$0.0603
T4 16GB4×64 GB82.232512$0.0885
V100 SXM2 32GB2×64 GB115.62775$0.1001
A100 40GB SXM42×80 GB199.24780$0.1046
L4 24GB4×96 GB77.093700$0.1201
AMD MI300X 192GB1×192 GB400.64807$0.1445

RTX 3080 10GB has no NVLink, so 8-way tensor parallelism runs over PCIe. tokcalc applies a flat 0.85× multi-GPU efficiency factor for this, which is realistic for NVLink-equipped datacenter cards but optimistic for PCIe scale-out — treat the throughput and cost figures for this configuration as an upper bound, and measure before committing.

How concurrency changes with context length
On 8× RTX 3080 10GB at FP8 (E4M3). KV cache scales linearly with context, so long contexts cut the number of users you can serve from the same GPUs.
ContextConcurrent usersKV per requestFits?
4K62549.8 MB✓ fits
8K311.07 GB✓ fits
33K74.29 GB✓ fits
66K38.59 GB✓ fits
131K117.18 GB✓ fits
Frequently asked questions

Can Mixtral 8x7B (MoE) run on a single GPU?

Yes. At FP8 (E4M3) the model needs 47.2 GB of VRAM, so it fits on a single Mac M3 Max (64GB) or anything with more VRAM. At full FP16 it needs 93.9 GB, which fits on one TPU v5p (per-chip, 95GB) or larger.

How many GPUs does Mixtral 8x7B (MoE) need?

At FP8 (E4M3) with 8,192-token context and 8 concurrent users, the cheapest configuration that fits is 8× RTX 3080 10GB (80 GB total VRAM). $0.0119 per 1M output tokens.

How fast can Mixtral 8x7B (MoE) be served?

On 2× Cerebras CS-3 (WSE-3) at FP8 (E4M3), the aggregate throughput is 61674 tokens/sec with continuous batching. That hardware has no published rental price, so its cost per million tokens is not comparable.

How much VRAM does Mixtral 8x7B (MoE) need?

Weights alone: 93.4 GB at FP16, 46.7 GB at FP8 (E4M3). Add KV cache for your context length and concurrency — every concurrent request at 8,192 tokens costs extra VRAM, so budget headroom above the weight figure.

Should I run Mixtral 8x7B (MoE) in FP8 or FP16?

FP8 halves the weight footprint (93 GB → 47 GB) and is faster than FP16 on Hopper and Blackwell, where it has native kernels. FP16 remains the reference quality bar; FP8 loses a small amount of accuracy in exchange for roughly doubling the batch size you can serve per GPU.

Why does Mixtral 8x7B (MoE) need more VRAM than its active parameter count suggests?

Mixtral 8x7B (MoE) is a mixture-of-experts model: 13B parameters are active per token, but all 47B experts must stay resident in VRAM. Capacity is bounded by the resident set, while decode throughput is bounded by the active set — so it serves fast per token but needs the full footprint.

Is it cheaper to self-host Mixtral 8x7B (MoE) than to call an API?

At 8× RTX 3080 10GB the marginal cost is $0.0119 per 1M output tokens, before you add idle capacity, ops staff, or egress. That is typically cheaper than frontier API pricing at sustained volume, but API calls win when your load is bursty or too small to keep a GPU busy. The Build vs Buy tab on tokcalc prices this against a specific provider for you.

Try Mixtral 8x7B (MoE) with your own context and concurrency

The calculator uses the exact same formulas as this page — change the workload and watch every number update.

Open in calculator

Mixtral 8x7B (MoE) on specific GPUs

Mixtral 8x7B (MoE) on RTX 5090 32GBMixtral 8x7B (MoE) on AMD MI300X 192GBMixtral 8x7B (MoE) on L40S 48GBMixtral 8x7B (MoE) on AMD MI325X 288GBMixtral 8x7B (MoE) on A100 80GB SXM4Mixtral 8x7B (MoE) on Intel Gaudi 3 128GBMixtral 8x7B (MoE) on H100 SXM5 80GBMixtral 8x7B (MoE) on B200 SXM 192GB
Model architecture
The numbers behind the KV-cache and throughput math.
Total parameters46.70B
Active parameters per token12.90B
ArchitectureMixture-of-Experts
Attention layoutFull attention (GQA)
Transformer layers32
Hidden dimension4096
Attention heads32 query / 8 KV (GQA)
Head dimension128
Vocabulary size32,000
Max context (catalog)33K
KV cache per token @ 8K128.0 KB

Every layer caches keys and values for the whole context, so the KV cache grows linearly with conversation length.