TextMixture-of-Experts20.91B params4B active131K max context

GPT-OSS 20B (MoE) VRAM Requirements: How Many GPUs & What It Costs

GPT-OSS 20B (MoE) needs 20.9 GB of VRAM for its weights at FP8 (E4M3) — 21.1 GB once a KV cache is loaded. Below is the full breakdown: which GPUs fit, how many you need, what it serves, and what it costs per million tokens. Every figure comes from tokcalc's open-source capacity model.

Weights (FP8 (E4M3))
20.9 GB
21.1 GB with KV cache
Fits on 1 GPU?
Yes
A10G 24GB or larger
Cheapest config
$0.006615
4× RTX 3080 10GB / 1M tokens
Fastest config
65000
tok/s on 1× Cerebras CS-3 (WSE-3)
VRAM requirement by quantization
Weight footprint = params × bytes per param. The “with KV cache” column adds one request at 4K context.
QuantizationWeightsWith KV cacheSmallest single GPU
FP1641.8 GB42.0 GBL40S 48GB (48 GB)
FP8 (E4M3)recommended20.9 GB21.1 GBA10G 24GB (24 GB)
AWQ 4-bit11.5 GB11.7 GBT4 16GB (16 GB)
Which GPU should you use?
Reference workload: 8K context, 8 concurrent users, continuous batching, FP8 (E4M3). Ranked by fitting first, then by tokens per dollar-hour.
GPUGPUsTotal VRAMDecode tok/sAggregate tok/sCost / 1M tokens
RTX 3080 10GB4×40 GB699.833592$0.006615
RTX 4080 16GB2×32 GB330.17923$0.0210
RTX 5090 32GB1×32 GB485.35824$0.0238
RTX 3090 24GB1×24 GB253.52472$0.0281
RTX A4000 16GB2×32 GB206.34950$0.0337
RTX 4090 24GB1×24 GB273.03276$0.0339
AMD MI300X 192GB1×192 GB143517225$0.0403
V100 SXM2 32GB1×32 GB243.82925$0.0475
T4 16GB2×32 GB147.32250$0.0494
A100 40GB SXM41×40 GB419.85038$0.0496
AMD MI325X 288GB1×288 GB162519500$0.0499
A100 80GB SXM41×80 GB541.76500$0.0513

RTX 3080 10GB has no NVLink, so 4-way tensor parallelism runs over PCIe. tokcalc applies a flat 0.85× multi-GPU efficiency factor for this, which is realistic for NVLink-equipped datacenter cards but optimistic for PCIe scale-out — treat the throughput and cost figures for this configuration as an upper bound, and measure before committing.

How concurrency changes with context length
On 4× RTX 3080 10GB at FP8 (E4M3). KV cache scales linearly with context, so long contexts cut the number of users you can serve from the same GPUs.
ContextConcurrent usersKV per requestFits?
4K183206.2 MB✓ fits
8K93412.3 MB✓ fits
33K231.61 GB✓ fits
66K113.22 GB✓ fits
131K56.44 GB✓ fits
Frequently asked questions

Can GPT-OSS 20B (MoE) run on a single GPU?

Yes. At FP8 (E4M3) the model needs 21.1 GB of VRAM, so it fits on a single A10G 24GB or anything with more VRAM. At full FP16 it needs 42.0 GB, which fits on one L40S 48GB or larger.

How many GPUs does GPT-OSS 20B (MoE) need?

At FP8 (E4M3) with 8,192-token context and 8 concurrent users, the cheapest configuration that fits is 4× RTX 3080 10GB (40 GB total VRAM). $0.006615 per 1M output tokens.

How fast can GPT-OSS 20B (MoE) be served?

On 1× Cerebras CS-3 (WSE-3) at FP8 (E4M3), the aggregate throughput is 65000 tokens/sec with continuous batching. That hardware has no published rental price, so its cost per million tokens is not comparable.

How much VRAM does GPT-OSS 20B (MoE) need?

Weights alone: 41.8 GB at FP16, 20.9 GB at FP8 (E4M3). Add KV cache for your context length and concurrency — every concurrent request at 8,192 tokens costs extra VRAM, so budget headroom above the weight figure.

Should I run GPT-OSS 20B (MoE) in FP8 or FP16?

FP8 halves the weight footprint (42 GB → 21 GB) and is faster than FP16 on Hopper and Blackwell, where it has native kernels. FP16 remains the reference quality bar; FP8 loses a small amount of accuracy in exchange for roughly doubling the batch size you can serve per GPU.

Why does GPT-OSS 20B (MoE) need more VRAM than its active parameter count suggests?

GPT-OSS 20B (MoE) is a mixture-of-experts model: 4B parameters are active per token, but all 21B experts must stay resident in VRAM. Capacity is bounded by the resident set, while decode throughput is bounded by the active set — so it serves fast per token but needs the full footprint.

Is it cheaper to self-host GPT-OSS 20B (MoE) than to call an API?

At 4× RTX 3080 10GB the marginal cost is $0.006615 per 1M output tokens, before you add idle capacity, ops staff, or egress. That is typically cheaper than frontier API pricing at sustained volume, but API calls win when your load is bursty or too small to keep a GPU busy. The Build vs Buy tab on tokcalc prices this against a specific provider for you.

Try GPT-OSS 20B (MoE) with your own context and concurrency

The calculator uses the exact same formulas as this page — change the workload and watch every number update.

Open in calculator

GPT-OSS 20B (MoE) on specific GPUs

GPT-OSS 20B (MoE) on RTX 5090 32GBGPT-OSS 20B (MoE) on RTX 3090 24GBGPT-OSS 20B (MoE) on RTX 4090 24GBGPT-OSS 20B (MoE) on AMD MI300X 192GBGPT-OSS 20B (MoE) on V100 SXM2 32GBGPT-OSS 20B (MoE) on A100 40GB SXM4GPT-OSS 20B (MoE) on AMD MI325X 288GBGPT-OSS 20B (MoE) on A100 80GB SXM4
Model architecture
The numbers behind the KV-cache and throughput math.
Total parameters20.91B
Active parameters per token3.60B
ArchitectureMixture-of-Experts
Attention layoutHybrid: 12 full + 12 sliding
Transformer layers24
Hidden dimension2880
Attention heads64 query / 8 KV (GQA)
Head dimension64
Vocabulary size201,088
Max context (catalog)131K
Native context4K
Context extensionRoPE scaling (yarn)
KV cache per token @ 8K24.4 KB
Native checkpoint size13.76 GB
Native weight formatmxfp4
Licenseapache-2.0

sliding layers stop growing once the context passes their window, so long-context cost is bounded.

Limitations of these numbers
Where the source does not fully establish the values above. Some of the figures hold only under conditions that may not apply to you.
  • •Active parameter count (3.6B) is taken from the model card; the checkpoint config does not state it. Weight bytes, parameter count and attention layout are exact.