Plan your LLM deployment
before you rent the GPUs.
An open-source LLM serving capacity planner. Estimate model fit, KV-cache, prefill/decode throughput, continuous batching, latency, multi-GPU scaling, cloud cost, API cost, and self-hosting break-even — with transparent formulas and cited benchmarks.
Not a GPU expert? Hover over any icon to see a plain-English explanation. Or scroll to theglossaryat the bottom for the full list.
Capacity planning questions tokcalc answers
- → Can I serve Qwen 2.5 72B at 128K context on 2× H100 with 20 concurrent users?
- → How many H200s for 1,000 req/min with P95 TTFT < 2s?
- → Does FP8 or AWQ save more money once quality + KV cache + engine support are included?
- → At what daily volume does an H100 beat GPT-4o pricing?
What makes tokcalc different
- ✓ Transparent formulas — no black-box throughput assumptions
- ✓ Engine-aware — continuous batching, paged KV, prefix caching
- ✓ Cited benchmarks — every multiplier is workload-specific, never universal
- ✓ Provider-neutral pricing — RunPod / Lambda / Modal / AWS / GCP side-by-side
1. Model
Which LLM are you running? Bigger = smarter but slower.
Total params8.03B
Active params8.03B
Layers32
2. GPU
The chip that runs the model. Memory speed matters most for LLMs.
Memory speed2000 GB/s
Math speed312 TF
Memory size80 GB
1×
3. Compression
Shrink the model to fit & run faster. Small accuracy cost.
16-bit float. Standard for inference, no dequant overhead.
4. Workload
How many users, and how long are the conversations?
1
Use a small model to predict ahead → big speedup
vLLM/TGI-style; serves requests mid-flight (1.5–4× typical)
Reuse a shared prefix (system prompt, RAG docs)
Generation speed
80.95
tok/s — 1 user
Total throughput
80.95
1 user
Wait per request
2.55 s
200-tok answer
Time to first token
51.5 ms
+ 12.4 ms per token
Memory budget
Will the model + conversation fit in the GPU's memory?
Model size
16.06 GB
8.03B × 2 bytes
Conversation memory
67.1 MB
128.0 MB/tok × 500 tok × 1
Total memory needed
16.13 GB
✓ fits in GPU memory
Available memory
80 GB
80 GB × 1 GPU
Memory used20.2%
Speed breakdown
Throughput & latency detail
Writing speed (1 user)
80.95 tok/s
12.4 ms per token
Total speed (batched)
80.95 tok/s
batch 1
Reading speed (prefill)
9714 tok/s
prompt in 51.5 ms
Batch sweet spot
120
users where adding more stops helping
Money
How much does it cost to run this?
GPU rent cost
$1.20/hr
per GPU
$ per 1M tokens
$4.12
standard LLM pricing unit
$ per request
$0.000851
200-token answer
How throughput grows with more users
More users = more total speed — until the GPU hits its math limit.
Compare GPUs — Llama 3 8B (FP16)
Speed of the same model on every GPU big enough to fit it. Green = your current pick.
Long-context capacity
How many concurrent users fit as context grows? KV cache scales linearly with context — at 128K+ you'll likely need multiple GPUs.
Recommended topology: Single GPU
16.1 GB ≤ 80 GB — fits in 1 A100
Max concurrent users at each context length80 GB total VRAM
Bars show how many concurrent users fit in VRAM at each context length. Green = closest to your current context (500). Red/dim = doesn't fit even at batch=1.
| Context | Max users | KV/request | Prefill time | Fits? |
|---|---|---|---|---|
| 4K | 119 | 549.8 MB | 425.2 ms | ✓ |
| 8K | 59 | 1.07 GB | 857.5 ms | ✓ |
Math: KV/request = 2 × layers × kv_heads × head_dim × bytes/KV × context. Prefill includes superlinear attention cost at long context (O(N²) attention dominates beyond ~32K). For RingAttention/blockwise attention, real prefill is lower than this estimate. Refs: RingAttention paper.
Observed benchmark — calibrate the formula
Paste a vLLM benchmark JSON output to see how tokcalc's theoretical estimates compare to real-world measured performance. The calibration verdict shows whether the formula is accurate, too conservative, or too optimistic.
Plain-English glossary
What all these words mean, without jargon.
Tokens
A 'token' is roughly 3/4 of an English word. So 1,000 tokens ≈ 750 words. The model reads and writes text in tokens, not characters.
Decode
'Decode' = the model WRITING new tokens, one at a time, after reading your prompt. This is the slow part where users wait.
Prefill
'Prefill' = the model READING your prompt before it starts answering. Happens once per request, in parallel.
Tokens Per Sec
How many tokens the model writes per second. Higher = faster responses for users.
Single Stream
When you serve just 1 user at a time. This is the max speed a single user will see.
Aggregate
Total tokens/sec when serving many users at once (batching). Always higher than single-stream because the GPU is shared efficiently.
Latency
Total wait time from when a user sends a prompt to when they get the full answer back.
Vram
Video RAM — the GPU's own memory. The model + its scratch space (KV cache) must fit entirely inside this. If it doesn't fit, the model won't run on this GPU.
Model Weights
The actual 'brain' of the model — billions of numbers stored in VRAM. Size = params × bytes per param. Bigger = smarter but slower and pricier.
Kv Cache
Scratch memory the model keeps while generating, so it doesn't re-read your prompt every token. Grows with conversation length and number of users.
Params
Number of 'neurons' (actually weight values) in the model. More params = smarter but slower & uses more VRAM. 'B' = billion. So 7B = 7,000,000,000 weights.
Active
For MoE (mixture-of-experts) models, only some experts activate per token. 'Active params' = what actually gets used per token, vs total params = the whole model size on disk.
Layers
How many 'stacked blocks' the model has. Deeper = smarter reasoning, but each token must pass through all of them, so it's slower.
Hbm Bw
How fast the GPU can read its own memory (in GB per second). This is the #1 factor for LLM speed — the GPU must load the entire model weights once per generated token.
Flops
Trillions of math operations per second (Tera-FLOPS). Determines how fast the model can process your prompt (prefill phase) and large batches.
Tensor Parallel
Splitting one model across multiple GPUs (each GPU holds part of the model). Lets you run models too big for 1 GPU, with near-linear speedup.
Quantization
Shrinking the model's weights from 16-bit numbers to 8-bit or 4-bit. Smaller = faster + cheaper, but slightly less accurate. INT4 = 4-bit, FP16 = 16-bit.
Batch Size
How many user requests the GPU serves at once. 1 = one user; 8 = eight users sharing one forward pass. Higher batch = more total throughput, but each user waits the same time.
Prompt Tokens
How long your input is. A 1-paragraph question is ~100 tokens; a 10-page document is ~3,000 tokens. Longer prompts take longer to read (prefill).
Output Tokens
How long the model's answer should be. 200 tokens ≈ 150 words. Longer answers take proportionally more time.
Speculative
A trick where a small 'draft' model guesses the next several tokens, then the big model verifies in one pass. Can 2-3x speed if the draft model is accurate.
Batch Crossover
The batch size where the GPU switches from 'memory-limited' to 'math-limited'. Below this, more users = more speed for free. Above this, you've maxed out the math capacity.
Cost Per M Tokens
What you pay to generate 1 million output tokens. The standard pricing unit for LLM APIs (e.g. GPT-4 charges $/1M tokens).
Moe
Mixture-of-Experts: a model with many 'expert' sub-networks, where only a few activate per token. Total size on disk is large, but per-token work is small. Example: Mixtral 8x7B has 47B total but only ~13B active per token.
Gqa
Grouped-Query Attention: an optimization that shares the same 'memory scratch space' (KV cache) across multiple attention heads. Cuts VRAM use significantly.
Continuous Batching
A serving trick where the GPU keeps generating tokens for active requests while NEW requests join mid-flight. Big speedup (1.5-4x typical) for production traffic. Used by vLLM, TGI, SGLang.
Ttft
Time-to-first-token: how long the user waits before seeing the first word. Equals prefill time (the model reading your prompt). Lower = better UX.
Itl
Inter-token-latency: how fast the model writes each token AFTER the first. This is what users perceive as 'streaming speed'.
Prompt Caching
Reuse the model's work from a repeated prompt prefix (system prompt, RAG docs, tool schemas). On cache hit, you skip the prefill computation for that prefix. Anthropic: 90% off cached tokens. OpenAI: 50% off.
Reasoning Tokens
Hidden tokens the model generates internally before answering (e.g. OpenAI o1, DeepSeek R1, Claude thinking). Billed as output but invisible to the user. Important for cost estimates of reasoning models.
Gguf
A file format used by llama.cpp for local inference. Has many sub-variants (Q2_K through Q8_0) trading size for quality. Q4_K_M is the recommended sweet spot for local Llama/Mistral.
Fp8
8-bit floating-point format. Native on NVIDIA H100/H200 — actually FASTER than FP16 thanks to dedicated FP8 tensor cores. Same VRAM as INT8 but better quality.
Nvfp4
NVIDIA's 4-bit floating-point format with block scaling. Native on Blackwell (B200/B300). Roughly 2x faster than FP8 on Blackwell per cited benchmarks.
Confidence levels
Measured
Sourced directly from official specs, model cards, API pricing pages, or HuggingFace config.json.
Modeled
Derived from a physics-based formula with cited sources (e.g., decode tok/s = HBM_BW × η_mem × quant_eff / model_size).
Inferred
Derived from a heuristic with known error bars (e.g., continuous batching multiplier default 1.5×, long-context attention O(N²) correction).
User-supplied
You provided this value directly.
Every metric on this page has a confidence dot (🟢 measured · 🟢 modeled · 🟡 inferred · ⚪ user-supplied). Honesty is the moat — we'd rather show you our uncertainty than pretend at a universal "tokens/sec" number. Real-world performance depends on engine, model revision, driver, and traffic distribution.
The math, transparently
Every number above comes from these formulas. No black boxes, no hidden "magic numbers." Optional reading for engineers.
Formulas verified against:vLLM docsNVIDIA cuBLAS benchmarksllama.cpp discussionsGPU spec sheetsartificialanalysis.ai