tokcalc

A 'token' is roughly 3/4 of an English word. So 1,000 tokens ≈ 750 words. The model reads and writes text in tokens, not characters.

'Decode' = the model WRITING new tokens, one at a time, after reading your prompt. This is the slow part where users wait.

'Prefill' = the model READING your prompt before it starts answering. Happens once per request, in parallel.

How many tokens the model writes per second. Higher = faster responses for users.

When you serve just 1 user at a time. This is the max speed a single user will see.

Total tokens/sec when serving many users at once (batching). Always higher than single-stream because the GPU is shared efficiently.

Total wait time from when a user sends a prompt to when they get the full answer back.

Video RAM — the GPU's own memory. The model + its scratch space (KV cache) must fit entirely inside this. If it doesn't fit, the model won't run on this GPU.

The actual 'brain' of the model — billions of numbers stored in VRAM. Size = params × bytes per param. Bigger = smarter but slower and pricier.

Scratch memory the model keeps while generating, so it doesn't re-read your prompt every token. Grows with conversation length and number of users. Not every model grows it the same way: sliding-window layers stop growing at their window, linear-attention layers keep a fixed-size state instead of a cache, and MLA layers store one compressed latent instead of a full key-value pair.

Number of 'neurons' (actually weight values) in the model. More params = smarter but slower & uses more VRAM. 'B' = billion. So 7B = 7,000,000,000 weights.

For MoE (mixture-of-experts) models, only some experts activate per token. 'Active params' = what actually gets used per token, vs total params = the whole model size on disk.

How many 'stacked blocks' the model has. Deeper = smarter reasoning, but each token must pass through all of them, so it's slower.

How fast the GPU can read its own memory (in GB per second). This is the #1 factor for LLM speed — the GPU must load the entire model weights once per generated token.

Trillions of math operations per second (Tera-FLOPS). Determines how fast the model can process your prompt (prefill phase) and large batches.

Splitting one model across multiple GPUs (each GPU holds part of the model). Lets you run models too big for 1 GPU, with near-linear speedup.

Shrinking the model's weights from 16-bit numbers to 8-bit or 4-bit. Smaller = faster + cheaper, but slightly less accurate. INT4 = 4-bit, FP16 = 16-bit.

How many user requests the GPU serves at once. 1 = one user; 8 = eight users sharing one forward pass. Higher batch = more total throughput, but each user waits the same time.

How long your input is. A 1-paragraph question is ~100 tokens; a 10-page document is ~3,000 tokens. Longer prompts take longer to read (prefill).

How long the model's answer should be. 200 tokens ≈ 150 words. Longer answers take proportionally more time.

A trick where a small 'draft' model guesses the next several tokens, then the big model verifies in one pass. Can 2-3x speed if the draft model is accurate.

The batch size where the GPU switches from 'memory-limited' to 'math-limited'. Below this, more users = more speed for free. Above this, you've maxed out the math capacity.

What you pay to generate 1 million output tokens. The standard pricing unit for LLM APIs (e.g. GPT-4 charges $/1M tokens).

Mixture-of-Experts: a model with many 'expert' sub-networks, where only a few activate per token. Total size on disk is large, but per-token work is small. Example: Mixtral 8x7B has 47B total but only ~13B active per token.

Grouped-Query Attention: an optimization that shares the same 'memory scratch space' (KV cache) across multiple attention heads. Cuts VRAM use significantly.

A serving trick where the GPU keeps generating tokens for active requests while NEW requests join mid-flight. Big speedup (1.5-4x typical) for production traffic. Used by vLLM, TGI, SGLang.

Estimated first-token latency: how long until the first token is available. Modeled as prefill time (the model reading your prompt) PLUS the first decode step, so it is slightly more than prefill alone — the old 'TTFT = prefill' claim was not quite right. This is an estimate of first-token availability on the serving side; it does not model your client's rendering or network path. Lower = better UX. For what a user actually perceives, see estimated readable-output latency.

The Verdict panel answers three questions that are easy to confuse. FIT: does the model and its cache fit in memory at all? RUNTIME: how fast does it actually run? SLO: does it clear the ceilings you set? A configuration can pass FIT and still be useless if it is three times too slow to serve anyone — which is why the three answers are reported separately.

Service-level objective: the ceiling your service has to clear. Usually a maximum time-to-first-token, a minimum throughput, and a maximum cost per million tokens. Set any of them in the calculator and the Verdict panel turns the runtime numbers into a pass or a fail you can act on.

Estimated readable-output latency: time until the user has something readable, not just a first token. It is the first-token latency plus the time to decode the first clause. Measured as the time to 12 generated tokens — a heuristic readable-output proxy, NOT semantic answer detection: the model may have emitted a preamble or a restatement instead of the answer, and hidden reasoning tokens are not separately accounted for unless the model profile explicitly supports that. A 200-token answer usually needs a few more tokens before a human can act on it, so this is usually several times the first-token latency. If you are writing an SLO, this is the number that matters.

Inter-token-latency: how fast the model writes each token AFTER the first. This is what users perceive as 'streaming speed'.

Reuse the model's work from a repeated prompt prefix (system prompt, RAG docs, tool schemas). On cache hit, you skip the prefill computation for that prefix. Anthropic: 90% off cached tokens. OpenAI: 50% off.

Hidden tokens the model generates internally before answering (e.g. OpenAI o1, DeepSeek R1, Claude thinking). Billed as output but invisible to the user. Important for cost estimates of reasoning models.

A file format used by llama.cpp for local inference. Has many sub-variants (Q2_K through Q8_0) trading size for quality. Q4_K_M is the recommended sweet spot for local Llama/Mistral.

8-bit floating-point format. Native on NVIDIA H100/H200 — actually FASTER than FP16 thanks to dedicated FP8 tensor cores. Same VRAM as INT8 but better quality.

NVIDIA's 4-bit floating-point format with block scaling. Native on Blackwell (B200/B300). Roughly 2x faster than FP8 on Blackwell per cited benchmarks.

Where a number came from — the spec sheet, the published price, an observed run, a formula, a rule of thumb, or your own input. Provenance is NOT a confidence ranking: an observed run can be perfectly genuine but irrelevant to the context length you are planning, and an official spec can pin memory capacity precisely while saying nothing about throughput. How far a number actually holds is stated separately, per record, as applicability and limitations.

How far the source behind a number actually establishes it: 'exact' (the source directly and completely establishes the value), 'partial' (it establishes it only under conditions that may not apply to you), or 'lower-bound' (the figure is a floor — real usage is at least this).