AI Hardware & Software Notes

The VRAM Budget for a 70B Model: Quantization, KV Cache, and Concurrency

A practical accounting of where VRAM actually goes when serving a 70B model, and how quantization level and concurrency constrain each other.

2 min read

The most common mistake when deploying a 70B-class model is assuming “if the weights fit, we’re fine.” Weights are only part of the picture. KV cache grows linearly with concurrency and context length, and it is usually what actually runs you out of memory.

Weight footprint

Parameter count times bytes per parameter gives the theoretical floor. For a 70B model:

Precision Bytes/param Weights Fits one 80GB card?
FP16 / BF16 2 ~140 GB No
FP8 1 ~70 GB Barely, no headroom
INT4 (GPTQ/AWQ) ~0.5 ~35 GB Yes, with room

Note that “~0.5” is not exact: 4-bit quantization carries per-group scales and zero-points, so real size runs a few percent above the theoretical figure.

KV cache is the real variable

Estimate KV cache with:

bytes = 2 (K and V) × layers × kv_heads × head_dim × seq_len × batch × dtype_bytes

For Llama-3 70B (80 layers, 8 KV heads with GQA, head_dim 128, BF16):

  • One 4K context: ~1.25 GB
  • One 32K context: ~10 GB
  • 8 concurrent × 32K: ~80 GB

That last line is the point: fitting the weights does not mean you can serve. It is why long-context workloads usually need PagedAttention or KV quantization.

Choosing a quantization level

INT4 saves a lot of VRAM, but the cost has to be measured on your actual task. In my experience:

  1. General chat and summarization: INT4 is usually within acceptable range of FP16
  2. Code generation and math reasoning: degradation is more visible; prefer FP8 or better
  3. Structured extraction: precision-sensitive, so run your own eval set

Do not optimize for perplexity alone. An A/B on 50–200 of your own real requests is more reliable than any public leaderboard.

Planning concurrency

Split the VRAM budget into three parts:

total = weights + KV cache(concurrency × context) + runtime overhead

Runtime overhead (CUDA context, activations, framework buffers) usually needs 3–6 GB reserved. Do not budget to the last byte. Keeping 10% headroom avoids request failures from OOM.

A practical check order

Run these in sequence before going live; it saves most of the rework:

  1. Single request, short context — confirms weights and runtime overhead
  2. Ramp context up to the target length, watch the VRAM curve
  3. Ramp concurrency, find the VRAM knee
  4. Set production config at 20% below that knee

If step 3 hits the wall early, the bottleneck is KV cache. Raising concurrency will not help; do KV quantization or prefix caching first.