Google's TurboQuant Compresses LLM Memory by 6x With Zero Accuracy Loss

The Memory Bottleneck Problem

Large language models are hungry for memory. Every conversation turn, every document analyzed, every agent action adds to the key-value (KV) cache, the data structure that stores attention state and allows the model to remember what came before. On production hardware, the KV cache frequently consumes more GPU memory than the model weights themselves, limiting batch sizes, context lengths, and ultimately the number of users a single GPU can serve.

Google Research has now published TurboQuant, a compression algorithm that reduces KV cache memory requirements by 6x while maintaining zero accuracy loss. The work, presented at ICLR 2026, achieves this by compressing cache values to just 3 bits per number, a level of compression that previous methods could not reach without degrading model outputs.

How TurboQuant Works

TurboQuant operates in two stages, each addressing a different aspect of the compression challenge.

Stage 1: PolarQuant

The first stage converts data vectors from standard Cartesian coordinates to polar coordinates, replacing X-Y-Z distance measurements with radius (data magnitude) and angle (semantic direction) pairs. This geometric transformation simplifies the data distribution, making it more amenable to aggressive quantization. Critically, PolarQuant eliminates the expensive data normalization step that most quantization methods require, mapping data onto a predictable circular grid instead.

Stage 2: QJL (Quantized Johnson-Lindenstrauss)

The second stage applies the Johnson-Lindenstrauss Transform to compress high-dimensional data while preserving essential distance relationships. Each vector component is reduced to a single sign bit (+1 or -1), achieving near-zero memory overhead for the error correction layer. A specialized estimator maintains accuracy for attention score calculations, ensuring that the compressed representation produces identical results to the uncompressed original.

The combination is powerful: PolarQuant handles the bulk compression efficiently, while QJL corrects any residual errors that would otherwise accumulate and degrade output quality.

Performance Results

The numbers are compelling across multiple dimensions.

TurboQuant was evaluated on Gemma, Mistral, and Llama-3.1-8B-Instruct across six benchmarks: LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, L-Eval, and GloVe. Across all models and benchmarks, the compressed models matched the accuracy of uncompressed baselines. The 4-bit variant achieved up to 8x performance improvement over 32-bit unquantized keys on NVIDIA H100 GPUs.

Why This Matters

TurboQuant's practical impact extends beyond benchmark scores. The algorithm works on existing models without any training or fine-tuning, meaning it can be applied as a post-processing step to any compatible LLM. This is a critical distinction from methods that require quantization-aware training, which adds weeks of compute cost and complexity.

For deployment operators, 6x memory reduction means a model that previously required 6 GPUs for a given context length can now run on 1, or that a single GPU can serve 6x more concurrent users. On expensive hardware like H100s (which rent for $2-3 per hour on cloud platforms), the cost savings are substantial.

The technique is also model-agnostic. Google tested it on both their own Gemma models and external models like Mistral and Llama, demonstrating portability across architectures.

Broader Context

TurboQuant arrives at a moment when the AI industry is grappling with inference cost as a strategic concern. As models grow larger and context windows extend to millions of tokens, the memory footprint of the KV cache scales linearly, making compression not just desirable but necessary for economic viability.

Previous compression methods like standard quantization (INT8, INT4) and pruning offered partial solutions but typically required accuracy trade-offs or model-specific calibration. TurboQuant's contribution is achieving extreme compression (3 bits) with zero accuracy loss and zero retraining, a combination that was previously considered impractical.

The publication timing, just ahead of its ICLR 2026 presentation, suggests Google views this as a significant contribution. The related PolarQuant paper will also be presented at AISTATS 2026, while the underlying QJL algorithm was published at AAAI 2025.

Limitations and Open Questions

As of March 2026, Google has published the research papers but has not yet released open-source code. The ICLR presentation is expected to coincide with or precede code availability, but until independent researchers can reproduce the results, the claims remain Google-verified only.

Additionally, the benchmarks focused on models up to 8B parameters. Performance characteristics at larger scales (70B+) may differ, and real-world serving environments introduce variables like batching strategies and multi-tenant interference that controlled benchmarks do not capture.

Conclusion

TurboQuant represents a meaningful advance in LLM inference efficiency. By combining PolarQuant's geometric compression with QJL's error correction, Google has achieved 3-bit KV cache compression with zero accuracy degradation and no retraining requirement. For anyone operating LLM inference at scale, whether on cloud GPUs or local hardware, this technique could significantly reduce costs and expand what is possible with existing silicon. The research community and deployment operators should watch closely for the open-source release.