Large language models (LLMs) lose little accuracy when their weights and activations are quantized to well below 16 bits. Moreover, the number of bits that an LLM needs varies across the model and with the input data. We design number formats and accelerators that exploit this low and variable precision to reduce the cost of LLM inference.
Research goal: Scaling LLMs to more parameters has improved their accuracy, but it also increases the memory and computation of every inference. Quantization can reduce both, and our goal is accelerators that realize these savings in hardware.
Gap in the SotA: Weight-only quantization keeps the activations in 16-bit floating point (FP16), and the arithmetic on these activations dominates the cost of the prefill phase. Methods that vary the activation precision between modules calibrate it offline, and the calibrated precision can violate the accuracy target on other data. For deep quantization formats like ternary {-1, 0, 1}, GPUs don't have native support and instead dequantize the values before using them, paying the full energy and area cost.
Recent results:
Anda, a variable-length activation format (HPCA 2025). In the Anda format, a group of activations shares one exponent, and an offline search determines the mantissa length for each type of module. On matrix multiplications with quantized weights, Anda is on average 3.1× more energy-efficient than a GPU-like FP16 accelerator, with less than 1% perplexity increase.
Kameleo, run-time selection of the activation precision. Because the Anda accelerator processes the mantissas from the most significant bit down, the results for shorter mantissas appear as intermediate values of the computation. Kameleo compares them with the full-precision results to select the mantissa lengths at run time. In the prefill phase, the matrix multiplications then take 2.1× to 2.3× fewer cycles than with FP16 activations, with less than 1% perplexity increase.
Lookup-table accelerators for ternary weights (ISPASS 2026, Best Paper Award). With an open-source hardware generator and a cost model validated against synthesis, we explored the design space of Lookup-table accelerators. This novel architecture can natively compute with ternary weights by pre-computing all ternary combinations of a group of activations. The exploration showed that the activation data type determines the best architecture. With FP16 activations, our optimized design reduces the area by 1.6× to 2.2× compared with multiplier-based baselines.
Chao Fang, Man Shi, Robin Geens, Arne Symons, Zhongfeng Wang, Marian Verhelst, “Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format,” IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2025. arXiv
Robin Geens, Joran Heldens, Joren Dumoulin, Marian Verhelst, “Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inference,” IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2026. Best Paper Award arXiv
Robin Geens, Thomas Debelle, Marian Verhelst, Chao Fang, “Kameleo: In-Situ Activation-Precision Selection for Bit-Serial LLM Accelerators,” arXiv preprint, 2026. arXiv