Hardware for low- and variable-precision Large Language Models

Robin Geens and Chao Fang , Marian Verhelst Hardware-efficient AI and ML

Large language models (LLMs) lose little accuracy when their weights and activations are quantized to well below 16 bits. Moreover, the number of bits that an LLM needs varies across the model and with the input data. We design number formats and accelerators that exploit this low and variable precision to reduce the cost of LLM inference.

Research goal: Scaling LLMs to more parameters has improved their accuracy, but it also increases the memory and computation of every inference. Quantization can reduce both, and our goal is accelerators that realize these savings in hardware.

Gap in the SotA: Weight-only quantization keeps the activations in 16-bit floating point (FP16), and the arithmetic on these activations dominates the cost of the prefill phase. Methods that vary the activation precision between modules calibrate it offline, and the calibrated precision can violate the accuracy target on other data. For deep quantization formats like ternary {-1, 0, 1}, GPUs don't have native support and instead dequantize the values before using them, paying the full energy and area cost.

Recent results:

Anda, a variable-length activation format (HPCA 2025). In the Anda format, a group of activations shares one exponent, and an offline search determines the mantissa length for each type of module. On matrix multiplications with quantized weights, Anda is on average 3.1× more energy-efficient than a GPU-like FP16 accelerator, with less than 1% perplexity increase.

Kameleo, run-time selection of the activation precision. Because the Anda accelerator processes the mantissas from the most significant bit down, the results for shorter mantissas appear as intermediate values of the computation. Kameleo compares them with the full-precision results to select the mantissa lengths at run time. In the prefill phase, the matrix multiplications then take 2.1× to 2.3× fewer cycles than with FP16 activations, with less than 1% perplexity increase.

Lookup-table accelerators for ternary weights (ISPASS 2026, Best Paper Award). With an open-source hardware generator and a cost model validated against synthesis, we explored the design space of Lookup-table accelerators. This novel architecture can natively compute with ternary weights by pre-computing all ternary combinations of a group of activations. The exploration showed that the activation data type determines the best architecture. With FP16 activations, our optimized design reduces the area by 1.6× to 2.2× compared with multiplier-based baselines.

Get in touch
Robin Geens
Phd student
Chao Fang
Postdoc
Marian Verhelst
Academic staff

Publications about this research topic

Chao Fang, Man Shi, Robin Geens, Arne Symons, Zhongfeng Wang, Marian Verhelst, “Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format,” IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2025. arXiv

Robin Geens, Joran Heldens, Joren Dumoulin, Marian Verhelst, “Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inference,” IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2026. Best Paper Award arXiv

Robin Geens, Thomas Debelle, Marian Verhelst, Chao Fang, “Kameleo: In-Situ Activation-Precision Selection for Bit-Serial LLM Accelerators,” arXiv preprint, 2026. arXiv

Other research topics in Hardware-efficient AI and ML

Adaptive hardware for embodied AI
Hardware-efficient AI and ML
Robin Geens and Jonas Crols | Marian Verhelst
Vertically-Integrated Logic Fabrics for Future 3D Computing Platforms
Hardware-efficient AI and ML
Jannes Willemen | Marian Verhelst
A Mix-Signal Ising Accelerator for Real-World Applications
Mixed-signal circuits and data converters, Hardware-efficient AI and ML
Jiacong Sun, Toon Bettens, Sofie De Weer, Axel Theunis, and Andreas Pedersen | Wim Dehaene and Marian Verhelst
Precision-Scalable Microscaling Hardware for Continual Learning at the Edge
Hardware-efficient AI and ML
Stef Cuyckens | Marian Verhelst
XDMA: A Distributed DMA for Flexible and Efficient Data Movement in Heterogeneous Multi-Accelerator SoCs
Hardware-efficient AI and ML
Yunhao Deng and Fanchen Kong | Marian Verhelst
Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
Hardware-efficient AI and ML
Man Shi, Arne Symons, Robin Geens, and Chao Fang | Marian Verhelst
Massive parallelism for combinatorial optimisation problems
Hardware-efficient AI and ML
Toon Bettens and Sofie De Weer | Wim Dehaene and Marian Verhelst
Carbon-aware Design Space Exploration for AI Accelerators
Hardware-efficient AI and ML
Jiacong Sun | Georges Gielen and Marian Verhelst
Decoupled Control Flow and Memory Orchestration in the Vortex GPGPU
Hardware-efficient AI and ML
Giuseppe Sarda | Marian Verhelst
Automated Causal CNN Scheduling Optimizer for Real-Time Edge Accelerators
Hardware-efficient AI and ML
Jun Yin | Marian Verhelst
A Scalable Heterogenous Multi-accelerator Platform for AI and ML
Hardware-efficient AI and ML
Ryan Antonio | Marian Verhelst
Uncertainty-Aware Design Space Exploration for AI Accelerators
Hardware-efficient AI and ML
Jiacong Sun and Fanchen Kong | Georges Gielen and Marian Verhelst
Integer GEMM Accelerator for SNAX
Hardware-efficient AI and ML
Xiaoling Yi | Marian Verhelst
Improving GPGPU micro architecture for future AI workloads
Hardware-efficient AI and ML
Giuseppe Sarda | Marian Verhelst
SRAM based digital in memory compute macro in 16nm
Hardware-efficient AI and ML
Weijie Jiang | Wim Dehaene
Scalable large array nanopore readouts for proteomics and next-generation sequencing
Analog and power management circuits, Hardware-efficient AI and ML, Biomedical circuits and sensor interfaces
Sander Crols | Filip Tavernier and Marian Verhelst
Design space exploration of in-memory computing DNN accelerators
Hardware-efficient AI and ML
Pouya Houshmand and Jiacong Sun | Marian Verhelst
Multi-core architecture exploration for layer-fused deep learning acceleration
Hardware-efficient AI and ML
Arne Symons | Marian Verhelst
HW-algorithm co-design for Bayesian inference of probabilistic machine learning
Ultra-low power digital SoCs and memories, Hardware-efficient AI and ML
Shirui Zhao | Marian Verhelst
Design space exploration for machine learning acceleration
Hardware-efficient AI and ML
Arne Symons | Marian Verhelst
Enabling Fast Exploration of the Depth-first Scheduling Space for DNN Accelerators
Hardware-efficient AI and ML
Arne Symons | Marian Verhelst
Optimized deployment of AI algorithms on rapidly-changing heterogeneous multi-core compute platforms
Ultra-low power digital SoCs and memories, Hardware-efficient AI and ML
Josse Van Delm | Marian Verhelst
High-throughput high-efficiency SRAM for neural networks
Ultra-low power digital SoCs and memories, Hardware-efficient AI and ML
Wim Dehaene and Marian Verhelst
Heterogeneous Multi-core System-on-Chips for Ultra Low Power Machine Learning Application at the Edge
Hardware-efficient AI and ML
Pouya Houshmand, Giuseppe Sarda, and Ryan Antonio | Marian Verhelst

Want to work with us?

Get in touch or discover the way we can collaborate.