Hardware-efficient AI and ML The rising demand for LLM deployment on edge devices could revolutionize how we interact with AI in daily life. Our work on Anda represents a significant step towards enabling efficient LLM inference on resource-constrained platforms, potentially bringing powerful language capabilities to smartphones and IoT devices while maintaining both performance and energy efficiency.
Research goal: Large Language Models (LLMs) have shown remarkable capabilities but face deployment challenges due to their massive computational demands. Weight-only quantization (keeping FP16 activations with INT4 weights) has emerged as a popular solution to reduce model size. However, processing FP activations remains a major bottleneck in terms of energy consumption and computational complexity, highlighting the need for more efficient activation processing solutions.
Gap in the SotA: Existing approaches to optimize FP activations face significant limitations. GPU implementations require costly conversions and FP computations, while dedicated FP-INT units suffer from high alignment and normalization overhead. Block Floating Point (BFP) solutions either need expensive retraining to maintain accuracy or use long mantissas that increase computation costs. No current solution effectively balances the critical triad of model accuracy, computational efficiency, and energy consumption.
Recent results: This work presents Anda, a comprehensive solution that achieves impressive efficiency gains while maintaining model accuracy. It introduces a variable-length grouped activation format with shared exponents and adjustable mantissa widths, coupled with a training-free adaptive precision search algorithm. The system includes hardware optimizations such as bit-plane data layout, bit-serial processing units, and runtime compression. Evaluations across OPT, LLaMA, and LLaMA-2 models demonstrate 2.4× speedup, 4.0× area efficiency, and 3.1× energy efficiency improvements over the GPU-like baseline while offering flexible accuracy-performance tradeoffs.
Fun fact: Anda is named after a lovely cat who brings good luck - just as we hope our work brings good fortune to efficient LLM deployment!
Chao Fang, Man Shi, Robin Geens, Arne Symons, Zhongfeng Wang and Marian Verhelst. “Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format.” to appear in 2025 IEEE International Symposium on High-Performance Computer Architecture (HPCA 2025).
