Channel-Wise Mixed-Precision Quantization for Large Language Models

arXiv:2410.13056v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated remarkable success across a wide range of language tasks, but their deployment on edge devices remains challenging due to the substantial memory requirements imposed by their large parameter sizes. Weight-only quantization presents a promising...

Läs hela artikeln hos källan.