LoRi: Low-Rank Distillation for Implicit Reasoning

arXiv:2606.05315v1 Announce Type: cross Abstract: Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation...

Read the full article at the source.