Cross-Layer Discrete Concept Discovery for Interpreting Language Models

arXiv:2506.20040v3 Announce Type: replace-cross Abstract: Interpreting language models remains challenging due to the existence of residual stream, which linearly mixes and duplicates features across adjacent layers, causing single-layer analyses to miss this cross-layer structure. Cross-layer sparse autoencoders (SAEs) address layer mixing but...

Read the full article at the source.