{"id":"arxiv:2505.22255","kind":"preprint","title":"Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders","authors":["Vadim Kurochkin","Yaroslav Aksenov","Daniil Laptev","Daniil Gavrilov","Nikita Balagansky"],"author_count":5,"year":2025,"published":"2025-05-28","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2505.22255","nct_id":null,"source_url":"https://arxiv.org/abs/2505.22255","archive_url":"https://space-kitty.com/papers/arxiv:2505.22255","updated":"2026-09-07","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2505.22255","abstract":"Sparse Autoencoders (SAEs) decompose language-model activations into sparse, interpretable features, but standard encoders usually treat the latent dictionary as a flat set of independent coordinates, leaving hierarchy and feature interactions to emerge only implicitly. We propose KronSAE, a design that factorizes the latent space into heads and forms post-latent features as pairwise compositions of lower-dimensional pre-latents using mAND, a differentiable AND-like interaction. This imposes a compositional co-activation prior while remaining compatible with standard SAE objectives and variants such as TopK, Matryoshka, and Switch SAEs. KronSAE matches strong baselines on the EV-FLOPs frontier, improves interpretability of the latents, better captures the underlying correlated feature structure, and reduces encoder computational cost as an additional benefit. Code is available at https://github.com/corl-team/kronsae.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.CL"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}