Cross-model Transfer of Matryoshka Sparse Autoencoders

A replication and extension of Matryoshka Sparse Autoencoders. I reproduced the paper's core result on the synthetic hierarchy and at scale on Gemma-2-2B, confirming that Matryoshka SAEs avoid the feature-absorption failure mode that vanilla SAEs exhibit.

I then transferred the setup to other model families — Llama-3.1-8B and Gemma-3-12B — to test how well the result generalizes beyond the original setting, and documented the full replication in a public writeup on LessWrong.

This project is part of my broader research direction on the generalizability of mechanistic interpretability methods: understanding when results established on one model transfer to others.