Replicating Matryoshka Sparse Autoencoders
Replicating feature recovery and comparing Matryoshka and standard sparse autoencoders
- Status
- Published Writeup (LessWrong)
- My role
- Independent Researcher
- Period
- May 2026 - June 2026
Overview
I independently replicated Matryoshka Sparse Autoencoders, reproducing controlled synthetic experiments that compare feature recovery with standard sparse autoencoders and extending the implementation to transformer activations.
The experiments examine feature splitting and absorption, and show why strong reconstruction performance does not necessarily establish faithful recovery of the underlying features. The public writeup documents the replication, implementation, results, and lessons from the tested settings.
I also investigated additional model settings to understand the scope of the method’s advantages over standard sparse autoencoders. The replication code and technical writeup are public.
Methods and Tools
- PyTorch
- Sparse Autoencoders
- Synthetic feature hierarchies / Transformer activations