Work

Replicating Matryoshka Sparse Autoencoders

Replicating feature recovery and comparing Matryoshka and standard sparse autoencoders

Status
Published Writeup (LessWrong)
My role
Independent Researcher
Period
May 2026 - June 2026

Overview

I independently replicated Matryoshka Sparse Autoencoders, reproducing controlled synthetic experiments that compare feature recovery with standard sparse autoencoders and extending the implementation to transformer activations.

The experiments examine feature splitting and absorption, and show why strong reconstruction performance does not necessarily establish faithful recovery of the underlying features. The public writeup documents the replication, implementation, results, and lessons from the tested settings.

I also investigated additional model settings to understand the scope of the method’s advantages over standard sparse autoencoders. The replication code and technical writeup are public.

Methods and Tools

  • PyTorch
  • Sparse Autoencoders
  • Synthetic feature hierarchies / Transformer activations