SNMF 1: How does the type of activations and number of top activating tokens affect concept detection in LLMs?

Motivation

My goal is to reproduce and extend the essential results in Shafran et al., 2025 paper on the decomposition of MLP activations using semi non-negative matrix factorization(SNMF) [1]. SNMF is an unsupervised method for improving the interpretability of LLMs. The paper claimed to have successfully used the technique for concept detection and causal steering. To test some of the results on concept evaluation, I will be evaluating 2 research questions. Firstly, how does training SNMF on mlp and residual stream impact concept detection? Secondly, how does the number of top-activated tokens used for generating feature or concept description impact concept detection?

Experimental procedure and Methodology

SNMF decomposes activations using, $A \approx ZY$. $Z \in \mathbb{R}^{mlp\_dim \times k}, Y \in \mathbb{R}_{\ge 0}^{k \times n}$. Its objective is to minimize reconstruction loss using $\mathcal{L}(Z, Y) = \left\| A - Z Y \right\|_F^2$.

The following are the series of steps required to test out the research questions [1].

• Train MLP and residual stream activations using SNMF on concepts dataset.
• Several k features were used such as 100, 200, 300 and 400.
• Using a predefined prompt, gpt-4o-mini is used to create natural language description of the concept encoded by MLP or residual stream features.
• For every concept description generated per feature, gpt-4o-mini is used to generate 5 activating and neutral sentences.
• Sentences are then fed into the model(gpt2-small) and MLP and residual stream activations are captured per layer.
• Lastly, concept detection score is computed, $\log\!\left(\frac{a_{\mathrm{activating}}}{a_{\mathrm{neutral}}}\right)$.

Research Question 1

The paper only tested SNMF on MLP activations. So, I thought it would be cool to see if training on residual stream is better. So I asked how does training SNMF on mlp and residual stream impact concept detection across different layers and number of features(k)?

Overall, training on MLP activations performed better than residual stream. This is surprising as residual stream is a more enriched representation. In addition, in spite of the number of features, concept detection did better at layer 0. The logic here is that, at early layer, concepts are disentangled so they are easier to separate out into different feature directions [1].

Research Question 2

Based on the findings here, it is believed the number of top-activating sequence has a significant impact on the quality of concept description of the explainer model(gpt-4o-mini in this case) [2]. Hence, I tried to ask the question, how does the number of top-activating tokens impact quality of feature description and ultimately concept detection?

In this context, across layer 0, 6, 11, there is no stark difference in concept detection scores from 0-10 top-activated tokens. However, in both layer 6 and 11, the most concept detection score are observed at top-2 and top-8 activated tokens and their respective scores approximately the same.

Probing Questions

I have a couple of probing questions and some hypothesis in relation to my results.

• Why is there a significant difference between concept detection scores in my experiment for gpt2-small versus one obtained in the paper? The biggest score is about 0.3 while the paper has its biggest score for gpt2-small at 3. This is about 10x magnitude difference. This is really a big difference. I believe enough to invalidate the claims of the paper. Will be holding off on this till I compare results with that of SAEs.
• Numbers were only reported for concept detection score. What are qualitative ways to show concepts truly encoded by the SNMF features and how can they be traced back to the original concept dataset. This way we know we are not just modelling, encoding and testing noise.

Further work

• Compare SNMF with SAE technique just as done in the paper [1].
• Use SVD instead of matrix factorization as used in Shafran et al., 2025.

Conclusion

MLP activations perform significantly better at early layers than residual stream. Finally, varying number of top-activated tokens has very minimal impact on concept detection.

Code

Available here

References

1. Shafran, O., Geiger, A., & Geva, M. (2025). Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization. ArXiv, abs/2506.10920.
2. Bills, et al., "Language models can explain neurons in language models", 2023.

Citation

If you find this article useful, please cite it as:

BibTex

@article{abdulhakeem2026snmf-1,
  title={How does the type of activations and number of top activating tokens affect concept detection in LLMs},
  author={Abdulhakeem, Adefioye},
  year={2026},
  month={February},
  url={https://adefioye.github.io/blog/snmf-1}
}