feature-lens

A research engineering toolkit for understanding how SAE features relate to each other, and to upstream / downstream components.

For projects supported by feature-lens, see projects

Quickstart

git clone https://github.com/dtch1997/feature-lens
cd feature-lens
pip install -e .

Development

Refer to Setup for how to set up development environment.

Implementation Details

Techniques for finding relevant feature associations:

• Activation patching (employed in Causal Graphs)
• (Total) attribution patching (employed in Sparse Feature Circuits)
• Direct attribution patching (employed in MLP Transcoders, Attention-out SAEs)

Note: "Total" attribution patching calculates the full effect of one feature on another via gradient backpropagation. "Direct" attribution patching estimates only the direct effect, which can be calculated analytically using matrix multiplication.

Tools which will be implemented.

• SAE features. We will use SAEs from SAE-Lens, which are annotated and have Neuronpedia visualizations.
• Feature dashboards. By visualizing the "functional connectome" of SAE features, we may obtain novel insight about what an SAE feature is doing. For each target SAE feature, we can create a dashboard of all relevant upstream / downstream features.

Ideas under consideration.

• Feature clustering. Features with similar upstream and downstream features could be hypothesized to be performing a similar role. Clustering features based on their connections may reveal novel insight about the general types of "functional role" played by SAE features.
• Linear direction features. Steering vectors are like an SAE feature because they are added to the residual activations.