Llama-3.1-8B-Instruct Brain Atlas — First Look

I ran a full activation-census brain atlas on Llama-3.1-8B-Instruct. Not a benchmark. Just opening the hood while it runs to see where the compute lives, where the style lives, and which parts you can safely edit.

Model: meta-llama/Llama-3.1-8B-Instruct
Layers: 32
Attention: 32 query heads, 8 KV heads (GQA)
Corpus: 8,965 prompts
Dataset: juiceb0xc0de/llama-3.1-8b-instruct-atlas

The big picture

Llama-3.1-8B-Instruct is a textbook dense transformer, and the atlas reads like one. Attention does distributed computation across ~57 directions per head. Early layers handle copy/induction bookkeeping. Late layers route vocabulary through the SwiGLU gates. Style is encoded into queries early and re-read at the very end. Most of the network is safe to touch surgically, but there are two specific directions you do not want to hit blind.

Headline numbers

• Feature taxonomy: 45.2% partial_shared, 32.4% broadly_shared, 15.0% non_activated, 7.4% all_shared, ~0.01% clean specific_* directions.
• OV circuits: spectral conc 0.055, effective rank ~57. Attention is doing weighted computation, not sparse lookup.
• Induction: strongest at layer 0 (avg 0.89, top head 1.26). Copy-and-continue lives right at the input.
• Logit lens: peak is layer 17 gate f7761 at F-stat 522, followed by a narrative-fiction feature at layer 18 up f9883 and a business/operations axis at layer 31 gate f2604.
• Code analysis: gate is 98.4% selective, up 93.6%, mlp 89.8%. Code is walled off in the MLP path.
• Coactivation: business is the tightest cluster (avg correlation 0.78), followed by introspection (0.67), community (0.65), writing (0.62), and roleplay (0.62).
• Sub-Zero: 76.9% of tested axes pass the capability fence. Average damage is low, but two directions are genuinely load-bearing.
• Compliance/behavior: linearly readable at every layer (classifier accuracy 0.92–1.00), concentrated in q and k, strongest at layer 0 and layers 30–31.

The geometry that matters

Layer 0 is the induction workhorse

The highest induction scores in the whole model are at layer 0. Head 22 scores 1.26, head 5 scores 1.24, and several others are above 1.0. By the late layers induction has dropped to ~0.27. This is the classic dense-transformer pattern: early layers copy and track tokens, deeper layers do the structured reasoning.

The MLP gate is sparse, selective, and fragile

gate_proj fires on only ~17% of inputs and carries 40% dead directions, but it is also where the cleanest concept detectors live. Almost all of the rare monosemantic features are gate features: narrative/fiction, business/operations, data, core-technical, introspection.

It is also the most dangerous projection to edit. Only 57.8% of gate directions pass the capability fence, versus 86.0% for up_proj and 83.3% for down_proj. The gate is doing a lot with a little.

Two landmines, one at each end

• Layer 1 down_proj axis 1: damage up to 2.64 nats/token on code and multilingual, yet it explains only 24% of its block's variance. Low-energy, critical, easy to miss.
• Layer 31 gate_proj axis 0: damage 1.67 on code, but it explains 99.8% of its block's variance. Load-bearing but clean — exactly the kind of direction you can reason about once you know it is there.

So the danger is not only late. It is bookended.

Style lives in queries, and refusal flips at the last layer

The compliance/behavior probe finds its cleanest signal in q and k. Average separation is highest at layer 0 q and again at layers 30–31 k/q. Llama writes "how to behave" into the query very early, then re-reads it at the exit.

The refusal geometry is even sharper. The angle between the corporate/assistant direction and the refusal direction stays near 160–179° through most of the network — nearly opposite. Then at layer 31 it collapses to 82°. Whatever refusal does in Llama, the final layer is where the geometry changes.

The logit lens tells a story

The strongest output-token predictors line up with real semantic axes:

• Layer 18 up f9883 pushes for plot, novel, character, director — a clean narrative/fiction direction.
• Layer 31 gate f2604 pushes for Operational, Efficiency, Operations, Maintenance while suppressing genre, genres, fiction — a business/operations ↔ creative-writing tradeoff parked right before output.
• Layer 16 gate f2572 actively suppresses novels, Novel, Fiction.

A creative/technical tension runs through the mid-to-late gate features and lands, fully formed, at the output vocabulary.

What surprised me

The business cluster is the tightest coactivation group in the model. In a balanced corpus you might expect technical or code clusters to dominate, and those are present, but the business bucket has the highest internal correlation (0.78). There is a strong professional-style subnetwork in this checkpoint.

Also, the refusal-angle collapse at layer 31 is unusually clean. If you are doing abliteration or refusal-steering work, that layer is the obvious place to intervene.

What I skipped

• sae_features is empty — no SAE pass was run, so dictionary-learning features are not corroborating the census taxonomy.
• compliance_score in the OV table is unpopulated, so attention-side compliance read comes from query/key feature stats, not the OV circuits.
• Tokens in the logit lens are raw BPE pieces, so read them as fragments, not whole words.

Bottom line

Llama-3.1-8B-Instruct is a clean, readable dense transformer with real surgical headroom. ~77% of MLP directions are safe to ablate, the attention is distributed and induction-heavy, and the behavior/style structure is linearly readable throughout the stack. The two things you absolutely need a map to avoid are a diffuse early down-projection at layer 1 and a concentrated late gate direction at layer 31.

If you are poking around inside this model, start with the SwiGLU gates for clean concept detectors and layer 31 for the refusal/behavior geometry. Has anyone else mapped this checkpoint? Would love to compare notes.

Llama-3.1-8B-Instruct Brain Atlas — First Look

I ran a full activation-census brain atlas on Llama-3.1-8B-Instruct. Not a benchmark. Just opening the hood while it runs to see where the compute lives, where the style lives, and which parts you can safely edit.

Model: meta-llama/Llama-3.1-8B-Instruct
Layers: 32
Attention: 32 query heads, 8 KV heads (GQA)
Corpus: 8,965 prompts
Dataset: juiceb0xc0de/llama-3.1-8b-instruct-atlas

The big picture

Llama-3.1-8B-Instruct is a textbook dense transformer, and the atlas reads like one. Attention does distributed computation across ~57 directions per head. Early layers handle copy/induction bookkeeping. Late layers route vocabulary through the SwiGLU gates. Style is encoded into queries early and re-read at the very end. Most of the network is safe to touch surgically, but there are two specific directions you do not want to hit blind.

Headline numbers

• Feature taxonomy: 45.2% partial_shared, 32.4% broadly_shared, 15.0% non_activated, 7.4% all_shared, ~0.01% clean specific_* directions.
• OV circuits: spectral conc 0.055, effective rank ~57. Attention is doing weighted computation, not sparse lookup.
• Induction: strongest at layer 0 (avg 0.89, top head 1.26). Copy-and-continue lives right at the input.
• Logit lens: peak is layer 17 gate f7761 at F-stat 522, followed by a narrative-fiction feature at layer 18 up f9883 and a business/operations axis at layer 31 gate f2604.
• Code analysis: gate is 98.4% selective, up 93.6%, mlp 89.8%. Code is walled off in the MLP path.
• Coactivation: business is the tightest cluster (avg correlation 0.78), followed by introspection (0.67), community (0.65), writing (0.62), and roleplay (0.62).
• Sub-Zero: 76.9% of tested axes pass the capability fence. Average damage is low, but two directions are genuinely load-bearing.
• Compliance/behavior: linearly readable at every layer (classifier accuracy 0.92–1.00), concentrated in q and k, strongest at layer 0 and layers 30–31.

The geometry that matters

Layer 0 is the induction workhorse

The highest induction scores in the whole model are at layer 0. Head 22 scores 1.26, head 5 scores 1.24, and several others are above 1.0. By the late layers induction has dropped to ~0.27. This is the classic dense-transformer pattern: early layers copy and track tokens, deeper layers do the structured reasoning.

The MLP gate is sparse, selective, and fragile

gate_proj fires on only ~17% of inputs and carries 40% dead directions, but it is also where the cleanest concept detectors live. Almost all of the rare monosemantic features are gate features: narrative/fiction, business/operations, data, core-technical, introspection.

It is also the most dangerous projection to edit. Only 57.8% of gate directions pass the capability fence, versus 86.0% for up_proj and 83.3% for down_proj. The gate is doing a lot with a little.

Two landmines, one at each end

• Layer 1 down_proj axis 1: damage up to 2.64 nats/token on code and multilingual, yet it explains only 24% of its block's variance. Low-energy, critical, easy to miss.
• Layer 31 gate_proj axis 0: damage 1.67 on code, but it explains 99.8% of its block's variance. Load-bearing but clean — exactly the kind of direction you can reason about once you know it is there.

So the danger is not only late. It is bookended.

Style lives in queries, and refusal flips at the last layer

The compliance/behavior probe finds its cleanest signal in q and k. Average separation is highest at layer 0 q and again at layers 30–31 k/q. Llama writes "how to behave" into the query very early, then re-reads it at the exit.

The refusal geometry is even sharper. The angle between the corporate/assistant direction and the refusal direction stays near 160–179° through most of the network — nearly opposite. Then at layer 31 it collapses to 82°. Whatever refusal does in Llama, the final layer is where the geometry changes.

The logit lens tells a story

The strongest output-token predictors line up with real semantic axes:

• Layer 18 up f9883 pushes for plot, novel, character, director — a clean narrative/fiction direction.
• Layer 31 gate f2604 pushes for Operational, Efficiency, Operations, Maintenance while suppressing genre, genres, fiction — a business/operations ↔ creative-writing tradeoff parked right before output.
• Layer 16 gate f2572 actively suppresses novels, Novel, Fiction.

A creative/technical tension runs through the mid-to-late gate features and lands, fully formed, at the output vocabulary.

What surprised me

The business cluster is the tightest coactivation group in the model. In a balanced corpus you might expect technical or code clusters to dominate, and those are present, but the business bucket has the highest internal correlation (0.78). There is a strong professional-style subnetwork in this checkpoint.

Also, the refusal-angle collapse at layer 31 is unusually clean. If you are doing abliteration or refusal-steering work, that layer is the obvious place to intervene.

What I skipped

• sae_features is empty — no SAE pass was run, so dictionary-learning features are not corroborating the census taxonomy.
• compliance_score in the OV table is unpopulated, so attention-side compliance read comes from query/key feature stats, not the OV circuits.
• Tokens in the logit lens are raw BPE pieces, so read them as fragments, not whole words.

Bottom line

Llama-3.1-8B-Instruct is a clean, readable dense transformer with real surgical headroom. ~77% of MLP directions are safe to ablate, the attention is distributed and induction-heavy, and the behavior/style structure is linearly readable throughout the stack. The two things you absolutely need a map to avoid are a diffuse early down-projection at layer 1 and a concentrated late gate direction at layer 31.

If you are poking around inside this model, start with the SwiGLU gates for clean concept detectors and layer 31 for the refusal/behavior geometry. Has anyone else mapped this checkpoint? Would love to compare notes.