What Residual Streams Look Like

I have been playing around with GPT-2 to see what state the transformer is in after every layer, and I wanted to document what I tried and what came out of it. GPT-2 small is obviously too small to say anything about real LLMs. I just wanted to see if it’s usable as a testbed for the control questions from Part 1, and whether it actually behaves like a dynamical system when you look at it that way.

Setting up GPT-2

Let’s start with a really simple setup: make it answer “The name of the capital of the Republic of Korea is”, and at every hidden state apply the model’s readout to see what it would predict from that state.

The results are quite fascinating. Earlier layers just attempt syntactic guesses like ‘now’, with no actual answer in there. Layers 10 and 11 read Seoul. And then the final output itself picks ". So the intermediate states look like they know Seoul and the final output doesn’t say it.

What the readout sees

I tried this with two other prompts (Japan → Tokyo, France → Paris) and printed some plots along with it.

A few things that came to my attention:

• Seoul and Tokyo hit rank 1 at states 10–11 and then fall at 12. Paris only reaches rank 1 at 12.
• The log-prob of the answer shoots up after the first layer, from $-40$ to about $-16$. But $-16$ is still way below random because random probability is $\log (1/50257)\approx -10.8$, so the state doesn’t see the answer at all until layer 9, where it finally crosses the line.

Then I added entropy, because rank and log-prob only tell you about the one token you’re tracking.

Entropy is in nats, and $e^H$ is roughly how many tokens the distribution is spread over. It drops from 3.6 to 2 over layers 1–7 (≈37 → 7 tokens) while Seoul is still at rank ~10⁴. So the state commits to something, syntax, before it has any semantics. Then it spikes back to 3.6 at layer 9, which is exactly where Seoul crosses uniform. I think we can read that as the old commitment getting released. It bottoms out at 1.2 nats at layer 10 (≈3 tokens, Seoul at $p\approx 0.66$), and then block 12 blows it out to 6.5 nats, ≈650 tokens. So the final output isn’t “wrong”. GPT-2 small is genuinely unsure, " wins with very little probability, and Seoul is rank 7.

The system behaves differently per layer

Step size, rotation and state norm all tell the same story: block 1 is a big jump, blocks 2–11 are small steps that barely rotate the state, and block 12 is another big jump.

I initially wanted to call this division “regimes”, but that’s the wrong word, as in control theory a regime is a region of state space, and these boundaries are in the layer index. It’s perhaps more a time-varying system $x_{l+1}=f_l(x_l)$ where the middle ten $f_l$ happen to look alike.

For blocks 2–12 the three lines sit on top of each other, which means the gain doesn’t depend on $\Vert ϵ\Vert$ across a 100x range, which is the definition of near-linear over that range. Against $\Vert x_{12}\Vert \approx 480$, a poke of norm 10 is a 2% nudge. I guess a small perturbation in a 768-dimensional vector space is almost always going to be invisible.

Block 1 is nonlinear and amplifying, but not for the reason I wanted. $\Vert x_0\Vert \approx 3$, so $\Vert ϵ\Vert =10$ can’t really be called a perturbation…

I still don’t know where linearity breaks.

Limitations

I did try three prompts but they’re all “capital of X”, so I’m playing with a tiny neighbourhood of the state space. The random $ϵ$ is basically orthogonal to anything the model cares about, so that doesn’t really work either.