62 vectors — 27 token embeddings (wte), 27 unembedding rows (lm_head),
8 position vectors (wpe) — drawn inside the one 32-dimensional residual stream
they all literally occupy.
Why one plot is a literal statement, not an
analogy: during inference each wte row and each wpe row is
added into the model’s 32-dimensional residual stream, and each
lm_head row is dotted with the final stream vector to produce
that token’s logit. All three tensors address the same 32 coordinates
of the same vector space, so they are plotted in one shared room.
Coordinates are Q16.48 fixed point: value = raw / 248, kept
BigInt-exact everywhere; floats appear only as screen geometry.
wte — filled circle (added at position 0..T)lm_head — ring (dotted with final stream)wpe — diamond (added by position)gold ring = selection (dashed = compare)
mode
families
assign strip clicks to:
derived axes — basis from wte only: exact integer scatter
G = ∑ (27·xᵢ − S)(27·xᵢ − S)ᵗ over the 27 wte rows;
eigenvectors are IEEE-754 float64 display geometry.
lm_head and wpe are projected onto this wte-derived basis.
camera
wpe polyline: the line connects consecutive
position indices p0→p1→…→p7 (an ordered sequence, nothing more).
family-ordered (wte, lm_head, wpe) with separator lines and
family-colored strips. cell color: cos < 0 → blue #4dabf7,
cos > 0 → orange #ffa94d, 0 → neutral — the color is a
display float; hover or arrow to a cell for the exact integer readout.
click a cell to select the pair.
hover or arrow to a cell…
space — help
62 vectors from the committed checkpoint share one literal
32-dimensional room: wte rows (tokens a..z plus ⌀ = BOS) and
wpe rows (positions p0..p7) are added into the residual
stream; lm_head rows are dotted with the final stream vector
to make logits. Same 32 coordinates, same space — that is the whole
justification, stated literally. This page makes no semantic claims;
it shows exact quantities and labeled display geometry only.
literal vs derived axes
MODE 1 plots three of the
32 literal stored coordinates (a banner counts the 29 dims not
drawn; solid axis lines, “literal, exact” tags).
MODES 2/3 plot derived
PCA geometry: the basis comes from wte’s exact integer scatter, its
eigenvectors are float64 display geometry, and lm_head/wpe are projected
onto that wte-derived basis (dashed axis lines, “derived” tags,
violet provenance banner). Mode 3 shows four 3-PC subsets of PC1..PC4;
each panel names the PC it omits and that PC’s share of wte variance.
encodings
#74c0fc filled circle
wte row (token embedding)
#f783ac ring
lm_head row (unembedding)
#ffd43b diamond
wpe row (position); optional polyline joins p0→p7 in index order
text label
every mark carries its literal label: a..z, ⌀ (BOS), p0..p7
mark size / brightness
3D depth (nearer = larger, brighter) — display geometry
gold ring / dashed gold
selection / compare, mirrored in every view and panel
matrix cell color
display-float cosine: blue negative, orange positive, neutral zero
exactness contract
Exact (BigInt, never floats): raw Q16.48 coordinates, hex
and decimal readouts, dot products (integers at Q32.96 scale), squared
norms, the recomputed wte scatter G, and the ordering of the
nearest-by-cosine list (sign classes + cross-multiplied comparisons).
Display-only floats (labeled in place): screen projections, PCA
eigenvectors/coordinates, tick values, matrix cell colors, and every
rounded cosine numeral. The startup self-check re-parses the embedded
checkpoint, re-hashes all 9 tensors, and re-derives G in this browser.
keyboard
123
mode: literal axes / derived PCA / PC tetrad
TUP
toggle families wte / lm_head / wpe
L
label density: all ↔ selection only
O
wpe polyline on/off
D
pair matrix panel
K
tetrad: link ↔ unlink the four cameras (linked = orbit one panel, all four turn)
A
auto-rotate on/off — a gentle orbit, one revolution per 48 s; grabbing a camera (drag, wheel, orbit keys, reset) pauses it, mode/family/matrix switches do not
walk
(button) cycle the selection through all 62 vectors, one per second — gold ring and inspector follow; composes with auto-rotate; never starts by itself
←↑→↓
orbit the focused canvas (yaw/pitch; linked tetrad: all four together); in the matrix: move the active cell
+−
zoom the focused canvas (linked tetrad: all four)
R
reset the focused canvas view (linked tetrad: the shared view)
Tab, Enter, Shift-Enter
walk the vector chips; select / set compare
Esc
close this overlay (or the matrix panel)
?
toggle this overlay
mode-1 extras: x/y/z selects pick any of the 32 residual
dims; the strip assigns a clicked dim to the armed axis.