Inference trace — 20 exact generations, every integer visible
Q16.48 fixed-point GPT (32d / 4 heads / 1 layer), replayed step by step from the C harness record and
re-derived in exact BigInt arithmetic in this page. Press ? for keys and encodings.
Samples
⌀ = BOS (token 26): the start marker; a step that chooses ⌀ ends the sample (that step is recorded but ⌀ never enters the text).
Residual stream — six 32-cell vectors, exactly as computed
hover a cell, or focus the grid and use arrow keys.
Attention — per head: weight bars over key positions; thin strip = pre-softmax scores
hover a bar, or focus a head and use ← →.
MLP hidden units — h_pre = fc1 · xn_mlp, 128 units (16×8, row-major)
hover a unit, or focus the grid and use arrow keys. R toggles pre/post ReLU.
Output & sampling — 27 logits, exact ×2 temperature scale, integer softmax, one RNG draw
hover a row, or focus the grid and use ↑ ↓.
Verification
Every panel shows the recorded values written by the C harness (tools/trace_harness.c, which runs the
unmodified upstream runtime as its oracle). This page independently recomputes the full integer forward pass and the
sampling chain in BigInt — same shifts, same truncations, same iteration counts as fp_math.h / microgpt_int.c — and asserts
equality field by field. So every number on screen is simultaneously a recorded value and a recomputed one.
Help — encodings & keys
Number format: Q16.48
Every model scalar is an int64 "raw" value; its meaning is
raw / 248. FP_ONE = 248 = 281474976710656 represents 1.0. Inspectors show three exact forms:
the raw integer, its 16-digit two's-complement hex image (the bytes on disk), and the exact decimal
(2-48 has a terminating decimal expansion, so at most 48 fraction digits — nothing is rounded).
Labels shortened for space always end with "…" and are prefixes of the exact decimal; inspectors show every digit.
Recorded vs recomputed
All displayed numbers are the C harness record. On load, this page parses the embedded model.mgw,
re-verifies all 9 tensor FNV-1a64 checksums, re-verifies the whole xorshift64 chain (every step's rng_before, r, and the
final state), and recomputes sample 1 completely in BigInt. Selecting any step recomputes that whole sample from scratch
(KV cache rebuilt by recomputation, not from the record) and compares every field; the Verification panel reports the
exact count of equal fields. Bar lengths and colors are float display geometry only — never the numbers.
Color encodings
Signed values use one diverging ramp: negative #4dabf7 ↔ dark neutral ↔ positive #ffa94d, symmetric
about zero; each strip labels its endpoints with the actual max |value| in view. Probabilities and attention weights in
[0,1] use a sequential ramp from the panel surface to gold #ffd166 (bars also encode value by height/length).
Heads keep fixed colors — h0 #fa5252, h1 #fab005, h2 #38d9a9, h3 #4dabf7 — used only in head labels, never for data
values, and always accompanied by text. Numbers are always drawn in ink, never in ramp colors; color is never the only
encoding.
Panels
Residual stream — the exact stage order of inference_forward():
emb = wte[token] + wpe[pos] (raw int64 add) → RMSNorm gives x0 (scale_init) → RMSNorm gives xn_attn (scale_attn) which
feeds Wq/Wk/Wv → x_mid = x0 + Wo(ao) (residual add) → RMSNorm gives xn_mlp (scale_mlp) which feeds fc1 →
x_out = x_mid + fc2(ReLU(h_pre)) (residual add). RMSNorm here is: ms = Σ fp_mul(x[i],x[i]) / 32 (integer divide),
scale = fp_inv_sqrt(ms + FP_ONE/100000), out[i] = fp_mul(x[i], scale). The left lane arrow is the data flow; right lane
brackets mark the two residual adds. There is no norm after x_out: lm_head reads x_out directly.
Attention — head h owns dims 8h..8h+7 of q/k/v. score(s) = fp_mul(Σ_j fp_mul(q_j, k_j(s)), attn_scale)
with attn_scale = fp_inv_sqrt(8) = 0.353553… (raw 99516432383215). Softmax per head: subtract the max score, fp_safe_exp
(clamped to ±10), sum, then weight(s) = fp_mul(e(s), fp_div(FP_ONE, sum)). Bars: height and gold ramp = weight; the thin
strip below is the signed pre-softmax score (diverging ramp); the letter under each bar is the input character at that key
position (⌀ = BOS). Σ weights is the exact BigInt sum of the recorded weights — integer truncation usually leaves it a few
ulp below FP_ONE, and it is displayed exactly as the integers give it.
MLP — h_pre[i] = Σ_c fp_mul(fc1[i·32+c], xn_mlp[c]) (fc1 row i), shown pre-ReLU (diverging) or
post-ReLU (post = max(0, h_pre); units with h_pre ≤ 0 are extinguished — outline only). "N/128 fire" counts h_pre > 0,
the exact C predicate.
Output & sampling — logits = lm_head · x_out. Temperature 0.5: the C code multiplies each
logit by inv_temperature = fp_div(FP_ONE, FP_ONE/2) = 2.0 exactly, so logits_scaled is an exact doubling (asserted per
element every step). Softmax over the scaled logits gives probs; prob_total is their exact integer sum — truncation
leaves it a few ulp below FP_ONE, and the callout shows the exact gap. Sampling draws ONE xorshift64 value
(s^=s≪13; s^=s≫7; s^=s≪17), takes the top 48 bits as uniform ∈ [0,1), and computes r = fp_mul(uniform, prob_total).
The chosen token is the first whose cumulative probability exceeds r — verbatim C: cum += w[i]; if (r < cum) return i;
(fallback n−1 if the loop completes). The roulette strip shows the 27 cumulative segments with the marker at r
(marker position is display geometry; the label carries the exact r).
Keyboard
[]
previous / next sample
←→
previous / next step (when no grid is focused)
Space
autoplay (1 step/s; runs on into the next sample, wrapping at the end)
R
toggle MLP pre-ReLU / post-ReLU
Tab
move focus between controls and grids
arrows in a grid
move that grid's cell cursor (residual and MLP: 2-D; attention: ← →; output: ↑ ↓; roulette: ← →)