Exact anatomy of a tiny integer GPT

An independent visual exploration of the complete model.mgw checkpoint — 14,272 int64 weights in Q16.48 fixed point, a 27-token vocabulary, one transformer layer — where every displayed value is either bit-exact or explicitly labeled as display geometry.

1 · Weight atlas

All nine weight tensors as heatmaps in literal matrix coordinates — every one of the 14,272 parameters at its exact row and column, with head boundaries marked, per-tensor histograms, a bit-occupancy view, and an exact round-to-F-bits preview. Below the panels, two derived comparison sections: four MLP neuron layouts and the residual coordinate swap.

Exact: raw int64, hex, and full decimal (≤ 48 fraction digits) for every cell; checksums tie the in-browser parse to the C loader, bit for bit.

open with 1

2 · Inference trace

The model's complete recorded output — 20 generated names — replayed step by step: residual stream, four attention heads, 128 ReLU units, logits, probabilities, and the exact roulette draw that picked each character.

Exact: every number was produced by the original C runtime, and your browser re-derives each step in BigInt integer arithmetic and verifies it matches.

open with 2

3 · Embedding space

62 vectors that live in the same 32-dimensional residual stream — 27 token embeddings, 27 output directions, 8 position vectors — in a hand-rolled 3-D view with literal coordinate axes, a derived PCA mode, and a four-panel PC1–4 treatment.

Exact: literal axes are stored coordinates; derived axes are labeled as such, and the basis provenance (an exact integer scatter matrix) is re-verified in the page.

open with 3

4 · Weight explorer

Any weight matrix as a 3-D point cloud, a heatmap with exact readouts, and the checkpoint's literal byte layout — plus an exact contribution mode (each stored weight × the recorded input it met at a chosen trace step) and a 4-D shadows view: four projections of (column, row, weight, contribution) under one shared camera. The address bar carries the whole scene as a deep link, and a captioned tour advances only by hand.

Exact: every readout is the stored int64 or the exact Q16.48 product fp_mul(w, x); each contribution row re-sums in BigInt to the recorded C-oracle output vector.

open with 4

…and one checkpoint in four MLP neuron orders

The sibling int-llm-coordinate-permutation repo reorders this exact checkpoint's 128 MLP neurons into three alternative physical layouts — pure row/column re-indexings that change neuron adjacency only, never a stored value. This repo re-proves the whole set on every test run (tools/derive_layouts.py): each variant checkpoint is reconstructed from the committed original and must hash to the recorded sha-256, every adjacency score is recomputed in exact integer arithmetic, and the C oracle replays all variants bit-identically — each step's 128-wide pre-activation vector is exactly the original's, reindexed. Compare them side by side in the atlas: smooth · spread · random.

The same sibling generator also swaps two coordinates of the 32-wide residual basis (0 ↔ 31) through every learned matrix at once — 892 of the 14,272 stored positions move, none changes value, and this repo's C oracle replays the swapped checkpoint bit-identically (tools/derive_hidden.py re-proves it on every test run). All nine matrix pairs are drawn side by side in the atlas — follow one coordinate across the swap: residual coordinate swap.

The artifact

The checkpoint is a single MGW v1 container holding eleven tensors: nine weight matrices, the tokenizer's character table (a…z, plus a beginning-of-sequence token), and the training run's final RNG state. Weights are int64 values on the Q16.48 scale: the stored integer is the value times 2⁴⁸. Nothing in this project ever rounds those integers — pages parse the checkpoint themselves and keep every scalar as a BigInt.

sha-256

The model's complete public output

Sampling with the checkpoint's stored RNG state at temperature 0.5 yields exactly these 20 names. The repo's C harness generated them with the original runtime, re-derived every intermediate value with an instrumented replica, and compared bit-for-bit: .

How exactness works here

Stored values never touch floating point. Pages decode the embedded checkpoint into BigInt64Arrays; tooltips and inspectors render raw integers, two's-complement hex, and exact terminating decimals.
Display geometry is labeled. Pixel positions, colors, and projections use float64 — a conversion that is itself exact for every weight in this file (all |raw| < 2⁵³; the build asserts it) — and every such use is marked as display-only.
The C runtime is the oracle. The trace harness compiles the unmodified original microgpt_int.c and compares its outputs against the instrumented replay: 122 steps, zero mismatches, identical final RNG state.
Pages verify themselves. On load, each page re-runs its exactness checks (checksum equality with the C loader, BigInt recomputation) and shows the result in the header badge. The same checks run DOM-less in CI via tools/page_check.js.
Everything regenerates deterministically. make test rebuilds every artifact twice and compares bytes.

Sibling experiments on the same checkpoint

This repo is one of a family built around the same integer-exact model. The copy of model.mgw vendored here is byte-identical (same sha-256) to the copies in the sibling repos — verified during construction and recorded in SOURCE_PROVENANCE.md. Results below are the siblings' own documented findings, cited — not re-verified here.

int-llm — the source: a GPT trained and run entirely in Q16.48 int64 arithmetic (github.com/nmicic/int-llm, pinned at commit 1b80e024). Its validation records report bit-identical inference across 14 hardware targets — down to an 8-bit AVR — with golden output hash c0d933ea340452ec. Its runtime is vendored here unmodified as the oracle.

int-llm-precision-ladder — descends the fraction-bit ladder from Q16.48: its records report the large-model evaluation's exact top-1 answers surviving down to F12, first changing at F11. The atlas page's round-to-F preview echoes the idea on stored weights only.

int-llm-coordinate-permutation — permutes internal coordinates coherently across all tensors: its records report 80/80 outputs identical after permutation. The checkpoint read here is that repo's committed copy.