what a number means to the person who measured it, and to the chip that stores it
A micrometer sextant reads to a tenth of an arcminute. When the navigator writes down 25°14.3′, that is not “approximately 25.238°” — it is an exact encoding of the recorded reading: this many whole tenths of arcminutes on the drum. (The physical observation behind it still carries observational error, of course — the point here is the recorded value.) Human measurement is quantized by the instrument, and the human knows it. The honest way to store that record is to keep it on its lattice:
The exact rational value of the written record 25°14.3′, expressed in decimal degrees, is 25.2383333… (the 3 repeats forever) — a rational number, 1514300/60000. A binary double cannot hold most decimal fractions, so it stores the nearest sum of powers of two. The red tail shows the exact binary64 value after conversion. The integer form retains the written decimal value with zero representation error; neither representation removes uncertainty from the original observation.
Neither is “wrong” for navigation — the double's error here is ~5×10−17 degrees, physically meaningless. The point is about bookkeeping: the integer stores exactly the recorded value; the double stores the nearest representable neighbor and rounds it back to look the same when printed. The difference is invisible on the page — and visible the moment you hash the bits.
In goes 1514300 — the CLI takes the drum reading as a scaled integer, and there is no decimal parser anywhere in the program. Out comes an altitude stored as 2901 centidegrees and printed as “29.01” by integer division: quotient 29, remainder 01, a period typeset between them. The dot is inserted for the reader — exactly as the log-table navigator kept the decimal point in the margin of the worked page while the columns underneath carried only digits. Floating point moved the point into the data itself, as a movable per-value binary exponent; fixed point moves it back out into the design: agreed once, written down, never carried at runtime.
Same shape as the navigator's toolkit: degrees-and-minutes on the notebook page, five-place mantissas inside the log table, natural values at the end. Coarse lattices where humans read, fine lattices where the arithmetic runs — and every conversion between rungs is an integer operation with a defined rounding rule, not a format change with silent loss.
A 1950s navigator knew the table step (0.1′), the almanac step (0.1′), and worked inside them. Binary64 usually makes arithmetic quantization negligible at navigation scales, but an application still needs an explicit model and output error budget. This library states those scales and gates directly. Even “unit length” is stated honestly: the Vega vector on page 1 squares to |v|² = 1152921504834171165 against 260 = 1152921504606846976 — unit to one part in 5×109, an error that is known, bounded, and gated, not hidden.
| DECLARED STEP / GATE | WHAT IT PINS |
|---|---|
| 0.01° output grid | printed altitude, lat/lon, and azimuth round to centidegrees — at most 0.30′ from this final rounding alone; Method B's separately measured azimuth disagreement reaches 0.577′ |
| arcminute-level almanac budget | truncated ephemeris series carry a declared error budget, measured against oracles rather than assumed |
| --ephemeris-check · 362 rows | star (270), Sun (45), and Moon (47) places vs an independent ephemeris stack (Skyfield + JPL DE421) |
| --external-check · ~1,250 rows | printed Nautical Almanac / Air Almanac pages and published worked examples |
| --cross-check · 411 rows | a separate computed implementation (PyEphem/libastro, with shared-model caveats reported by the checker) plus independent intercept geometry |
| --scenario-check · 29 scenarios | whole-pipeline recovery against Skyfield scenarios, supplemented by separate published examples and intercept rows |
$ ./sight_reduction --golden golden determinism hash: 0x3d6ddb7b86f2c7d2 (4096 cases + 9 edges) bit-exact against committed golden value: PASS
4096 pseudo-random sight reductions plus 9 hand-picked edge cases,
every output folded into one hash. The committed value reproduces bit-for-bit across the
CI matrix, on both arithmetic backends (native __int128 and portable two-limb):
And the same library, cross-compiled freestanding for five 32-bit MCU targets, runs six per-feature test schedules under QEMU whose 30 hashes all match the host bit-for-bit:
The Cortex-M4 has an optional FPU on silicon but is deliberately built soft-float; every embedded image is audited for zero FP opcodes and soft-float helpers. For the six committed profile schedules, the freestanding targets produce the same bits as their host counterparts. This is a measured property of the stated matrix and schedules, not a promise about every future compiler or processor.