What the numbers on every trace mean, how they are computed, and — just as
important — what they do not claim. The single implementation is
j7scope/rigor.py;
the frontend only displays what was baked into each trace at capture time.
1. What a read-out is
At each generated token we take the residual stream h_l at layer
l, map it through the model's expected input–output Jacobian
J_l into final-layer coordinates, and decode with the model's own
unembedding — the J-lens of
Anthropic's global-workspace paper.
The result is a top-k list of tokens the model is inclined to say next.
Because the tokenizer is bilingual, that list naturally contains both Chinese and
English tokens; we bucket it by script into a 中文 column and an English column.
The zh/en split is a display heuristic (by character script), not a
linguistic judgment. Mixed tokens, proper nouns, code, and byte fragments fall to
an other bucket and are ignored by the rigor layer.
2. Concept space
Raw "欺骗" and "deception" never string-match, so agreement is measured in
concept space: each read-out token is mapped to a concept via a lexicon
(the probe corpus's expected lists, plus synonyms). Tokens with no
concept (fillers, punctuation) are dropped. Overlap between two concept sets uses
the overlap coefficient |A∩B| / min(|A|,|B|).
3. Two rigor questions
| Metric | Question | Where shown |
|---|---|---|
| Intra-token | Within one session, do the token's zh column and en column point at the same concept? | Replay + each Compare panel |
| Cross-trace | Across two position-aligned sessions (e.g. a Chinese and an English run), do they land on the same concept at the same position? | Compare (top strip) |
Cross-trace is the stronger cross-lingual signal — it compares two genuinely different-language sessions rather than one bilingual read-out.
4. The null baseline — and why raw overlap is meaningless without it
An observed overlap is compared against a shuffled-pairing null: pair this position's concepts with other positions' concepts and recompute. The null's p05–p95 band is drawn behind every bar; only a bar past the band is a real signal.
When samples are far fewer than dimensions (here n≈30, d≈3584), chance-level
similarity is high, not zero — the raw CKA between two independent random
matrices can reach ~0.7. So we never bare-report an overlap or CKA number;
it always appears against its null. This is enforced in
metrics.py
and is the reason the null band exists.
5. Sharedness
Normalizing the observed overlap to the range between chance and the achievable ceiling:
- observed — the concept overlap at this position.
- null.mean — chance level from shuffled pairings.
- same_lang_baseline — the within-language ceiling (how well one language agrees with itself).
- CI — a 95% interval from bootstrap resampling of the concept lists.
0 means at chance; 1 means as shared as a language is with itself; a wide CI means the signal is uncertain (typically concrete entities with short read-outs).
6. Honest limitations (provisional-for-P1/P5)
- same_lang_baseline and CI are estimated from a single trace here (tokens as pseudo-prompts). With a real multi-prompt corpus (roadmap P3) they become proper cross-prompt resampling.
- sharedness is a definition, not a truth. CKA, SVCCA, top-k overlap, and activation-patching would each rank things slightly differently; the definition above is one choice, stated so it can be argued with.
- Demo traces are synthetic (clearly marked DEMO). They are run through the real rigor pipeline, but carry no empirical claim. Real, DOI-bearing traces arrive with roadmap P3.
- The concept lexicon is finite; concepts outside it are invisible to the rigor layer.
7. Reproducing a number
Every trace records its provenance (model + revision, layer, J_l corpus
and seed) in manifest.json, and every per-token/per-position rigor block
in tokens.jsonl / align.json. Schema:
j7scope/trace.py;
full field spec:
docs/platform-plan.md §3.