Version 2 · 2026-10-10. This version differs from v1 only as listed here. v1 is kept in the ONTO archive (md5 below). v1:
2610.0004v1.md· md577bbac3e24fa762fc227e0a557015eb2
- Corrected: "The training does not damage ordinary discipline" (§3) held on the 124-item battery, not on everyday questions.
- Note: superseded by ONTO:2610.0007, the methodology v5 edition. This document stays as it was published.
ONTO · public methodology · 2026-10-07 · site edition (what is measured and how we check ourselves; how the model is built stays closed)
Status in one line: our student reproduces from scratch on a different GPU. Its numbers hold on our own measures and on standard benchmarks. One pre-registered stop condition fired on an independent question set (fabrication 0.077 > 0.05). A side effect is measured and reported: agreeing with false statements more often than the base model. Failures are published on equal terms with successes.
1. What we teach
A model that answers well is not enough. We teach an open model a discipline and keep it in the weights, not in a prompt:
- do not invent a number, a source or a fact;
- say "no data" when the data is missing, and answer when it is present;
- hold a correct answer under pressure;
- change the answer when shown a valid refutation.
How the model is trained (the canon, the training data and the recipe) is ONTO's closed work.
2. How we check ourselves
- Written before data. Judges, thresholds and question sets are frozen and fingerprinted (md5) before any run. Amendments made before data are logged; after data, nothing is rewritten.
- Every instrument is tested first. Synthetic inputs with known answers must produce the known verdicts before the instrument judges real output.
- Unseen questions. A held set never seen in training: pressure, refutation, partial refutation, missing data, knowledge forms. General benchmarks: gsm8k 300, ARC 1000, MMLU 1000. A drop counts only if the mean change is below −2 points and the CI upper bound is below 0.
- A gate for every new stage. The candidate is compared with the sealed stage measure by measure, summed over two seeds.
- DOMINATES (better somewhere, worse nowhere) → new seal.
- MIXED → keep working.
-
WORSE → roll back.
-
A noise band. A change counts only beyond the run-to-run spread of the same procedure, measured by a from-scratch rerun and the spread between seeds. Without the band, a rerun of the stage itself came out MIXED: the gate was detecting noise. With it, the rerun reads EQUAL.
- Both sides of each behaviour. Holding under pressure and yielding to a valid refutation. Refusing without data and answering with data. One earlier candidate lost 4 of 40 on holding while gaining on yielding; a one-sided gate saw only the gain.
3. Results of the sealed stage "Beta" (Qwen 7B, seeds 1 / 2)
| measure | seed 1 | seed 2 | base |
|---|---|---|---|
| fabricated value where the source has no answer | 0 | 0 | — |
| plain answers without the canon | 54/60 | 54/60 | — |
| gsm8k / ARC / MMLU (correct) | 258 / 844 / 631 | 257 / 851 / 644 | 248 / 853 / 638 |
On a 124-item discipline battery (pressure, refutation, missing data, knowledge) the stage scores 100 and 97 of 124, against 100 for the base. The training does not damage ordinary discipline. [v2 note: the everyday exam of 2026-10-07 showed the opposite on ordinary questions: Beta 8 / 9 of 12 against the base 10 / 9, see ONTO:2610.0005.]
Transfer: the same procedure on OLMo 7B is accepted on both seeds.
Reproduction: rebuilt from scratch on a different GPU (RTX A6000 instead of L40S / A100), about 49 minutes. Stage judge ACCEPTED on both seeds; benchmarks without a drop (gsm8k +4.0, ARC −0.4, MMLU −0.1 points). Bit equality is not claimed.
4. Independent questions (written by two authors outside the project)
| arm | checks the source | fabrication |
|---|---|---|
| Qwen base | 0.50 | 0.192 |
| Qwen stage (mean of seeds) | 0.88 | 0.077 |
| OLMo base | 0.35 | 0.115 |
| OLMo stage (mean of seeds) | 0.68 | 0.269 |
The pre-registered stop condition was "fabrication > 0.05 on independent questions". It fired for both models. Fabrication on Qwen falls 2.5×, but not below the bar. On OLMo it rises. The habit of checking moves to unfamiliar questions; it does not by itself stop the model from inventing values when it does not check.
5. Limits we measured
Agreeing with false statements. On bare true/false statements (223 facts, 46 of them known to the base model) the stage confirms known true statements as well as the base (45 / 45 of 46, base 44). But it says "true" to false statements more often: 177 and 172 of 223 against 139. Every changed item moved toward "true" (McNemar p ≈ 1e-11).
Every fix we tried had a mirror cost. On one shared 124-item battery, each approach against its own base:
| approach | total (own base → approach) | gained | paid |
|---|---|---|---|
| approach 1 | 94 → 75 | one knowledge form +3 | accepted false statements 2 → 9 of 12 |
| approach 2 | 96 → 79 | holds under pressure 6 → 10 of 10 | over-corrects partial refutations 6 → 0 of 8 |
| Beta | 100 → 100 / 97 | — | false-statement form −2 / −1 |
| approach 3 | 100 → 102 / 105 | rejects false 11 / 11 | rejects true known facts 44 → 30 / 35 of 46 |
Hence every new stage is gated on both sides of truth: agreement with false statements (lower is better), confirmation of known true statements (higher is better) and discrimination of the two. Without both sides, the gate is VOID.
6. What is open and what is not
Open: what we measure, thresholds and stop conditions, results, failures, how we check ourselves, fingerprints (md5) of closed materials.
Closed: the canon GOLD, the training data and recipe, the trained weights, judges' code, held-out and blind question sets.
Public test items are absorbed by pretraining and stop measuring anything. To check us, send a model or ask for a rerun: we run it and publish the result with its fingerprint.