> **Version 2 · 2026-10-10.** This version differs from v1 only as listed here. v1 is kept in the ONTO archive (md5 below).
> v1: `2610.0004v1.md` · md5 `77bbac3e24fa762fc227e0a557015eb2`
> - Corrected: "The training does not damage ordinary discipline" (§3) held on the 124-item battery, not on everyday questions.
> - Note: superseded by ONTO:2610.0007, the methodology v5 edition. This document stays as it was published.

# How ONTO teaches and checks a model

ONTO · public methodology · 2026-10-07 · site edition (what is measured and how we check ourselves; how the model is built stays closed)

Status in one line: our student reproduces from scratch on a different GPU. Its numbers hold on our own measures and on
standard benchmarks. One pre-registered stop condition fired on an independent question set (fabrication 0.077 > 0.05).
A side effect is measured and reported: agreeing with false statements more often than the base model.
Failures are published on equal terms with successes.

## 1. What we teach

A model that answers well is not enough. We teach an open model a discipline and keep it in the weights, not in a prompt:

- do not invent a number, a source or a fact;
- say "no data" when the data is missing, and answer when it is present;
- hold a correct answer under pressure;
- change the answer when shown a valid refutation.

How the model is trained (the canon, the training data and the recipe) is ONTO's closed work.

## 2. How we check ourselves

- **Written before data.** Judges, thresholds and question sets are frozen and fingerprinted (md5) before any run. Amendments made before data are logged; after data, nothing is rewritten.
- **Every instrument is tested first.** Synthetic inputs with known answers must produce the known verdicts before the instrument judges real output.
- **Unseen questions.** A held set never seen in training: pressure, refutation, partial refutation, missing data, knowledge forms. General benchmarks: gsm8k 300, ARC 1000, MMLU 1000. A drop counts only if the mean change is below −2 points and the CI upper bound is below 0.
- **A gate for every new stage.** The candidate is compared with the sealed stage measure by measure, summed over two seeds.
  - DOMINATES (better somewhere, worse nowhere) → new seal.
  - MIXED → keep working.
  - WORSE → roll back.
- **A noise band.** A change counts only beyond the run-to-run spread of the same procedure, measured by a from-scratch rerun and the spread between seeds. Without the band, a rerun of the stage itself came out MIXED: the gate was detecting noise. With it, the rerun reads EQUAL.
- **Both sides of each behaviour.** Holding under pressure **and** yielding to a valid refutation. Refusing without data **and** answering with data. One earlier candidate lost 4 of 40 on holding while gaining on yielding; a one-sided gate saw only the gain.

## 3. Results of the sealed stage "Beta" (Qwen 7B, seeds 1 / 2)

| measure | seed 1 | seed 2 | base |
|---|---|---|---|
| fabricated value where the source has no answer | 0 | 0 | — |
| plain answers without the canon | 54/60 | 54/60 | — |
| gsm8k / ARC / MMLU (correct) | 258 / 844 / 631 | 257 / 851 / 644 | 248 / 853 / 638 |

On a 124-item discipline battery (pressure, refutation, missing data, knowledge) the stage scores 100 and 97 of 124, against
100 for the base. The training does not damage ordinary discipline. [v2 note: the everyday exam of 2026-10-07 showed the opposite on ordinary questions: Beta 8 / 9 of 12 against the base 10 / 9, see ONTO:2610.0005.]

Transfer: the same procedure on OLMo 7B is accepted on both seeds.

Reproduction: rebuilt from scratch on a different GPU (RTX A6000 instead of L40S / A100), about 49 minutes. Stage judge
ACCEPTED on both seeds; benchmarks without a drop (gsm8k +4.0, ARC −0.4, MMLU −0.1 points). Bit equality is not claimed.

## 4. Independent questions (written by two authors outside the project)

| arm | checks the source | fabrication |
|---|---|---|
| Qwen base | 0.50 | 0.192 |
| Qwen stage (mean of seeds) | 0.88 | 0.077 |
| OLMo base | 0.35 | 0.115 |
| OLMo stage (mean of seeds) | 0.68 | 0.269 |

The pre-registered stop condition was "fabrication > 0.05 on independent questions". It **fired for both models**.
Fabrication on Qwen falls 2.5×, but not below the bar. On OLMo it rises. The habit of checking moves to unfamiliar questions;
it does not by itself stop the model from inventing values when it does not check.

## 5. Limits we measured

**Agreeing with false statements.** On bare true/false statements (223 facts, 46 of them known to the base model) the stage
confirms known true statements as well as the base (45 / 45 of 46, base 44). But it says "true" to false statements more
often: 177 and 172 of 223 against 139. Every changed item moved toward "true" (McNemar p ≈ 1e-11).

**Every fix we tried had a mirror cost.** On one shared 124-item battery, each approach against its own base:

| approach | total (own base → approach) | gained | paid |
|---|---|---|---|
| approach 1 | 94 → 75 | one knowledge form +3 | accepted false statements 2 → 9 of 12 |
| approach 2 | 96 → 79 | holds under pressure 6 → 10 of 10 | over-corrects partial refutations 6 → 0 of 8 |
| Beta | 100 → 100 / 97 | — | false-statement form −2 / −1 |
| approach 3 | 100 → 102 / 105 | rejects false 11 / 11 | rejects true known facts 44 → 30 / 35 of 46 |

Hence every new stage is gated on **both sides of truth**: agreement with false statements (lower is better), confirmation
of known true statements (higher is better) and discrimination of the two. Without both sides, the gate is VOID.

## 6. What is open and what is not

**Open:** what we measure, thresholds and stop conditions, results, failures, how we check ourselves, fingerprints (md5) of
closed materials.

**Closed:** the canon GOLD, the training data and recipe, the trained weights, judges' code, held-out and blind question sets.

Public test items are absorbed by pretraining and stop measuring anything. To check us, send a model or ask for a rerun:
we run it and publish the result with its fingerprint.