For researchers
What is measured, how, and how to check it.
What is the goal?
A model that goes to the source by itself, answers from it and says “no data” when the source is silent, without losing what it knew. ONTO:2610.0007, section 5.
What is measured?
Answers from the source on held questions; invented values where the source is silent; both sides of each behaviour (holds without proof and accepts a proof; refuses without data and answers with data); gsm8k, ARC, MMLU. ONTO:2610.0007, sections 2 and 3.
How is fitting to the test excluded?
Judges, thresholds and question sets are frozen and fingerprinted before any run; 100 pre-registrations on file. Held questions; 110 questions by two outside authors; base model in the same run; two seeds; noise band from a from-scratch rerun. ONTO:2610.0007, section 2; ONTO:2610.0008.
Is the habit in the weights?
Rerun with the source removed at answer time: the answers stay the same. ONTO:2610.0005.
General abilities?
gsm8k, ARC, MMLU on Qwen and OLMo, two seeds: −0.6 to +3.2 points. ONTO:2610.0008.
Transfer and reproduction?
OLMo 7B: accepted on both seeds. Rebuilt from scratch on an RTX A6000 in about 49 minutes: accepted on both seeds. ONTO:2610.0008.
What is closed, and how to check it?
Closed: the canon, training data and recipe, weights, judges' code, held-out sets; public test items are absorbed by pretraining. To check: send a model or request a rerun; the result is published with its fingerprint. ONTO:2610.0007, section 6.
How to cite?
ONTO:id and version, for example ONTO:2610.0007v1. Every version carries its md5 in the archive.
Words used here
- Source
- the material the model checks its answer against: the canon GOLD, or another body of documents
- Base model
- the same model before ONTO training; every result is compared with it in the same run
- Seed
- one independent training run; two seeds = the result is obtained twice
- Held questions
- questions the model never saw in training
- Threshold
- the pass mark, written and fingerprinted before the data
- Stage, seal
- a model state that passed all its thresholds; the next stage replaces it only if it is better somewhere and worse nowhere
- Invented answer
- a fact, number or check result that is not in the question and not in any source