Atlas of the limits of silicon AI models

ONTO · Atlas · version 2 · 2026-10-31

ONTO keeps a closed canon, GOLD. This document is an atlas of a different kind: it maps where today's probabilistic models stop, and what training can and cannot move. "Silicon" names that class of models, as distinct from the technology ONTO is building to work differently.

The scope is what we measured, on two open model families: Qwen2.5-Coder-Instruct at 3B, 7B, 14B and 32B parameters (most rows at 7B) and OLMo-3-7B-Instruct. They were measured in the ONTO lab in October 2026 with ONTO's instruments. Each row gives:

• a number with its denominator;
• what it is compared with, and whether that comparison comes from a different run;
• the date and fingerprint of the lab record that states the result.

Many rows read both sides of one behaviour, because in this lab most repairs of one side moved the other.

These are limits of the technology measured with our method, not claims about every possible method. A later measurement that overturns a row is published as a new version of this atlas.

At a glance

How to read the numbers

• Seeds. "Seed 1 / seed 2" are two independent trainings of the same recipe; "a / b" in a row means seed 1 / seed 2. A value ending in .5 is a mean: of two seeds in row 15, of two passes of the same untrained model in rows 4 and 17.
• Source. The material a model consults to answer: the closed canon GOLD, or another body of documents.
• Held questions. 103 questions never seen in training. "Goes to the source by itself" is the share of them answered with the value taken from the source, with nothing in the prompt asking for it.
• Stage Beta is the sealed stage of ONTO:2610.0008.
• Outside authors wrote questions outside the lab. Most of them are AI models (Kimi, Grok, and Claude through a separate account); they are named where it matters.
• Comparisons. A base is the same model untrained. Where it was measured in a different run, the row says so.
• Records. A date is the day the result was recorded. A fingerprint is the first 8 characters of the md5 of the lab record that states it. Copies of these records were archived on 2026-10-30; the raw files behind each row are fingerprinted in the companion file 2610.0009.json.

1. Knowledge: what the substrate holds

1. The model knows little and cannot show which part. We asked Qwen 7B, a code model, 1,007 Russian-language questions from RuBQ 2.0, a public set of facts with checked answers. It knows 223 (22 %): its first answer is right and at least 3 of 5 samples agree. On 698 (69 %) its first answer is wrong and at most 1 of 5 samples is right; it declines 9 of them and answers the rest. 86 more are unstable. Of the 166 "which year" questions it knows 62 (37 %); other question types have no reproducible split. Training on new facts is known to raise invention (Gekhman et al. 2024), so this is not fixed by teaching facts.

Record: 2026-10-05 · a0f798d7.

2. Its sense of "I know" does not reach the answer. The probability the model gives its own first answer separates known from unknown facts with an AUROC of 0.79 (the 86 unstable facts excluded). The signal exists; the answers do not show it. We recomputed three refusal settings of one trained recipe (two trainings), offline, on stored samples, after the data. Across the three:

• facts the model does not know that drew at least one wrong verdict fell from 40 and 36 to 13 and 12 of 177 (base 143);
• known facts judged right on both the true and the false statement fell from 32 and 36 to 24 and 24 of 46 (base 23);
• refusals of known truths rose from 9 to 20 of 46.

Each cut in invention cost known answers. Without a source to check against, the setting is a choice of policy, not a defect that training removes.

Record: 2026-10-05 · a0f798d7.

3. Discipline cannot reject what the model does not know. On 18 false statements by outside authors (AI models), the base said "true" to 9. Every recipe we trained left between 1 and 4. Our reading of those that remain: facts the 7B does not know, or knows in part (in one, it knows the year of an event but not the country). n = 18.

Record: 2026-10-05 · a0f798d7.

4. Size improves judgement of false statements, at a cost on true ones. Untrained, mean of two passes (gsm8k: one pass):

Larger models accepted far fewer false statements and confirmed somewhat fewer true ones. The true statements were chosen as known to the 7B, which tilts that side against the larger models. The untrained models were not told the source exists, so at every size none went to it (0.00 by construction).

Record: 2026-10-12 · 4ce3773a.

2. Judgement: two sides of one behaviour

5. Repairing one side moves the other. One trained candidate cut "true" on 223 false statements from 139 (base) to 13 and 42. On 223 true statements, its "true" fell from 187 to 56 and 85. Correct verdicts over all 446: base 271, candidate 266 and 266. A measure that reads only the false side records progress. Both sides together show none.

Record: 2026-10-05 · 24f37ffc.

6. Holding firm and accepting proof move together. In a training run aimed at holding a correct answer under pressure, holding moved from 5 to 6 of 8 on both seeds, inside the noise. In the same run, acceptance of a valid proof fell from 8 of 8 to 4 and 6. The same change also altered other training content, so the two effects cannot be separated. n = 8.

Record: 2026-10-09 · 244751fb.

7. A disciplined model can still agree too easily. Stage Beta confirmed 45 of 46 known truths on both seeds and called 1 false (base: 44 confirmed, 2 called false). But it said "true" to 177 and 172 of 223 false statements, against the base's 139 (McNemar p = 7e-12 and 2e-10, toward "true"). A differently trained model, whose verdict comes from a check rather than a word, said "true" to 9 and 3 of the 223 false statements. It confirmed 42 and 36 of the 46 known truths and refused 4 of them on each seed.

Record: 2026-10-05 · d7c91b30.

8. Checking its own words is the bottleneck. On three fresh sets of 48 questions, untrained Qwen 7B (4-bit, run locally) answered, then wrote checks of its own answers against the limits.

• Verdicts: when a check gave one, it was right 34 of 38 times. Errors were 0, 2 and 2 by set; the lab assigns the third set's 2 errors to its own instrument.
• Own impossible claims: it caught 11 of 45 (5 of 14, 2 of 15, 4 of 16).
• Impossible premises: on the third set its answers rejected none of 13. When the check also read the question's premise, it marked 3 premises impossible and flagged 2 of those answers.

The pre-registered bar failed and the line was closed. Others report the same "blind solver" pattern on faulty maths problems (Rahman et al. 2024). Training on correction traces alone is often insufficient to teach self-correction (Kumar et al. 2024).

Record: 2026-10-16 · fb403483, 75c9ca2e.

3. Habit: what training moves, and at what price

9. Going to the source by itself is trainable. With nothing in the prompt asking for it, stage Beta (Qwen 7B) answers from the source on 0.90 and 0.88 of the 103 held questions (base 0.00). It invents 0 values where the source has no answer. The same procedure on OLMo-3-7B-Instruct gives 0.84 and 0.97, and a rebuild of the Qwen stage from scratch on another GPU gives 0.88 and 0.91 (ONTO:2610.0008). Separately, a later candidate was rerun on everyday questions with the source removed at answer time. It gave the same answers, so its everyday behaviour does not depend on the source being present (ONTO:2610.0005).

10. The habit stays with its source. We gave stage Beta 840 public maths problems with a key number missing (UMWP; Sun et al. 2024) and no source to consult. It invented the missing number as often as its base.

• Pre-registered judge: 366 and 355 against 348, one-sided McNemar for a reduction p ≥ 0.5 on both seeds. This judge left about 400 of the 840 unresolved.
• Stricter rule (an assumed value followed by a final number counts as invented), applied after the data as an exploratory count: base 504, stage Beta 514 and 509.

The habit is going to the source. Where there is none, it did not become a general caution.

Record: 2026-10-11 · db488024.

11. Refusal taught without a source spread to solvable problems. We trained the 7B to say when a number is missing, with no source. On the same 840 problems, scored with the stricter rule of row 10 and with the base measured again in this run, its invention fell from 501 (0.596) to 49 and 64. Each problem has a solvable twin. On those 840 twins, its correct answers fell from 758 to 604 and 638 (−18.3 and −14.3 points), and its refusals rose from 4 to 190 and 140. It called "missing" what could be derived: three weeks is 21 days. The behaviour moved by an order of magnitude, but at this price it is not usable. One training mix was tested; others were not.

Record: 2026-10-12 · 869e0ec1.

12. A larger dose of one kind of example is not a lever. We gave one larger dose of one kind of training example (one dose, two seeds). Held questions answered with the value from the source fell from 89 and 86 (the previous candidate, a separate run) to 60 and 76 of 103. The model consulted the source less: its calls to it on these questions fell from 102 and 101 of 103 (the previous candidate) to 92 and 92.

Record: 2026-10-07 · cd171604.

13. A self-description did not move into the weights. A model trained to describe itself gave that description on 5 and 7 of 15 questions with nothing in the prompt. The untrained base scored 6 of 15 on the same judge, which the lab records as lenient. With the description in the prompt: 11 of 15. The training added nothing; the prompt carried it.

Record: 2026-10-05 · becf3ce4.

14. Consulting the limits of physics can be trained into the weights. The questions ask for a quantity where the given figures break a law of physics: Carnot, Betz, Shannon, Landauer and others. Examples are the efficiency of a power plant and the heating coefficient of a heat pump.

• What is counted: the claim is routed to the deciding limit in the source and flagged as breaking it. The limit itself stays in the source; the training puts into the weights the habit of consulting it.
• The sets, each of 45 questions, 15 of them impossible:
  • set A was frozen before the last five attempts and used to accept each of them;
  • set B was written after the first attempt, a day before set A, and measured after every later attempt without deciding it;
  • set C is an earlier set written the same way.

Results on the 15 impossible claims:

• Untrained Qwen 7B, nothing in the prompt, set A (a separate run): 0 of 15. This is zero by construction, since it does not know the source; in its own words it called none of the 15 impossible either. Each time, it computed the asked quantity and reported a number.
• Untrained, with a one-page manual of the source in the prompt, set C (quantized models run locally): Qwen 7B 3 of 15, with 2 possible claims wrongly flagged; OLMo-3-7B-Instruct 0 of 15. Gemma 4 12B returned no answer within its token budget on all 30 limit questions (void); DeepSeek-R1 14B was stopped at 6 of 45.
• Trained Qwen 7B, from the question alone:
  • set A 13 and 14 of 15; set B 11 and 12 of 15;
  • possible claims and everyday questions flagged as impossible on set A: 0 and 1 of 30;
  • in its own words it flagged the claim as breaking a limit on 14 and 15 of the 15.
• The same training on Qwen 3B:
  • set A 14 and 14, set B 10 and 13;
  • but its general maths (gsm8k) fell by 31 and 25 of 300 against its base, where the 7B lost 7 and 5.

3B and 7B reached the same score on set A.

Record: 2026-10-17, corrected 2026-10-30 · a5b005c9; 2026-10-30 · 45d23ebf, 2fd9d76f.

15. Multi-step everyday arithmetic stays with the base. The test has 60 new everyday problems of several steps (money, time, kitchen, travel), each reference answer recomputed independently. A trained model computes in a program it writes, then answers in words; both are scored. Trained values are means of two seeds.

The last training stage raised the 7B's program score from the previous stage's 23 to 30.5, not past its base. By its words the trained 7B is 3 above its base, and the 3B equals its base. When the program is wrong, the words often recompute and correct it. The untrained values come from a separate run, on the same questions with the same grader.

Record: 2026-10-30 · 45d23ebf, 2fd9d76f.

16. The physics skill cost everyday answers in every attempt. Each attempt was judged blind on 40 everyday questions against the same model before the physics training. A fresh model agent read the question and the two answers, in an order shuffled by a fixed seed, and chose the better answer or a tie. Net (wins − losses) per seed:

From attempt 2 on, the physics passed and the everyday answers decided the verdict. The bar of −8 tolerates a loss; the two accepted attempts are within the judge's noise of each other.

Record: 2026-10-30 · 45d23ebf.

4. Substrate: what belongs to the technology itself

17. Instructions in the prompt buy part of the habit, at a price. We gave untrained models the same instructions and manual in the prompt. Mean of two passes:

What the table shows:

• Gains: instructions cut false "true" at every size.
• Costs: they cost true answers at every size (2, 14.5 and 7 of 46) and raised invention on outside questions at every size; the invention rises were not tested for significance, and at 14B it is inside the noise of row 18.
• Noise: two passes on the source measure agree within ±0.03, so the 7B and 14B values cannot be told apart.
• Our defect at 32B: the 32B value is lowered by a defect in our own documentation of the source; 34 of its 103 programs failed on it.

A trained 7B with no instructions answers 0.90 from the source (row 9, a separate run). Trained models of 14B and 32B are being measured now.

Record: 2026-10-12 · 4ce3773a.

18. Small sets have a noise floor. On 82 questions by four outside authors (three AI models named above and one more), 6 inventions (0.073, 95 % CI 0.034–0.151) and 4 (0.049, 0.019–0.119) cannot be told apart, so a fixed bar such as "below 0.05" is noise at this size. What can be told apart: the base's 18 of 82 (0.220) against 8 and 8 (0.098) for a trained candidate, McNemar p 0.003 and 0.001. For invention on outside questions we require "better than the base on both seeds, beyond the noise". Other measures use pre-registered bars, named in each row.

Record: 2026-10-05 · a0f798d7.

5. Where the limits meet: the missing piece

Read together, the rows describe one machine:

• It knows part of the world and cannot say which part (1–3).
• Its judgements come in pairs that move together (5–8).
• It learns habits. A habit held where a source existed (9), and failed or overreached where none did (10–11).
• Size improves its judgement (4); instructions give part of the habit of consulting a source, at a price (17).

The largest gains came where the model had a boundary to consult outside its own guesses (rows 9, 14 and 7 below). None came free, and the one gain without a source (row 11) cost the most:

A code model without the boundary is a calculator. On 15 impossible claims the untrained 7B computed every asked quantity and flagged none. Trained to consult the boundary, it computes the same number and flags the claim with the limit it breaks and by how much. The boundary is not knowledge to be memorised. It is held in the closed canon GOLD in a form the machine can consult, and that is the piece the substrate does not have.

6. What the next step has to meet

ONTO is building its own AI technology with the source at its core. From this atlas, it has to meet six requirements:

1. The boundary is part of the machine and consulted, not recited in a prompt (14, 17).
2. Knowledge comes from outside the weights; the weights carry the habit of going to it (1, 9, 10).
3. Every behaviour is measured on both sides as one measure (5, 6, 7; ONTO:2610.0007).
4. Arithmetic and checks are computed, not guessed, and the check reads the question as well as the answer (8, 15).
5. A refusal is taught only together with the twin that must be answered (11).
6. Every claim is set against the base on the same questions, on two seeds, with the noise named (18).

Who this atlas is for

• Builders of general AI: where training stops and the architecture has to take over.
• Teams that fine-tune models for one field: what a tuned model will and will not do, and what to test before it ships.
• Everyone else: a confident voice from an AI is not evidence, and why.

Limits of this atlas

• Models: Qwen2.5-Coder-Instruct 3B, 7B, 14B, 32B and OLMo-3-7B-Instruct. Trained and pod measurements in bf16 on single GPUs; row 8 and the screen in row 14 ran quantized models locally.
• Questions:
  • RuBQ 2.0, a public set, in Russian: rows 1, 2, 4, 5, 7 and 17;
  • UMWP, a public set, in English: rows 10–11;
  • outside authors: rows 3 and 18, and the outside questions of row 17;
  • written for these measurements, and closed so that they keep measuring: rows 6 and 13 (largely Russian), rows 9, 12 and 17 (held questions, mostly Russian), rows 8 and 14–16 (English).
• Reuse: set A in row 14 was the acceptance test of five successive attempts, so the attempts were chosen against it. Set B did not decide any of them, but its results were seen after each one.
• Samples: each row states its sample size; several are small, and row 18 says what that allows.
• Instruments: ours, each first tested on inputs with known answers. Where a base was measured in a different run, the row says so.
• Pending: trained models of 14B and 32B are being measured now, under a plan published before the first measurement.

Records and sources

Each fingerprint below names a lab record as archived on 2026-10-30. The records stay closed; the fingerprint shows they were not changed after this date. The raw files behind each row are fingerprinted in 2610.0009.json.

ONTO:2610.0005 Everyday questions: the student against its base. ONTO:2610.0007 How ONTO teaches and checks a model: methodology v5. ONTO:2610.0008 Stage Beta, cycle by cycle.

Gekhman, Z. et al. (2024). Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? arXiv:2405.05904. Kumar, A. et al. (2024). Training Language Models to Self-Correct via Reinforcement Learning. arXiv:2409.12917. Rahman, A. M. M. et al. (2024). From Blind Solvers to Logical Thinkers: Benchmarking LLMs' Logical Integrity on Faulty Mathematical Problems. arXiv:2410.18921. Rybin, I., Korablinov, V., Efimov, P., Braslavski, P. (2021). RuBQ 2.0: An Innovated Russian Question Answering Dataset. ESWC 2021. doi:10.1007/978-3-030-77385-4_32. Licence CC BY-SA 4.0. Sun, Y. et al. (2024). Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem. arXiv:2403.03558.