Research · 2026
What a model grew up on matters more than its size.
Two small open models, the same training, the same 100 claims. One catches twice as many impossible claims as the other.
SmolLM2 360M20 of 47
Gemma 3 270M10 of 47
One square = one impossible claim, 47 in all. Filled: caught.
20 vs 10impossible claims caught, of 47 (p = 0.02)
7 vs 8false alarms, of 36 possible claims
4 vs 9missing figure named, of 17: here Gemma is better
How we measured
Claims100 real-world claims: 47 impossible, 36 possible, 17 need data to decide.
TrainingThe same file of worked checks, the same script, two epochs, the same seed, on one laptop GPU.
The only differenceThe base model. SmolLM2 was pretrained on open educational text, code and maths (FineWeb-Edu, Stack-Edu, FineMath).
ScoringTwo blind judges per answer, a third on disagreements. They never see which model wrote it.
What it gives
Pick the base model by what it was trained on, not by its size. A 360M model is enough to learn the habit.