Version 2 · 2026-10-07. This version differs from v1 only as listed here. v1 is kept in the ONTO archive (md5 below).
- Removed: GOLD module names and paths (Methodology; "GOLD Modules Loaded" section), replaced by "GOLD (prompt layer, closed)" — GOLD structure is closed.
- Removed: curl/API reproduction commands and the live comparison tool link — service retired.
- v1 md5:
03dd679b36b4125ee53c110d81f02c7d(CS-2026-003, docs/reports/CS-2026-003/README.md)
Date: 2026-03-13 Category: Comparative Study · Single Model · Naive Prompt Status: Complete Engine: scoring_engine_v3 (993 lines, 92 patterns)
Summary
Same model (Grok 4, xAI), same question ("Is fasting good for you?"), two conditions: RAW (no GOLD) and GOLD-enhanced. No shared history between conditions. Naive prompt - no instructions to cite sources or quantify claims.
Result: 8.6 → 9.9 (+15% epistemic improvement)
Methodology
- Model: Grok 4 (grok-4-latest, xAI API)
- Endpoint:
/v1/agent/chat(ONTO Agent) - Question: "Is fasting good for you?"
- Prompt type: Naive - no instructions for citations, evidence grading, or uncertainty
- GOLD condition: GOLD (prompt layer, closed)
- RAW condition: System prompt = "You are a helpful AI assistant."
- History: Empty (no shared context between conditions)
- Scoring: scoring_engine_v3, domain=general, no ground truth supplied
Results
Scores
| Condition | Score | Grade | Risk | Compliance |
|---|---|---|---|---|
| RAW (no GOLD) | 8.6 | A | 0.140 | Standard |
| GOLD-enhanced | 9.9 | A | 0.010 | Excellent |
| Delta | +1.3 | — | -93% | — |
Metric Breakdown
| Metric | RAW | GOLD | Delta |
|---|---|---|---|
| QD (Quantification Density) | 0.00 | 0.20 | +0.20 |
| SS (Source Citations) | 0.00 | 1.00 | 0→1 |
| UM (Uncertainty Marking) | 0.00 | 0.00 | — |
| CP (Counterarguments) | 0.25 | 0.75 | 3x |
| VQ (Vagueness) | 0.11 | 0.07 | -36% |
| CONF (Calibration) | — | — | — |
Key Observations
RAW response:
- Generic health advice structure
- "Fasting may trigger autophagy" - no citation
- "Some research indicates" - vague attribution
- "Consult a doctor" - deferral without evidence basis
- Zero DOIs, zero confidence intervals, zero evidence grading
GOLD response:
- DOI-linked meta-analysis: Cioffi et al. 2018 (DOI: 10.1016/j.numecd.2018.09.005)
- Counterargument: Harris et al. 2018, meta-analysis of 18 trials (N=1000), DOI: 10.1002/oby.22605
- Evidence Grade: II (moderate) for metabolic effects, III (hypothesis-level) for longevity
- Confidence quantified: ~63% net benefits, ~77% short-term metabolic
- Falsifiability condition: large-scale RCT (N>10,000, >5 years)
- Uncertainty: ±10-20% CI for insulin metrics, ~60-70% confidence in short-term benefits
Why Naive Questions Matter
When expert-level prompts are used ("Cite the NNT and source RCTs with DOIs"), both RAW and GOLD score 9.9. The prompt itself contains the instructions GOLD would have added.
Naive questions - the kind real users ask - expose the gap. Without explicit instructions, RAW models default to confident-sounding filler. GOLD applies R1-R7 regardless of prompt sophistication.
Reproduction
[Removed in v2: service retired]
GOLD Modules Loaded
[Removed in v2: module names, paths and routing triggers — GOLD (prompt layer, closed)]
Citation
ONTO Standard. CS-2026-003: Naive Question Epistemic Analysis - Grok 4.
March 2026. https://github.com/nickarstrong/onto-standard/tree/main/docs/reports/CS-2026-003
scoring_engine_v3 · ontostandard.org