ONTO Epistemic Risk Standard, version 1.0

Version 1.0

Publication Date: January 2026 Status: Active Maintained by: ONTO Standards Council

Document Information

1. SCOPE

1.1 Purpose

This standard establishes requirements and recommendations for measuring epistemic calibration in artificial intelligence systems—the degree to which an AI system accurately recognizes the boundaries of its knowledge.

1.2 Applicability

This standard applies to:

• Large language models (LLMs)
• AI assistants and chatbots
• Decision support systems
• Automated reasoning systems
• Any AI system that generates factual claims or recommendations

1.3 Normative References

This standard aligns with:

• EU AI Act (Regulation 2024/1689), Articles 9, 13, 15
• NIST AI Risk Management Framework (AI RMF 1.0)
• ISO/IEC 23894:2023 — Guidance on AI risk management
• ISO/IEC 42001:2023 — AI management system

2. TERMS AND DEFINITIONS

2.1 Epistemic Calibration

The alignment between an AI system's expressed confidence and its actual accuracy, particularly regarding the recognition of knowledge boundaries.

2.2 Unknown Detection

The ability of an AI system to correctly identify questions or queries that have no established answer or fall outside its reliable knowledge.

2.3 Hallucination

The generation of confident but incorrect or unfounded responses by an AI system, particularly when the system should instead express uncertainty.

2.4 Calibration Error

A quantitative measure of the discrepancy between expressed confidence and observed accuracy, typically measured as Expected Calibration Error (ECE).

2.5 Epistemic Risk

The potential for harm arising from an AI system's failure to accurately represent the boundaries of its knowledge.

3. REQUIREMENTS

3.1 Measurement Requirements

3.1.1 Unknown Detection Rate (Mandatory)

Organizations SHALL measure the rate at which their AI system correctly identifies genuinely unanswerable questions.

Metric: Unknown Recall (U-Recall)

U-Recall = True Unknown Detections / Total Unknown Questions

Minimum Threshold:

• Low Risk Applications: ≥ 30%
• Medium Risk Applications: ≥ 50%
• High Risk Applications: ≥ 70%

3.1.2 Calibration Error (Mandatory)

Organizations SHALL measure the calibration error of their AI system.

Metric: Expected Calibration Error (ECE)

ECE = Σ |accuracy(bin) - confidence(bin)| × n(bin) / N

Maximum Threshold:

• Low Risk Applications: ≤ 0.20
• Medium Risk Applications: ≤ 0.15
• High Risk Applications: ≤ 0.10

3.1.3 Overconfidence Rate (Recommended)

Organizations SHOULD measure the rate of overconfident predictions.

Metric: Overconfidence Rate

Overconfidence = Cases where confidence > accuracy / Total cases

Target: ≤ 20%

3.2 Evaluation Requirements

3.2.1 Benchmark Coverage

Evaluation SHALL include:

• Known facts (established ground truth)
• Unknown questions (genuinely open problems)
• Contradictions (topics with legitimate expert disagreement)

3.2.2 Sample Size

Minimum evaluation samples:

• Initial certification: 50 questions per category
• Continuous monitoring: 20 questions per category per evaluation

3.2.3 Evaluation Frequency

3.3 Documentation Requirements

3.3.1 Risk Assessment (Mandatory)

Organizations SHALL document:

• Epistemic risk level (Critical/High/Medium/Low)
• Measurement methodology
• Evaluation results
• Identified limitations

3.3.2 Mitigation Measures (Mandatory)

Organizations SHALL document implemented measures to:

• Detect knowledge boundaries
• Express appropriate uncertainty
• Prevent confident hallucination
• Enable human oversight

3.3.3 Audit Trail (Recommended)

Organizations SHOULD maintain:

• Historical evaluation results
• Model version tracking
• Change documentation

4. COMPLIANCE LEVELS

4.1 Level 1: Basic Compliance

Requirements:

• Unknown Detection ≥ 30%
• ECE ≤ 0.20
• Annual evaluation
• Basic documentation

Suitable for: Low-risk applications, internal tools

4.2 Level 2: Standard Compliance

Requirements:

• Unknown Detection ≥ 50%
• ECE ≤ 0.15
• Quarterly evaluation
• Full documentation

Suitable for: Customer-facing applications, B2B tools

4.3 Level 3: Advanced Compliance

Requirements:

• Unknown Detection ≥ 70%
• ECE ≤ 0.10
• Monthly evaluation
• Complete audit trail
• Third-party verification

Suitable for: High-stakes applications, regulated industries

5. REGULATORY MAPPING

5.1 EU AI Act Alignment

5.2 NIST AI RMF Alignment

5.3 ISO/IEC 23894 Alignment

6. CERTIFICATION

6.1 Certification Process

1. Application: Submit evaluation request
2. Evaluation: Third-party assessment using ONTO-Bench
3. Review: Standards Council review of results
4. Certification: Issuance of compliance certificate
5. Maintenance: Ongoing monitoring per risk level

6.2 Certification Tiers

6.3 Certification Mark

Certified organizations may display the ONTO Trust Mark per usage guidelines:

• Verification URL required
• Expiration date displayed
• Revocation for non-compliance

7. IMPLEMENTATION GUIDANCE

7.1 Getting Started

1. Assess current AI system capabilities
2. Run ONTO-Bench evaluation
3. Identify gaps against target compliance level
4. Implement mitigation measures
5. Re-evaluate and document

7.2 Common Mitigations

7.3 Continuous Improvement

• Regular re-evaluation
• Model version tracking
• Feedback integration
• Benchmark updates

8. REFERENCES

Normative References

• EU AI Act (Regulation 2024/1689)
• NIST AI RMF 1.0
• ISO/IEC 23894:2023
• ISO/IEC 42001:2023

Informative References

• Guo et al., "On Calibration of Modern Neural Networks" (2017)
• Kamath et al., "Selective Question Answering" (2020)
• ONTO-Bench: Benchmark for Epistemic Calibration (2026)

9. VERSION POLICY

9.1 Versioning Scheme

ONTO-ERS follows semantic versioning:

• Major (X.0): Breaking changes requiring re-certification
• Minor (X.Y): Backward-compatible additions
• Patch (X.Y.Z): Clarifications, no metric changes

9.2 Review Cycle

9.3 Deprecation Policy

• Deprecated versions supported for 24 months
• 12-month advance notice for breaking changes
• Transition guidance provided for major updates

9.4 Current Status

10. REGULATORY TRACEABILITY MATRIX

10.1 EU AI Act Mapping

10.2 NIST AI RMF Mapping

10.3 ISO/IEC 23894 Mapping

10.4 Legal Citation Format

For regulatory submissions:

Per ONTO Epistemic Risk Standard v1.0 (ONTO-ERS-1.0), 
Section [X], the AI system demonstrates compliance with 
[EU AI Act Article Y / NIST AI RMF Function Z] through 
[specific measurement/documentation].

Reference: ONTO Standards Council. (2026). ONTO Epistemic 
Risk Standard (Version 1.0). https://onto-bench.org/standard

11. REVISION HISTORY

10. CONTACT

ONTO Standards Council standards@onto-bench.org onto-bench.org/standard

APPENDIX A: COMPLIANCE CHECKLIST

A.1 Level 1 (Basic) Checklist

• [ ] Unknown Detection measured (≥30%)
• [ ] Calibration Error measured (≤0.20)
• [ ] Annual evaluation scheduled
• [ ] Risk level documented
• [ ] Limitations documented
• [ ] Mitigation measures documented

A.2 Level 2 (Standard) Checklist

All Level 1 requirements plus:

• [ ] Unknown Detection ≥50%
• [ ] Calibration Error ≤0.15
• [ ] Quarterly evaluation scheduled
• [ ] Overconfidence Rate measured
• [ ] Audit trail initiated

A.3 Level 3 (Advanced) Checklist

All Level 2 requirements plus:

• [ ] Unknown Detection ≥70%
• [ ] Calibration Error ≤0.10
• [ ] Monthly evaluation scheduled
• [ ] Third-party verification
• [ ] Complete audit trail
• [ ] Model version tracking

APPENDIX B: CITATION

To cite this standard:

ONTO Standards Council. (2026). ONTO Epistemic Risk Standard 
(Version 1.0). ONTO-ERS-1.0. https://onto-bench.org/standard

For legal/regulatory reference:

Compliance with ONTO Epistemic Risk Standard v1.0 (ONTO-ERS-1.0), 
as maintained by the ONTO Standards Council, demonstrates adherence 
to epistemic calibration requirements aligned with EU AI Act 
Articles 9 and 13, and NIST AI RMF MEASURE function.

ONTO Epistemic Risk Standard v1.0 © 2026 ONTO Standards Council This standard is publicly available for adoption