Validator Expert
Hallucination detection and claim validation
Purpose
The Validator Expert detects hallucinations, validates factual claims, and assesses the reliability of expert outputs within the BeastBullet reasoning system.
Key Function: Cross-validate expert findings to prevent false high-confidence errors (ISL principle: zero false high-confidence).
Architecture
- Input: 256-dim embeddings (from shared encoder)
- Hidden: 256-dim, 3 layers with dropout (0.3)
- Output: 3 classes (VALID, INVALID, UNCERTAIN)
- Parameters: 1.3M
- Size: 5.2MB (.pt), 308KB (.onnx)
Model Type: Feed-forward neural network with ReLU activation
Training
Dataset
- Primary: SNLI (Stanford Natural Language Inference) - 570K samples
- Adversarial: Custom hardened dataset - 1500+ adversarial examples
- Validation: 5-fold cross-validation
Methodology
- Teacher Model: Ollama/Mistral (knowledge distillation)
- Training Samples: 2000+ (including adversarial)
- Validation Samples: 500+
- Epochs: 20 (with early stopping)
- Batch Size: 64
- Learning Rate: 1e-4
- Weight Decay: 1e-3 (aggressive for hardening)
Hardening Process
- Initial training on SNLI (baseline: 75% accuracy)
- Adversarial dataset generation (physics, code, math edge cases)
- Retraining with 100× weight decay
- 5-fold cross-validation for generalization
- Final accuracy: 90.2%
Performance
Validation Metrics
- Accuracy: 90.2% (5-fold CV average)
- Precision: 92.1% (VALID class)
- Recall: 88.7% (INVALID class)
- F1-Score: 90.3%
ISL Metrics
- False High-Confidence Rate: 0% (critical for ISL)
- Confidence Calibration: 95%+ (high confidence → high accuracy)
- Inference Time: <10ms (CPU)
Adversarial Robustness
- Physics Constants: 95% detection
- Code Definitions: 92% detection
- Advanced Math: 88% detection
- Contradiction Pairs: 94% detection
Output Format
{
"class": "VALID", # or "INVALID", "UNCERTAIN"
"confidence": 0.92,
"logits": [0.05, 0.03, 0.92] # [INVALID, UNCERTAIN, VALID]
}
Class Definitions:
- VALID: Claim is factually correct and supported
- INVALID: Claim contains hallucinations or contradictions
- UNCERTAIN: Insufficient information to validate
Limitations
Known Constraints
- English Only: Trained exclusively on English text
- Factual Claims: Best for factual statements, not opinions
- Context-Dependent: Requires sufficient context for validation
- Domain Coverage: May struggle with highly specialized domains (medical, legal)
Edge Cases
- Ambiguous Statements: May return UNCERTAIN for vague claims
- Sarcasm/Irony: Not designed to detect non-literal language
- Temporal Claims: Limited ability to validate time-sensitive facts
Use Cases
✅ Good Use Cases
- Validating expert outputs in reasoning pipelines
- Detecting hallucinations in generated text
- Cross-checking factual claims across sources
- Quality control for AI-generated content
❌ Not Suitable For
- Standalone fact-checking (requires BeastBullet system)
- Opinion validation or sentiment analysis
- Real-time streaming validation
- Multilingual claim validation
Integration Example
from beastbullet.experts import ValidatorExpert
from beastbullet.core import SharedEncoder
# Load models
encoder = SharedEncoder("shared_encoder_v1.0.pt")
validator = ValidatorExpert("validator_expert_v1.0.pt")
# Validate a claim
claim = "The speed of light is 299,792,458 m/s"
embedding = encoder.encode(claim)
result = validator.validate(embedding)
print(result)
# {
# "class": "VALID",
# "confidence": 0.95,
# "logits": [0.02, 0.03, 0.95]
# }
# Detect hallucination
false_claim = "The speed of light is 500,000,000 m/s"
embedding = encoder.encode(false_claim)
result = validator.validate(embedding)
print(result)
# {
# "class": "INVALID",
# "confidence": 0.88,
# "logits": [0.88, 0.10, 0.02]
# }
Training Logs
5-Fold Cross-Validation Results:
Fold 1: 89.8% accuracy
Fold 2: 90.5% accuracy
Fold 3: 90.1% accuracy
Fold 4: 90.7% accuracy
Fold 5: 89.9% accuracy
Average: 90.2% ± 0.35%
Adversarial Test Results:
Spicy Hallucinations (100 examples):
- Physics: 95/100 detected
- Code: 92/100 detected
- Math: 88/100 detected
- Overall: 91.7% detection rate
Model Versions
v1.0 (Current)
- Initial release
- 90.2% accuracy on validation
- Hardened with 1500+ adversarial examples
- 5-fold cross-validation
Planned Improvements (v1.1)
- Expand adversarial dataset to 3000+ examples
- Add domain-specific validators (medical, legal)
- Improve temporal claim validation
- Optimize for mobile deployment
Citation
@model{validator_expert2025,
title={BeastBullet Validator Expert: Hallucination Detection},
author={Bhosale, Shrikant},
year={2025},
url={https://huggingface.co/beastbullet/beastbullet-experts},
note={Trained via knowledge distillation with adversarial hardening}
}
Links
- Main Repository: https://huggingface.co/beastbullet/beastbullet-experts
- Source Code: https://codeberg.org/ishrikantbhosale/beastbullet-core
- Training Code: https://codeberg.org/ishrikantbhosale/beastbullet-experts
- Documentation: https://potatobullet.com/beastbullet-project/
Zero false high-confidence. Always. 🧠