| # Validator Expert |
|
|
| **Hallucination detection and claim validation** |
|
|
| --- |
|
|
| ## Purpose |
|
|
| The Validator Expert detects hallucinations, validates factual claims, and assesses the reliability of expert outputs within the BeastBullet reasoning system. |
|
|
| **Key Function**: Cross-validate expert findings to prevent false high-confidence errors (ISL principle: zero false high-confidence). |
|
|
| --- |
|
|
| ## Architecture |
|
|
| - **Input**: 256-dim embeddings (from shared encoder) |
| - **Hidden**: 256-dim, 3 layers with dropout (0.3) |
| - **Output**: 3 classes (VALID, INVALID, UNCERTAIN) |
| - **Parameters**: 1.3M |
| - **Size**: 5.2MB (.pt), 308KB (.onnx) |
|
|
| **Model Type**: Feed-forward neural network with ReLU activation |
|
|
| --- |
|
|
| ## Training |
|
|
| ### Dataset |
| - **Primary**: SNLI (Stanford Natural Language Inference) - 570K samples |
| - **Adversarial**: Custom hardened dataset - 1500+ adversarial examples |
| - **Validation**: 5-fold cross-validation |
|
|
| ### Methodology |
| - **Teacher Model**: Ollama/Mistral (knowledge distillation) |
| - **Training Samples**: 2000+ (including adversarial) |
| - **Validation Samples**: 500+ |
| - **Epochs**: 20 (with early stopping) |
| - **Batch Size**: 64 |
| - **Learning Rate**: 1e-4 |
| - **Weight Decay**: 1e-3 (aggressive for hardening) |
|
|
| ### Hardening Process |
| 1. Initial training on SNLI (baseline: 75% accuracy) |
| 2. Adversarial dataset generation (physics, code, math edge cases) |
| 3. Retraining with 100× weight decay |
| 4. 5-fold cross-validation for generalization |
| 5. Final accuracy: **90.2%** |
|
|
| --- |
|
|
| ## Performance |
|
|
| ### Validation Metrics |
| - **Accuracy**: 90.2% (5-fold CV average) |
| - **Precision**: 92.1% (VALID class) |
| - **Recall**: 88.7% (INVALID class) |
| - **F1-Score**: 90.3% |
|
|
| ### ISL Metrics |
| - **False High-Confidence Rate**: 0% (critical for ISL) |
| - **Confidence Calibration**: 95%+ (high confidence → high accuracy) |
| - **Inference Time**: <10ms (CPU) |
|
|
| ### Adversarial Robustness |
| - **Physics Constants**: 95% detection |
| - **Code Definitions**: 92% detection |
| - **Advanced Math**: 88% detection |
| - **Contradiction Pairs**: 94% detection |
|
|
| --- |
|
|
| ## Output Format |
|
|
| ```python |
| { |
| "class": "VALID", # or "INVALID", "UNCERTAIN" |
| "confidence": 0.92, |
| "logits": [0.05, 0.03, 0.92] # [INVALID, UNCERTAIN, VALID] |
| } |
| ``` |
|
|
| **Class Definitions**: |
| - **VALID**: Claim is factually correct and supported |
| - **INVALID**: Claim contains hallucinations or contradictions |
| - **UNCERTAIN**: Insufficient information to validate |
|
|
| --- |
|
|
| ## Limitations |
|
|
| ### Known Constraints |
| 1. **English Only**: Trained exclusively on English text |
| 2. **Factual Claims**: Best for factual statements, not opinions |
| 3. **Context-Dependent**: Requires sufficient context for validation |
| 4. **Domain Coverage**: May struggle with highly specialized domains (medical, legal) |
|
|
| ### Edge Cases |
| - **Ambiguous Statements**: May return UNCERTAIN for vague claims |
| - **Sarcasm/Irony**: Not designed to detect non-literal language |
| - **Temporal Claims**: Limited ability to validate time-sensitive facts |
|
|
| --- |
|
|
| ## Use Cases |
|
|
| ### ✅ Good Use Cases |
| - Validating expert outputs in reasoning pipelines |
| - Detecting hallucinations in generated text |
| - Cross-checking factual claims across sources |
| - Quality control for AI-generated content |
|
|
| ### ❌ Not Suitable For |
| - Standalone fact-checking (requires BeastBullet system) |
| - Opinion validation or sentiment analysis |
| - Real-time streaming validation |
| - Multilingual claim validation |
|
|
| --- |
|
|
| ## Integration Example |
|
|
| ```python |
| from beastbullet.experts import ValidatorExpert |
| from beastbullet.core import SharedEncoder |
| |
| # Load models |
| encoder = SharedEncoder("shared_encoder_v1.0.pt") |
| validator = ValidatorExpert("validator_expert_v1.0.pt") |
| |
| # Validate a claim |
| claim = "The speed of light is 299,792,458 m/s" |
| embedding = encoder.encode(claim) |
| result = validator.validate(embedding) |
| |
| print(result) |
| # { |
| # "class": "VALID", |
| # "confidence": 0.95, |
| # "logits": [0.02, 0.03, 0.95] |
| # } |
| |
| # Detect hallucination |
| false_claim = "The speed of light is 500,000,000 m/s" |
| embedding = encoder.encode(false_claim) |
| result = validator.validate(embedding) |
| |
| print(result) |
| # { |
| # "class": "INVALID", |
| # "confidence": 0.88, |
| # "logits": [0.88, 0.10, 0.02] |
| # } |
| ``` |
|
|
| --- |
|
|
| ## Training Logs |
|
|
| **5-Fold Cross-Validation Results**: |
| ``` |
| Fold 1: 89.8% accuracy |
| Fold 2: 90.5% accuracy |
| Fold 3: 90.1% accuracy |
| Fold 4: 90.7% accuracy |
| Fold 5: 89.9% accuracy |
| Average: 90.2% ± 0.35% |
| ``` |
|
|
| **Adversarial Test Results**: |
| ``` |
| Spicy Hallucinations (100 examples): |
| - Physics: 95/100 detected |
| - Code: 92/100 detected |
| - Math: 88/100 detected |
| - Overall: 91.7% detection rate |
| ``` |
|
|
| --- |
|
|
| ## Model Versions |
|
|
| ### v1.0 (Current) |
| - Initial release |
| - 90.2% accuracy on validation |
| - Hardened with 1500+ adversarial examples |
| - 5-fold cross-validation |
|
|
| ### Planned Improvements (v1.1) |
| - Expand adversarial dataset to 3000+ examples |
| - Add domain-specific validators (medical, legal) |
| - Improve temporal claim validation |
| - Optimize for mobile deployment |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @model{validator_expert2025, |
| title={BeastBullet Validator Expert: Hallucination Detection}, |
| author={Bhosale, Shrikant}, |
| year={2025}, |
| url={https://huggingface.co/beastbullet/beastbullet-experts}, |
| note={Trained via knowledge distillation with adversarial hardening} |
| } |
| ``` |
|
|
| --- |
|
|
| ## Links |
|
|
| - **Main Repository**: https://huggingface.co/beastbullet/beastbullet-experts |
| - **Source Code**: https://codeberg.org/ishrikantbhosale/beastbullet-core |
| - **Training Code**: https://codeberg.org/ishrikantbhosale/beastbullet-experts |
| - **Documentation**: https://potatobullet.com/beastbullet-project/ |
|
|
| --- |
|
|
| **Zero false high-confidence. Always.** 🧠 |
|
|
