# Validator Expert

**Hallucination detection and claim validation**

---

## Purpose

The Validator Expert detects hallucinations, validates factual claims, and assesses the reliability of expert outputs within the BeastBullet reasoning system.

**Key Function**: Cross-validate expert findings to prevent false high-confidence errors (ISL principle: zero false high-confidence).

---

## Architecture

- **Input**: 256-dim embeddings (from shared encoder)
- **Hidden**: 256-dim, 3 layers with dropout (0.3)
- **Output**: 3 classes (VALID, INVALID, UNCERTAIN)
- **Parameters**: 1.3M
- **Size**: 5.2MB (.pt), 308KB (.onnx)

**Model Type**: Feed-forward neural network with ReLU activation

---

## Training

### Dataset
- **Primary**: SNLI (Stanford Natural Language Inference) - 570K samples
- **Adversarial**: Custom hardened dataset - 1500+ adversarial examples
- **Validation**: 5-fold cross-validation

### Methodology
- **Teacher Model**: Ollama/Mistral (knowledge distillation)
- **Training Samples**: 2000+ (including adversarial)
- **Validation Samples**: 500+
- **Epochs**: 20 (with early stopping)
- **Batch Size**: 64
- **Learning Rate**: 1e-4
- **Weight Decay**: 1e-3 (aggressive for hardening)

### Hardening Process
1. Initial training on SNLI (baseline: 75% accuracy)
2. Adversarial dataset generation (physics, code, math edge cases)
3. Retraining with 100× weight decay
4. 5-fold cross-validation for generalization
5. Final accuracy: **90.2%**

---

## Performance

### Validation Metrics
- **Accuracy**: 90.2% (5-fold CV average)
- **Precision**: 92.1% (VALID class)
- **Recall**: 88.7% (INVALID class)
- **F1-Score**: 90.3%

### ISL Metrics
- **False High-Confidence Rate**: 0% (critical for ISL)
- **Confidence Calibration**: 95%+ (high confidence → high accuracy)
- **Inference Time**: <10ms (CPU)

### Adversarial Robustness
- **Physics Constants**: 95% detection
- **Code Definitions**: 92% detection
- **Advanced Math**: 88% detection
- **Contradiction Pairs**: 94% detection

---

## Output Format

```python
{
  "class": "VALID",  # or "INVALID", "UNCERTAIN"
  "confidence": 0.92,
  "logits": [0.05, 0.03, 0.92]  # [INVALID, UNCERTAIN, VALID]
}
```

**Class Definitions**:
- **VALID**: Claim is factually correct and supported
- **INVALID**: Claim contains hallucinations or contradictions
- **UNCERTAIN**: Insufficient information to validate

---

## Limitations

### Known Constraints
1. **English Only**: Trained exclusively on English text
2. **Factual Claims**: Best for factual statements, not opinions
3. **Context-Dependent**: Requires sufficient context for validation
4. **Domain Coverage**: May struggle with highly specialized domains (medical, legal)

### Edge Cases
- **Ambiguous Statements**: May return UNCERTAIN for vague claims
- **Sarcasm/Irony**: Not designed to detect non-literal language
- **Temporal Claims**: Limited ability to validate time-sensitive facts

---

## Use Cases

### ✅ Good Use Cases
- Validating expert outputs in reasoning pipelines
- Detecting hallucinations in generated text
- Cross-checking factual claims across sources
- Quality control for AI-generated content

### ❌ Not Suitable For
- Standalone fact-checking (requires BeastBullet system)
- Opinion validation or sentiment analysis
- Real-time streaming validation
- Multilingual claim validation

---

## Integration Example

```python
from beastbullet.experts import ValidatorExpert
from beastbullet.core import SharedEncoder

# Load models
encoder = SharedEncoder("shared_encoder_v1.0.pt")
validator = ValidatorExpert("validator_expert_v1.0.pt")

# Validate a claim
claim = "The speed of light is 299,792,458 m/s"
embedding = encoder.encode(claim)
result = validator.validate(embedding)

print(result)
# {
#   "class": "VALID",
#   "confidence": 0.95,
#   "logits": [0.02, 0.03, 0.95]
# }

# Detect hallucination
false_claim = "The speed of light is 500,000,000 m/s"
embedding = encoder.encode(false_claim)
result = validator.validate(embedding)

print(result)
# {
#   "class": "INVALID",
#   "confidence": 0.88,
#   "logits": [0.88, 0.10, 0.02]
# }
```

---

## Training Logs

**5-Fold Cross-Validation Results**:
```
Fold 1: 89.8% accuracy
Fold 2: 90.5% accuracy
Fold 3: 90.1% accuracy
Fold 4: 90.7% accuracy
Fold 5: 89.9% accuracy
Average: 90.2% ± 0.35%
```

**Adversarial Test Results**:
```
Spicy Hallucinations (100 examples):
- Physics: 95/100 detected
- Code: 92/100 detected
- Math: 88/100 detected
- Overall: 91.7% detection rate
```

---

## Model Versions

### v1.0 (Current)
- Initial release
- 90.2% accuracy on validation
- Hardened with 1500+ adversarial examples
- 5-fold cross-validation

### Planned Improvements (v1.1)
- Expand adversarial dataset to 3000+ examples
- Add domain-specific validators (medical, legal)
- Improve temporal claim validation
- Optimize for mobile deployment

---

## Citation

```bibtex
@model{validator_expert2025,
  title={BeastBullet Validator Expert: Hallucination Detection},
  author={Bhosale, Shrikant},
  year={2025},
  url={https://huggingface.co/beastbullet/beastbullet-experts},
  note={Trained via knowledge distillation with adversarial hardening}
}
```

---

## Links

- **Main Repository**: https://huggingface.co/beastbullet/beastbullet-experts
- **Source Code**: https://codeberg.org/ishrikantbhosale/beastbullet-core
- **Training Code**: https://codeberg.org/ishrikantbhosale/beastbullet-experts
- **Documentation**: https://potatobullet.com/beastbullet-project/

---

**Zero false high-confidence. Always.** 🧠