RAGOne: A Lightweight and fast RAG Evaluation Model

RAGOne is a fine-tuned ModernBERT-based model designed to evaluate retrieved context in Retrieval-Augmented Generation (RAG) pipelines. It scores candidate labels for four evaluation tasks: answerability, evidence quality, contradiction detection, and relevance classification.

The project aims to provide a compact, reusable evaluation component that can help developers and researchers assess retrieval quality and context suitability before or during answer generation.

Model: Hariharan05/ragone
Python library: ragone-ai (imported as ragone)
Base model: answerdotai/ModernBERT-base
License: MIT — research-oriented release; see the license terms and restrictions below.

Overview

RAGOne receives a query, retrieved context, an evaluation task, and a set of candidate labels. It encodes the input using its fine-tuned ModernBERT encoder and scores the candidate labels using a trained classification head.

Supported evaluation tasks:

Task	Purpose	Example labels
answerability	Determines whether the context contains sufficient information to answer the query	yes, no
evidence_quality	Evaluates the strength of the retrieved context as evidence	strong, medium, weak
contradiction	Detects whether a statement conflicts with the retrieved context	yes, no
relevance	Determines whether the retrieved context is relevant to the query	relevant, irrelevant

Candidate labels should follow the conventions used during training. The model ranks the supplied options and returns normalized scores.

Key Features

• Four RAG evaluation tasks in a single model.
• ModernBERT-based encoder fine-tuned for context evaluation.
• Candidate-label scoring with a trained neural scoring head.
• Reusable Python interface through the ragone-ai package.
• Hugging Face Hub distribution for loading the model without manually downloading the base checkpoint.
• Potential integration with RAG pipelines, retrieval evaluation workflows, and research experiments.

Inference Latency Benchmarks

RAGOne performs task-specific classification using a ModernBERT-base encoder and a lightweight scoring head.

Per-Task Inference Latency CPU VS GPU

Task	Prediction Example	Latency on CPU	Latency on GPU
All tasks	lables	>100ms	<50ms

Recorded Inference Example

• Query: What is the refund period?
• Context: Customers can request a refund within 30 days.
• Predicted label: yes
• Yes score: 0.8938 (89.38%)
• No score: 0.1062 (10.62%)
• Measured inference latency: 19.03 ms

Benchmark Notes

• The reported latency is from recorded inference runs of multiple samples and real world data's.
• Actual latency depends on hardware, input length, runtime configuration, and whether model-loading time is included.
• CPU latency > 200ms and GPU latency <50ms
• Even 10ms on GPU

Model Architecture

RAGOne uses the following architecture:

1. Input construction: Combines the task instruction, query, retrieved context, and candidate options.
2. Encoder: A fine-tuned answerdotai/ModernBERT-base encoder produces contextual representations.
3. Option representation: Hidden states at the candidate-option marker positions are extracted.
4. Scoring head: A multilayer neural network scores each candidate.
5. Prediction: A softmax over the supplied candidate scores produces normalized scores, which are used to rank the options.

The scoring head is:

Linear(hidden_size, 256)
    → GELU
    → Dropout(0.1)
    → Linear(256, 1)

The model uses a maximum input length of 1,024 tokens. Longer contexts are truncated to fit the available token budget.

The exported model package contains the fine-tuned encoder, tokenizer, scoring head, configuration, and inference implementation. The encoder weights are stored in FP16, while the scoring head is stored in FP32 in the current export.

Dataset

RAGOne was trained using a dataset containing 5,074 records loaded from data.jsonl.

Dataset source: Hariharasuthana/RAGOne dataset

Task distribution

Task	Samples	Percentage
Contradiction	1,406	27.7%
Answerability	1,326	26.1%
Evidence quality	1,326	26.1%
Relevance	1,016	20.0%
Total	5,074	100%

The dataset covers multiple dimensions of retrieved-context quality rather than treating RAG evaluation as a single binary classification problem.

The dataset counts above describe the loaded dataset. They do not, by themselves, establish that the data are free of duplication, leakage, or annotation bias.

Training

RAGOne was fine-tuned for five epochs. Training completed in approximately 5.5 minutes in the reported Kaggle environment.

The best checkpoint was retained based on validation loss, with a reported best validation loss of 0.2697.

Training metrics

Epoch	Training loss	Validation loss	Validation accuracy
1	0.6552	0.4962	83.509%
2	0.2869	0.2697	91.425%
3	0.1530	0.3231	92.876%
4	0.0649	0.3371	94.063%
5	0.0206	0.3248	94.591%

Reported final validation accuracy: 94.591%.

Training accuracy and validation loss should be interpreted together. The decrease in training loss alongside the rise in validation loss after epoch 2 suggests possible overfitting. The best checkpoint was reported as the one with the lowest validation loss, not necessarily the final epoch.

These metrics are specific to the reported dataset split and training setup; they should not be interpreted as a guarantee of performance on unseen real-world RAG systems.

Evaluation

A separate evaluation suite tested RAGOne on 40 samples, with 10 samples for each of the four tasks. The reported evaluation used Groq with the openai/gpt-oss-120b model as the provider/model in the evaluation workflow.

Overall results

Metric	Result
Correct predictions	37 / 40
Overall accuracy	92.5%
Answerability	10 / 10 (100%)
Evidence quality	9 / 10 (90%)
Contradiction	10 / 10 (100%)
Relevance	8 / 10 (80%)

Per-task interpretation

• Answerability — 100%: All 10 evaluated samples matched their expected labels.
• Evidence quality — 90%: Nine of the 10 samples matched their expected labels.
• Contradiction — 100%: All 10 evaluated samples matched their expected labels.
• Relevance — 80%: Eight of the 10 samples matched their expected labels, making relevance a priority for further evaluation and improvement.

These results come from a small evaluation set and are not sufficient to establish generalization across domains, retrieval systems, or production workloads. The evaluation set's independence from the training data should be verified before treating these results as an independent benchmark.

The reported Groq evaluation identifies the provider and model used in the evaluation workflow. It should not be interpreted as evidence that RAGOne itself runs on Groq; the model can be loaded locally through its Python interface.

Installation

Install the Python package:

pip install ragone-ai

The Python import name is ragone.

The library requires PyTorch, Transformers, Hugging Face Hub, and Safetensors. Dependencies are installed automatically through the package metadata.

Quick Start

from ragone import RAGOne

model = RAGOne.from_pretrained("Hariharan05/ragone")

result = model.predict(
    query="What is the refund period?",
    context="Customers can request a refund within 30 days.",
    task="answerability",
    options=["yes", "no"],
)

print(result)

Example response structure:

{
    "prediction": "yes",
    "scores": [
        {"label": "yes", "score": 0.89},
        {"label": "no", "score": 0.11}
    ],
    "latency_ms": 85.0,
    "task": "answerability"
}

The scores and latency shown above are illustrative. Actual values depend on the model output and execution environment.

Usage Examples

1. Answerability

result = model.predict(
    query="What is the refund period?",
    context="Customers can request a refund within 30 days.",
    task="answerability",
    options=["yes", "no"],
)

2. Evidence quality

result = model.predict(
    query="What is the refund period?",
    context="The policy permits refunds within 30 days.",
    task="evidence_quality",
    options=["strong", "medium", "weak"],
)

3. Contradiction detection

result = model.predict(
    query="Refunds are not permitted.",
    context="Customers can request a refund within 30 days.",
    task="contradiction",
    options=["yes", "no"],
)

4. Relevance classification

result = model.predict(
    query="What is the refund period?",
    context="The company was founded in 2012.",
    task="relevance",
    options=["relevant", "irrelevant"],
)

These examples illustrate the interface, not guaranteed predictions. For contradiction detection in particular, ensure that the input statement and the context follow the same interpretation and formatting conventions as the training data.

Integration with RAG Pipelines

RAGOne can be incorporated into a RAG workflow as an evaluation stage:

User query
    ↓
Retriever
    ↓
Retrieved context
    ↓
RAGOne evaluation
    ├── Answerability
    ├── Evidence quality
    ├── Contradiction
    └── Relevance
    ↓
Application-defined filtering or reranking
    ↓
Answer generation

Applications can use the returned candidate scores to implement their own thresholds, context filtering, reranking, or fallback behavior.

RAGOne does not itself perform document retrieval, generate final answers, or guarantee factual correctness. Those behaviors must be implemented by the surrounding application.

Intended Use

RAGOne is intended for research, experimentation, and development involving:

• RAG evaluation and context-quality assessment.
• Retrieval relevance and answerability studies.
• Evidence-quality and contradiction-detection experiments.
• Prototyping evaluation components for RAG pipelines.
• Comparative experiments involving smaller task-specific evaluators.

It is in early stage, We will update it soon.

License

RAGOne is released under the License:apache-2.0 license designation for research-oriented use, subject to the actual license terms included with this repository.

Acknowledgements

• Answer.AI ModernBERT — base encoder.
• Groq — provider used in the reported separate evaluation workflow.
• OpenAI gpt-oss-120b — model identified in that evaluation workflow.
• The RAGOne dataset contributors and the open-source model ecosystem.

Citation

If you use RAGOne in your research, cite the model repository:

Hariharan Suthan. RAGOne: A Lightweight RAG Evaluation Model.
Hugging Face Model Hub.
https://huggingface.co/Hariharan05/ragone

Author

Hariharan Suthan

• Hugging Face: Hariharan05
• Model: Hariharan05/ragone

---

RAGOne is an ongoing research project. Feedback, reproducible evaluations, and contributions that improve generalization, robustness, and transparency are welcome.