GiovanniIacuzzo02/SemEval-2027-Task-1-subtrack-2b

RETECO Sub-track 2b: Gold-Passage Grounded Generation

An experimental PEFT/LoRA adapter for evidence-grounded conversational answer generation.

Overview

This repository contains an experimental language-model adapter developed for RETECO SemEval-2027 Task 1, Sub-track 2b (Gold-Passage Grounded Generation). The task supplies the evidence passages so that answer generation can be evaluated separately from the retrieval component.

Given a target user turn, the preceding conversation, and organizer-provided gold passages, the model generates a response intended to be correct, complete, relevant, coherent with the dialogue, and supported by the evidence.

This is not an end-to-end retrieval-augmented generation system. It assumes that the supporting passages are already available and does not retrieve them from a corpus.

Task definition

Item	Description
Benchmark	RETECO / RECOR, Track 2
Sub-track	2b — Gold-Passage Grounded Generation
Input	Current turn, conversation history, and provided gold evidence
Training target	Reference answer, used only as the supervised target
Inference output	Generated answer for the target turn
Main evaluation	Five generation-quality dimensions in the official RETECO protocol

See the official task definition and evaluation protocol.

Method

Evidence-grounded SFT

The training pipeline uses supervised fine-tuning of an instruction-tuned causal language model with PEFT/LoRA. The prompt is constructed from the conversation context, current question, and resolved gold passages. During training, prompt tokens are masked from the loss (-100 labels), so the language-model objective is applied to the reference-answer tokens.

The reference answer is a training target only. It must not be inserted into the inference prompt, used to select candidate outputs, or included in a submission input.

Prompt construction

The implementation supports configurable input ablations:

Setting	Context
A	Current query only
B	Query and conversation history
C	Query and gold evidence
D	Query, conversation history, and gold evidence
E	Full context with optional auxiliary sub-question reasoning metadata

The default project setup uses the query, conversation history, and gold passages while leaving auxiliary reasoning metadata disabled. Long contexts are handled using configurable input/output budgets and history limits. For benchmark-aligned inference, reproduce the exact prompt construction and truncation rules used by the project implementation.

Training implementation

The pipeline supports:

• LoRA/PEFT fine-tuning;
• optional 4-bit NF4 QLoRA on compatible CUDA setups;
• gradient checkpointing and gradient accumulation;
• conversation-level train/internal-validation splitting;
• optional domain-balanced sampling;
• early stopping based on internal validation loss.

The adapter and its base model must be loaded together. The tokenizer and adapter configuration included in this repository are the source of truth for the published artifact; the project YAML and training logs should be consulted for the precise run configuration.

Loading and generating

The code below is a minimal example for an adapter based on Qwen/Qwen2.5-0.5B-Instruct. Replace the repository ID with the ID of this model repository. Confirm that base_model in the Hub adapter_config.json matches the base model below before using this example; if it differs, update the base-model ID to the exact recorded value.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE_MODEL_ID = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER_ID = "GiovanniIacuzzo02/SemEval-2027-Task-1-subtrack-2b"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_ID)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.float16 if device.type == "cuda" else torch.float32

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL_ID,
    torch_dtype=dtype,
)
model = PeftModel.from_pretrained(base_model, ADAPTER_ID)
model.to(device)
model.eval()

history = "Earlier turns in the conversation go here."
question = "The current user's question goes here."
evidence = "The provided gold passage text goes here."

messages = [
    {
        "role": "system",
        "content": (
            "Answer the current question using the provided evidence. "
            "Do not add unsupported factual claims."
        ),
    },
    {
        "role": "user",
        "content": (
            f"Conversation history:\n{history}\n\n"
            f"Current question:\n{question}\n\n"
            f"Provided gold passages:\n{evidence}"
        ),
    },
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

new_tokens = output_ids[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())

This example illustrates the Hugging Face loading and generation API. The message wording above is generic; for reproduction or benchmark submission, use the exact prompt builder and generation settings from the project's dataset and inference.py modules rather than substituting a different template.

Checkpoint status and evaluation

The recorded completed run for Qwen/Qwen2.5-0.5B-Instruct was an end-to-end smoke test, not a full-data model-quality experiment. It used 8 training examples and 4 internal-validation examples for one epoch. The recorded training loss was 1.9218, internal-validation loss was 1.3842, and internal-validation perplexity was 3.99. These values only establish that the data-loading, tokenization, forward/backward, validation, and checkpoint-saving path executed; they are not benchmark performance results.

A separate full-data attempt using Qwen/Qwen2.5-7B-Instruct with 4-bit QLoRA on a T4 terminated with CUDA out-of-memory after 8 of 1,933 training micro-batches. That run did not complete and produced no full-training result. No official hidden-test score or validated full-data generation score is claimed in this model card.

Before citing this model as a trained RETECO system, verify that the uploaded adapter corresponds to the intended completed run and report scores only from an identified checkpoint, split, prompt configuration, and evaluator version.

Evaluation protocol

The official RETECO protocol assesses generated answers along five independent dimensions:

1. Correctness — factual accuracy of the response.
2. Completeness — coverage of the required information.
3. Relevance — how directly the answer addresses the current turn.
4. Conversational coherence — consistency with the dialogue context.
5. Faithfulness — support from the supplied evidence passages.

The task reports these generation judgments separately. Automatic metrics such as ROUGE-L, METEOR, or BERTScore can be useful diagnostics, but they should not be presented as substitutes for the official evaluation.

Data and intended use

The project uses the RECOR conversational data distributed through the official RETECO dataset. The eleven Track 2 domains are Biology, Drones, Earth Science, Economics, Hardware, Law, Medical Sciences, Politics, Psychology, Robotics, and Sustainable Living.

Intended uses include research on evidence-grounded response generation, reproducibility checks for parameter-efficient fine-tuning, and controlled experiments where gold evidence is given as input. The adapter should not be interpreted as a substitute for evidence retrieval or as a guarantee that every generated statement is supported.

Limitations and safeguards

• The recorded 0.5B run is a small smoke test, not a representative estimate of answer quality.
• The documented 7B full-data attempt did not complete due to GPU memory constraints.
• Grounding depends on the quality, relevance, and completeness of the supplied gold passages and the prompt construction.
• Long conversation histories and evidence can be truncated to fit the context window.
• Reference answers must remain excluded from inference inputs and system-side candidate selection.
• No official hidden-test result is reported here.

References

• RETECO — SemEval-2027 Task 1
• RETECO task definition
• RETECO evaluation protocol
• RETECO dataset on Hugging Face
• Qwen2.5-0.5B-Instruct model card
• Hugging Face PEFT documentation
• Hugging Face bitsandbytes quantization documentation
• Ali et al. (2026), RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark, Findings of ACL 2026.