joestacey/gemma4-e4b-nli-10k

Gemma 4 E4B NLI 10k

LoRA adapter for three-way natural language inference, trained on 10,000 examples with 2,000 sampled from each listed dataset using seed 0. FEVER inputs use evidence as the premise and the claim as the hypothesis. Labels are entailment, neutral and contradiction. The adapter and tokenizer are included; the base weights are loaded separately at the revision pinned in adapter_config.json.

Training uses one epoch, batch size 8, AdamW with learning rate 0.0001, linear warmup over 5% of steps, and a 256-token premise cap. QLoRA uses 4-bit NF4 with double quantization and bfloat16 compute. LoRA rank is 16, alpha 32 and dropout 0.05 on text attention and MLP projections. Loss applies only to answer tokens. Both adapters use the same training examples and recipe, with their own model tokenizer and chat template.

The scoring implementation normalizes the logits for the distinct first tokens of the three label words. It returns probabilities in entailment, neutral, contradiction order. These are not full-word sequence likelihoods.

Code and scoring commands are in understanding-probe-robustness. From its correctness_models directory, score saved generations with

python run_scores.py --artifacts_dir /path/to/artifacts --model llama --dataset sciq --setting ID --correctness_function gemma --correctness_model joestacey/gemma4-e4b-nli-10k

This scores both training and evaluation answers by default without regenerating answers or hidden states. The saved NLI score uses AlignScore's sentence and reference aggregation. Evaluation of the trained adapter on held-out NLI datasets is in progress; no final evaluation results are claimed here.