banking77-llama-1b-lora

LoRA adapter for unsloth/Llama-3.2-1B-Instruct, fine-tuned on the mteb/banking77 77-class banking intent classification benchmark using QLoRA + rsLoRA + NEFTune.

90.21% exact-match accuracy on the Banking77 test split (2,775 / 3,076), 90.28% with trie-constrained decoding (2,777 / 3,076) Prompted zero-shot with the same instruction, the base model produced no correct labels (0 / 3,076) One free-tier T4 GPU · 3 epochs · 25 minutes of training (train_runtime 1,507 s)

---

Evaluation Results

Setting	Accuracy	Correct / Total
Base model (prompted zero-shot, same instruction)	0.00%	0 / 3,076
Fine-tuned, free generation	90.21%	2,775 / 3,076
Fine-tuned, constrained decoding (trie)	90.28%	2,777 / 3,076

All rows are greedy-decoded (do_sample=False) exact-match scores on the full Banking77 test split (3,076 queries, 77 classes, ~40 per class) from a single 3-epoch training run. Only the 9,993 train rows are written to the training data, so the model never trains on a test row. The run has no separate validation split: the test split is also the trainer's evaluation set, its loss by epoch was 0.4100, 0.3817 and 0.3839, and load_best_model_at_end on eval_loss kept the epoch-2 checkpoint, which is the model the fine-tuned rows score. During training, a generation check on a fixed 500-query random subset of the same split scored 80.80%, 89.00% and 90.60% after epochs 1, 2 and 3.

Exact match needs the precise snake_case label string. Given the same instruction, the base model never produced one, so the 90.2 pp gain covers both learning the output format and learning to tell 77 intents apart.

Constrained decoding builds a token-level prefix trie over all 77 valid label names and, at every decoding step, allows only tokens that continue a valid label, so every output is a valid label. On the full test split it fixed 2 samples and broke 0 (+0.07 pp). One fix replaced the invalid get_virtual_card with the correct get_disposable_virtual_card; in the other, the per-step restriction moved greedy decoding from one valid label (getting_spare_card) to the correct one (disposable_card_limits).

---

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "unsloth/Llama-3.2-1B-Instruct",
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "rajo0113/banking77-llama-1b-lora")
model = model.merge_and_unload()  # optional: merge weights for faster inference
tokenizer = AutoTokenizer.from_pretrained("rajo0113/banking77-llama-1b-lora")

The reported figures were measured in the training notebook with the base model loaded in 4-bit NF4 through Unsloth (adapter_config.json names unsloth/llama-3.2-1b-instruct-unsloth-bnb-4bit). Loading the base in 16-bit as above can change a small number of predictions.

Inference

The model was trained with Llama's native chat template and the instruction below. Use both unchanged at inference:

INSTRUCTION = (
    "Classify the following banking customer query into one of 77 intent categories. "
    "Respond with only the intent label name."
)

def classify(query: str) -> str:
    messages = [{"role": "user", "content": f"{INSTRUCTION}\n\n{query}"}]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(
            **inputs, max_new_tokens=20, do_sample=False,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(
        out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
    ).strip()

# Labels recorded for these queries by the notebook's inference test (Section 8):
print(classify("I lost my card, what should I do?"))
# → lost_or_stolen_card
print(classify("How do I change my PIN?"))
# → change_pin
print(classify("My transfer hasn't arrived yet."))
# → transfer_not_received_by_recipient

---

Training Configuration

Parameter	Value
Base model	unsloth/Llama-3.2-1B-Instruct
Quantisation	4-bit NF4 (bitsandbytes)
LoRA rank	16
LoRA alpha	32 (2 × rank)
LoRA dropout	0.05
rsLoRA	✓ (scales the adapter by alpha/√rank instead of alpha/rank: 8 rather than 2 at rank 16)
NEFTune noise alpha	5
Optimizer	adamw_8bit
Learning rate	2e-4 (cosine + 3% warmup), weight decay 0.01
Effective batch size	16 (per_device=8, grad_accum=2)
Max sequence length	512
Sequence packing	✓ (9,993 training rows packed into 1,610 sequences; 303 steps over 3 epochs)
Epochs	3 (epoch-2 checkpoint kept, lowest eval loss)
Precision	fp16 on the T4 (no bf16 support)
Trainable parameters	11,272,192 of 1,247,086,592 (0.90%)
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training time	1,507 s (train_runtime), one T4

Why the native chat template?

Using Llama's native chat format rather than Alpaca ### Instruction / ### Response was a central architectural decision. SFT builds on the format the model was already instruction-tuned with, instead of teaching it a new one, and the same template is used for training, evaluation and inference.

---

Dataset

mteb/banking77: 13,069 samples across 77 banking intent classes. Train split: 9,993 samples, the only rows written to the training data. Test split: 3,076 samples, used for evaluation and, as the trainer's only evaluation set, for checkpoint selection by eval loss. Labels are integer IDs (0-76) resolved to snake_case strings (e.g. card_arrival, transfer_not_received_by_recipient).

The task is framed as generative classification: the model outputs the label string directly, leveraging its text generation capability without requiring a classification head.

---

Worst-Class Analysis

Class	Accuracy	Root cause
topping_up_by_card	0/40 (0%)	Confused with top_up_by_card_charge (13 errors), a near-identical surface form, and top_up_reverted (10); the model produced the invalid label top_up_by_card on 9/40 samples
card_arrival	21/40 (52.5%)	Dataset-level ambiguity with card_delivery_estimate: 18 of its 19 errors come from that single confused pair
transfer_not_received_by_recipient	27/39 (69.2%)	Overlaps with transfer_timing (5 of its 12 errors)

Constrained decoding rules out the invalid top_up_by_card, but those 9 samples remain wrong: it cannot fix semantic confusion the model never learned to resolve during training. That points to the training data (for example, targeted topping_up_by_card examples) rather than model capacity.

---

Training Code & Full Methodology

Full training notebook (Google Colab T4) with the evaluation suite (per-class accuracy, confused pairs, zero-shot baseline, constrained decoding): github.com/rajo69/banking77-llama-1b-lora

---

Citation

@inproceedings{casanueva2020efficient,
  title     = {Efficient Intent Detection with Dual Sentence Encoders},
  author    = {Casanueva, I{\~n}igo and others},
  booktitle = {Proceedings of the 2nd Workshop on NLP for ConvAI},
  year      = {2020}
}
@article{hu2021lora,
  title  = {LoRA: Low-Rank Adaptation of Large Language Models},
  author = {Hu, Edward J and others},
  year   = {2021}
}
@article{dettmers2023qlora,
  title  = {QLoRA: Efficient Finetuning of Quantized LLMs},
  author = {Dettmers, Tim and others},
  year   = {2023}
}