StudyAbroadGPT-7B-LoRa-Kaggle

Parameter-efficient LoRA-adapted Mistral-7B-Instruct-v0.3 fine-tuned on synthetic study-abroad conversational data for domain-specific academic advising guidance.

Status: Generation and source-verified factuality audit complete ✅ (per arXiv:2504.15610v4 §4.3) | Caveat Phrase Usage regression flagged in §5.1

⚠️ Important Disclaimers (read first)

This model is a research artifact documenting a failure mode, not a deployable advising system. By the v4 §4.3.1 source-verified audit, the model produces 4 confident factual errors on policy-sensitive topics (healthcare, admissions, scholarships) against zero for the un-fine-tuned base on the same prompts. The training data is the cause, not the fine-tuning method (see v4 §4.4).

Do not deploy as a student-advising system without prior factuality-gating or retrieval grounding against authoritative sources. Institutional deployment is particularly discouraged.

🔗 Project Ecosystem

Resource	Link
Dataset	millat/StudyAbroadGPT-Dataset
Training Code	codermillat/StudyAbroadGPT
Dataset Generation	codermillat/study-abroad-dataset
Evaluation Artifacts	LoRA Paper evaluation workspace
Research Paper	arXiv:2504.15610
Author ORCID	0009-0005-7198-9893

📊 Model Details

Architecture

Component	Specification
Base Model	mistralai/Mistral-7B-Instruct-v0.3
Base Model Size	7.24 billion parameters
Quantization	4-bit NF4 (via Unsloth)
Fine-Tuning Method	LoRA (Low-Rank Adaptation)
LoRA Rank (r)	16
LoRA Alpha (α)	32
Scaling Factor	α / r = 2.0

Trainable Parameters (verified against adapter_config.json)

• Total Model Parameters: 7,241,731,200
• LoRA Trainable Parameters: 41,943,040 (0.60% of base model)
• LoRA Adapters: Applied to all linear projections (q, k, v, o, gate, up, down) of all 32 transformer layers
• Note: an earlier v1/v2-era model card reported ~4.7M trainable params (attention-only); the v4 paper and adapter_config.json confirm the full 41.9M across all 7 target modules per layer.

Quantization Details

• Method: 4-bit NF4 quantization via Unsloth (with double quantization)
• Memory: ~5.5 GB at 4-bit (vs ~14 GB at fp16)
• Benefit: Enables fine-tuning on a single 16 GB GPU

🎯 Training Details

Training Data

• Dataset: millat/StudyAbroadGPT-Dataset
• Training Samples: 2,274 conversations
• Test Samples: 402 conversations
• Topics Covered: Admissions, scholarships, visas, accommodation, cultural adaptation
• Generator: Gemini 1.0 Pro (December 2023-era); see the dataset card and arXiv:2504.15610v4 §4.4 for the source-verified factuality audit (28–40% data error rate, n=40).

Training Configuration (verified, v4 §3.2 + adapter_config.json)

Parameter	Value
Total Epochs	3 (1 P100 + 2 T4)
Batch Size (P100 / T4)	2 / 4 per device
Gradient Accumulation	4 / 8
Effective Batch Size	8 / 32
Learning Rate	2 × 10⁻⁴
Warmup Ratio	0.03
LR Scheduler	Linear
Max Sequence Length	2,048 tokens
Optimizer	AdamW 8-bit (bitsandbytes)
Max Gradient Norm	0.3
Weight Decay	0.0
Mixed Precision	bfloat16
Gradient Checkpointing	Enabled
Seed	42
LoRA dropout	0
LoRA bias	False (none)

Note: an earlier v1/v2-era model card reported 4 epochs; the v4 paper and training reports confirm 3 epochs total (1 P100 + 2 T4). The 4-epoch figure is retracted.

Hardware & Resources

Setting	Value
Phase 1 GPU	Tesla P100-16GB (Kaggle)
Phase 1 time	5h 47m 25s (284 steps)
Phase 1 peak VRAM	15.888 GB
Phase 2 GPU	Tesla T4-16GB (Kaggle)
Phase 2 time	5h 26m 18s (142 steps × 2 epochs)
Phase 2 peak VRAM	14.741 GB
Adapter-only handoff	Yes; optimizer and scheduler re-initialized on the second GPU

🚀 Usage

Option 1: Using Unsloth (Recommended for Inference)

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="millat/StudyAbroadGPT-7B-LoRa-Kaggle",
    max_seq_length=2048,
    dtype=torch.float16,
    load_in_4bit=True,
)

FastLanguageModel.for_inference(model)

prompt = "What documents do I need for a UK student visa?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.0,
        do_sample=False,
        top_p=1.0
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Option 2: Using Transformers Library (merged weights)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "millat/StudyAbroadGPT-7B-LoRa-Kaggle",
    subfolder="merged",
    torch_dtype="auto",
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(
    "millat/StudyAbroadGPT-7B-LoRa-Kaggle",
    subfolder="merged"
)

prompt = "How much should I budget for accommodation in London?"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.0,
    do_sample=False
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Option 3: Using LoRA Adapter (Continued Training)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    torch_dtype="auto",
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    "millat/StudyAbroadGPT-7B-LoRa-Kaggle"
)

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")

📊 Evaluation Results (arXiv:2504.15610v4 §4.3)

Source-Verified Factuality Audit (the load-bearing signal)

Metric	Base	LoRA	Direction
Source-verified factual errors on the 18/50 divergent prompts (4 highest-sensitivity cases verified)	0	4	LoRA worse on 16/18
Sample size	50 prompts	50 prompts	(same prompts, deterministic decoding)
Generation settings	do_sample=False, T=0.0, top_p=1.0, max_new_tokens=512	(same)	(matched)

The 4 verified LoRA errors (all on policy-sensitive topics):

• Harvard Medical School testing: LoRA states HMS "requires all applicants to submit GRE scores"; base correctly states MCAT, not GRE. HMS source
• Australian healthcare: LoRA states students are "eligible for Medicare after at least six months"; base correctly states OSHC is compulsory under visa condition 8501. Study Australia
• "Bachelor of Medicine at Stanford": LoRA elaborates a program that does not exist; base correctly states Stanford offers no direct undergraduate medicine degree.
• Brazil→Bangladesh scholarships: LoRA fabricates named scholarship programs; base correctly states such scholarships are not common. British Council

Blind LLM-as-Judge (single judge family, different from data generator)

Metric	Base	LoRA	Δ (LoRA − Base)	95% CI on Δ
Domain accuracy (0–3)	2.14 ± 0.53	1.74 ± 0.69	−0.40	[−0.62, −0.18]
Helpfulness (0–3)	2.14 ± 0.49	1.82 ± 0.59	−0.32	[−0.52, −0.12]
Preference (3-way)	23/50 (46%)	9/50 (18%)	(tie: 18/50)	McNemar-Bowker χ²(2)=9.3, p=0.0095

Reference-Based Metrics (max_new_tokens=512)

Metric	Base	LoRA	Δ	95% CI on Δ
SacreBLEU (corpus)	5.71	9.05	+3.34	corpus-level
ROUGE-L F1	0.1937	0.2125	+0.019	[+0.010, +0.028]
BERTScore F1 (rescaled)	0.0981	0.1611	+0.063	[+0.047, +0.078]

⚠️ These reference-based gains measure fidelity to the synthetic training distribution, not ground-truth quality. The synthetic references themselves contain the same factual errors documented in the dataset card and v4 §4.4. Reporting BERTScore alone would invert the conclusion.

Caveat Phrase Usage (a safety regression flagged in v4 §5.1)

Indicator	Base	LoRA
Caveat Phrase Usage	2.0%	0.0%

The LoRA model dropped its safety hedges compared to base. In any deployment setting this would systematically reduce the rate at which the model flags uncertainty, which is a substantive concern for an advising model. v4 §5.1 calls this out as a finding, not just a quality metric.

⚠️ Important Limitations (v4 §5.2)

1. Training and evaluation data are fully synthetic (Gemini 1.0 Pro).
2. The LLM-as-judge is a single model, single pass, on synthetic-split prompts.
3. The factuality audit is curated (4 highest-sensitivity cases), not exhaustive.
4. The dataset prevalence rate (28–40%) rests on a single judge at n=40.
5. The two LLM-judged signals (model and dataset) are not independent.
6. The reliability finding rests on a single fine-tuning run (no seed variance).
7. Reference-based metrics use synthetic references.
8. The two-phase schedule is a recipe, not an ablation.
9. Single language and domain (English, study-abroad).

Recommended Usage

✅ Safe to use for:

• Educational chatbot prototyping under controlled conditions
• Research on domain adaptation and synthetic-data failure modes
• Fine-tuning experiments (continued training)
• Data augmentation for related research tasks

❌ NOT safe to use for:

• Direct immigration, healthcare, or visa advice
• Official policy interpretation
• Time-sensitive information (visas, deadlines)
• High-stakes decision making without expert review and factuality-gating

🔄 Model Variants

Variant	Format	Size	Use Case
/merged	Full merged weights	~14 GB	Inference, GGUF conversion
Adapter-only (this repo)	LoRA weights	~50 MB	Continued training, fine-tuning

📚 Citation

If you use this model or the v4 findings, cite the paper:

@article{hosen2026finetuning,
  author = {Hosen, Md Millat},
  title = {Fine-Tuning a 7B Advisor on Free-Tier GPUs: An Adapter-Handoff Recipe and a Synthetic-Data Reliability Caution},
  journal = {arXiv preprint arXiv:2504.15610v4},
  year = {2026},
  doi = {10.48550/arXiv.2504.15610}
}

🔐 License

Apache License 2.0 — the upstream Mistral-7B-Instruct-v0.3 license.

Note: an earlier v1/v2-era model card stated "Mistral Research License"; the v4 paper and the upstream model card confirm Apache 2.0.

📊 Performance / Hardware Compatibility

Device	Status	Notes
NVIDIA T4 (16 GB)	✅ Tested	Kaggle (Phase 2)
NVIDIA P100 (16 GB)	✅ Tested	Kaggle (Phase 1)
NVIDIA A100 (40 GB)	Should work	Not tested
CPU only	❌ Not recommended	Too slow
Mac M1/M2	⚠️ Requires setup	MPS acceleration possible

🤝 Contributing & Feedback

• Bug Reports: Open an issue on training repo
• Model Improvements: PRs with new training runs or data are welcome
• Questions: Check the v4 paper §5 (Discussion + Limitations + Future Work) and the GitHub workspace

---

Model Version: 1.0 (v4 reconciliation, September 2026)
Adapter SHA: matches data/v4-adapter-config.json
Training Framework: Unsloth + Transformers + bitsandbytes
Base Model Lineage: Mistral-7B → Instruct-v0.3 → 4-bit NF4 quantized → LoRA fine-tuned