--- base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 library_name: peft license: apache-2.0 datasets: - stanfordnlp/imdb language: - en pipeline_tag: text-classification tags: - lora - knowledge-distillation - sentiment-analysis --- # TinyLlama 1.1B, distilled from Mistral 7B, IMDB sentiment LoRA adapter for `TinyLlama-1.1B-Chat-v1.0`, trained by knowledge distillation from [the Mistral 7B teacher](https://huggingface.co/Falcot922/mistral-nf4-sentiment). Its control, trained on hard labels only, is [tinyllama-sentiment-hard-labels](https://huggingface.co/Falcot922/tinyllama-sentiment-hard-labels). Part of [mistral-optimisation](https://github.com/Falcot922/mistral-optimisation). ## Results 1,000 IMDB test reviews (shuffled with seed 42, first 400 characters), answer read from the class-token logits. | Model | Accuracy (95% Wilson CI) | Reviews / s (T4) | Peak VRAM | |---|---|---|---| | Teacher: Mistral 7B NF4 + QLoRA | 91.6% (89.7 to 93.2) | 0.87 | 4.1 GiB | | TinyLlama, no adapter | 65.4% (62.4 to 68.3) | 24.4 | 2.1 GiB | | TinyLlama, hard labels | 89.7% (87.7 to 91.4) | 24.9 | 2.2 GiB | | TinyLlama, distilled | 91.3% (89.4 to 92.9) | 24.9 | 2.2 GiB | This adapter: **91.3% (89.4 to 92.9)**. Distillation beats hard labels by 1.6 points (paired test, p = 0.033); an earlier run of the same code gave 1.1 points (p = 0.14), so the gain is small. The distilled student is statistically indistinguishable from the teacher (p = 0.81) and 29 times faster. ## Training - Base: `TinyLlama/TinyLlama-1.1B-Chat-v1.0` in fp16 - LoRA r = 16, alpha = 32, on all attention and MLP projections - 1,000 IMDB train reviews, 3 epochs, lr 2e-4, effective batch 16 - Loss: 0.7 x KL divergence to the teacher's class distribution at temperature 2 (scaled by T²) + 0.3 x cross-entropy on the true label ## Usage ```python import torch from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer base = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" tok = AutoTokenizer.from_pretrained(base) model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.float16, device_map="auto") model = PeftModel.from_pretrained(model, "Falcot922/tinyllama-sentiment-distilled").eval() prompt = ("Analyze the sentiment of this movie review and respond with exactly one word: " "positive or negative.\n\nReview: {text}\nSentiment:") text = prompt.format(text="A wonderful, moving film.") n = len(tok(text)["input_ids"]) pos, neg = (tok(text + w)["input_ids"][n] for w in (" positive", " negative")) # first token of each answer with torch.no_grad(): last = model(**tok(text, return_tensors="pt").to(model.device)).logits[0, -1] print("positive" if last[pos] > last[neg] else "negative") ```