Instructions to use Falcot922/tinyllama-sentiment-distilled with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Falcot922/tinyllama-sentiment-distilled with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") model = PeftModel.from_pretrained(base_model, "Falcot922/tinyllama-sentiment-distilled") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from Falcot922/tinyllama-sentiment-distilled: direct link, hf CLI and curl.
- Browser
- Download file 2.79 kB
-
https://huggingface.co/Falcot922/tinyllama-sentiment-distilled/resolve/main/README.md
- Command line
-
hf download hf://Falcot922/tinyllama-sentiment-distilled/README.md
-
curl -L -o README.md https://huggingface.co/Falcot922/tinyllama-sentiment-distilled/resolve/main/README.md
2.79 kB
metadata
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
library_name: peft
license: apache-2.0
datasets:
- stanfordnlp/imdb
language:
- en
pipeline_tag: text-classification
tags:
- lora
- knowledge-distillation
- sentiment-analysis
TinyLlama 1.1B, distilled from Mistral 7B, IMDB sentiment
LoRA adapter for TinyLlama-1.1B-Chat-v1.0, trained by knowledge distillation from the Mistral 7B teacher. Its control, trained on hard labels only, is tinyllama-sentiment-hard-labels. Part of mistral-optimisation.
Results
1,000 IMDB test reviews (shuffled with seed 42, first 400 characters), answer read from the class-token logits.
| Model | Accuracy (95% Wilson CI) | Reviews / s (T4) | Peak VRAM |
|---|---|---|---|
| Teacher: Mistral 7B NF4 + QLoRA | 91.6% (89.7 to 93.2) | 0.87 | 4.1 GiB |
| TinyLlama, no adapter | 65.4% (62.4 to 68.3) | 24.4 | 2.1 GiB |
| TinyLlama, hard labels | 89.7% (87.7 to 91.4) | 24.9 | 2.2 GiB |
| TinyLlama, distilled | 91.3% (89.4 to 92.9) | 24.9 | 2.2 GiB |
This adapter: 91.3% (89.4 to 92.9). Distillation beats hard labels by 1.6 points (paired test, p = 0.033); an earlier run of the same code gave 1.1 points (p = 0.14), so the gain is small. The distilled student is statistically indistinguishable from the teacher (p = 0.81) and 29 times faster.
Training
- Base:
TinyLlama/TinyLlama-1.1B-Chat-v1.0in fp16 - LoRA r = 16, alpha = 32, on all attention and MLP projections
- 1,000 IMDB train reviews, 3 epochs, lr 2e-4, effective batch 16
- Loss: 0.7 x KL divergence to the teacher's class distribution at temperature 2 (scaled by T²) + 0.3 x cross-entropy on the true label
Usage
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, "Falcot922/tinyllama-sentiment-distilled").eval()
prompt = ("Analyze the sentiment of this movie review and respond with exactly one word: "
"positive or negative.\n\nReview: {text}\nSentiment:")
text = prompt.format(text="A wonderful, moving film.")
n = len(tok(text)["input_ids"])
pos, neg = (tok(text + w)["input_ids"][n] for w in (" positive", " negative")) # first token of each answer
with torch.no_grad():
last = model(**tok(text, return_tensors="pt").to(model.device)).logits[0, -1]
print("positive" if last[pos] > last[neg] else "negative")