Instructions to use Falcot922/tinyllama-sentiment-distilled with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Falcot922/tinyllama-sentiment-distilled with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") model = PeftModel.from_pretrained(base_model, "Falcot922/tinyllama-sentiment-distilled") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from Falcot922/tinyllama-sentiment-distilled: direct link, hf CLI and curl.
- Browser
- Download file 2.79 kB
-
https://huggingface.co/Falcot922/tinyllama-sentiment-distilled/resolve/main/README.md
- Command line
-
hf download hf://Falcot922/tinyllama-sentiment-distilled/README.md
-
curl -L -o README.md https://huggingface.co/Falcot922/tinyllama-sentiment-distilled/resolve/main/README.md
2.79 kB
| base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 | |
| library_name: peft | |
| license: apache-2.0 | |
| datasets: | |
| - stanfordnlp/imdb | |
| language: | |
| - en | |
| pipeline_tag: text-classification | |
| tags: | |
| - lora | |
| - knowledge-distillation | |
| - sentiment-analysis | |
| # TinyLlama 1.1B, distilled from Mistral 7B, IMDB sentiment | |
| LoRA adapter for `TinyLlama-1.1B-Chat-v1.0`, trained by knowledge distillation from [the Mistral 7B teacher](https://huggingface.co/Falcot922/mistral-nf4-sentiment). Its control, trained on hard labels only, is [tinyllama-sentiment-hard-labels](https://huggingface.co/Falcot922/tinyllama-sentiment-hard-labels). Part of [mistral-optimisation](https://github.com/Falcot922/mistral-optimisation). | |
| ## Results | |
| 1,000 IMDB test reviews (shuffled with seed 42, first 400 characters), answer read from the class-token logits. | |
| | Model | Accuracy (95% Wilson CI) | Reviews / s (T4) | Peak VRAM | | |
| |---|---|---|---| | |
| | Teacher: Mistral 7B NF4 + QLoRA | 91.6% (89.7 to 93.2) | 0.87 | 4.1 GiB | | |
| | TinyLlama, no adapter | 65.4% (62.4 to 68.3) | 24.4 | 2.1 GiB | | |
| | TinyLlama, hard labels | 89.7% (87.7 to 91.4) | 24.9 | 2.2 GiB | | |
| | TinyLlama, distilled | 91.3% (89.4 to 92.9) | 24.9 | 2.2 GiB | | |
| This adapter: **91.3% (89.4 to 92.9)**. Distillation beats hard labels by 1.6 points (paired test, p = 0.033); an earlier run of the same code gave 1.1 points (p = 0.14), so the gain is small. The distilled student is statistically indistinguishable from the teacher (p = 0.81) and 29 times faster. | |
| ## Training | |
| - Base: `TinyLlama/TinyLlama-1.1B-Chat-v1.0` in fp16 | |
| - LoRA r = 16, alpha = 32, on all attention and MLP projections | |
| - 1,000 IMDB train reviews, 3 epochs, lr 2e-4, effective batch 16 | |
| - Loss: 0.7 x KL divergence to the teacher's class distribution at temperature 2 (scaled by T²) + 0.3 x cross-entropy on the true label | |
| ## Usage | |
| ```python | |
| import torch | |
| from peft import PeftModel | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| base = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" | |
| tok = AutoTokenizer.from_pretrained(base) | |
| model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.float16, device_map="auto") | |
| model = PeftModel.from_pretrained(model, "Falcot922/tinyllama-sentiment-distilled").eval() | |
| prompt = ("Analyze the sentiment of this movie review and respond with exactly one word: " | |
| "positive or negative.\n\nReview: {text}\nSentiment:") | |
| text = prompt.format(text="A wonderful, moving film.") | |
| n = len(tok(text)["input_ids"]) | |
| pos, neg = (tok(text + w)["input_ids"][n] for w in (" positive", " negative")) # first token of each answer | |
| with torch.no_grad(): | |
| last = model(**tok(text, return_tensors="pt").to(model.device)).logits[0, -1] | |
| print("positive" if last[pos] > last[neg] else "negative") | |
| ``` | |