| --- |
| license: apache-2.0 |
| language: |
| - en |
| - de |
| - es |
| base_model: |
| - google/flan-t5-small |
| pipeline_tag: text-generation |
| tags: |
| - text-generation-inference |
| - summarization |
| - bitsandbytes |
| - text2text |
| - small |
| - inference |
| --- |
| |
|
|
|
|
| # t5-smaller |
|
|
| I made this as a smaller, more optimized version of |
| [`google/flan-t5-small`](https://huggingface.co/google/flan-t5-small). The |
| weights use bitsandbytes NF4 4-bit quantization with double quantization and |
| FP16 compute. |
|
|
| The name refers to the checkpoint size. This is still the original FLAN-T5 |
| Small architecture: no layers were removed, and it still has roughly 77 million |
| parameters, but it is around 61.7% smaller. |
|
|
| ## What changed |
|
|
| | | Original FLAN-T5 Small | This checkpoint | |
| |---|---:|---:| |
| | Stored weights | 307.9 MB | 118.1 MB | |
| | Architecture | FLAN-T5 Small | FLAN-T5 Small | |
| | Weight format | FP32 | NF4 4-bit | |
|
|
| That makes the weight file about 61.7% smaller. This number only describes file |
| size; runtime memory and speed depend on your GPU, PyTorch build, and |
| bitsandbytes version. |
|
|
| Also, much less RAM is used |
|
|
| I did not keep the exact base-model commit or the original conversion |
| environment. If you need a fully reproducible build, treat this checkpoint as a |
| starting point rather than a reference conversion. |
|
|
| ## Install |
|
|
| Install a PyTorch build that matches your CUDA setup, then run: |
|
|
| ```bash |
| pip install transformers accelerate bitsandbytes sentencepiece safetensors |
| ``` |
|
|
| ## Use it |
|
|
| The model files currently live in `optimized-flan-t5-small`, so both loader |
| calls need the `subfolder` argument: |
|
|
| ```python |
| from transformers import AutoModelForSeq2SeqLM, AutoTokenizer |
| |
| model_id = "ShinpacheShimura/t5-smaller" |
| subfolder = "optimized-flan-t5-small" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_id, subfolder=subfolder) |
| model = AutoModelForSeq2SeqLM.from_pretrained( |
| model_id, |
| subfolder=subfolder, |
| device_map="auto", |
| ) |
| |
| prompt = "translate English to German: How old are you?" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| output = model.generate(**inputs, max_new_tokens=64, do_sample=False) |
| |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| I plan to move the model and tokenizer files to the repository root. After that, |
| the two `subfolder` arguments can be removed. |
|
|
| ## Benchmark |
|
|
| `benchmark.py` compares this checkpoint with the FP32 base model on a small, |
| fixed set of prompts: |
|
|
| ```bash |
| python benchmark.py \ |
| --base-model google/flan-t5-small \ |
| --quantized-model ShinpacheShimura/t5-smaller \ |
| --quantized-subfolder optimized-flan-t5-small \ |
| --output-dir benchmark-results |
| ``` |
|
|
| It records loading time, model memory footprint, peak CUDA allocation, |
| generation latency, throughput, and the outputs from both models. Output |
| agreement is useful for spotting quantization changes, but it is not an accuracy |
| score. |
|
|
| I have not published GPU benchmark numbers yet. When I do, I will include the |
| GPU model, package versions, decoding settings, and raw result files here. |
|
|
| ## Where it may be useful |
|
|
| FLAN-T5 is a text-to-text model, so this checkpoint can be tried on translation, |
| summarization, question answering, and prompted classification. It is mainly |
| intended for experiments where checkpoint size or GPU memory matters. |
|
|
| All this considered, the model is best used when fine-tuned for a specific task, |
| and thanks to it's size, the post trtaining will not consume a lot of resources |
|
|
| The checkpoint is usually faster than the original on most machines. |
| But, small models can lose some of the expected speed benefit to quantization |
| overhead. Check the outputs on your own task before relying on it. |
|
|
| ## Credits |
|
|
| This checkpoint comes from |
| [`google/flan-t5-small`](https://huggingface.co/google/flan-t5-small) and keeps |
| its Apache-2.0 license. The original training work is described in: |
|
|
| ```bibtex |
| @article{chung2022scaling, |
| title={Scaling Instruction-Finetuned Language Models}, |
| author={Chung, Hyung Won and Hou, Le and Longpre, Shayne and others}, |
| journal={arXiv preprint arXiv:2210.11416}, |
| year={2022} |
| } |
| ``` |
|
|
| Quantized checkpoint published by |
| [`ShinpacheShimura`](https://huggingface.co/ShinpacheShimura). |
