Text Generation
Transformers
Safetensors
English
llama
tinyllama
4bit
bitsandbytes
nf4
quantized
conversational
4-bit precision
Instructions to use PaddySeahorse/tinyllama-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use PaddySeahorse/tinyllama-4bit with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="PaddySeahorse/tinyllama-4bit") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("PaddySeahorse/tinyllama-4bit") model = AutoModelForCausalLM.from_pretrained("PaddySeahorse/tinyllama-4bit", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use PaddySeahorse/tinyllama-4bit with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "PaddySeahorse/tinyllama-4bit" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "PaddySeahorse/tinyllama-4bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/PaddySeahorse/tinyllama-4bit
- SGLang
How to use PaddySeahorse/tinyllama-4bit with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "PaddySeahorse/tinyllama-4bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "PaddySeahorse/tinyllama-4bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "PaddySeahorse/tinyllama-4bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "PaddySeahorse/tinyllama-4bit", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use PaddySeahorse/tinyllama-4bit with Docker Model Runner:
docker model run hf.co/PaddySeahorse/tinyllama-4bit
|
Download README.md from PaddySeahorse/tinyllama-4bit: direct link, hf CLI and curl.
- Browser
- Download file 5.34 kB
-
https://huggingface.co/PaddySeahorse/tinyllama-4bit/resolve/main/README.md
- Command line
-
hf download hf://PaddySeahorse/tinyllama-4bit/README.md
-
curl -L -o README.md https://huggingface.co/PaddySeahorse/tinyllama-4bit/resolve/main/README.md
5.34 kB
| language: | |
| - en | |
| license: apache-2.0 | |
| base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 | |
| tags: | |
| - tinyllama | |
| - 4bit | |
| - bitsandbytes | |
| - nf4 | |
| - quantized | |
| - transformers | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| # TinyLlama-1.1B-Chat-v1.0 – 4-bit NF4 | |
| This repository contains a **4-bit quantized version of | |
| https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0\*\*. | |
| The model was created as a personal experiment while I was first learning about | |
| large language models and model quantization around **July–August 2025**. | |
| The quantization was performed on **Google Colab using an NVIDIA T4 GPU**, with | |
| Hugging Face Transformers and `bitsandbytes`. | |
| ## Model Details | |
| - **Base model:** `TinyLlama/TinyLlama-1.1B-Chat-v1.0` | |
| - **Architecture:** TinyLlama / Llama-style causal language model | |
| - **Task:** Text generation / chat | |
| - **Quantization:** 4-bit | |
| - **Quantization backend:** bitsandbytes | |
| - **Quantization type:** NF4 | |
| - **Compute dtype:** `torch.bfloat16` | |
| - **Original model:** https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0 | |
| ## Quantization | |
| The model was loaded using the following `BitsAndBytesConfig`: | |
| ```python | |
| quantization_config = BitsAndBytesConfig( | |
| load_in_4bit=True, | |
| bnb_4bit_compute_dtype=torch.bfloat16, | |
| bnb_4bit_quant_type="nf4" | |
| ) | |
| ``` | |
| It was then loaded with: | |
| ```python | |
| model = AutoModelForCausalLM.from_pretrained( | |
| "TinyLlama/TinyLlama-1.1B-Chat-v1.0", | |
| quantization_config=quantization_config, | |
| device_map="auto" | |
| ) | |
| ``` | |
| and uploaded to the Hugging Face Hub with `model.push_to_hub()`. | |
| ### Original quantization script | |
| For reproducibility and historical reference, the relevant script used to | |
| create this repository was: | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig | |
| from huggingface_hub import login | |
| from accelerate import Accelerator | |
| import torch | |
| # Authenticate with Hugging Face. | |
| # Do not hard-code access tokens in public scripts. | |
| login(token="YOUR_HF_TOKEN") | |
| accelerator = Accelerator() | |
| quantization_config = BitsAndBytesConfig( | |
| load_in_4bit=True, | |
| bnb_4bit_compute_dtype=torch.bfloat16, | |
| bnb_4bit_quant_type="nf4" | |
| ) | |
| model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" | |
| tokenizer = AutoTokenizer.from_pretrained(model_name) | |
| model = AutoModelForCausalLM.from_pretrained( | |
| model_name, | |
| quantization_config=quantization_config, | |
| device_map="auto" | |
| ) | |
| model = accelerator.prepare(model) | |
| repo_id = "PaddySeahorse/tinyllama-4bit" | |
| model.push_to_hub( | |
| repo_id, | |
| commit_message="Push 4-bit quantized TinyLlama" | |
| ) | |
| tokenizer.push_to_hub( | |
| repo_id, | |
| commit_message="Push tokenizer" | |
| ) | |
| ``` | |
| ## Usage | |
| This repository was produced using Hugging Face Transformers and | |
| `bitsandbytes`. A typical loading configuration is: | |
| ```python | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig | |
| repo_id = "PaddySeahorse/tinyllama-4bit" | |
| quantization_config = BitsAndBytesConfig( | |
| load_in_4bit=True, | |
| bnb_4bit_compute_dtype=torch.bfloat16, | |
| bnb_4bit_quant_type="nf4" | |
| ) | |
| tokenizer = AutoTokenizer.from_pretrained(repo_id) | |
| model = AutoModelForCausalLM.from_pretrained( | |
| repo_id, | |
| quantization_config=quantization_config, | |
| device_map="auto", | |
| ) | |
| ``` | |
| For chat-style inference, refer to the usage instructions and chat template of | |
| the original TinyLlama model. | |
| ## Intended Use | |
| This repository was created primarily as a **learning experiment** for: | |
| - exploring LLM quantization; | |
| - learning the Hugging Face Transformers ecosystem; | |
| - experimenting with `bitsandbytes` 4-bit loading; | |
| - reducing the memory required to run TinyLlama; | |
| - testing model deployment and sharing through the Hugging Face Hub. | |
| It is not an independently trained or fine-tuned model. The underlying model | |
| and its capabilities originate from `TinyLlama/TinyLlama-1.1B-Chat-v1.0`. | |
| ## Limitations | |
| This repository should be considered an **experimental quantization artifact**. | |
| No systematic evaluation was performed after quantization, so no claims are | |
| made regarding accuracy, perplexity, benchmark performance, or output quality | |
| relative to the original model. | |
| The repository was produced by loading the base model through the | |
| `bitsandbytes` 4-bit quantization path and subsequently uploading it. It should | |
| therefore not be confused with separately converted **GPTQ, AWQ, or GGUF** | |
| quantizations. | |
| Compatibility may depend on the versions of `transformers`, `accelerate`, | |
| `bitsandbytes`, PyTorch, and the available hardware. | |
| For the original model's intended uses, limitations, training information, and | |
| other details, please refer to the | |
| https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0. | |
| ## Hardware | |
| The original quantization experiment was performed in **Google Colab** with an | |
| **NVIDIA T4 GPU**. | |
| Exact package versions from the original July–August 2025 environment were not | |
| recorded. | |
| ## Acknowledgements | |
| All credit for the base model goes to the **TinyLlama** project and its | |
| contributors. | |
| Base model: | |
| - https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0 | |
| This repository only provides the experimental 4-bit quantized artifact | |
| created from that model. | |
| ## License | |
| Please refer to the license and usage terms of the original | |
| `TinyLlama/TinyLlama-1.1B-Chat-v1.0` model. This repository does not introduce | |
| a new license for the underlying model. | |