---
language:
- en
license: apache-2.0
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
tags:
- tinyllama
- 4bit
- bitsandbytes
- nf4
- quantized
- transformers
library_name: transformers
pipeline_tag: text-generation
---

# TinyLlama-1.1B-Chat-v1.0 – 4-bit NF4

This repository contains a **4-bit quantized version of
https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0\*\*.

The model was created as a personal experiment while I was first learning about
large language models and model quantization around **July–August 2025**.

The quantization was performed on **Google Colab using an NVIDIA T4 GPU**, with
Hugging Face Transformers and `bitsandbytes`.

## Model Details

- **Base model:** `TinyLlama/TinyLlama-1.1B-Chat-v1.0`
- **Architecture:** TinyLlama / Llama-style causal language model
- **Task:** Text generation / chat
- **Quantization:** 4-bit
- **Quantization backend:** bitsandbytes
- **Quantization type:** NF4
- **Compute dtype:** `torch.bfloat16`
- **Original model:** https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0

## Quantization

The model was loaded using the following `BitsAndBytesConfig`:

```python
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4"
)
```

It was then loaded with:

```python
model = AutoModelForCausalLM.from_pretrained(
    "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    quantization_config=quantization_config,
    device_map="auto"
)
```

and uploaded to the Hugging Face Hub with `model.push_to_hub()`.

### Original quantization script

For reproducibility and historical reference, the relevant script used to
create this repository was:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from huggingface_hub import login
from accelerate import Accelerator
import torch

# Authenticate with Hugging Face.
# Do not hard-code access tokens in public scripts.
login(token="YOUR_HF_TOKEN")

accelerator = Accelerator()

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4"
)

model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=quantization_config,
    device_map="auto"
)

model = accelerator.prepare(model)

repo_id = "PaddySeahorse/tinyllama-4bit"

model.push_to_hub(
    repo_id,
    commit_message="Push 4-bit quantized TinyLlama"
)

tokenizer.push_to_hub(
    repo_id,
    commit_message="Push tokenizer"
)
```

## Usage

This repository was produced using Hugging Face Transformers and
`bitsandbytes`. A typical loading configuration is:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

repo_id = "PaddySeahorse/tinyllama-4bit"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4"
)

tokenizer = AutoTokenizer.from_pretrained(repo_id)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    quantization_config=quantization_config,
    device_map="auto",
)
```

For chat-style inference, refer to the usage instructions and chat template of
the original TinyLlama model.

## Intended Use

This repository was created primarily as a **learning experiment** for:

- exploring LLM quantization;
- learning the Hugging Face Transformers ecosystem;
- experimenting with `bitsandbytes` 4-bit loading;
- reducing the memory required to run TinyLlama;
- testing model deployment and sharing through the Hugging Face Hub.

It is not an independently trained or fine-tuned model. The underlying model
and its capabilities originate from `TinyLlama/TinyLlama-1.1B-Chat-v1.0`.

## Limitations

This repository should be considered an **experimental quantization artifact**.

No systematic evaluation was performed after quantization, so no claims are
made regarding accuracy, perplexity, benchmark performance, or output quality
relative to the original model.

The repository was produced by loading the base model through the
`bitsandbytes` 4-bit quantization path and subsequently uploading it. It should
therefore not be confused with separately converted **GPTQ, AWQ, or GGUF**
quantizations.

Compatibility may depend on the versions of `transformers`, `accelerate`,
`bitsandbytes`, PyTorch, and the available hardware.

For the original model's intended uses, limitations, training information, and
other details, please refer to the
https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0.

## Hardware

The original quantization experiment was performed in **Google Colab** with an
**NVIDIA T4 GPU**.

Exact package versions from the original July–August 2025 environment were not
recorded.

## Acknowledgements

All credit for the base model goes to the **TinyLlama** project and its
contributors.

Base model:

- https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0

This repository only provides the experimental 4-bit quantized artifact
created from that model.

## License

Please refer to the license and usage terms of the original
`TinyLlama/TinyLlama-1.1B-Chat-v1.0` model. This repository does not introduce
a new license for the underlying model.