Instructions to use mikecovlee/tinymixtral-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mikecovlee/tinymixtral-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mikecovlee/tinymixtral-it", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("mikecovlee/tinymixtral-it", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mikecovlee/tinymixtral-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mikecovlee/tinymixtral-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mikecovlee/tinymixtral-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mikecovlee/tinymixtral-it

SGLang

How to use mikecovlee/tinymixtral-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mikecovlee/tinymixtral-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mikecovlee/tinymixtral-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mikecovlee/tinymixtral-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mikecovlee/tinymixtral-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mikecovlee/tinymixtral-it with Docker Model Runner:
```
docker model run hf.co/mikecovlee/tinymixtral-it
```

tinymixtral-it / README.md

mikecovlee

Unify model card: shared English template, remove internal identifiers

98fc327 about 1 month ago

preview code

Raw History Blame Contribute Delete

6.4 kB

---
license: mit
library_name: transformers
pipeline_tag: text-generation
language: en
tags:
  - tinymixtral
  - moe
  - mixture-of-experts
  - instruction-tuned
  - sft
  - text-generation
  - custom_code
model_type: tinymixtral
base_model: mikecovlee/tinymixtral
datasets:
  - allenai/tulu-3-sft-mixture
  - teknium/OpenHermes-2.5
  - Open-Orca/SlimOrca
  - Open-Orca/OpenOrca
  - HuggingFaceH4/ultrachat_200k
  - meta-math/MetaMathQA
  - microsoft/orca-math-word-problems-200k
  - nvidia/OpenMathInstruct-2
  - rajpurkar/squad_v2
  - mandarjoshi/trivia_qa
---
# tinymixtral-it — instruction-tuned MoE

`tinymixtral-it` (a.k.a. **v3.0-it**) is the **instruction-tuned** version of the MoE flagship
[`mikecovlee/tinymixtral`](https://huggingface.co/mikecovlee/tinymixtral): 477.5M total / 276.1M
active parameters (top-2 of 4 experts), 2048 context, 32k SentencePiece vocabulary.

It is the MoE counterpart of the dense
[`mikecovlee/tinymistral-276m-it`](https://huggingface.co/mikecovlee/tinymistral-276m-it): the
**exact same 3M-tier SFT recipe** (same data, hyper-parameters and schedule) is applied to both
the MoE and the dense base, so the two instruction-tuned models can be compared at the same
active-parameter count and the same SFT budget.

## Model details

| Parameter | Value |
|-----------|-------|
| hidden_size | 1024 |
| num_layers | 16 |
| Attention | Grouped Query Attention (16 heads / 4 KV heads) |
| Head dim | 64 |
| RoPE theta | 1,000,000 |
| Norm | RMSNorm + per-head QK-Norm (pre-RoPE) |
| Experts | 4 routed (top-2), aux loss 1e-3 |
| Expert FFN | SwiGLU, intermediate = 2048 |
| Vocab size | 32,000 (tied embeddings) |
| Max position | 2,048 |
| **Total params** | **~477.5M** |
| **Active params** | **~276.1M** |

## Training

- **Initialization:** the public base `mikecovlee/tinymixtral` (v3.0).
- **Data:** 2,168,835 deduplicated and eval-decontaminated English instruction conversations,
  blended from 10 public sources (Tulu3, OpenHermes, SlimOrca, OpenOrca, UltraChat, MetaMath,
  OrcaMath, OpenMathInstruct-2, SQuAD2, TriviaQA).
- **Recipe:** 1 epoch, sequence packing to 1024 tokens, batch 24, bf16, AdamW, lr 2e-5
  (cosine, 100-step warmup), weight decay 0.1, seed 42.

## Evaluation

Measured with `lm-evaluation-harness` v0.4.12, 0-shot (cuda, bf16).

Harness = mean of the 7 primary metrics (hellaswag acc_norm, piqa acc, winogrande acc,
arc_easy acc, arc_challenge acc_norm, openbookqa acc_norm, lambada acc).
| Model | 7-task harness | 7-task all-acc | GSM8K strict / flex | IFEval prompt / inst |
|---|---|---|---|---|
| **`tinymixtral-it`** (MoE + 3M SFT) | **0.3994** | 0.3691 | 0.0182 / 0.0205 | 0.1756 / 0.2782 |
| `tinymixtral` (MoE base) | 0.3992 | — | 0.0000 / 0.0159 | — |
| `tinymistral-276m-it` (dense + 3M SFT) | 0.3892 | 0.3631 | 0.0174 / 0.0205 | 0.1460 / 0.2602 |
| `tinymistral-276m` (dense base) | 0.3904 | 0.3890 | 0.0000 / 0.0136 | — |
> **Harness note.** Means quoted here use the **7-task** harness (excludes BoolQ); the base
> v1.0/v3.0 cards report an **8-task** mean (includes BoolQ). The two are not directly comparable.
Per-task (`tinymixtral-it`): hellaswag 0.340, piqa 0.634, winogrande 0.528, arc_easy 0.448,
arc_challenge 0.260, openbookqa 0.302, lambada 0.289 — plus **BoolQ 0.426**, reported separately
(BoolQ is excluded from the 7-task mean above, to match the dense iso-active comparison).
Additional metric:
| Metric | Score |
|---|---|
| Open-ended answer quality (LLM rubric, 4,955 held-out prompts, 0–100) | 15.0 ± 0.3 |
Note: relative to the base model, instruction following and open-ended answer quality improve
substantially while multiple-choice common-sense accuracy drops (BoolQ is the most sensitive task).
Arithmetic remains far below practical use. The dense counterpart receives the same SFT recipe and
lands close on the 7-task harness (0.3892 vs 0.3994) but is less steerable (IFEval 0.1460 / 0.2602).
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "mikecovlee/tinymixtral-it"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype="bfloat16", device_map="auto")
messages = [{"role": "user", "content": "What is 12% of 250?"}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
```
Conversational use via the bundled chat template (`tokenizer.apply_chat_template`); greedy
decoding works well at this scale.
## Limitations
- Absolute numbers are bounded by the 477.5M MoE total / 276.1M active scale and the
  8.05B-token pretraining budget; arithmetic remains far below practical use.
- English-centric, no safety alignment.
- Multiple-choice common-sense accuracy drops slightly after SFT (BoolQ most sensitive).
## Family
- [`mikecovlee/tinymixtral`](https://huggingface.co/mikecovlee/tinymixtral) — 477.5M MoE base (v3.0 flagship)
- **`mikecovlee/tinymixtral-it`** — this model (MoE instruction-tuned, v3.0-it, 3M SFT)
- [`mikecovlee/tinymistral-276m`](https://huggingface.co/mikecovlee/tinymistral-276m) — 276M dense base (iso-active ablation)
- [`mikecovlee/tinymistral-276m-it`](https://huggingface.co/mikecovlee/tinymistral-276m-it) — 276M dense instruction-tuned (3M SFT)
- [`mikecovlee/tinymixtral-v1.1-1b`](https://huggingface.co/mikecovlee/tinymixtral-v1.1-1b) — 1B MoE (earlier flagship)
- [`mikecovlee/tinymixtral-v1.1-0.5b`](https://huggingface.co/mikecovlee/tinymixtral-v1.1-0.5b) — 0.5B-class MoE (data-quality ablation)
- [`mikecovlee/tinymixtral-v2.0-beta`](https://huggingface.co/mikecovlee/tinymixtral-v2.0-beta) — shared-expert experiment (beta)
- [`mikecovlee/tinymixtral-v1.0`](https://huggingface.co/mikecovlee/tinymixtral-v1.0) — legacy (C4)
> **Naming.** The MoE family is published under `tinymixtral`; the dense 276M iso-active
> ablation companions use the `tinymistral` spelling. Both belong to the same project.
## Citation
```bibtex
@misc{tinymixtralit2026,
  title  = {TinyMixtral: a small Mixture-of-Experts language-model family},
  author = {Michael Lee},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/mikecovlee/tinymixtral-it}}
}
```
## License
MIT (Copyright (C) 2026 Michael Lee).

	---
	license: mit
	library_name: transformers
	pipeline_tag: text-generation
	language: en
	tags:
	- tinymixtral
	- moe
	- mixture-of-experts
	- instruction-tuned
	- sft
	- text-generation
	- custom_code
	model_type: tinymixtral
	base_model: mikecovlee/tinymixtral
	datasets:
	- allenai/tulu-3-sft-mixture
	- teknium/OpenHermes-2.5
	- Open-Orca/SlimOrca
	- Open-Orca/OpenOrca
	- HuggingFaceH4/ultrachat_200k
	- meta-math/MetaMathQA
	- microsoft/orca-math-word-problems-200k
	- nvidia/OpenMathInstruct-2
	- rajpurkar/squad_v2
	- mandarjoshi/trivia_qa
	---

	# tinymixtral-it — instruction-tuned MoE

	`tinymixtral-it` (a.k.a. v3.0-it) is the instruction-tuned version of the MoE flagship
	[`mikecovlee/tinymixtral`](https://huggingface.co/mikecovlee/tinymixtral): 477.5M total / 276.1M
	active parameters (top-2 of 4 experts), 2048 context, 32k SentencePiece vocabulary.

	It is the MoE counterpart of the dense
	[`mikecovlee/tinymistral-276m-it`](https://huggingface.co/mikecovlee/tinymistral-276m-it): the
	exact same 3M-tier SFT recipe (same data, hyper-parameters and schedule) is applied to both
	the MoE and the dense base, so the two instruction-tuned models can be compared at the same
	active-parameter count and the same SFT budget.

	## Model details

	\| Parameter \| Value \|
	\|-----------\|-------\|
	\| hidden_size \| 1024 \|
	\| num_layers \| 16 \|
	\| Attention \| Grouped Query Attention (16 heads / 4 KV heads) \|
	\| Head dim \| 64 \|
	\| RoPE theta \| 1,000,000 \|
	\| Norm \| RMSNorm + per-head QK-Norm (pre-RoPE) \|
	\| Experts \| 4 routed (top-2), aux loss 1e-3 \|
	\| Expert FFN \| SwiGLU, intermediate = 2048 \|
	\| Vocab size \| 32,000 (tied embeddings) \|
	\| Max position \| 2,048 \|
	\| Total params \| ~477.5M \|
	\| Active params \| ~276.1M \|

	## Training

	- Initialization: the public base `mikecovlee/tinymixtral` (v3.0).
	- Data: 2,168,835 deduplicated and eval-decontaminated English instruction conversations,
	blended from 10 public sources (Tulu3, OpenHermes, SlimOrca, OpenOrca, UltraChat, MetaMath,
	OrcaMath, OpenMathInstruct-2, SQuAD2, TriviaQA).
	- Recipe: 1 epoch, sequence packing to 1024 tokens, batch 24, bf16, AdamW, lr 2e-5
	(cosine, 100-step warmup), weight decay 0.1, seed 42.

	## Evaluation

	Measured with `lm-evaluation-harness` v0.4.12, 0-shot (cuda, bf16).

	Harness = mean of the 7 primary metrics (hellaswag acc_norm, piqa acc, winogrande acc,
	arc_easy acc, arc_challenge acc_norm, openbookqa acc_norm, lambada acc).

	\| Model \| 7-task harness \| 7-task all-acc \| GSM8K strict / flex \| IFEval prompt / inst \|
	\|---\|---\|---\|---\|---\|
	\| `tinymixtral-it` (MoE + 3M SFT) \| 0.3994 \| 0.3691 \| 0.0182 / 0.0205 \| 0.1756 / 0.2782 \|
	\| `tinymixtral` (MoE base) \| 0.3992 \| — \| 0.0000 / 0.0159 \| — \|
	\| `tinymistral-276m-it` (dense + 3M SFT) \| 0.3892 \| 0.3631 \| 0.0174 / 0.0205 \| 0.1460 / 0.2602 \|
	\| `tinymistral-276m` (dense base) \| 0.3904 \| 0.3890 \| 0.0000 / 0.0136 \| — \|

	> Harness note. Means quoted here use the 7-task harness (excludes BoolQ); the base
	> v1.0/v3.0 cards report an 8-task mean (includes BoolQ). The two are not directly comparable.

	Per-task (`tinymixtral-it`): hellaswag 0.340, piqa 0.634, winogrande 0.528, arc_easy 0.448,
	arc_challenge 0.260, openbookqa 0.302, lambada 0.289 — plus BoolQ 0.426, reported separately
	(BoolQ is excluded from the 7-task mean above, to match the dense iso-active comparison).

	Additional metric:

	\| Metric \| Score \|
	\|---\|---\|
	\| Open-ended answer quality (LLM rubric, 4,955 held-out prompts, 0–100) \| 15.0 ± 0.3 \|

	Note: relative to the base model, instruction following and open-ended answer quality improve
	substantially while multiple-choice common-sense accuracy drops (BoolQ is the most sensitive task).
	Arithmetic remains far below practical use. The dense counterpart receives the same SFT recipe and
	lands close on the 7-task harness (0.3892 vs 0.3994) but is less steerable (IFEval 0.1460 / 0.2602).

	## Usage

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model_id = "mikecovlee/tinymixtral-it"
	tok = AutoTokenizer.from_pretrained(model_id)
	model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype="bfloat16", device_map="auto")

	messages = [{"role": "user", "content": "What is 12% of 250?"}]
	ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
	out = model.generate(ids, max_new_tokens=256, do_sample=False)
	print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
	```

	Conversational use via the bundled chat template (`tokenizer.apply_chat_template`); greedy
	decoding works well at this scale.

	## Limitations

	- Absolute numbers are bounded by the 477.5M MoE total / 276.1M active scale and the
	8.05B-token pretraining budget; arithmetic remains far below practical use.
	- English-centric, no safety alignment.
	- Multiple-choice common-sense accuracy drops slightly after SFT (BoolQ most sensitive).

	## Family

	- [`mikecovlee/tinymixtral`](https://huggingface.co/mikecovlee/tinymixtral) — 477.5M MoE base (v3.0 flagship)
	- `mikecovlee/tinymixtral-it` — this model (MoE instruction-tuned, v3.0-it, 3M SFT)
	- [`mikecovlee/tinymistral-276m`](https://huggingface.co/mikecovlee/tinymistral-276m) — 276M dense base (iso-active ablation)
	- [`mikecovlee/tinymistral-276m-it`](https://huggingface.co/mikecovlee/tinymistral-276m-it) — 276M dense instruction-tuned (3M SFT)
	- [`mikecovlee/tinymixtral-v1.1-1b`](https://huggingface.co/mikecovlee/tinymixtral-v1.1-1b) — 1B MoE (earlier flagship)
	- [`mikecovlee/tinymixtral-v1.1-0.5b`](https://huggingface.co/mikecovlee/tinymixtral-v1.1-0.5b) — 0.5B-class MoE (data-quality ablation)
	- [`mikecovlee/tinymixtral-v2.0-beta`](https://huggingface.co/mikecovlee/tinymixtral-v2.0-beta) — shared-expert experiment (beta)
	- [`mikecovlee/tinymixtral-v1.0`](https://huggingface.co/mikecovlee/tinymixtral-v1.0) — legacy (C4)

	> Naming. The MoE family is published under `tinymixtral`; the dense 276M iso-active
	> ablation companions use the `tinymistral` spelling. Both belong to the same project.

	## Citation

	```bibtex
	@misc{tinymixtralit2026,
	title = {TinyMixtral: a small Mixture-of-Experts language-model family},
	author = {Michael Lee},
	year = {2026},
	howpublished = {\url{https://huggingface.co/mikecovlee/tinymixtral-it}}
	}
	```

	## License

	MIT (Copyright (C) 2026 Michael Lee).