Instructions to use mikecovlee/tinymixtral-v2.0-beta with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mikecovlee/tinymixtral-v2.0-beta with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mikecovlee/tinymixtral-v2.0-beta", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("mikecovlee/tinymixtral-v2.0-beta", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mikecovlee/tinymixtral-v2.0-beta with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mikecovlee/tinymixtral-v2.0-beta"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mikecovlee/tinymixtral-v2.0-beta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mikecovlee/tinymixtral-v2.0-beta

SGLang

How to use mikecovlee/tinymixtral-v2.0-beta with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mikecovlee/tinymixtral-v2.0-beta" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mikecovlee/tinymixtral-v2.0-beta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mikecovlee/tinymixtral-v2.0-beta" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mikecovlee/tinymixtral-v2.0-beta",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mikecovlee/tinymixtral-v2.0-beta with Docker Model Runner:
```
docker model run hf.co/mikecovlee/tinymixtral-v2.0-beta
```

TinyMixtral v2

A 498M-parameter Mixture-of-Experts language model with a DeepSeek-style shared expert, trained from scratch on 4B tokens (FineWeb-Edu + Cosmopedia v2) and post-trained on 1B tokens (Wikipedia + Cosmopedia v2).

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mikecovlee/tinymixtral", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mikecovlee/tinymixtral")

text = "The capital of France is"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Architecture

Parameter	Value
hidden_size	896
num_layers	10
Attention	GQA (16 heads / 4 KV heads, head_dim=56)
Experts	1 shared (always active) + 6 routed (top-2)
Expert FFN	SwiGLU, intermediate = 2389
Vocab size	32,000 (TinyLlama SentencePiece)
Max position	2,048
Total params	~498M
Active params/token	~241M

The shared expert provides general-purpose features and is always computed. Routed experts specialize via top-2 gating with load-balancing auxiliary loss.

Training

Phase	Data	Tokens	LR	Schedule	Time
Pretrain	FineWeb-Edu (89%) + Cosmopedia v2 (11%)	4B	7e-4	WSD	~96h
Post-train	Wikipedia (50%) + Cosmopedia v2 (50%)	1B	2e-5	WSD	24h

Hardware: single NVIDIA RTX A5000 24GB
Precision: bf16 (model + autocast)
Optimizer: AdamW (β=0.9,0.95, wd=0.1)
Batch: 22 × 1024 = 22,528 tokens/step
Gradient clipping: 1.0
Activation checkpointing: enabled

Results

Standard Benchmarks (lm-evaluation-harness, 0-shot)

Task	Metric	Score
HellaSwag	acc_norm	0.326
PIQA	acc	0.631
WinoGrande	acc	0.506
ARC-Easy	acc	0.474
ARC-Challenge	acc_norm	0.272
OpenBookQA	acc_norm	0.290
BoolQ	acc	0.455
LAMBADA	acc	0.224

Performance is comparable to SmolLM-360M on PIQA, ARC-C, and WinoGrande. HellaSwag and LAMBADA are weaker, reflecting limited next-token prediction capability at this scale.

GLUE (zero-shot, conditional log-likelihood)

Task	Score
SST2	0.586
MRPC (F1)	0.809
QQP (F1)	0.519
QNLI	0.458
RTE	0.520
MNLI	0.348
MNLI-mm	0.368
Mean	0.515

Comparison with baseline (v1.1, 432M / 176M active)

Metric	v1.1	v2 (this model)
GLUE Mean	0.513	0.515
ARC-C 0-shot	0.249	0.264
ARC-E 0-shot	0.365	0.388
SST2	0.568	0.586

v2 improves ARC (+1.5–2.3pp) and SST2 (+1.8pp) over v1.1, with GLUE mean roughly equivalent. The shared expert design increases knowledge capacity at the cost of slightly more parameters.

Limitations

At ~241M active parameters, performance on multi-step reasoning tasks (QNLI, MNLI, RTE) remains near random
Trained on English data only
Not instruction-tuned; best used as a base model or for text completion
2048 token context window

Citation

@misc{tinymixtral2026,
  author = {Michael Lee},
  title = {TinyMixtral: Small-scale MoE Language Model Research},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/mikecovlee/tinymixtral}}
}

License

Downloads last month: 21

Datasets used to train mikecovlee/tinymixtral-v2.0-beta

Collection including mikecovlee/tinymixtral-v2.0-beta

Tiny Mixtral

Collection

A small Mixtral-style Mixture-of-Experts causal language model for pretraining research on a single consumer GPU. • 3 items • Updated 3 days ago