shibatch/tinygemma3-2m

TinyStories Gemma3 2M (HF + Q4_K_M GGUF)

This repository contains a tiny Gemma 3 text-only model trained with official Hugging Face Transformers classes, together with a llama.cpp-compatible Q4_K_M GGUF conversion.

The model has 1,560,064 parameters (the 2m name is an approximate size label). It is intended for inference-engine validation and small-model experiments, not for production language quality.

The Hugging Face artifact was downloaded from shibatch/tinygemma3-2m at revision 7aab2f4e525707e046799eb5674f344dc74853d6.

Repository contents

• hf/: original Hugging Face model and tokenizer
• gguf/tinygemma3-2m-Q4_K_M.gguf: ready-to-run Q4_K_M GGUF
• convert_to_gguf.py: reproducible HF -> F16 -> Q4_K_M conversion wrapper
• conversion_metadata.json: source revisions, quantization details, and validation result
• reference/: original deterministic Hugging Face reference tensors
• gemma3_text_config_dump.json: normalized configuration dump
• safetensors_keys.json: source tensor names and shapes
• artifact_metadata.json: original training metadata
• SHA256SUMS: checksums for the distributed binary artifacts

GGUF quick start

Install a recent llama.cpp build that supports Gemma 3. On Debian systems with the llama.cpp tools package installed, run:

llama-completion \
  -m gguf/tinygemma3-2m-Q4_K_M.gguf \
  -p "Once upon" \
  -n 100 \
  --temp 0.8 \
  --top-p 0.95 \
  --seed 1234

The GGUF was load- and generation-tested with Debian llama.cpp build 8681. For a deterministic smoke test:

llama-completion \
  -m gguf/tinygemma3-2m-Q4_K_M.gguf \
  -p "Once upon" \
  -n 40 \
  --temp 0 \
  --seed 1234 \
  --no-display-prompt

One validated completion was:

 a time, there was a little girl named Lily. She loved to play with her toys
 and her favorite thing to do was to go to the park. One day, Lily's

Q4_K_M details

The distributed GGUF reports general.file_type = 15 (Q4_K_M) and contains 80 tensors. Because this deliberately tiny architecture has a hidden width of 128 while K-quants use 256-value blocks, llama.cpp applies its normal Q4_K_M fallback rules to tensors that cannot use Q4_K directly:

Tensor type	Count
F32	37
Q5_0	34
Q4_K	4
Q8_0	3
Q6_K	2

This mixed tensor layout is the expected llama.cpp representation of the Q4_K_M preset for this model shape. The file is about 1.1 MB, compared with about 3.1 MB for the intermediate F16 GGUF.

Reproduce the GGUF

Requirements:

• a recent llama.cpp source checkout containing convert_hf_to_gguf.py
• llama-quantize on PATH, or its path passed explicitly
• the Python packages required by llama.cpp's converter

Run from this repository:

python convert_to_gguf.py \
  --llama-cpp /path/to/llama.cpp \
  --quantizer /path/to/llama-quantize

The wrapper handles two properties of this validation checkpoint that generic conversion currently does not infer correctly:

1. The custom tokenizer is a GPT-2-style ByteLevel BPE whose probe hash is not in llama.cpp's generated pre-tokenizer table.
2. The tokenizer has 1,003 entries, but the model intentionally pads its embedding and logits vocabulary to 1,024 rows. Those rows must be retained for Gemma 3 runtime shape checks and for compatibility with the reference logits.

The script validates the tokenizer structure before applying the tokenizer override. It creates the F16 intermediate inside gguf/ and removes it after Q4_K_M quantization, including when quantization fails.

Hugging Face usage

from pathlib import Path

import torch
from transformers import Gemma3ForCausalLM, PreTrainedTokenizerFast

model_dir = Path("hf")
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_dir)
model = Gemma3ForCausalLM.from_pretrained(
    model_dir,
    dtype=torch.float32,
).eval()

prompt = "Once upon"
input_ids = torch.tensor(
    [[tokenizer.bos_token_id] + tokenizer.encode(prompt, add_special_tokens=False)]
)

with torch.no_grad():
    output = model.generate(
        input_ids,
        max_new_tokens=100,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

The same example is available as example_generate.py.

Official classes used

• Gemma3TextConfig
• Gemma3ForCausalLM
• Trainer

No custom Gemma 3 modeling code is used by the original HF artifact.

Architecture

model_type: gemma3_text
architecture: Gemma3ForCausalLM
parameter_count: 1,560,064
vocab_size: 1,024
tokenizer_entries: 1,003
hidden_size: 128
intermediate_size: 512
num_hidden_layers: 6
num_attention_heads: 4
num_key_value_heads: 1
head_dim: 32
max_position_embeddings: 256
sliding_window: 32
layer_types:
  - sliding_attention
  - sliding_attention
  - sliding_attention
  - sliding_attention
  - sliding_attention
  - full_attention
tie_word_embeddings: true

The model exercises local/global attention, sliding-window attention, GQA, per-head q_norm / k_norm, Gemma 3's four-norm decoder structure, a gated MLP, and a tied output head.

Limitations

This is a synthetic tiny checkpoint. It is not an official Google model and does not contain weights from an original Gemma checkpoint. It is not intended for instruction following, chat, factual recall, safety-critical use, or production deployment. Quantized output may differ from the float32 Hugging Face checkpoint.