| ---
|
| language:
|
| - en
|
| license: apache-2.0
|
| library_name: transformers
|
| pipeline_tag: text-generation
|
| tags:
|
| - causal-lm
|
| - base-model
|
| - small-language-model
|
| - custom_code
|
| - muon
|
| - hummingbird
|
| - hummingbird-v2
|
| datasets:
|
| - HuggingFaceFW/fineweb-edu
|
| - HuggingFaceTB/smollm-corpus
|
| - HuggingFaceTB/finemath
|
| - mlfoundations/dclm-baseline-1.0
|
| - allenai/dolma3_dolmino_mix-100B-1025
|
| - allenai/dolma3_dolmino_mix-10B-1025
|
| - HuggingFaceFW/finepdfs-edu
|
| --- |
| |
|  |
|
|
| # Hummingbird-V2 |
|
|
| Hummingbird-V2 is a 9.6M-parameter English base language model trained from scratch on |
| 10 billion tokens. It is designed for text completion and continuation scoring. |
|
|
| ## Model series
|
|
|
| Hummingbird-V2 follows [Hummingbird-V1](https://huggingface.co/juinron/Hummingbird-V1) |
| in the same small language model series. V2 was trained from a fresh initialization. |
|
|
| ## Architecture
|
|
|
| | Property | Value | |
| |---|---:| |
| | Parameters | 9,592,720 | |
| | Layers / hidden size | 14 / 240 | |
| | Vocabulary | 4,096-token digit-aware byte-level BPE | |
| | Maximum context | 2,048 tokens | |
| | Architecture | Decoder-only Transformer with grouped-query attention and SwiGLU | |
|
|
| ## Training and data |
|
|
| The model was trained on this balanced mix: |
|
|
| | Source | Share | |
| |---|---:| |
| | FineWeb-Edu sources | 55% | |
| | Cosmopedia v2 | 15% | |
| | FineMath 4+ | 10% | |
| | DCLM baseline | 10% | |
| | Dolma 3 science, question answering, and code | 7% | |
| | FinePDFs-Edu | 3% | |
|
|
| Training used Muon and AdamW with a 512-token context. See |
| [training data](TRAINING_DATA.md) for source details and |
| [training provenance](training/provenance.json) for the full recipe. |
|
|
| ## Zero-shot evaluation
|
|
|
| Author-run zero-shot results for the 10B-token checkpoint. Task scores are normalized |
| continuation accuracy in percent; the Intelligence Index is a chance-normalized composite. |
|
|
| | Benchmark | Score |
|
| |---|---:|
|
| | HellaSwag | 27.63 |
|
| | ARC-Easy | 39.39 |
|
| | ARC-Challenge | 21.16 |
|
| | PIQA | 57.40 |
|
| | ArithMark-3 | 36.00 |
|
| | Chance-normalized Intelligence Index | **9.556** |
|
|
|
| These evaluations helped select the released checkpoint; results have not been |
| independently verified. |
|
|
| ## Use
|
|
|
| Load `juinron/Hummingbird-V2` with Transformers and the packaged custom code:
|
|
|
| ```python
|
| import torch
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
| model_id = "juinron/Hummingbird-V2"
|
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
|
| model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()
|
|
|
| inputs = tokenizer("The color of the sky is", return_tensors="pt")
|
| with torch.inference_mode():
|
| output = model.generate(**inputs, max_new_tokens=32, do_sample=False, use_cache=False)
|
| print(tokenizer.decode(output[0], skip_special_tokens=True))
|
| ```
|
|
|
| `trust_remote_code=True` loads the packaged model code. Review it before use. |
|
|
| ## Limitations
|
|
|
| Hummingbird-V2 is small and English-focused. It can make factual or reasoning errors and |
| is not instruction-tuned or safety-aligned. Do not rely on it for consequential decisions. |
|
|
| ## License |
|
|
| The model package is released under [Apache-2.0](LICENSE). Third-party dataset notices |
| are included in [NOTICE](NOTICE) and [TRAINING_DATA.md](TRAINING_DATA.md). |
|
|
|
|
