German–English base language model · 99.1M parameters · 20B pretraining tokens · up to 8K context
 > [!IMPORTANT] > **Hanse2-100M-Base is a base model, not a chat assistant.** It has not been instruction-tuned, preference-tuned, or safety-aligned. Use it as a text-completion model or as a starting point for continued pretraining and post-training. Hanse2-100M-Base is a **99,144,320-parameter decoder-only causal language model** trained from scratch on **19,999,752,192 German and English tokens**. It uses a compact Llama-style architecture, a custom 32K byte-level BPE tokenizer, grouped-query attention, and a three-stage pretraining curriculum that progressively extends context length from 2K to 8K. The model was trained on a **single AMD Radeon RX 9070 XT 16 GB**, and the repository includes the tokenizer-training, pretraining, and inference scripts used for the project. ## Highlights - **99.1M parameters**, trained from scratch - **19.999B pretraining tokens** (~201.7 tokens per parameter) - **German + English**, approximately 56% / 44% by planned token share - **2K → 4K → 8K** progressive context curriculum - **32K custom byte-level BPE tokenizer** - **Single-GPU training** on an AMD Radeon RX 9070 XT 16 GB - Reproducible project code for tokenizer training, pretraining, and inference - Apache-2.0 release ## Model overview | | | | --- | --- | | **Model type** | Decoder-only causal language model | | **Architecture** | Llama-style Transformer | | **Parameters** | 99,144,320 | | **Languages** | German, English | | **Vocabulary** | 32,000 | | **Training tokens** | 19,999,752,192 | | **Tokens / parameter** | ~201.7 | | **Configured context** | 8,192 tokens | | **Training precision** | bfloat16 | | **Position encoding** | RoPE, θ = 10,000 | | **Embeddings** | Tied input/output embeddings | | **Instruction tuned** | No | | **Safety aligned** | No | ### Architecture | Hyperparameter | Value | | --- | ---: | | Hidden size | 640 | | Layers | 16 | | Attention heads | 10 | | KV heads | 2 | | Head dimension | 64 | | Intermediate size | 2,048 | | Vocabulary size | 32,000 | | RoPE θ | 10,000 | | Tied embeddings | Yes | ## Pretraining Pretraining used three consecutive phases while keeping the effective optimizer batch at approximately **131,072 tokens per optimizer step**. | Phase | Purpose | Tokens | Context | Micro batch | Grad. accum. | Learning rate | | --- | --- | ---: | ---: | ---: | ---: | --- | | **1** | Broad bilingual pretraining | ~15B | 2,048 | 2 | 32 | 1% warmup → constant `6e-4` | | **2** | Quality-focused annealing | ~4B | 4,096 | 1 | 32 | 1% warmup, `3e-4` → `3e-5`, `1-sqrt` decay | | **3** | Cooldown + context extension | ~1B | 8,192 | 1 | 16 | cosine `3e-5` → `1e-5` |  ### Data mixture Approximate token share over the full curriculum: | Source | Share | | --- | ---: | | German FineWeb | 43.5% | | English FineWeb-Edu | 39.0% | | German FineWiki | 12.5% | | English FineWiki | 5.0% | This corresponds to approximately **56% German / 44% English**. ### Optimization | Setting | Value | | --- | --- | | Optimizer | AdamW | | β1 / β2 | 0.9 / 0.95 | | Weight decay | 0.1 | | Gradient clipping | 1.0 | | Effective batch | ~131,072 tokens / optimizer step | | Training precision | bfloat16 | | Framework | PyTorch + Transformers | | Attention | SDPA / AOTriton on ROCm | | `torch.compile` | Disabled | | Hardware | 1× AMD Radeon RX 9070 XT 16 GB | | Platform | Windows + ROCm | Final held-out evaluation loss after Phase 3 was approximately **2.825** on the fixed bilingual evaluation split. ### Tokenizer Hanse2 uses a custom **32,000-token byte-level BPE tokenizer** trained on a German–English mixture. Several tokens were reserved for later instruction/tool post-training. They were **not used as a chat format during base pretraining**, and this checkpoint should not be prompted with an assumed chat template. ## Evaluation The following model-comparison results were produced with the **EleutherAI LM Evaluation Harness** using the same local setup for Hanse2-100M and Supra-50M: - 0-shot - no chat template - bfloat16 - identical task implementations - identical random seeds | Benchmark | Metric | Hanse2-100M | Supra-50M | | --- | --- | ---: | ---: | | ARC Easy | `acc_norm` | 0.4306 | **0.4609** | | ARC Challenge | `acc_norm` | 0.2415 | **0.2534** | | HellaSwag | `acc_norm` | 0.3122 | **0.3171** | | WinoGrande | `acc` | 0.4980 | **0.5107** | | PIQA | `acc_norm` | 0.6083 | **0.6219** | | OpenBookQA | `acc_norm` | 0.2960 | **0.3080** | | BoolQ | `acc` | **0.5966** | 0.5294 | | SciQ | `acc_norm` | 0.6380 | **0.6770** | | BLiMP | `acc` | **0.8003** | 0.7778 | | MultiBLiMP German | `acc` | **0.9852** | 0.7698 | | MultiBLiMP German | `acc_norm` | **0.9774** | 0.6710 | Hanse2 remains close to Supra-50M on several English benchmarks while substantially improving the German linguistic evaluation. ### Long-context diagnostic The model is configured for **8,192 tokens** and produces finite forward passes at ~7.7K tokens. This does **not** imply reliable retrieval across the full context window. A small base-LM continuation test placed a one-token identifier earlier in a synthetic context and measured its next-token rank against nine decoys. | Context | Needle position | Accuracy | Mean rank | Mean margin | | ---: | ---: | ---: | ---: | ---: | | 512 | 10% | 1.00 | 1.00 | 10.08 | | 512 | 50% | 1.00 | 1.00 | 10.35 | | 512 | 90% | 1.00 | 1.00 | 9.31 | | 1,024 | 10% | 1.00 | 1.00 | 9.82 | | 1,024 | 50% | 1.00 | 1.00 | 10.68 | | 1,024 | 90% | 1.00 | 1.00 | 8.14 | | 2,048 | 10% | 1.00 | 1.00 | 8.11 | | 2,048 | 50% | 1.00 | 1.00 | 10.19 | | 2,048 | 90% | 1.00 | 1.00 | 11.40 | | 4,096 | 10% | 0.20 | 6.70 | -4.22 | | 4,096 | 50% | 1.00 | 1.00 | 10.34 | | 4,096 | 90% | 1.00 | 1.00 | 11.35 | | 7,680 | 10% | 0.00 | 5.90 | -2.58 | | 7,680 | 50% | 0.00 | 5.40 | -3.46 | | 7,680 | 90% | 1.00 | 1.00 | 11.13 | The model can process long inputs, but retrieval degrades when relevant information is several thousand tokens away from the prediction point. Treat this as a **small synthetic diagnostic**, not a standardized long-context benchmark. ## Quickstart ### Transformers ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "Evicka/Hanse2-100M-Base" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", ) model.eval() prompt = "Artificial intelligence is" inputs = tokenizer( prompt, return_tensors="pt", return_token_type_ids=False, ).to(model.device) with torch.inference_mode(): output = model.generate( **inputs, max_new_tokens=150, do_sample=True, temperature=0.5, top_k=25, top_p=0.9, repetition_penalty=1.2, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, ) print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` > [!NOTE] > This is a completion model. Prompts such as sentence beginnings, paragraphs, or documents to continue are more appropriate than chat-style `system/user/assistant` messages. ## Reproducibility The repository includes the core scripts needed to reproduce or adapt the project: | File | Purpose | | --- | --- | | `train_tokenizer.py` | Train the custom byte-level BPE tokenizer | | `train.py` | Run the three-phase pretraining pipeline | | `inference.py` | Minimal local text generation | The training pipeline includes exact token-budget accounting, resumable checkpoints, deterministic held-out evaluation, and separate context/batch settings for each phase. The initial training/tokenizer code was based on the Apache-2.0-licensed **Supra-50M** training scripts and was substantially rewritten and extended for Hanse2. **No Supra model weights were used to initialize Hanse2.** > Re-running the scripts does not by itself guarantee bit-identical reproduction across different hardware, driver, PyTorch, ROCm, dataset revisions, or nondeterministic kernels. ## Intended use Hanse2-100M-Base is intended primarily for: - research on small language models - German–English pretraining experiments - bilingual data-mixture studies - tokenizer and architecture experiments - progressive context-extension experiments - benchmarking and ablations - continued pretraining - supervised fine-tuning / post-training research - education and consumer-hardware experimentation ## Limitations At ~99M parameters, Hanse2 is a small research model. Expect: - weak factual recall and hallucinations - weak arithmetic and multi-step reasoning - repetition or topic drift - brittle knowledge of specific entities - limited long-range coherence and retrieval - unstable behavior outside its training distribution The checkpoint has **no instruction tuning, preference tuning, RLHF, refusal training, or production safety guardrails**. It may generate inaccurate, biased, offensive, harmful, or otherwise undesirable continuations. The configured 8K context length should not be interpreted as reliable 8K retrieval capability; see the long-context diagnostic above. ## License Released under the **Apache License 2.0**. See [`LICENSE`](LICENSE). ## Acknowledgements - [Hugging Face](https://huggingface.co/) — Transformers, Datasets, Tokenizers, FineWeb, FineWeb-Edu, and FineWiki - [EleutherAI](https://www.eleuther.ai/) — LM Evaluation Harness - [SupraLabs](https://huggingface.co/SupraLabs) — original Apache-2.0 Supra-50M training/tokenizer scripts used as the starting point for the project code ## Citation If you use Hanse2-100M-Base in a project, a link to the model repository is appreciated. ```bibtex @misc{brauer2026hanse2base100m, author = {Erik Brauer}, title = {Hanse2-100M-Base}, year = {2026}, url = {https://huggingface.co/Evicka/Hanse2-100M-Base} } ```