README.md

---
license: apache-2.0
language:
  - de
  - en
library_name: transformers
pipeline_tag: text-generation
datasets:
  - HuggingFaceFW/fineweb-2
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceFW/finewiki
tags:
  - llama
  - causal-lm
  - base-model
  - pretrained
  - from-scratch
  - german
  - english
  - bilingual
  - small-language-model
---

<h1 align="center">Hanse2-100M Base</h1>

<p align="center">
  German–English base language model · 99.1M parameters · 20B pretraining tokens · up to 8K context
</p>

![Hanse2-100M Base](assets/modelcard.jpg)

> [!IMPORTANT]
> **Hanse2-100M-Base is a base model, not a chat assistant.** It has not been instruction-tuned, preference-tuned, or safety-aligned. Use it as a text-completion model or as a starting point for continued pretraining and post-training.

Hanse2-100M-Base is a **99,144,320-parameter decoder-only causal language model** trained from scratch on **19,999,752,192 German and English tokens**. It uses a compact Llama-style architecture, a custom 32K byte-level BPE tokenizer, grouped-query attention, and a three-stage pretraining curriculum that progressively extends context length from 2K to 8K.

The model was trained on a **single AMD Radeon RX 9070 XT 16 GB**, and the repository includes the tokenizer-training, pretraining, and inference scripts used for the project.

## Highlights

- **99.1M parameters**, trained from scratch
- **19.999B pretraining tokens** (~201.7 tokens per parameter)
- **German + English**, approximately 56% / 44% by planned token share
- **2K → 4K → 8K** progressive context curriculum
- **32K custom byte-level BPE tokenizer**
- **Single-GPU training** on an AMD Radeon RX 9070 XT 16 GB
- Reproducible project code for tokenizer training, pretraining, and inference
- Apache-2.0 release

## Model overview

| | |
| --- | --- |
| **Model type** | Decoder-only causal language model |
| **Architecture** | Llama-style Transformer |
| **Parameters** | 99,144,320 |
| **Languages** | German, English |
| **Vocabulary** | 32,000 |
| **Training tokens** | 19,999,752,192 |
| **Tokens / parameter** | ~201.7 |
| **Configured context** | 8,192 tokens |
| **Training precision** | bfloat16 |
| **Position encoding** | RoPE, θ = 10,000 |
| **Embeddings** | Tied input/output embeddings |
| **Instruction tuned** | No |
| **Safety aligned** | No |

### Architecture

| Hyperparameter | Value |
| --- | ---: |
| Hidden size | 640 |
| Layers | 16 |
| Attention heads | 10 |
| KV heads | 2 |
| Head dimension | 64 |
| Intermediate size | 2,048 |
| Vocabulary size | 32,000 |
| RoPE θ | 10,000 |
| Tied embeddings | Yes |

## Pretraining

Pretraining used three consecutive phases while keeping the effective optimizer batch at approximately **131,072 tokens per optimizer step**.

| Phase | Purpose | Tokens | Context | Micro batch | Grad. accum. | Learning rate |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| **1** | Broad bilingual pretraining | ~15B | 2,048 | 2 | 32 | 1% warmup → constant `6e-4` |
| **2** | Quality-focused annealing | ~4B | 4,096 | 1 | 32 | 1% warmup, `3e-4` → `3e-5`, `1-sqrt` decay |
| **3** | Cooldown + context extension | ~1B | 8,192 | 1 | 16 | cosine `3e-5` → `1e-5` |

![Hanse2 pretraining curriculum](assets/training_curriculum.png)

### Data mixture

Approximate token share over the full curriculum:

| Source | Share |
| --- | ---: |
| German FineWeb | 43.5% |
| English FineWeb-Edu | 39.0% |
| German FineWiki | 12.5% |
| English FineWiki | 5.0% |

This corresponds to approximately **56% German / 44% English**.

### Optimization

| Setting | Value |
| --- | --- |
| Optimizer | AdamW |
| β1 / β2 | 0.9 / 0.95 |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Effective batch | ~131,072 tokens / optimizer step |
| Training precision | bfloat16 |
| Framework | PyTorch + Transformers |
| Attention | SDPA / AOTriton on ROCm |
| `torch.compile` | Disabled |
| Hardware | 1× AMD Radeon RX 9070 XT 16 GB |
| Platform | Windows + ROCm |

Final held-out evaluation loss after Phase 3 was approximately **2.825** on the fixed bilingual evaluation split.

### Tokenizer

Hanse2 uses a custom **32,000-token byte-level BPE tokenizer** trained on a German–English mixture.

Several tokens were reserved for later instruction/tool post-training. They were **not used as a chat format during base pretraining**, and this checkpoint should not be prompted with an assumed chat template.

## Evaluation

The following model-comparison results were produced with the **EleutherAI LM Evaluation Harness** using the same local setup for Hanse2-100M and Supra-50M:

- 0-shot
- no chat template
- bfloat16
- identical task implementations
- identical random seeds

| Benchmark | Metric | Hanse2-100M | Supra-50M |
| --- | --- | ---: | ---: |
| ARC Easy | `acc_norm` | 0.4306 | **0.4609** |
| ARC Challenge | `acc_norm` | 0.2415 | **0.2534** |
| HellaSwag | `acc_norm` | 0.3122 | **0.3171** |
| WinoGrande | `acc` | 0.4980 | **0.5107** |
| PIQA | `acc_norm` | 0.6083 | **0.6219** |
| OpenBookQA | `acc_norm` | 0.2960 | **0.3080** |
| BoolQ | `acc` | **0.5966** | 0.5294 |
| SciQ | `acc_norm` | 0.6380 | **0.6770** |
| BLiMP | `acc` | **0.8003** | 0.7778 |
| MultiBLiMP German | `acc` | **0.9852** | 0.7698 |
| MultiBLiMP German | `acc_norm` | **0.9774** | 0.6710 |

Hanse2 remains close to Supra-50M on several English benchmarks while substantially improving the German linguistic evaluation.

### Long-context diagnostic

The model is configured for **8,192 tokens** and produces finite forward passes at ~7.7K tokens. This does **not** imply reliable retrieval across the full context window.

A small base-LM continuation test placed a one-token identifier earlier in a synthetic context and measured its next-token rank against nine decoys.

| Context | Needle position | Accuracy | Mean rank | Mean margin |
| ---: | ---: | ---: | ---: | ---: |
| 512 | 10% | 1.00 | 1.00 | 10.08 |
| 512 | 50% | 1.00 | 1.00 | 10.35 |
| 512 | 90% | 1.00 | 1.00 | 9.31 |
| 1,024 | 10% | 1.00 | 1.00 | 9.82 |
| 1,024 | 50% | 1.00 | 1.00 | 10.68 |
| 1,024 | 90% | 1.00 | 1.00 | 8.14 |
| 2,048 | 10% | 1.00 | 1.00 | 8.11 |
| 2,048 | 50% | 1.00 | 1.00 | 10.19 |
| 2,048 | 90% | 1.00 | 1.00 | 11.40 |
| 4,096 | 10% | 0.20 | 6.70 | -4.22 |
| 4,096 | 50% | 1.00 | 1.00 | 10.34 |
| 4,096 | 90% | 1.00 | 1.00 | 11.35 |
| 7,680 | 10% | 0.00 | 5.90 | -2.58 |
| 7,680 | 50% | 0.00 | 5.40 | -3.46 |
| 7,680 | 90% | 1.00 | 1.00 | 11.13 |

The model can process long inputs, but retrieval degrades when relevant information is several thousand tokens away from the prediction point. Treat this as a **small synthetic diagnostic**, not a standardized long-context benchmark.

## Quickstart

### Transformers

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Evicka/Hanse2-100M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "Artificial intelligence is"
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    return_token_type_ids=False,
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=150,
        do_sample=True,
        temperature=0.5,
        top_k=25,
        top_p=0.9,
        repetition_penalty=1.2,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

> [!NOTE]
> This is a completion model. Prompts such as sentence beginnings, paragraphs, or documents to continue are more appropriate than chat-style `system/user/assistant` messages.

## Reproducibility

The repository includes the core scripts needed to reproduce or adapt the project:

| File | Purpose |
| --- | --- |
| `train_tokenizer.py` | Train the custom byte-level BPE tokenizer |
| `train.py` | Run the three-phase pretraining pipeline |
| `inference.py` | Minimal local text generation |

The training pipeline includes exact token-budget accounting, resumable checkpoints, deterministic held-out evaluation, and separate context/batch settings for each phase.

The initial training/tokenizer code was based on the Apache-2.0-licensed **Supra-50M** training scripts and was substantially rewritten and extended for Hanse2. **No Supra model weights were used to initialize Hanse2.**

> Re-running the scripts does not by itself guarantee bit-identical reproduction across different hardware, driver, PyTorch, ROCm, dataset revisions, or nondeterministic kernels.

## Intended use

Hanse2-100M-Base is intended primarily for:

- research on small language models
- German–English pretraining experiments
- bilingual data-mixture studies
- tokenizer and architecture experiments
- progressive context-extension experiments
- benchmarking and ablations
- continued pretraining
- supervised fine-tuning / post-training research
- education and consumer-hardware experimentation

## Limitations

At ~99M parameters, Hanse2 is a small research model. Expect:

- weak factual recall and hallucinations
- weak arithmetic and multi-step reasoning
- repetition or topic drift
- brittle knowledge of specific entities
- limited long-range coherence and retrieval
- unstable behavior outside its training distribution

The checkpoint has **no instruction tuning, preference tuning, RLHF, refusal training, or production safety guardrails**. It may generate inaccurate, biased, offensive, harmful, or otherwise undesirable continuations.

The configured 8K context length should not be interpreted as reliable 8K retrieval capability; see the long-context diagnostic above.

## License

Released under the **Apache License 2.0**. See [`LICENSE`](LICENSE).

## Acknowledgements

- [Hugging Face](https://huggingface.co/) — Transformers, Datasets, Tokenizers, FineWeb, FineWeb-Edu, and FineWiki
- [EleutherAI](https://www.eleuther.ai/) — LM Evaluation Harness
- [SupraLabs](https://huggingface.co/SupraLabs) — original Apache-2.0 Supra-50M training/tokenizer scripts used as the starting point for the project code

## Citation

If you use Hanse2-100M-Base in a project, a link to the model repository is appreciated.

```bibtex
@misc{brauer2026hanse2base100m,
  author = {Erik Brauer},
  title  = {Hanse2-100M-Base},
  year   = {2026},
  url    = {https://huggingface.co/Evicka/Hanse2-100M-Base}
}
```