README.md

---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- teutonic
- cortexlm
- chat
- reasoning
- hybrid-attention
- fp8
- long-context
base_model:
- Qwen/Qwen3.5-9B
---

> ⚠️ **Benchmarks:** Benchmarks on this page may differ from other evals; they are **not fully verified** and may change.
>
> ⚠️ **Preview only:** **Training is in progress**; uploaded checkpoints are **preview only**, **not for production**.

# Teutonic-1-Chat-Preview

**CortexLM Teutonic-1-Chat-Preview** is a hybrid linear+softmax ~9B-class chat model with thinking-style reasoning, trained for long-context conversation and verifiable math/code problem solving.

> **Preview — still under active training.** Expect weight updates, better alignment, and longer reliable generations. This card is a snapshot, not a final release.

## Highlights

- Hybrid architecture (linear attention + softmax attention) in the Qwen3.5 text lineage
- Default weights: **FP8** (block-scaled `e4m3`, Qwen-style `quantization_config`) for efficient inference
- RoPE context capacity up to **256k** input tokens
- **Default checkpoint already shows strong long-context retrieval** in internal needle smokes (**1.0** single-needle @**128k** and @**256k**), alongside strong verifiable-math behavior
- Chat template with optional `<think>…</think>` reasoning blocks and tool-calling tags
- Trained with continued long-context mid-training, supervised chat+thinking+tools, and RL on verifiable rewards

## What’s done vs what’s still cooking

| Done in this preview | Still cooking |
| --- | --- |
| Long-context continued pretraining (up to 64k packs) | More preference / alignment optimization |
| Supervised chat, thinking, and tools | Broader RL beyond math/code verifiers |
| RL on verifiable math (and code-oriented) rewards | Reliable **64k** free-form generation (in progress) |
| Long-output continuation (16k→32k response budgets) | Multimodal later |
| **256k RoPE** on default weights + strong single-needle @128k/256k (internal smokes) | Harder multikey / multi-hop long-ctx retrieval |
| Chat + math quality retained on the shipped snapshot | Further alignment / preference passes |

**Practical limits today:** the **default FP8 weights** are the chat + math snapshot (strong internal math + single-needle @128k/256k). A later **pure long-context co-train** that pushed harder on 256k packs improved needle-style retrieval but **collapsed thinking/math** — that experiment is **not shipped** here and will not be promoted without math recovery. Generation defaults target long answers (up to 64k in `generation_config`), but **stable 64k-out training is still underway**.

## How it was trained (high level)

1. **Mid-training** — continued pretraining with a long-context curriculum (8k → 32k → 64k packs) to extend useful context under FP8 training.
2. **Supervised chat fine-tune** — chat, thinking traces, and tool-use style data.
3. **Reinforcement learning** — policy optimization against verifiable math/code rewards (boxed-answer style grading).
4. **Long-context / long-output continuation** — further supervised + RL passes to keep needle-style long-context behavior and raise response-length budgets.

### Training charts (real logged metrics)

![Training overview](assets/training_overview.png)

*Overview of mid-training, supervised chat fine-tune, RL verifiable rewards, and long-output continuation.*

![Mid-training loss](assets/midtrain_loss.png)

*Mid-training train loss (smoothed) with held-out val points across the long-context curriculum.*

![Supervised fine-tune loss](assets/sft_loss.png)

*Supervised chat+thinking fine-tune loss, plus a later long-context SFT continuation.*

![RL verifiable reward](assets/rlvr_reward.png)

*RL mean verifiable reward and response clip ratio; lower panel shows long-output continuation runs.*

## Quick start

### Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "CortexLM/Teutonic-1-Chat-Preview"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype="auto",  # loads FP8 weights via quantization_config
)

messages = [
    {"role": "system", "content": "You are Teutonic, a helpful reasoning assistant."},
    {"role": "user", "content": "Solve: 17 * 19. Put the final answer in \\boxed{}."},
]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,  # if supported by the template path you use
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=4096,
    temperature=0.6,
    top_p=0.95,
    do_sample=True,
)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=False))
```

Thinking content appears inside `<think>…</think>` when the model opts to reason before the final answer. Turns end with `<|im_end|>`.

### vLLM

```bash
vllm serve CortexLM/Teutonic-1-Chat-Preview \
  --trust-remote-code \
  --dtype auto \
  --max-model-len 32768
```

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="CortexLM/Teutonic-1-Chat-Preview",
    messages=[{"role": "user", "content": "Write a short haiku about glaciers."}],
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

### Recommended sampling

| Mode | temperature | top_p | notes |
| --- | ---: | ---: | --- |
| Reasoning / math | 0.5–0.7 | 0.90–0.95 | Prefer `\boxed{}` for graded answers |
| Chat / creative | 0.7–0.9 | 0.90–0.95 | — |
| Deterministic smoke | 0.0 | — | `do_sample=False` |

## Context & generation

- **Input capacity (config):** up to **256k** tokens (RoPE `max_position_embeddings=262144`)
- **Long-context retrieval (default weights, internal smokes):** single-needle pass rate **1.0** at **128k** and **256k**
- **Generation target:** moving toward **64k** free-form outputs; this preview is strongest at moderate-to-long answers (multi-k tokens). Treat 64k-out as aspirational until a later drop.

## Long-context experiments (not shipped)

We also ran a **pure 256k-input continued-training** co-train after the chat/RL path (128k → 256k packs). That path kept strong needle retrieval but **broke thinking/math** (internal private math collapsed; model often never closed `</think>` even at large think budgets). **Those weights are not published** and are **not** candidates for promotion without a clear math recovery (≥ strong chat/math gate).

| Capability | Default weights (this repo) | Pure LC co-train (internal only) |
| --- | --- | --- |
| Single-needle @128k / @256k | **1.0** (internal smokes) | Strong (also ~1.0) |
| Multi-key needle @128k+ | Still cooking / uneven | Weak–mixed |
| Verifiable math (internal private probe) | **Strong** (~chat/RL quality) | **Regressed hard** — do not ship |
| 64k free-form **output** training | Still **in progress** | Orthogonal track |

**Default weights on this repo** remain the stronger chat + math + long-ctx retrieval snapshot (FP8). We will only publish a dedicated “harder long-ctx” weight drop if math stays ≥ the promotion gate after any further long-context work.

## Evaluation honesty

Internal math/reasoning probes improved substantially after verifiable-reward RL relative to the supervised merge. We **do not** claim contaminated public-bench SOTA (e.g. treat GSM8K-style numbers with caution). Prefer private or carefully decontaminated suites when comparing. Behavioural chat smokes (format, tools, basic instruction following) pass on this snapshot; broader alignment and preference quality are still in progress.

## Limitations

- Preview checkpoint — APIs and weights will change
- Long free-form generation beyond ~32k tokens is not fully hardened
- Not fully preference-aligned; may over-think or under-refuse
- Hybrid kernels need recent `transformers` / `vLLM` with `trust_remote_code`
- FP8 inference quality tracks the Qwen-style block FP8 path; if your stack lacks FP8 support, request a BF16 variant or dequantize

## License & acknowledgements

Released under **Apache 2.0**.

This model continues the **Qwen3.5** text architecture lineage (hybrid Teutonic conversion). Please respect the Qwen / Alibaba base model terms and citations where applicable:

- [Qwen](https://huggingface.co/Qwen)
- CortexLM / Dendrite training stack

## Citation

```bibtex
@misc{teutonic1chatpreview2026,
  title        = {Teutonic-1-Chat-Preview},
  author       = {CortexLM},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/CortexLM/Teutonic-1-Chat-Preview}},
  note         = {Preview checkpoint; under active training}
}
```




## Benchmarks

See [`benchmarks.md`](./benchmarks.md) for the **corrected** same-harness comparison vs Qwen3.5-9B (chat + thinking on, fixed 2026-09-01). The first published Qwen column (MMLU-Pro ~21 / MATH ~20) was an **invalid harness artifact** (thinking ON with 1k–8k caps → mid-think truncation).

| Benchmark | Teutonic | Qwen3.5-9B (fixed harness) | Δ |
|---|---:|---:|---:|
| MMLU-Pro (n=2000) | 69.8 | **73.2** | −3.4 |
| MMLU-Redux (generative) | 87.0 | **91.3** | −4.3 |
| MATH-500 | **88.6** | 68.2 | **+20.4** |
| HumanEval+ | **82.9** | 81.1 | **+1.8** |
| MBPP+ | **75.1** | 64.6 | **+10.5** |
| IFEval | **68.6** | 65.4 | **+3.2** |
| GPQA-Diamond (fair boxed) | 46.0 | **55.1** | −9.1 |
| LiveCodeBench (v5+v6) | 26.9 | 26.9 | 0.0 |
| RULER (4k–64k) | **89.5** | 88.4 | **+1.1** |

Protocol: chat · thinking on · raised think budgets · seed 20260830. GPQA = generative boxed-letter (not card shuffle). See benchmarks.md.