---
license: apache-2.0
language:
  - en
tags:
  - text-generation
  - causal-lm
  - pytorch
  - sft
  - instruction-tuned
  - chat
  - reasoning
  - tool-use
  - hybrid
  - gated-deltanet-2
  - parallax-attention
  - n-gram-memory
  - engram
  - senryu
  - senryu-2
pipeline_tag: text-generation
library_name: tiny_gdn
datasets:
  - HuggingFaceTB/smoltalk2
  - HuggingFaceTB/smoltalk
  - nvidia/Nemotron-Cascade-2-SFT-Data
  - nvidia/Nemotron-Post-Training-Dataset-v1
  - nvidia/Nemotron-Agentic-v1
  - nvidia/Nemotron-SFT-Agentic-v2
  - nvidia/Nemotron-SFT-Instruction-Following-Chat-v2
  - nvidia/Nemotron-Instruction-Following-Chat-v1
  - nvidia/OpenMathReasoning
  - nvidia/OpenCodeReasoning
  - nvidia/When2Call
  - Agent-Ark/Toucan-1.5M
  - interstellarninja/hermes_reasoning_tool_use
  - interstellarninja/tool-use-multiturn-reasoning
  - NousResearch/Hermes-3-Dataset
  - glaiveai/glaive-function-calling-v2
  - allenai/WildChat-4.8M
  - QuixiAI/SystemChat-2.0
  - Open-Orca/SlimOrca
  - HuggingFaceH4/ultrachat_200k
  - ise-uiuc/Magicoder-Evol-Instruct-110K
  - bigcode/self-oss-instruct-sc2-exec-filter-50k
  - OpenAssistant/oasst2
  - allenai/tulu-3-sft-personas-instruction-following
  - HuggingFaceH4/no_robots
  - mrfakename/identity
  - modelscope/self-cognition
---

<div align="center">

# Senryu-2-1B

### Hybrid Gated DeltaNet-2 + Parallax chat model (1.09B) with 2.17B parameters of n-gram memory

[![Model](https://img.shields.io/badge/Model-1.09B_core_params-blue)](.)
[![Memory](https://img.shields.io/badge/N--gram_memory-2.17B_params-blue)](.)
[![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](https://www.apache.org/licenses/LICENSE-2.0)
[![Demo](https://img.shields.io/badge/Demo-ZeroGPU_Space-orange.svg)](https://huggingface.co/spaces/KerzgrAI/senryu-2-1b-demo)

*By [KerzgrAI](https://huggingface.co/KerzgrAI): a 28-layer hybrid trained from random weights on 2.5B tokens of reasoning, tool-call and chat data*

</div>

---

## What this is

**Senryu-2-1B** is the first model on the Senryu-2 architecture and the first release from [KerzgrAI](https://huggingface.co/KerzgrAI). It succeeds [Senryu](https://huggingface.co/kerzgrr/Senryu) (803M).

- **1.09B core parameters**, embeddings included, plus **2.17B parameters of hashed 2- and 3-gram memory**, read by lookup rather than matrix multiplies.
- **No pretraining stage.** It trained from random weights directly on chat data, at a 32,768-token sequence length on one RTX PRO 6000 Blackwell GPU.
- **Thinking on or off**, and **Hermes JSON tool calls**.

Try it in the [demo Space](https://huggingface.co/spaces/KerzgrAI/senryu-2-1b-demo).

---

## Chat contract

Each assistant turn starts with a control token:

| Mode | Prefix | Typical body |
|------|--------|--------------|
| think | `<\|think\|>\n` | `<think>…</think>` then the answer |
| no-think | `<\|no_think\|>\n` | answer only |

Tool schemas go in a Hermes-style `<tools>` block in the system turn, and the model calls them with:

```
<tool_call>
{"name": "web-search", "arguments": {"query": "…"}}
</tool_call>
```

Each observation comes back as a `tool` turn starting with `<|tool_response|>\n`.

---

## Install & run

```bash
pip install torch safetensors tokenizers huggingface_hub
hf download KerzgrAI/Senryu-2-1B inference.py --local-dir .
python inference.py
```

```bash
python inference.py --prompt "Explain recursion in two sentences."
python inference.py --tools web-search,python,calculator
python inference.py --no-think --prompt "Reply in one sentence."
python inference.py --ngram-device cpu
```

`inference.py` downloads the weights, n-gram tables, tokenizer and `tiny_gdn/`, and installs `flash-linear-attention` from git at the pinned commit, so Git must be on `PATH`. A CUDA GPU is required: Gated DeltaNet-2 and Parallax run on Triton kernels.

| Flag | Default | Description |
|------|---------|-------------|
| `--prompt` | — | One-shot user message |
| `--system` | — | System prompt, used verbatim |
| `--think` / `--no-think` | think | Assistant control prefix |
| `--tools` | — | Built-ins: `web-search`, `python`, `calculator` (comma-separated) |
| `--tools-json` | — | Extra tool schemas (JSON list) for the `<tools>` block |
| `--temperature` | `0.6` | Sampling temperature (0.3 suits tool-calling turns) |
| `--top-p` | `0.95` | Nucleus sampling |
| `--top-k` | `50` | Top-k sampling |
| `--repetition-penalty` | `1.08` | Over the last `--repetition-window` (256) tokens |
| `--max-new-tokens` | `4096` | Max generation length |
| `--context-length` | `32768` | Conversation tokens kept (oldest dropped first) |
| `--device` | `cuda` | CUDA device, e.g. `cuda:1` |
| `--ngram-device` | same as `--device` | Where the n-gram tables live; `cpu` saves 4.3 GB of VRAM |

Interactive commands: `/think` `/no_think` `/system …` `/reset` `/exit`. Ctrl+C stops the current reply.

---

## Model architecture

| | |
|--|--|
| **Core parameters** | 1,094,961,660 (embeddings included) |
| **N-gram memory** | 2,166,398,464 table parameters: 2- and 3-gram hashes, four hash heads per order |
| **Layers** | 28: 7 × (3 Gated DeltaNet-2 + 1 Parallax attention) |
| **Hidden** | 1,536 |
| **MLP** | SiTU-GLU 3,584 |
| **Attention** | Parallax: 12 query and 4 KV heads of 128, RoPE on half of each head (θ = 10⁶) |
| **Linear** | Gated DeltaNet-2, 12 heads × 128, with a short convolution |
| **Residuals** | Four-branch gated residual stream (rank-128 gate) |
| **Memory read** | Before block 1, with a context-aware gate per residual branch, as in Engram |
| **Vocab** | 65,536 BPE, tied embeddings |
| **Context** | 32,768 |

---

## Training

From random weights, with no pretraining, on a fully shuffled mixture of the datasets listed in this card's metadata: 16 sequences of 32,768 tokens per step, Muon v2 + AdamW at 5×10⁻⁴, and sparse Adam at 2.5×10⁻³ for the n-gram tables. About 2.5B tokens in 56 hours. `sft_config.json` holds the run's settings.

---

## Files

| File | Contents |
|------|----------|
| `model.safetensors` | Core weights, bfloat16 |
| `ngram_tables.safetensors` | N-gram tables (bfloat16), canonical ids and hash multipliers |
| `config.json` | Architecture |
| `tokenizer.json`, `special_token_ids.json` | Tokenizer and control-token ids |
| `inference.py`, `requirements.txt`, `tiny_gdn/` | Standalone chat CLI and the model code it imports |
| `sft_config.json` | Training settings |
| `windows_fla_patches/` | Import patches `inference.py` applies to fla on Windows |

---

## Limitations

- **Scale.** 1.09B core parameters and 2.5B training tokens with no pretraining: expect thin world knowledge, factual errors and invented details.
- **Language.** Training data is mostly English.
- **Hardware.** A CUDA GPU is required.

Weights are Apache-2.0. Each training dataset keeps its own license; some carry attribution or non-commercial terms, so check them before commercial use.

---

## Model family

| Model | Notes | Hub |
|-------|-------|-----|
| **Senryu-2-1B** | **Senryu-2 architecture** | **this repo** |
| Senryu | Previous generation (803M) | [`kerzgrr/Senryu`](https://huggingface.co/kerzgrr/Senryu) |
| Haiku | Earlier generation (~655M) | [`kerzgrr/Haiku`](https://huggingface.co/kerzgrr/Haiku) |

---

## References

- Gated DeltaNet-2: <https://arxiv.org/abs/2605.22791>
- Parallax attention: <https://arxiv.org/abs/2605.29157>
- Engram conditional memory: <https://arxiv.org/abs/2601.07372>
- flash-linear-attention, pinned at [`cbb0a72`](https://github.com/fla-org/flash-linear-attention/tree/cbb0a72efb55c18ca0ef4f298298317573ad2cb3)

```bibtex
@misc{senryu2_1b_2026,
  title={Senryu-2-1B: A 1.1B Hybrid Gated DeltaNet-2 + Parallax Chat Model with Hashed N-gram Memory},
  author={KerzgrAI},
  year={2026},
  url={https://huggingface.co/KerzgrAI/Senryu-2-1B}
}
```