README.md · KerzgrAI/Senryu-2-1B at main

Upload Senryu-2-1B (EMA weights)

0a4bb1e verified 11 days ago

7.84 kB

---
license: apache-2.0
language:
  - en
tags:
  - text-generation
  - causal-lm
  - pytorch
  - sft
  - instruction-tuned
  - chat
  - reasoning
  - tool-use
  - hybrid
  - gated-deltanet-2
  - parallax-attention
  - n-gram-memory
  - engram
  - senryu
  - senryu-2
pipeline_tag: text-generation
library_name: tiny_gdn
datasets:
  - HuggingFaceTB/smoltalk2
  - HuggingFaceTB/smoltalk
  - nvidia/Nemotron-Cascade-2-SFT-Data
  - nvidia/Nemotron-Post-Training-Dataset-v1
  - nvidia/Nemotron-Agentic-v1
  - nvidia/Nemotron-SFT-Agentic-v2
  - nvidia/Nemotron-SFT-Instruction-Following-Chat-v2
  - nvidia/Nemotron-Instruction-Following-Chat-v1
  - nvidia/OpenMathReasoning
  - nvidia/OpenCodeReasoning
  - nvidia/When2Call
  - Agent-Ark/Toucan-1.5M
  - interstellarninja/hermes_reasoning_tool_use
  - interstellarninja/tool-use-multiturn-reasoning
  - NousResearch/Hermes-3-Dataset
  - glaiveai/glaive-function-calling-v2
  - allenai/WildChat-4.8M
  - QuixiAI/SystemChat-2.0
  - Open-Orca/SlimOrca
  - HuggingFaceH4/ultrachat_200k
  - ise-uiuc/Magicoder-Evol-Instruct-110K
  - bigcode/self-oss-instruct-sc2-exec-filter-50k
  - OpenAssistant/oasst2
  - allenai/tulu-3-sft-personas-instruction-following
  - HuggingFaceH4/no_robots
  - mrfakename/identity
  - modelscope/self-cognition
---
<div align="center">

# Senryu-2-1B

### Hybrid Gated DeltaNet-2 + Parallax chat model (1.09B) with 2.17B parameters of n-gram memory

[![Model](https://img.shields.io/badge/Model-1.09B_core_params-blue)](.)
[![Memory](https://img.shields.io/badge/N--gram_memory-2.17B_params-blue)](.)
[![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](https://www.apache.org/licenses/LICENSE-2.0)
[![Demo](https://img.shields.io/badge/Demo-ZeroGPU_Space-orange.svg)](https://huggingface.co/spaces/KerzgrAI/senryu-2-1b-demo)

*By [KerzgrAI](https://huggingface.co/KerzgrAI): a 28-layer hybrid trained from random weights on 2.5B tokens of reasoning, tool-call and chat data*

</div>

---

## What this is

**Senryu-2-1B** is the first model on the Senryu-2 architecture and the first release from [KerzgrAI](https://huggingface.co/KerzgrAI). It succeeds [Senryu](https://huggingface.co/kerzgrr/Senryu) (803M).

- **1.09B core parameters**, embeddings included, plus **2.17B parameters of hashed 2- and 3-gram memory**, read by lookup rather than matrix multiplies.
- **No pretraining stage.** It trained from random weights directly on chat data, at a 32,768-token sequence length on one RTX PRO 6000 Blackwell GPU.
- **Thinking on or off**, and **Hermes JSON tool calls**.

Try it in the [demo Space](https://huggingface.co/spaces/KerzgrAI/senryu-2-1b-demo).

---

## Chat contract

Each assistant turn starts with a control token:

| Mode | Prefix | Typical body |
|------|--------|--------------|
| think | `<\|think\|>\n` | `<think>…</think>` then the answer |
| no-think | `<\|no_think\|>\n` | answer only |

Tool schemas go in a Hermes-style `<tools>` block in the system turn, and the model calls them with:

```
<tool_call>
{"name": "web-search", "arguments": {"query": "…"}}
</tool_call>
```

Each observation comes back as a `tool` turn starting with `<|tool_response|>\n`.

---

## Install & run

```bash
pip install torch safetensors tokenizers huggingface_hub
hf download KerzgrAI/Senryu-2-1B inference.py --local-dir .
python inference.py
```

```bash
python inference.py --prompt "Explain recursion in two sentences."
python inference.py --tools web-search,python,calculator
python inference.py --no-think --prompt "Reply in one sentence."
python inference.py --ngram-device cpu
```

`inference.py` downloads the weights, n-gram tables, tokenizer and `tiny_gdn/`, and installs `flash-linear-attention` from git at the pinned commit, so Git must be on `PATH`. A CUDA GPU is required: Gated DeltaNet-2 and Parallax run on Triton kernels.

| Flag | Default | Description |
|------|---------|-------------|
| `--prompt` | — | One-shot user message |
| `--system` | — | System prompt, used verbatim |
| `--think` / `--no-think` | think | Assistant control prefix |
| `--tools` | — | Built-ins: `web-search`, `python`, `calculator` (comma-separated) |
| `--tools-json` | — | Extra tool schemas (JSON list) for the `<tools>` block |
| `--temperature` | `0.6` | Sampling temperature (0.3 suits tool-calling turns) |
| `--top-p` | `0.95` | Nucleus sampling |
| `--top-k` | `50` | Top-k sampling |
| `--repetition-penalty` | `1.08` | Over the last `--repetition-window` (256) tokens |
| `--max-new-tokens` | `4096` | Max generation length |
| `--context-length` | `32768` | Conversation tokens kept (oldest dropped first) |
| `--device` | `cuda` | CUDA device, e.g. `cuda:1` |
| `--ngram-device` | same as `--device` | Where the n-gram tables live; `cpu` saves 4.3 GB of VRAM |

Interactive commands: `/think` `/no_think` `/system …` `/reset` `/exit`. Ctrl+C stops the current reply.

---

## Model architecture

| | |
|--|--|
| **Core parameters** | 1,094,961,660 (embeddings included) |
| **N-gram memory** | 2,166,398,464 table parameters: 2- and 3-gram hashes, four hash heads per order |
| **Layers** | 28: 7 × (3 Gated DeltaNet-2 + 1 Parallax attention) |
| **Hidden** | 1,536 |
| **MLP** | SiTU-GLU 3,584 |
| **Attention** | Parallax: 12 query and 4 KV heads of 128, RoPE on half of each head (θ = 10⁶) |
| **Linear** | Gated DeltaNet-2, 12 heads × 128, with a short convolution |
| **Residuals** | Four-branch gated residual stream (rank-128 gate) |
| **Memory read** | Before block 1, with a context-aware gate per residual branch, as in Engram |
| **Vocab** | 65,536 BPE, tied embeddings |
| **Context** | 32,768 |

---

## Training

From random weights, with no pretraining, on a fully shuffled mixture of the datasets listed in this card's metadata: 16 sequences of 32,768 tokens per step, Muon v2 + AdamW at 5×10⁻⁴, and sparse Adam at 2.5×10⁻³ for the n-gram tables. About 2.5B tokens in 56 hours. `sft_config.json` holds the run's settings.

---

## Files

| File | Contents |
|------|----------|
| `model.safetensors` | Core weights, bfloat16 |
| `ngram_tables.safetensors` | N-gram tables (bfloat16), canonical ids and hash multipliers |
| `config.json` | Architecture |
| `tokenizer.json`, `special_token_ids.json` | Tokenizer and control-token ids |
| `inference.py`, `requirements.txt`, `tiny_gdn/` | Standalone chat CLI and the model code it imports |
| `sft_config.json` | Training settings |
| `windows_fla_patches/` | Import patches `inference.py` applies to fla on Windows |

---

## Limitations

- **Scale.** 1.09B core parameters and 2.5B training tokens with no pretraining: expect thin world knowledge, factual errors and invented details.
- **Language.** Training data is mostly English.
- **Hardware.** A CUDA GPU is required.

Weights are Apache-2.0. Each training dataset keeps its own license; some carry attribution or non-commercial terms, so check them before commercial use.

---

## Model family

| Model | Notes | Hub |
|-------|-------|-----|
| **Senryu-2-1B** | **Senryu-2 architecture** | **this repo** |
| Senryu | Previous generation (803M) | [`kerzgrr/Senryu`](https://huggingface.co/kerzgrr/Senryu) |
| Haiku | Earlier generation (~655M) | [`kerzgrr/Haiku`](https://huggingface.co/kerzgrr/Haiku) |

---

## References

- Gated DeltaNet-2: <https://arxiv.org/abs/2605.22791>
- Parallax attention: <https://arxiv.org/abs/2605.29157>
- Engram conditional memory: <https://arxiv.org/abs/2601.07372>
- flash-linear-attention, pinned at [`cbb0a72`](https://github.com/fla-org/flash-linear-attention/tree/cbb0a72efb55c18ca0ef4f298298317573ad2cb3)

```bibtex
@misc{senryu2_1b_2026,
  title={Senryu-2-1B: A 1.1B Hybrid Gated DeltaNet-2 + Parallax Chat Model with Hashed N-gram Memory},
  author={KerzgrAI},
  year={2026},
  url={https://huggingface.co/KerzgrAI/Senryu-2-1B}
}
```

	---
	license: apache-2.0
	language:
	- en
	tags:
	- text-generation
	- causal-lm
	- pytorch
	- sft
	- instruction-tuned
	- chat
	- reasoning
	- tool-use
	- hybrid
	- gated-deltanet-2
	- parallax-attention
	- n-gram-memory
	- engram
	- senryu
	- senryu-2
	pipeline_tag: text-generation
	library_name: tiny_gdn
	datasets:
	- HuggingFaceTB/smoltalk2
	- HuggingFaceTB/smoltalk
	- nvidia/Nemotron-Cascade-2-SFT-Data
	- nvidia/Nemotron-Post-Training-Dataset-v1
	- nvidia/Nemotron-Agentic-v1
	- nvidia/Nemotron-SFT-Agentic-v2
	- nvidia/Nemotron-SFT-Instruction-Following-Chat-v2
	- nvidia/Nemotron-Instruction-Following-Chat-v1
	- nvidia/OpenMathReasoning
	- nvidia/OpenCodeReasoning
	- nvidia/When2Call
	- Agent-Ark/Toucan-1.5M
	- interstellarninja/hermes_reasoning_tool_use
	- interstellarninja/tool-use-multiturn-reasoning
	- NousResearch/Hermes-3-Dataset
	- glaiveai/glaive-function-calling-v2
	- allenai/WildChat-4.8M
	- QuixiAI/SystemChat-2.0
	- Open-Orca/SlimOrca
	- HuggingFaceH4/ultrachat_200k
	- ise-uiuc/Magicoder-Evol-Instruct-110K
	- bigcode/self-oss-instruct-sc2-exec-filter-50k
	- OpenAssistant/oasst2
	- allenai/tulu-3-sft-personas-instruction-following
	- HuggingFaceH4/no_robots
	- mrfakename/identity
	- modelscope/self-cognition
	---

	<div align="center">

	# Senryu-2-1B

	### Hybrid Gated DeltaNet-2 + Parallax chat model (1.09B) with 2.17B parameters of n-gram memory

	[![Model](https://img.shields.io/badge/Model-1.09B_core_params-blue)](.)
	[![Memory](https://img.shields.io/badge/N--gram_memory-2.17B_params-blue)](.)
	[![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](https://www.apache.org/licenses/LICENSE-2.0)
	[![Demo](https://img.shields.io/badge/Demo-ZeroGPU_Space-orange.svg)](https://huggingface.co/spaces/KerzgrAI/senryu-2-1b-demo)

	By [KerzgrAI](https://huggingface.co/KerzgrAI): a 28-layer hybrid trained from random weights on 2.5B tokens of reasoning, tool-call and chat data

	</div>

	---

	## What this is

	Senryu-2-1B is the first model on the Senryu-2 architecture and the first release from [KerzgrAI](https://huggingface.co/KerzgrAI). It succeeds [Senryu](https://huggingface.co/kerzgrr/Senryu) (803M).

	- 1.09B core parameters, embeddings included, plus 2.17B parameters of hashed 2- and 3-gram memory, read by lookup rather than matrix multiplies.
	- No pretraining stage. It trained from random weights directly on chat data, at a 32,768-token sequence length on one RTX PRO 6000 Blackwell GPU.
	- Thinking on or off, and Hermes JSON tool calls.

	Try it in the [demo Space](https://huggingface.co/spaces/KerzgrAI/senryu-2-1b-demo).

	---

	## Chat contract

	Each assistant turn starts with a control token:

	\| Mode \| Prefix \| Typical body \|
	\|------\|--------\|--------------\|
	\| think \| `<\\|think\\|>\n` \| `<think>…</think>` then the answer \|
	\| no-think \| `<\\|no_think\\|>\n` \| answer only \|

	Tool schemas go in a Hermes-style `<tools>` block in the system turn, and the model calls them with:

	```
	<tool_call>
	{"name": "web-search", "arguments": {"query": "…"}}
	</tool_call>
	```

	Each observation comes back as a `tool` turn starting with `<\|tool_response\|>\n`.

	---

	## Install & run

	```bash
	pip install torch safetensors tokenizers huggingface_hub
	hf download KerzgrAI/Senryu-2-1B inference.py --local-dir .
	python inference.py
	```

	```bash
	python inference.py --prompt "Explain recursion in two sentences."
	python inference.py --tools web-search,python,calculator
	python inference.py --no-think --prompt "Reply in one sentence."
	python inference.py --ngram-device cpu
	```

	`inference.py` downloads the weights, n-gram tables, tokenizer and `tiny_gdn/`, and installs `flash-linear-attention` from git at the pinned commit, so Git must be on `PATH`. A CUDA GPU is required: Gated DeltaNet-2 and Parallax run on Triton kernels.

	\| Flag \| Default \| Description \|
	\|------\|---------\|-------------\|
	\| `--prompt` \| — \| One-shot user message \|
	\| `--system` \| — \| System prompt, used verbatim \|
	\| `--think` / `--no-think` \| think \| Assistant control prefix \|
	\| `--tools` \| — \| Built-ins: `web-search`, `python`, `calculator` (comma-separated) \|
	\| `--tools-json` \| — \| Extra tool schemas (JSON list) for the `<tools>` block \|
	\| `--temperature` \| `0.6` \| Sampling temperature (0.3 suits tool-calling turns) \|
	\| `--top-p` \| `0.95` \| Nucleus sampling \|
	\| `--top-k` \| `50` \| Top-k sampling \|
	\| `--repetition-penalty` \| `1.08` \| Over the last `--repetition-window` (256) tokens \|
	\| `--max-new-tokens` \| `4096` \| Max generation length \|
	\| `--context-length` \| `32768` \| Conversation tokens kept (oldest dropped first) \|
	\| `--device` \| `cuda` \| CUDA device, e.g. `cuda:1` \|
	\| `--ngram-device` \| same as `--device` \| Where the n-gram tables live; `cpu` saves 4.3 GB of VRAM \|

	Interactive commands: `/think` `/no_think` `/system …` `/reset` `/exit`. Ctrl+C stops the current reply.

	---

	## Model architecture

	\| \| \|
	\|--\|--\|
	\| Core parameters \| 1,094,961,660 (embeddings included) \|
	\| N-gram memory \| 2,166,398,464 table parameters: 2- and 3-gram hashes, four hash heads per order \|
	\| Layers \| 28: 7 × (3 Gated DeltaNet-2 + 1 Parallax attention) \|
	\| Hidden \| 1,536 \|
	\| MLP \| SiTU-GLU 3,584 \|
	\| Attention \| Parallax: 12 query and 4 KV heads of 128, RoPE on half of each head (θ = 10⁶) \|
	\| Linear \| Gated DeltaNet-2, 12 heads × 128, with a short convolution \|
	\| Residuals \| Four-branch gated residual stream (rank-128 gate) \|
	\| Memory read \| Before block 1, with a context-aware gate per residual branch, as in Engram \|
	\| Vocab \| 65,536 BPE, tied embeddings \|
	\| Context \| 32,768 \|

	---

	## Training

	From random weights, with no pretraining, on a fully shuffled mixture of the datasets listed in this card's metadata: 16 sequences of 32,768 tokens per step, Muon v2 + AdamW at 5×10⁻⁴, and sparse Adam at 2.5×10⁻³ for the n-gram tables. About 2.5B tokens in 56 hours. `sft_config.json` holds the run's settings.

	---

	## Files

	\| File \| Contents \|
	\|------\|----------\|
	\| `model.safetensors` \| Core weights, bfloat16 \|
	\| `ngram_tables.safetensors` \| N-gram tables (bfloat16), canonical ids and hash multipliers \|
	\| `config.json` \| Architecture \|
	\| `tokenizer.json`, `special_token_ids.json` \| Tokenizer and control-token ids \|
	\| `inference.py`, `requirements.txt`, `tiny_gdn/` \| Standalone chat CLI and the model code it imports \|
	\| `sft_config.json` \| Training settings \|
	\| `windows_fla_patches/` \| Import patches `inference.py` applies to fla on Windows \|

	---

	## Limitations

	- Scale. 1.09B core parameters and 2.5B training tokens with no pretraining: expect thin world knowledge, factual errors and invented details.
	- Language. Training data is mostly English.
	- Hardware. A CUDA GPU is required.

	Weights are Apache-2.0. Each training dataset keeps its own license; some carry attribution or non-commercial terms, so check them before commercial use.

	---

	## Model family

	\| Model \| Notes \| Hub \|
	\|-------\|-------\|-----\|
	\| Senryu-2-1B \| Senryu-2 architecture \| this repo \|
	\| Senryu \| Previous generation (803M) \| [`kerzgrr/Senryu`](https://huggingface.co/kerzgrr/Senryu) \|
	\| Haiku \| Earlier generation (~655M) \| [`kerzgrr/Haiku`](https://huggingface.co/kerzgrr/Haiku) \|

	---

	## References

	- Gated DeltaNet-2: <https://arxiv.org/abs/2605.22791>
	- Parallax attention: <https://arxiv.org/abs/2605.29157>
	- Engram conditional memory: <https://arxiv.org/abs/2601.07372>
	- flash-linear-attention, pinned at [`cbb0a72`](https://github.com/fla-org/flash-linear-attention/tree/cbb0a72efb55c18ca0ef4f298298317573ad2cb3)

	```bibtex
	@misc{senryu2_1b_2026,
	title={Senryu-2-1B: A 1.1B Hybrid Gated DeltaNet-2 + Parallax Chat Model with Hashed N-gram Memory},
	author={KerzgrAI},
	year={2026},
	url={https://huggingface.co/KerzgrAI/Senryu-2-1B}
	}
	```