---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- kent
- recurrent
- linear-time
- frequency-domain
- dct
- text-generation
- custom_code
- fp16
- base
---

# KenT-30M-Base

KenT is a recurrent language model. This ~30M-parameter version reads **one token per step** and stacks 24 identical layers;
each layer expands the token vector into `M` resolution positions, mixes them with a fixed DCT-II, transforms the features
with a SwiGLU, and keeps a learned-forgetting memory (EMA) whose decay is a learned **matrix** (one coefficient per
resolution position and feature). The memory is read back through a DCT-II and projected to the token dimension.

> This repo contains `trust_remote_code` code; weights are stored in **FP16**.

## Layer

```
Z   = ResProj_in(Norm(X))              # 1 -> M: token vector n -> M x d
Z   = Z + SwiGLU(DCT_M(RMSNorm(Z)))    # WNN block: fixed DCT-II over the M axis, SwiGLU over the d axis
H_t = (1 - A) * H_{t-1} + A * Z_t      # EMA Learner Matrix: A = sigmoid(logits), shape M x d, per layer
X'  = X + ResProj_out(DCT_M(Norm(H_t)))  # DCT-II Proj, then M -> 1; residual with the layer input
```

| | value |
|---|---|
| hidden size `n` | 256 (tied embedding / `lm_head`, vocab 50257) |
| layers | 24 |
| resolution `M` x features `d` | 12 x 96 (M·d = 1152 = 4.5n) |
| SwiGLU hidden | 384 (4d) |
| EMA decay | learned matrix `(12, 96)` per layer (`ema_per_feature=false` -> vector `(12,)`) |
| recurrent state | `H`: `(12, 96)` per layer, constant size |
| parameters | ~29.75M (embedding 12.9M + layers 16.9M) |

## Quickstart

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

repo = "Vi-LAB/KenT-30M-Base"  # or a local directory
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float16).to("cuda").eval()
gen_cfg = GenerationConfig.from_pretrained(repo)

inputs = tok("Hello, today", return_tensors="pt").to("cuda")
out = model.generate(input_ids=inputs.input_ids, attention_mask=inputs.attention_mask, generation_config=gen_cfg)
print(tok.decode(out[0], skip_special_tokens=True))
```

Long documents: `python infer.py --model ./KenT-30M-Base --prompt-file doc.txt --chunk-size 1024`
(chunks are processed in parallel and merged with an affine scan; the result equals sequential processing).

## Notes

- Research model, trained from scratch; evaluate before real-world use.
- The tokenizer and weights must match `vocab_size` in `config.json`.