Kestrel-34M-base / README.md
lillycham's picture
Upload Kestrel-34M-base
a2ddbe0 verified
|
Raw History Blame Contribute Delete
1.83 kB
---
license: mit
library_name: transformers
tags:
- kestrel
- titans
- titans-mag
- looped-transformer
- small-language-model
---
# Kestrel-34M-base
Base pretrained Looped Titans-MAG model (prelude + 3x4 looped core + coda, chunked neural memory + sliding-window attention, 16k byte-level BPE).
- **Repository:** [lillycham/kestrel](https://github.com/lillycham/kestrel)
- **Stage:** Pretraining (25,000 steps, ~819M tokens)
- **Parameters:** 34,262,272 (~34.3M)
- **Architecture:** Looped Titans-MAG (`prelude=1`, `core=4 x 3 loops`, `coda=1`, `hidden_size=512`, `attention_window=256`, `memory_chunk_size=64`, `memory_heads=4`, `vocab_size=16384`)
- **Final metrics:** step 25000, train loss 3.263, val loss 3.230
- **Open SLM Eval:** intelligence index 8.55 (raw 9.29) — hellaswag: 26.6% (norm 25.7%), arc_easy: 42.1% (norm 39.6%), arc_challenge: 18.9% (norm 22.3%), piqa: 58.7% (norm 57.6%), arithmark3: 33.2% (norm 33.4%)
## Usage
This model uses a custom `KestrelForCausalLM` architecture (`trust_remote_code=True`):
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "lillycham/Kestrel-34M-base"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
prompt = tokenizer.apply_chat_template([{"role": "user", "content": "Hello!"}], tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(inputs.input_ids, max_new_tokens=64, temperature=0.7, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=False))
```
## Training config summary
- Steps: `25000`, Batch size: `8` x grad_accum `4`, Sequence length: `1024`
- Optimizers: Muon (`lr=0.02`) + AdamW (`lr=0.001`), weight decay `0.01`
- Precision: `bf16` on `cuda`