--- license: mit library_name: transformers tags: - kestrel - titans - titans-mag - looped-transformer - small-language-model --- # Kestrel-34M-base Base pretrained Looped Titans-MAG model (prelude + 3x4 looped core + coda, chunked neural memory + sliding-window attention, 16k byte-level BPE). - **Repository:** [lillycham/kestrel](https://github.com/lillycham/kestrel) - **Stage:** Pretraining (25,000 steps, ~819M tokens) - **Parameters:** 34,262,272 (~34.3M) - **Architecture:** Looped Titans-MAG (`prelude=1`, `core=4 x 3 loops`, `coda=1`, `hidden_size=512`, `attention_window=256`, `memory_chunk_size=64`, `memory_heads=4`, `vocab_size=16384`) - **Final metrics:** step 25000, train loss 3.263, val loss 3.230 - **Open SLM Eval:** intelligence index 8.55 (raw 9.29) — hellaswag: 26.6% (norm 25.7%), arc_easy: 42.1% (norm 39.6%), arc_challenge: 18.9% (norm 22.3%), piqa: 58.7% (norm 57.6%), arithmark3: 33.2% (norm 33.4%) ## Usage This model uses a custom `KestrelForCausalLM` architecture (`trust_remote_code=True`): ```python from transformers import AutoModelForCausalLM, AutoTokenizer repo_id = "lillycham/Kestrel-34M-base" tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True) prompt = tokenizer.apply_chat_template([{"role": "user", "content": "Hello!"}], tokenize=False, add_generation_prompt=True) inputs = tokenizer(prompt, return_tensors="pt") out = model.generate(inputs.input_ids, max_new_tokens=64, temperature=0.7, top_p=0.9) print(tokenizer.decode(out[0], skip_special_tokens=False)) ``` ## Training config summary - Steps: `25000`, Batch size: `8` x grad_accum `4`, Sequence length: `1024` - Optimizers: Muon (`lr=0.02`) + AdamW (`lr=0.001`), weight decay `0.01` - Precision: `bf16` on `cuda`