--- license: mit library_name: transformers tags: - kestrel - titans - titans-mag - looped-transformer - small-language-model --- # Kestrel-34M-instruct Supervised fine-tuning run initialized from `Kestrel-34M-base` using assistant-only loss. - **Repository:** [lillycham/kestrel](https://github.com/lillycham/kestrel) - **Stage:** SFT (2,000 steps from Kestrel-34M-base, assistant-only loss) - **Initialized from:** [lillycham/Kestrel-34M-base](https://huggingface.co/lillycham/Kestrel-34M-base) - **Parameters:** 34,262,272 (~34.3M) - **Architecture:** Looped Titans-MAG (`prelude=1`, `core=4 x 3 loops`, `coda=1`, `hidden_size=512`, `attention_window=256`, `memory_chunk_size=64`, `memory_heads=4`, `vocab_size=16384`) - **Final metrics:** step 2000, train loss 1.711, val loss 1.832 ## Usage This model uses a custom `KestrelForCausalLM` architecture (`trust_remote_code=True`): ```python from transformers import AutoModelForCausalLM, AutoTokenizer repo_id = "lillycham/Kestrel-34M-instruct" tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True) prompt = tokenizer.apply_chat_template([{"role": "user", "content": "Hello!"}], tokenize=False, add_generation_prompt=True) inputs = tokenizer(prompt, return_tensors="pt") out = model.generate(inputs.input_ids, max_new_tokens=64, temperature=0.7, top_p=0.9) print(tokenizer.decode(out[0], skip_special_tokens=False)) ``` ## Training config summary - Steps: `2000`, Batch size: `8` x grad_accum `4`, Sequence length: `1024` - Optimizers: Muon (`lr=0.004`) + AdamW (`lr=0.0002`), weight decay `0.01` - Precision: `bf16` on `cuda`