lillycham's picture
Upload Kestrel-34M-instruct
ff2813d verified
|
Raw History Blame Contribute Delete
1.69 kB
---
license: mit
library_name: transformers
tags:
- kestrel
- titans
- titans-mag
- looped-transformer
- small-language-model
---
# Kestrel-34M-instruct
Supervised fine-tuning run initialized from `Kestrel-34M-base` using assistant-only loss.
- **Repository:** [lillycham/kestrel](https://github.com/lillycham/kestrel)
- **Stage:** SFT (2,000 steps from Kestrel-34M-base, assistant-only loss)
- **Initialized from:** [lillycham/Kestrel-34M-base](https://huggingface.co/lillycham/Kestrel-34M-base)
- **Parameters:** 34,262,272 (~34.3M)
- **Architecture:** Looped Titans-MAG (`prelude=1`, `core=4 x 3 loops`, `coda=1`, `hidden_size=512`, `attention_window=256`, `memory_chunk_size=64`, `memory_heads=4`, `vocab_size=16384`)
- **Final metrics:** step 2000, train loss 1.711, val loss 1.832
## Usage
This model uses a custom `KestrelForCausalLM` architecture (`trust_remote_code=True`):
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "lillycham/Kestrel-34M-instruct"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
prompt = tokenizer.apply_chat_template([{"role": "user", "content": "Hello!"}], tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(inputs.input_ids, max_new_tokens=64, temperature=0.7, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=False))
```
## Training config summary
- Steps: `2000`, Batch size: `8` x grad_accum `4`, Sequence length: `1024`
- Optimizers: Muon (`lr=0.004`) + AdamW (`lr=0.0002`), weight decay `0.01`
- Precision: `bf16` on `cuda`