| --- |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - weevil |
| - gdn2 |
| - custom_code |
| --- |
| |
| # Weevil-3M |
|
|
| Pretrained causal language model. 2,993,576 parameters, eight parallel GDN2/SwiGLU blocks, |
| and a shared 4,096-token BPE tokenizer. Uses the shared Weevil Transformers implementation. |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo = "ELBennett/Weevil-3M" |
| tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True) |
| inputs = tokenizer("The purpose of this article is", return_tensors="pt") |
| output = model.generate(**inputs, max_new_tokens=64, do_sample=True, |
| temperature=0.8, top_k=40) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| Requires PyTorch and Transformers (`pip install torch 'transformers>=4.57,<6'`). |
| Weights are safetensors; generation uses fixed-size recurrent/convolution caches. |
| Source checkpoint: `step-00038147`. This is a continuation model, with no chat template. |
|
|
