Add Weevil Transformers model, tokenizer and weights

0f70a95 verified 4 days ago

1.06 kB

library_name: transformers
pipeline_tag: text-generation
tags:
  - weevil
  - gdn2
  - custom_code

Weevil-3M

Pretrained causal language model. 2,993,576 parameters, eight parallel GDN2/SwiGLU blocks, and a shared 4,096-token BPE tokenizer. Uses the shared Weevil Transformers implementation.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ELBennett/Weevil-3M"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)
inputs = tokenizer("The purpose of this article is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=True,
                        temperature=0.8, top_k=40)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Requires PyTorch and Transformers (pip install torch 'transformers>=4.57,<6'). Weights are safetensors; generation uses fixed-size recurrent/convolution caches. Source checkpoint: step-00038147. This is a continuation model, with no chat template.