Mini

A small English chatbot trained from scratch. It is a decoder-only transformer with 6 layers, 4 attention heads, an embedding size of 192, and a context window of 512 tokens. The vocabulary is 507 words. It answers short questions it has seen in its training dialogues. It is not a general-purpose assistant.

Load

model.py, tokenizer.py, and pretrained.py from this repo need to be on the Python path.

from model import TinyGPT
from tokenizer import Tokenizer

model = TinyGPT.from_pretrained("StrongDev2024/mini", trust_remote_code=True)
tokenizer = Tokenizer.from_pretrained("StrongDev2024/mini")

ids = tokenizer.encode("<user> what is 2 + 2 <bot>")
import torch
out = model.generate(
    torch.tensor([ids]),
    max_new_tokens=40,
    temperature=0.0,
    stop_ids={tokenizer.token_to_id["<end>"], tokenizer.token_to_id["<user>"]},
)
print(tokenizer.decode(out[0, len(ids) :].tolist()))

The prompt is <user> your question <bot>. Generation stops at <end>.

License

MIT