--- license: mit language: [en, es, fr] tags: [trilingual, from-scratch, pytorch, text-generation] datasets: [HuggingFaceFW/fineweb-edu, epfml/FineWeb2-HQ] --- # Trilingua (base) A 70.5M-parameter decoder-only language model for English, Spanish and French, trained from scratch (tokenizer included) for the Arbiter Hacks V1 hackathon. Architecture: pre-norm transformer, RMSNorm, RoPE, SwiGLU, tied embeddings, 10 layers, d_model 640, 10 heads, context 1024. This is NOT a `transformers` model; load it with the code at the project repository: ```python from trilingua.utils import load_model from trilingua.tokenizer import TrilinguaTokenizer model = load_model("path/to/this/folder") tok = TrilinguaTokenizer.from_file("path/to/this/folder/tokenizer.json") ``` Prompt format: start with a language tag `<|en|>`, `<|es|>` or `<|fr|>`. Training step: 2385. See the repository README for data, metrics and limitations. Small model: expect fluent-looking but often factually wrong text.