🛑 Important Notice from the Creator

please dont make qanizations of models I make because these are very hard bulit

---

📈 Training Improvements (Nov 2026)

ANTRU-Nano was retrained on CIFAR-10 with several enhancements:

• Horizontal-flip augmentation — effectively doubled the training dataset with random horizontal flips
• LR schedule — linear warmup over first 500 steps, then cosine decay to 5% of initial LR
• EMA decay warmup — exponential moving average decay ramped from 0.99 to 0.999 over first 2000 steps
• Richer preview grids — now shows all 6 CIFAR classes (airplane, automobile, bird, dog, ship, truck) instead of 3
• Format bump — checkpoint format updated to v1.1 for backward/forward compatibility

Results: Loss improved from ~3.3 (original) → 2.91 (retrained), with steady decrease throughout training. 50 epochs completed on NVIDIA GeForce 930MX (Maxwell, sm_50) with CUDA 12.6.

Model card now includes both the creator's notice and training improvements for transparency.

🙏 A note from the author

These models were hand-trained by me on a 2GB consumer laptop GPU over many long nights — they are hard to build and mean a lot to me. Please don't create quantizations or re-uploads of my models. If you build on them anyway, please clearly credit ANTRUhub and link back to the original model. Thank you.

ANTRU-Nano

A 79.5M-parameter causal decoder-only transformer trained 100% from scratch (random initialization, no pre-trained base) on a consumer laptop: an Intel i7-8550U CPU and a GeForce MX150 (2GB) GPU, in one ~40-minute GPU session at ~761 tokens/sec.

Built as part of the ANTRU project — a family of fully local, self-trained models.

Model details

Architecture	Decoder-only transformer (pre-norm LayerNorm, GELU FFN, learned positional embeddings, tied input/output embeddings)
Parameters	79,538,688
Hidden dim	768 · 12 layers · 12 heads (head_dim 64) · FFN 2048
Vocabulary	16,384 word-level BPE-free tokens (frequency-built, byte/char fallback — zero unknown-token failures)
Context	1,024-token attention window + up to 200K-token application-level RAM retrieval buffer
Checkpoint	fp32 + optimizer state (~911MB)

Training

• Corpus (~1.79M tokens): English Wikipedia (70%) + a curated ANTRU curriculum (identity & persona statements, ~25 verified Python/coding explanations, step-by-step math reasoning) + local knowledge-base facts + 30k synthetic arithmetic drills (repeated 6×, 4× respectively so the small model could actually learn them).
• Recipe: AdamW (lr 3e-4, cosine decay, 100-step warmup), grad clip 1.0, batch 1 × seq 512, 1 epoch, loss 9.7 → ~3.3.
• Hardware: MX150 2GB — weights + gradients + optimizer fit entirely in VRAM (~1.8GB at 99% utilization).

What it can do (honest)

✅ Generates English-like token structure, learned word ordering, answers simple arithmetic from its drills ("25 + 17" patterns), reproduces its identity curriculum ("I am ANTRU...") when prompted in-style, never crashes on unknown characters (byte fallback).

⚠️ This is an educational artifact, not a chat assistant. At ~22 training tokens per parameter it is heavily under-trained (Chinchilla-optimal would be ~20× more data). Expect loosely-coherent local text rather than fluent answers. Best results: low temperature (0.2–0.3), top_k ≤ 5, short prompts in its curriculum style.

Usage

The model uses a custom (transformers-independent) PyTorch class. Full code ships in the code/ folder:

import sys; sys.path.insert(0, "code")
from antru_model_torch import AntruTorchModel
from antru_tokenizer import AntruTokenizer
from antru_config import AntruConfig
import torch

tok = AntruTokenizer.load("vocab.json")
cfg = AntruConfig(vocab_size=tok.vocab_size)
model = AntruTorchModel.load_checkpoint("antru_nano.pt")  # 911MB fp32+optimizer
ids = tok.encode("What is 25 + 17?", add_special_tokens=False)
out = model.generate(ids, max_new_tokens=24, temperature=0.25, top_k=3)
print(tok.decode(out[len(ids):]))

License & attribution

Apache-2.0. 100% of the weights were trained by the ANTRU project's own hardware — no pre-trained base was used. Corpus text from English Wikipedia (CC BY-SA 3.0).