JunkyAI
A 15.7-million-parameter language model trained from scratch.
JunkyAI is a small English language model built end to end without any pretrained weights: its own byte-level BPE vocabulary, its own transformer implementation, its own training loop, and its own GGUF writer. It speaks simple, fluent English — roughly the register of a children's book — and answers short questions. It does nothing else, by design and by capacity.
It exists to be understood completely rather than to be capable. Every component is a few hundred lines of readable Python with no ML framework above PyTorch tensors: no transformers, no tokenizers, no peft, no gguf.
At a glance
Parameters 15,735,168 (~15.7M)
Architecture Decoder-only transformer, Llama-shaped
Layers / hidden dim 8 / 384
Attention 6 query heads, 2 KV heads (grouped-query)
Feed-forward SwiGLU, 1024 inner
Normalisation RMSNorm, pre-norm
Positions RoPE (interleaved pairs), θ=10000
Context length 512 tokens
Vocabulary 8,192 byte-level BPE, t…
