Nirca Mini
Input bytes
vocabulary 265: 256 bytes and 9 special tokens
Embedding
width 512
2 transformer layers, run once
8 heads, RoPE, SwiGLU 1408
Merge latent with embedded input
each pass starts here
Recurrent block: attention + mixture of experts
32 experts, top 8, width 2048, ternary weights
Final norm and output head
a prediction at every pass (tied to the embedding)
Halt?
learned, 1 to 24 passes
no: another pass
yes
Next-byte prediction