ks46

/

username-generator

Like 0

Hi ks46,

Running an exhaustive depth vs width Pareto sweep (from 2x32 up to 16x1024), evaluating on held-out bits/char via hash splits, and shipping a custom single-threaded x86-64 batch engine with INT7 quantization (ndgen) is exceptional systems work.

A few technical observations on your scaling data and inference setup:

1. The clean 256-byte vocabulary efficiency:
   Unlike standard subword models where embedding tables burn 30% to 60% of total parameters in sub-100M budgets, your 256-byte vocabulary keeps embedding overhead virtually at zero (256 * d_model is negligible). Literally 99%+ of your parameter budget directly funds active layer compute, which explains how your 0.8M and 4.9M models achieve such tight bits/char numbers.

2. Depth dominance in byte-level composition:
   Your grid clearly demonstrates that depth consistently beats width at matched parameter budgets:
   g3-12x256 (9.7M) achieves 3.2891 bits/char, outperforming g3-6x384 (10.7M) at 3.2958 despite having fewer parameters.
   g3-8x192 (3.7M) beats g3-4x384 (7.2M) at 3.3369 despite having roughly half the weights.
   In byte-level generation, character transition dynamics and phonetic/numeric clustering require multi-step sequential composition. Deeper computation graphs give the network the necessary non-linear hops to resolve local n-gram transitions that shallow wide models cannot capture.

3. KV-cache overhead in custom CPU batch engines:
   For short sequences like usernames, standard transformer attention requires allocating and updating key-value vectors across every layer. In custom CPU kernels like ndgen, managing dynamic attention buffers and computing softmax across sequential decode steps introduces memory traffic overhead, especially when running batched inference on single-threaded CPU loops.

In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we explore sub-quadratic state transitions for compact models:
By routing 75% of depth through linear recurrence (DGDA) paired with specialized latent sparse attention (MABA-SA), the network maintains an associative state matrix that updates in constant O(1) memory.
For byte-level character streams, linear recurrence naturally acts as a continuous n-gram state machine. It eliminates per-step KV-cache lookups entirely across recurrent layers, which maps exceptionally well to lightweight SIMD/AVX2 CPU execution.

If you explore further architecture variations for ndgen, checking out the recurrent state formulation in the Maba v2 repo might provide some useful ideas for squeezing even higher batch throughput on x86.

In your ndgen INT7 kernel, is batch decoding primarily memory-bandwidth bound when streaming weights, or does attention score computation take the majority of CPU cycles?

Best,
Andrew