Factorized embeddings for RepOps deterministic training budgets
Hi Gensyn team,
Your model card separates the 1.61B total parameter count from the 1.08B non-embedding backbone, which points to a 530M parameter allocation (roughly 33% of the total budget) locked in untied token projection tables.
In verifiable runs where every step is audited and compute efficiency is paramount, spending a third of the weights on static vocabulary lookup limits active backbone capacity.
In an open architecture I work on called Maba (101M-30B specs: https://huggingface.co/AndrewThompson1233/maba-v1-architecture; weights: https://huggingface.co/AndrewThompson1233/maba-101m), we decouple vocabulary size from hidden dimension using a low-rank bottleneck:
Projecting the ~131k vocabulary through a rank-128 bottleneck reduces the combined untied embedding and LM-head footprint from ~530M to roughly 34M parameters.
This recovers ~496M parameters within the exact same 1.61B total envelope.
Reinvesting that budget into the backbone allows increasing non-embedding capacity from 1.08B to ~1.57B parameters (roughly a 45% increase in transformer layers).
Because low-rank factorized embeddings rely entirely on standard linear GEMM operations, they introduce no custom scan kernels or state drift, remaining fully compatible with bit-exact RepOps verification and quantization-aware training (QAT).
Did the open-1b recipe evaluate low-rank embedding bottlenecks before locking the untied full-rank token projection matrices?
Best,
Andrew