Cascade-1B-Logic

Public experimental byte-level causal language model. Training is in progress and checkpoints are uploaded continuously.

Architecture

• Approximately 1B stored parameters and approximately one third active per token.
• UTF-8 bytes, vocabulary 256, no external tokenizer.
• Four local causal Transformer layers process every byte.
• Entropy boundaries compress the stream to about one patch per eight bytes, with capacity for 192 patches per 1024-byte sequence. Over-cap boundaries are uniformly coalesced across the sequence instead of truncating its tail.
• Three independent eight-layer global routes are trained cyclically, one full batch per route.
• The previous completed global patch conditions the local prediction, preventing future-token leakage.
• Muon optimizes hidden matrices; AdamW optimizes embeddings and normalization parameters.
• BF16 training on one NVIDIA RTX PRO 6000 Blackwell, with batch size selected by a live memory/throughput probe.

Default context is 1024 bytes. This is a research checkpoint, not an instruction model.

Data mixture

The stream is not redistributed in this repository:

• 58% FineWeb-Edu sample-10BT
• 22% FineMath finemath-4plus
• 10% Cosmopedia v2
• 7% OpenR1-Math-220k
• 3% decontaminated Codeforces reasoning traces

Review each upstream dataset card and license before redistribution or commercial use.

Checkpoints

checkpoints/step-XXXXXXXX/model.safetensors contains BF16 model weights. training_state.json reports live loss, throughput, token count, batch size, routing and patch statistics.

Evidence and limits

The precursor 998.7M-parameter prototype reached validation byte loss 1.235 versus 1.360 for a 988.1M dense baseline on the same TinyStories test. It used about 4.15x fewer estimated model FLOPs and ran 2.06x faster wall-clock. This short controlled result motivated full training; it is not evidence of 100x speedup, broad reasoning ability or production readiness.

Training code and architecture are included for auditability. Results may change while training is active.