---
language: [en, th]
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags: [faber-genesis, gqa, rope, swiglu, weight-sharing, cass, bf16, a100, th]
---

# FABER-Genesis-A100-Max (13.86B architecture)

Decoder-only transformer with weight sharing + CASS state/value/halt heads.

- hidden 6144, heads 48, KV 8, head-dim 128, SwiGLU FFN 16384
- vocab 32768, tied embeddings, RMSNorm, RoPE, GQA
- 35 physical blocks -> 42 logical transformations + 1 recurrent pass
- total params: **13.860526082B** (backbone + CASS)
- training recipe: BF16, AdamW8bit, activation checkpointing, microbatch=1, ctx=512

## Profiles

| profile | physical | params | status |
|---|---|---:|---|
| production-13.08B | 33 | 13.0804B | default for long runs (~73.55 GiB) |
| safe-13.47B | 34 | 13.4704B | squeeze with safety (~75.75 GiB) |
| max-13.86B | 35 | 13.8605B | capacity experiments (~77.94 GiB) |
| oom-14.25B | 36 | 14.2506B | OOM on single A100 AdamW8bit |

## What this repo contains

`config.json` (max profile), `modeling_faber_genesis.py` (self-contained HF
model code), `genesis_profiles.json`, `tokenizer_config.json` (placeholder
config, vocab 32768 — train a real tokenizer before pretraining),
`benchmark_genesis_a100.py`, `EVIDENCE_GENESIS_A100_MAX.md`.

No trained weights are included in this upload. `config.json` defines a
random-init architecture. To materialize weights:

```python
from modeling_faber_genesis import FaberGenesisConfig, FaberGenesisForCausalLM
cfg = FaberGenesisConfig()  # max-13.86B
model = FaberGenesisForCausalLM(cfg)  # random init, BF16 training recipe applies on A100
```

## Ceiling benchmark (run on A100 80GB)

```bash
pip install "torch==2.6.0" bitsandbytes transformers
python benchmark_genesis_a100.py --profile max-13.86B --steps 1
python benchmark_genesis_a100.py --sweep --n-list 25,29,32,33,34,35,36,37 --steps 1
```

Pass = fwd+bwd+optim step completes; n=36/37 must OOM. VRAM estimates in
`EVIDENCE_GENESIS_A100_MAX.md` are analytic, not a substitute for this run.

## Limitations

Random-init step-pass says nothing about Thai/code/reasoning quality.
Quality requires pretraining curriculum + CASS + verifier + agent training
and SWE/APEX evals. 14.25B+ is not a single-A100 resident AdamW8bit config.