-
Add DCNv2 + JointFusionModel (nuFormer-style Transformer + tabular fusion)
-
Add PLR embeddings (Gorishniy et al. 2022)
-
Add DomainTransformerForCausalLM — GPT-style NoPE model with SDPA attention, weight tying, HF Trainer compatible
-
Add DomainTransformerConfig with presets (24M/85M/330M)
-
Phase 2B: Model architecture — DomainTransformerForCausalLM (NoPE, GPT-style), PLR embeddings, DCNv2 + JointFusion, 105 passing tests
Implements the full model architecture following Nubank nuFormer patterns:
- configuration.py: DomainTransformerConfig with presets (24M/85M/330M)
- modeling.py: GPT-style causal decoder with NoPE, SDPA attention, pre-norm, weight tying
- plr_embeddings.py: PeriodicLinearReLU numerical embeddings (Gorishniy et al. 2022)
- joint_fusion.py: DCNv2 + PLR + Transformer joint fusion (nuFormer-style)
- test_model.py: 33 tests covering config, model, PLR, DCNv2, joint fusion, integration
- All 105 tests passing (72 tokenizer + 33 model)