-
Add data_pipeline.py — tokenize_user_sequences, pack_sequences, prepare_clm_dataset
-
Phase 2C: Pre-training pipeline — data pipeline, sequence packing, HF Trainer CLM, 124 total tests passing
Implements the pre-training framework:
- data_pipeline.py: tokenize_user_sequences, pack_sequences (run_clm.py pattern), prepare_clm_dataset
- pretrain.py: pretrain_domain_model with HF Trainer, DataCollatorForLanguageModeling, cosine schedule
- test_training.py: 19 tests covering tokenization, packing, collation, integration, 24-step smoke test
- All 124 tests passing (72 tokenizer + 33 model + 19 training) -
Update package init to v0.2.0 with model exports
-
Add DCNv2 + JointFusionModel (nuFormer-style Transformer + tabular fusion)
-
Add PLR embeddings (Gorishniy et al. 2022)
-
Add DomainTransformerForCausalLM — GPT-style NoPE model with SDPA attention, weight tying, HF Trainer compatible
-
Add DomainTransformerConfig with presets (24M/85M/330M)
-
Phase 2B: Model architecture — DomainTransformerForCausalLM (NoPE, GPT-style), PLR embeddings, DCNv2 + JointFusion, 105 passing tests
Implements the full model architecture following Nubank nuFormer patterns:
- configuration.py: DomainTransformerConfig with presets (24M/85M/330M)
- modeling.py: GPT-style causal decoder with NoPE, SDPA attention, pre-norm, weight tying
- plr_embeddings.py: PeriodicLinearReLU numerical embeddings (Gorishniy et al. 2022)
- joint_fusion.py: DCNv2 + PLR + Transformer joint fusion (nuFormer-style)
- test_model.py: 33 tests covering config, model, PLR, DCNv2, joint fusion, integration
- All 105 tests passing (72 tokenizer + 33 model) -
Add predefined schemas (FINANCE, ECOMMERCE, HEALTHCARE)
-
Add domain_tokenizer.py — DomainTokenizerBuilder (core assembler, HF integration)
-
Add field_tokenizers.py — Sign, MagnitudeBucket, Calendar, Categorical, DiscreteNumerical tokenizers
-
Add tokenizers package init
-
Add schemas package init
-
Add schema.py — DomainSchema, FieldSpec, FieldType definitions
-
Phase 2A: Core tokenizer library — schema, field tokenizers, composite builder, predefined schemas, 72 passing tests
Implements the domain tokenizer library following Nubank nuFormer patterns:
- schema.py: DomainSchema, FieldSpec, FieldType (declarative event schema)
- field_tokenizers.py: Sign, MagnitudeBucket, Calendar, Categorical, DiscreteNumerical
- domain_tokenizer.py: DomainTokenizerBuilder (assembles into HF PreTrainedTokenizerFast)
- predefined.py: FINANCE_SCHEMA (97 domain tokens, Nubank-compatible), ECOMMERCE_SCHEMA, HEALTHCARE_SCHEMA
- test_tokenizer.py: 72 tests covering schemas, individual tokenizers, full pipeline, end-to-end encoding
