May 5 16:32 CRITICAL FIX: Switch from ByteLevel to Whitespace pre-tokenizer — fixes 42% UNK rate on domain token sequences Root cause: ByteLevel pre-tokenizer splits space-separated special tokens like
'[EVT_000] [PRICE_16]' into byte-level fragments ('Ġ[', 'PRICE', '_', '16', ']')
which don't match the special token vocabulary → 42% UNK tokens.
Fix: Use Whitespace pre-tokenizer which splits on spaces and preserves each token
as a whole unit. Special tokens like [EVT_000] are kept intact. Text fields like
'electronics.smartphone' are split on dots by BPE merges (not by pre-tokenizer).
Verified: 0% UNK on realistic domain token sequences after fix.
Also added HF login cell to the e-commerce notebook. Apr 29 20:04 Update package to v0.4.0 with fine-tuning exports 20:01 Update training init with finetune exports 20:00 Add finetune.py — finetune_domain_model (HF Trainer Pattern A, auto tabular_features passthrough) 20:00 Phase 2D: Fine-tuning pipeline — DomainFinetuneDataset, finetune_domain_model, 139 total tests passing Implements the supervised fine-tuning pipeline for JointFusionModel:
- finetune_data.py: DomainFinetuneDataset (per-user padded sequences + tabular features + labels)
- finetune.py: finetune_domain_model (HF Trainer Pattern A — auto-detects tabular_features)
- test_finetune.py: 15 tests covering dataset, batching, forward/backward, Trainer smoke, multiclass
- All 139 tests passing (72 tokenizer + 33 model + 19 pre-training + 15 fine-tuning) 18:25 Update package to v0.3.0 with training exports 18:20 Add pretrain.py — pretrain_domain_model with HF Trainer, cosine schedule, DataCollatorForLanguageModeling 18:18 Add data_pipeline.py — tokenize_user_sequences, pack_sequences, prepare_clm_dataset 18:17 Phase 2C: Pre-training pipeline — data pipeline, sequence packing, HF Trainer CLM, 124 total tests passing Implements the pre-training framework:
- data_pipeline.py: tokenize_user_sequences, pack_sequences (run_clm.py pattern), prepare_clm_dataset
- pretrain.py: pretrain_domain_model with HF Trainer, DataCollatorForLanguageModeling, cosine schedule
- test_training.py: 19 tests covering tokenization, packing, collation, integration, 24-step smoke test
- All 124 tests passing (72 tokenizer + 33 model + 19 training) 17:49 Update package init to v0.2.0 with model exports 17:49 Add DCNv2 + JointFusionModel (nuFormer-style Transformer + tabular fusion) 17:49 Add PLR embeddings (Gorishniy et al. 2022) 17:48 Add DomainTransformerForCausalLM — GPT-style NoPE model with SDPA attention, weight tying, HF Trainer compatible 17:47 Add DomainTransformerConfig with presets (24M/85M/330M) 17:46 Phase 2B: Model architecture — DomainTransformerForCausalLM (NoPE, GPT-style), PLR embeddings, DCNv2 + JointFusion, 105 passing tests Implements the full model architecture following Nubank nuFormer patterns:
- configuration.py: DomainTransformerConfig with presets (24M/85M/330M)
- modeling.py: GPT-style causal decoder with NoPE, SDPA attention, pre-norm, weight tying
- plr_embeddings.py: PeriodicLinearReLU numerical embeddings (Gorishniy et al. 2022)
- joint_fusion.py: DCNv2 + PLR + Transformer joint fusion (nuFormer-style)
- test_model.py: 33 tests covering config, model, PLR, DCNv2, joint fusion, integration
- All 105 tests passing (72 tokenizer + 33 model) 16:24 Add predefined schemas (FINANCE, ECOMMERCE, HEALTHCARE) 16:23 Add domain_tokenizer.py — DomainTokenizerBuilder (core assembler, HF integration) 16:22 Add field_tokenizers.py — Sign, MagnitudeBucket, Calendar, Categorical, DiscreteNumerical tokenizers 16:20 Add tokenizers package init 16:20 Add schemas package init 16:19 Add schema.py — DomainSchema, FieldSpec, FieldType definitions 16:19 Phase 2A: Core tokenizer library — schema, field tokenizers, composite builder, predefined schemas, 72 passing tests Implements the domain tokenizer library following Nubank nuFormer patterns:
- schema.py: DomainSchema, FieldSpec, FieldType (declarative event schema)
- field_tokenizers.py: Sign, MagnitudeBucket, Calendar, Categorical, DiscreteNumerical
- domain_tokenizer.py: DomainTokenizerBuilder (assembles into HF PreTrainedTokenizerFast)
- predefined.py: FINANCE_SCHEMA (97 domain tokens, Nubank-compatible), ECOMMERCE_SCHEMA, HEALTHCARE_SCHEMA
- test_tokenizer.py: 72 tests covering schemas, individual tokenizers, full pipeline, end-to-end encoding