May 5 19:44 Fix label leakage: temporal split — use first 70% of events as input, predict purchase in last 30%. Remove n_purchases/purchase_rate from features. 18:25 Fix model loading: use from_pretrained() instead of torch.load() for safetensors format 18:07 Add 03_ecommerce_finetune.ipynb — next-purchase prediction with JointFusion, LightGBM baseline comparison 18:06 Add e-commerce pre-training report — successful demo, behavioral clusters found, future improvements noted 16:34 Update 02_ecommerce notebook: add HF login, memory-free cell, subsample option for <64GB RAM machines 16:32 CRITICAL FIX: Switch from ByteLevel to Whitespace pre-tokenizer — fixes 42% UNK rate on domain token sequences Root cause: ByteLevel pre-tokenizer splits space-separated special tokens like
'[EVT_000] [PRICE_16]' into byte-level fragments ('Ġ[', 'PRICE', '_', '16', ']')
which don't match the special token vocabulary → 42% UNK tokens.
Fix: Use Whitespace pre-tokenizer which splits on spaces and preserves each token
as a whole unit. Special tokens like [EVT_000] are kept intact. Text fields like
'electronics.smartphone' are split on dots by BPE merges (not by pre-tokenizer).
Verified: 0% UNK on realistic domain token sequences after fix.
Also added HF login cell to the e-commerce notebook. Apr 30 22:37 Add 02_ecommerce_pretrain.ipynb — REES46 e-commerce pre-training with sequential entropy check, wandb, push to hub 22:26 Add finance pre-training report — honest analysis of results and lessons learned 21:43 Add .gitignore — Python, Jupyter, training artifacts, IDE files 21:38 Fix notebook: total_mem → total_memory, add hub_model_id push, add wandb logging support 16:14 Add 01_finance_pretrain.ipynb — Phase 3.1 notebook for pre-training on 5M Nigerian financial transactions 29 cells (18 code + 11 markdown):
- Loads electricsheepafrica/Nigerian-Financial-Transactions dataset from HF Hub
- Data profiling with matplotlib visualizations
- Converts to FINANCE_SCHEMA, groups by sender_account
- Builds hybrid domain tokenizer (97 special + BPE)
- Packs sequences, trains 24M DomainTransformer
- Loss curves, next-token predictions, t-SNE user embeddings
- Saves artifacts for fine-tuning notebook
Auto-detects GPU (L4/CPU) and adjusts batch size/epochs accordingly. 00:44 Phase 3.0: Pipeline validation demo on mindweave/bank-transactions-us — ALL 10 CHECKS PASSED Validated end-to-end on real public financial data:
- 3,232 transactions, 1 account, signed amounts, 4 description types
- 0% UNK tokens, 187 vocab (97 domain + BPE)
- 896 packed blocks × 64 tokens = 57,344 training tokens
- 815K param model: loss 5.38 → 1.09 in 30 epochs (78.7% reduction)
- Zero NaN/inf in losses or gradients 00:27 Add ADR-002: Dataset selection for Phase 3 demos — research findings, rationale, phased plan Apr 29 20:13 Update implementation report: add Phase 2D, update header to v0.4.0 / 139 tests, update cumulative summary and API 20:05 Add fine-tuning test suite — 15 tests covering dataset, batching, forward/backward, Trainer smoke, multiclass 20:04 Update package to v0.4.0 with fine-tuning exports 20:01 Update training init with finetune exports 20:00 Add finetune.py — finetune_domain_model (HF Trainer Pattern A, auto tabular_features passthrough) 20:00 Phase 2D: Fine-tuning pipeline — DomainFinetuneDataset, finetune_domain_model, 139 total tests passing Implements the supervised fine-tuning pipeline for JointFusionModel:
- finetune_data.py: DomainFinetuneDataset (per-user padded sequences + tabular features + labels)
- finetune.py: finetune_domain_model (HF Trainer Pattern A — auto-detects tabular_features)
- test_finetune.py: 15 tests covering dataset, batching, forward/backward, Trainer smoke, multiclass
- All 139 tests passing (72 tokenizer + 33 model + 19 pre-training + 15 fine-tuning) 19:30 Update README v0.3.0 — add usage example, update roadmap status, add implementation report link 19:28 Add Phase 2A-2C implementation report — technical decisions, architecture summary, test results 18:33 Add training test suite — 19 tests covering data pipeline, packing, collation, integration, Trainer smoke test 18:25 Update package to v0.3.0 with training exports 18:20 Add pretrain.py — pretrain_domain_model with HF Trainer, cosine schedule, DataCollatorForLanguageModeling 18:18 Add data_pipeline.py — tokenize_user_sequences, pack_sequences, prepare_clm_dataset 18:17 Phase 2C: Pre-training pipeline — data pipeline, sequence packing, HF Trainer CLM, 124 total tests passing Implements the pre-training framework:
- data_pipeline.py: tokenize_user_sequences, pack_sequences (run_clm.py pattern), prepare_clm_dataset
- pretrain.py: pretrain_domain_model with HF Trainer, DataCollatorForLanguageModeling, cosine schedule
- test_training.py: 19 tests covering tokenization, packing, collation, integration, 24-step smoke test
- All 124 tests passing (72 tokenizer + 33 model + 19 training) 17:50 Add model test suite — 33 tests covering config, model, PLR, DCNv2, joint fusion, integration 17:49 Update package init to v0.2.0 with model exports 17:49 Add DCNv2 + JointFusionModel (nuFormer-style Transformer + tabular fusion) 17:49 Add PLR embeddings (Gorishniy et al. 2022) 17:48 Add DomainTransformerForCausalLM — GPT-style NoPE model with SDPA attention, weight tying, HF Trainer compatible 17:47 Add DomainTransformerConfig with presets (24M/85M/330M) 17:46 Phase 2B: Model architecture — DomainTransformerForCausalLM (NoPE, GPT-style), PLR embeddings, DCNv2 + JointFusion, 105 passing tests Implements the full model architecture following Nubank nuFormer patterns:
- configuration.py: DomainTransformerConfig with presets (24M/85M/330M)
- modeling.py: GPT-style causal decoder with NoPE, SDPA attention, pre-norm, weight tying
- plr_embeddings.py: PeriodicLinearReLU numerical embeddings (Gorishniy et al. 2022)
- joint_fusion.py: DCNv2 + PLR + Transformer joint fusion (nuFormer-style)
- test_model.py: 33 tests covering config, model, PLR, DCNv2, joint fusion, integration
- All 105 tests passing (72 tokenizer + 33 model) 16:25 Add comprehensive test suite — 72 passing tests covering all components 16:24 Add predefined schemas (FINANCE, ECOMMERCE, HEALTHCARE) 16:23 Add domain_tokenizer.py — DomainTokenizerBuilder (core assembler, HF integration) 16:22 Add field_tokenizers.py — Sign, MagnitudeBucket, Calendar, Categorical, DiscreteNumerical tokenizers 16:20 Add tokenizers package init 16:20 Add schemas package init 16:19 Add schema.py — DomainSchema, FieldSpec, FieldType definitions 16:19 Phase 2A: Core tokenizer library — schema, field tokenizers, composite builder, predefined schemas, 72 passing tests Implements the domain tokenizer library following Nubank nuFormer patterns:
- schema.py: DomainSchema, FieldSpec, FieldType (declarative event schema)
- field_tokenizers.py: Sign, MagnitudeBucket, Calendar, Categorical, DiscreteNumerical
- domain_tokenizer.py: DomainTokenizerBuilder (assembles into HF PreTrainedTokenizerFast)
- predefined.py: FINANCE_SCHEMA (97 domain tokens, Nubank-compatible), ECOMMERCE_SCHEMA, HEALTHCARE_SCHEMA
- test_tokenizer.py: 72 tests covering schemas, individual tokenizers, full pipeline, end-to-end encoding 15:30 Update README: add ADR reference, update documentation table and repo structure 15:30 Add ADR-001: Implementation framework decision with detailed roadmap 15:10 Update README with Nubank case study and expanded repo structure 15:08 Add Nubank nuFormer reverse-engineering analysis — full pipeline reconstruction 14:48 Add README with project overview and vision 14:43 Add comprehensive research report on domain-specific tokenization 14:36 initial commit