Commit History

  1. CRITICAL FIX: Switch from ByteLevel to Whitespace pre-tokenizer β€” fixes 42% UNK rate on domain token sequences⁠

    Verified signature
    a9c4a62

  1. Add domain_tokenizer.py β€” DomainTokenizerBuilder (core assembler, HF integration)

    Verified signature
    818a2e9
  2. Add field_tokenizers.py β€” Sign, MagnitudeBucket, Calendar, Categorical, DiscreteNumerical tokenizers

    Verified signature
    511f3aa
  3. Add tokenizers package init

    Verified signature
    0b06df3