Hugging Face's logo

Spaces:
Ethosoft
/
NedoTokenizer-Demo
Running

App Files Community
NedoTokenizer-Demo / CORE_SOURCE_COMMIT

Commit History

  1. release: deploy 20GB max48 vocabulary

    Ethosoft TRUBA Deploy
    5f4471c

  1. release: deploy root-hard suffix-soft tokenizer

    Ethosoft TRUBA Deploy
    f6987ef

  2. fix: restore morphology tokens and readable unicode

    Ethosoft TRUBA Deploy
    f1d80ed

  3. release: deploy lexical byte-BPE tokenizer demo

    Ethosoft TRUBA Deploy
    7ae0777

  4. docs: identify real surface token semantics

    Ethosoft TRUBA Deploy
    a77c3b4

  1. Update demo to lexical-exception-free core

    Ethosoft TRUBA Deploy
    6805e05

  1. Deploy accepted NedoTokenizer core as static WASM demo

    Ethosoft TRUBA Deploy
    03058ad