Spaces:
Build error
Build error
Commit History
-
Add skin and tokenizer systems with parsing and tokenization functionalities
- Implemented `SkinSystem` and `SkinWriter` for skin prediction and data export. - Created `TokenizerPart` and associated classes for tokenization processes. - Added parsing functions for both skin and tokenizer configurations. - Introduced utility functions for discretizing and undiscertizing data. - Established a structure for handling different tokenizer methods. - Included error handling and assertions for robustness in data processing.11b119e