Commit History
-
Fix tokenizer configuration for proper MLX compatibility
- Add all special tokens to additional_special_tokens and added_tokens_decoder - Add special tokens: <|system|>, <|user|>, <|assistant|>, <|end|> - Embed chat_template directly in tokenizer_config.json - Fix tokenizer_class from LlamaTokenizer to PreTrainedTokenizerFast - Add proper token ID mappings for Phi-3 format - Ensure compatibility with and without system prompts This update ensures proper tokenization for MLX inference with the Phi-3 chat format, supporting flexible usage with or without system prompts in multi-turn conversations. Tested with mlx-lm and verified correct token encoding. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>eaba2fa
-
Upload folder using huggingface_hub
Verified signature6b84460 -
initial commit
Verified signaturea7da84b