Spaces:
Paused
Paused
Commit History
-
Fix empty text and incorrect is_special for Mistral control tokens
gary-boon +1: Claude Opus 4.6 (1M context)mistral-common's decode_token() returns "" for chat-template tokens (<s>, [INST], [/INST], [SYSTEM_PROMPT], [/SYSTEM_PROMPT], tool markers). Fall back to the HF tokenizer so every token arrives with a printable string form. Also widen special_token_ids from {eos, bos, pad, unk} to include all control-token IDs from mistral-common's Tekkenizer, fixing is_special for chat-template delimiters. Applied to both streaming and non-streaming endpoints, plus the logit-candidate decode helper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b5e4add
-
Fix MistralTokenizer not loaded during model switch
gary-boon +1: Claude Opus 4.5The switch_model endpoint was not creating the MistralTokenizer, causing special tokens to be decoded incorrectly when switching from CodeGen to Devstral. Also adds attention overlay feature changes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>2bdf299
-
Integrate mistral-common for correct Devstral tokenization
gary-boon +1: Claude Opus 4.5Root cause: Devstral's Tekken tokenizer is incompatible with HuggingFace's standard tokenization. When [INST]/[/INST] are formatted as text and tokenized with HF, the model receives corrupted tokens -> garbage output. Solution: - Add mistral-common>=1.5.0 dependency - Create MistralTokenizerWrapper using MistralTokenizer.from_hf_hub() - Use encode_chat_completion() for correct Tekken token encoding - Relax numpy/pydantic version constraints for compatibility The mistral-common library produces correct token sequences by encoding chat messages directly to token IDs, bypassing text-based formatting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>ed06dcb