Spaces:
Paused
Paused
Commit History
-
feat: Implement tier-based model filtering by device type
gary-boon +1: Claude Opus 4.5Add min_device field to model configs to specify minimum hardware requirement: - CodeGen 350M: min_device="cpu" (available to all users) - Code Llama 7B: min_device="gpu" (GPU users only) - Devstral Small 24B: min_device="gpu" (GPU users only) Update /models endpoint to filter models based on backend hardware: - GPU backends return all 3 models - CPU backends only return CodeGen 350M This ensures users only see models their backend can actually serve, preventing confusion and failed requests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>6bf9f5c
-
Add tokenSections boundaries and update system prompt
gary-boon +1: Claude Opus 4.5- Return tokenSections in research endpoint response with boundaries for system prompt, user prompt, and output sections - Estimate system prompt boundary for Devstral using MistralTokenizer - Update default system prompt to handle both code completion and instruction-style prompts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>c6f4cc5 -
Use mistral_common for proper Devstral prompt formatting
gary-boon +1: Claude Opus 4.5- Add mistral_common>=1.5.0 dependency - Update prompt_formatter to use MistralTokenizer for Devstral - Add recommended_temperature to model configs (0.15 for Devstral) - Remove hardcoded <s> from manual format (tokenizer adds BOS) The mistral_common library provides the correct chat template encoding that Devstral expects, which should fix the garbage token output. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>3e80769 -
Add system prompt support for instruction-tuned models
gary-boon +1: Claude Opus 4.5- Add prompt_style and system_prompt fields to ModelConfig - Create prompt_formatter.py service for unified prompt handling - Update research endpoint to use formatter with proper system prompts - Devstral now receives proper system + user message format This fixes the garbage token output from Devstral by properly formatting prompts for instruction-tuned models. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>2860768 -
fix: Add chat template support for Devstral instruct model
gary-boon +1: Claude Opus 4.5- Add uses_chat_template field to model configs - Apply tokenizer.apply_chat_template() for instruct models - Fallback to manual [INST] format if needed - Lower temperature to 0.15 for instruct models (better code output) Fixes garbage token generation when using Devstral without proper instruction formatting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>8d85da8
-
Add recommended_dtype to model configs
gary-boon +1: Claude Opus 4.5Each model now specifies its recommended dtype: - codegen-350m: fp16 - code-llama-7b: fp16 - devstral-small: bf16 (required for Mistral models) Model loader now uses recommended_dtype from config when TORCH_DTYPE env var is not explicitly set. This ensures Devstral automatically loads with bf16 without requiring manual configuration. Priority: TORCH_DTYPE env > model config > device default 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>62525b2 -
Phase 2: Add Devstral backend support
gary-boon +1: Claude Opus 4.5Add support for Mistral-based models (Devstral) with: - MistralAdapter in model_adapter.py for Mistral architecture - devstral-small config (40 layers, 32 Q heads, 8 KV heads, 131K vocab) - Percentage-based layer classification (works for any layer count) - Environment variable support: - DEFAULT_MODEL: which model to load (default: codegen-350m) - TORCH_DTYPE: bf16/fp16/fp32 (default: auto based on device) - MAX_CONTEXT: context length limit (default: 8192) - BATCH_SIZE: batch size (default: 1) - GET /models endpoint: list available models with hardware availability - GET /models/current endpoint: return currently loaded model info Devstral requires TORCH_DTYPE=bf16 when deployed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>9080f28
-
Add Code Llama 7B support with hardware-aware filtering and ICL timeout fixes
gary-boon +1: Claude- Added multi-architecture support to ICL components (attention extractor, service, induction detector) - Implemented hardware-aware model filtering for CPU/GPU spaces - Fixed Code Llama tokenizer padding token configuration - Updated model config with accurate Code Llama 7B specifications - Added model adapter pattern for seamless architecture switching 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>ed40a9a