Spaces:
Paused
Paused
Commit History
-
Backend: ephemeral flag on analyze stream — leave no cache trace
gary-boon +1: Claude Opus 4.7 (1M context)The What-if Rewrite & rerun re-uses /analyze/research/attention/stream to regenerate text with an edited prompt. That endpoint unconditionally called matrix_cache.clear_old_requests(new_id) on every invocation, evicting the original traced run's matrices and hidden states — so opening the Mechanism / Attention lenses afterwards surfaced "Matrix data not found. Cache may have expired." Adds an opt-in ephemeral=true field on the stream request body. When true, the handler skips every cache touch: - clear_old_requests (keep prior runs intact) - set_request_metadata - set_step_query_info - hidden_state_cache.store_step / store_input_ids - matrix_cache.store (per-layer, per-head) Default behaviour (no flag) is byte-identical to before, so every existing caller — the main "Generate trace" path, the non-streaming sibling, every other endpoint — is untouched. Only the What-if regen opts in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>88879ce -
Backend: add mask_positions intervention for What-if "Test this token"
gary-boon +1: Claude Opus 4.7 (1M context)The frontend's What-if lens "Test this token" mode posts intervention_type=mask_positions with a list of arbitrary prompt positions to mask, but the backend only ever implemented the three contiguous-region masks (mask_system / mask_user_span / mask_generated). Requests hit the "Unknown intervention type" fallthrough. Adds a mask_positions branch to /analyze/intervention: - params.positions: list[int] (clamped to valid range, deduped) - params.selected_token_id: int (optional; defaults to original step's winner — the token whose logit delta we track) - params.top_k: int (default 8, clamped 1–50) Zeroes those positions in the attention mask, re-runs the forward pass, returns the standard InterventionResponse with a rich details dict matching the frontend's AblationResult interface: mask_type, mask_positions_count, ablated_positions, seq_len, prompt_len, selected_token_id / selected_token, selected_logit_original / _ablated / _delta, and top_k_after_ablation with per-en…f578686
-
Backend: unembedding cosine emission + Devstral tokenizer fallback
Two changes that accumulated together in the working tree. 1. Unembedding cosine decomposition emitted per top-K alternative For Step 8's "How this token won the vote" disclosure, the backend now decomposes each alternative's logit into the geometric factors: logit_v = cos(W_E[v], h_final) × ||W_E[v]|| × ||h_final|| Cosine similarity and the unembedding row norm (||W_E[v]||) are appended to every entry in the logits payload — both in the regular and outlier paths, in both the non-SSE and SSE generation loops. Without this, the SSE endpoint (which the frontend uses) returned no cosine data while the non-SSE endpoint did, leading to a silent "cosine not available" state in the lens. Logging on cosine prep upgraded to info-level on success and warning-level on failure so future regressions are visible. 2. Tokenizer fallback for Devstral on tokenizers >= 0.20 Devstral-Small-2507 ships only tekken.json. AutoTokenizer with newer tokenizers is …3a10cf3
-
Emit per-token embedding L2 norms for RQ1 Step 3
Adds a `_compute_embedding_norms` helper that bounds memory by unique token IDs (`torch.unique` + `weight.index_select`) so long contexts don't allocate `[1, seq_len, hidden_size]` × 2 unnecessarily. Both analyse endpoints now return `embeddingNorms: List[float]` covering the prompt + generated tokens, used by the new code-first Mechanism lens to surface real per-token embedding magnitudes instead of synthetic illustrative values. Also extends MatrixCache with `get_aggregate_attention_matrix()` (mean / max pooling across heads) so the Step 4 attention view can fetch the aggregate matrix from a single endpoint instead of synthesising it on the client. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>336085f -
Emit per-layer top-K SwiGLU intermediate activations for RQ1 Step 5
Adds a forward hook on Mistral/LLaMA-style FFN modules that captures the top-K (=32) hidden neurons of `silu(W_gate · x) ⊙ (W_up · x)` at the predicting (last) position for each generation step. Stored on each layer entry as `ffn_top_neurons: { k, intermediate_size, neurons: [{idx, value}] }`. Memory-bounded: top-32 of 14,336 ≈ 256 bytes per layer per step, versus the 57KB-per-layer-per-step the full intermediate vector would require. The top-K is what the RQ1 frontend's "Behind the SwiGLU" disclosure needs to render an honest per-neuron view rather than synthesising activations or showing only summary stats. Only registers on layers exposing both `gate_proj` and `up_proj` (SwiGLU); other architectures fall back to the existing `gate_stats` / `mlp_output_norm` / `ffn_contribution` summary fields. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>019492b
-
Emit effective_probability and sampling eligibility per alternative
gary-boon`alternatives[].probability` is the post-temperature, PRE-filter probability — the model's preference. The sampler draws from `probs_filtered`, the post-top-k/top-p renormalised distribution. With filtering active, the panel was reporting margins and per-token probabilities that didn't reflect the actual decision. For each alternative now emit: - `effective_probability`: post-filter renormalised probability (what the sampler actually drew from). 0 if filtered out. Equals `probability` when no filter is active. - `is_eligible`: convenience flag (`effective_probability > 0`). Sampling metadata gains: - `is_filtered`: bool, true if top_k or top_p constrained the set. - `eligible_count`: count of tokens in the eligible set. The frontend can now show one distribution when defaults are in effect, or two (model preference vs sampling-effective) when filtering is active. Same change applied to both SSE and non-SSE analysis paths.6f43bd6 -
Expose query_position and query_token_id in /attention/row response
gary-boonThe cached attention[-1] is the row of the *predicting* query position (the position whose logits selected the step-s token), not the predicted token's self-attention. The frontend was labelling this row as "attention influences on this token", which implies a self-attention relationship the data does not represent. Record (query_position, query_token_id) per generation step on the MatrixCache during analysis (both SSE and non-SSE paths). Surface them in the /attention/row response so the Why this token? panel can show explicit provenance ("Attention used to predict this token, from query position N — `<token>`") instead of the misleading framing. This is a documentation/honesty fix, not a data fix — the underlying attention values are unchanged. It supports the project's stated "grounded hypotheses, not causal claims" framing.d2d962d -
Derive aggregate-row num_heads from request metadata, not live model
gary-boonThe /attention/row aggregate endpoint was reading num_attention_heads from manager.model.config at request time. After a model switch, that returns the *new* model's head count even when serving rows from the *previous* model's cached attention. The per-head row data is correct (stored under request_id keys) but the aggregation averaged the wrong number of heads, silently producing wrong values. Record per-request metadata (num_heads, num_layers, model_id) at analysis time on the MatrixCache. get_aggregate_row derives num_heads from this metadata; the explicit argument is retained as an optional override for tests. The endpoint no longer touches manager.model when serving aggregate rows. clear_request and clear_old_requests drop metadata alongside the per-head cache entries.3506846
-
Fix empty text and incorrect is_special for Mistral control tokens
gary-boon +1: Claude Opus 4.6 (1M context)mistral-common's decode_token() returns "" for chat-template tokens (<s>, [INST], [/INST], [SYSTEM_PROMPT], [/SYSTEM_PROMPT], tool markers). Fall back to the HF tokenizer so every token arrives with a printable string form. Also widen special_token_ids from {eos, bos, pad, unk} to include all control-token IDs from mistral-common's Tekkenizer, fixing is_special for chat-template delimiters. Applied to both streaming and non-streaming endpoints, plus the logit-candidate decode helper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b5e4add
-
Fix score-all classification in vectorised path
gary-boon +1: Claude Opus 4.6 (1M context)The vectorised GPU path (used for most layers) still had the old cascade with "semantic"/"positional" catch-alls. Updated to match the score-all-then-rank system with behaviour types + code cues. Also fixed self reference error — code runs in a standalone async function, not a class method. Use dict-based cache instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>e375e45
-
Refactor head classification from cascade to score-all-then-rank
gary-boon +1: Claude Opus 4.6 (1M context)Replace priority-ordered first-match cascade with two-dimension system: Behaviour type (attention geometry) — all scored simultaneously: - attention_sink, previous_token, local, induction, focused, diffuse - Primary = highest qualifying score (with min thresholds), secondary = runner-up - "focused" and "diffuse" replace weak "positional"/"semantic" catch-alls Code cue (separate dimension — code token relevance): - delimiter_sensitive: attention to (){}[]:;, - keyword_sensitive: attention to language keywords (def, return, if, etc.) - pattern_reuse: derived from induction signal (repeated spans) - Each scored by proportion of attention mass on target token class - Evidence text included (e.g. "34% attention on delimiter sensitive tokens") Per-head output now includes: - pattern (primary behaviour type + confidence) - secondary_behaviour (type + score) - code_cue (type + score + evidence text) - secondary_cue (type + score) Token texts decoded and cached per generation step for code-…10baadf -
Add FFN contribution metrics and gate activation stats to research endpoint
gary-boon +1: Claude Opus 4.6 (1M context)- Compute attn_contribution/ffn_contribution ratios from existing attention and MLP output norm hooks (no new forward passes needed) - Add gate activation hook for SwiGLU models (LLaMA/Mistral) capturing sparsity, mean, and max of gate values per layer per token - Expose ffnType and intermediateSize in modelInfo response - Clear gate_activation_stats each generation step alongside existing dicts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>82349c1
-
Add tuned lens as supplementary projection mode for logit lens
gary-boon +1: Claude Opus 4.6Introduces per-layer affine probes (trained to minimise KL divergence with the final layer) that correct for subspace mismatch in early transformer layers. Positioned alongside the existing raw logit lens with a frontend toggle, allowing developers to compare projections and assess whether observed commitment patterns are robust to projection choice. New: backend/tuned_lens.py (runtime), scripts/train_tuned_lens.py (training) Modified: model_service.py (load at startup, tuned lens computation, tuned commitment summary, modelInfo.tunedLensAvailable, health endpoint) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>6f48db0 -
Rename 'counterfactual' terminology to 'input ablation' in docstrings
gary-boon +1: Claude Opus 4.6Aligns with extension terminology update — these are ablation studies, not true counterfactuals. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>0d76811 -
Fix margin winner_token in margin_data and extract _classify_stability helper
gary-boon +1: Claude Opus 4.6The Decision Stability display was showing the sampled token as the winner rather than the logit-top token. When temperature changed the pick (e.g. "(number" was logit winner but "(n" was sampled), both sides showed the same token. Now includes winner_token in margin_data from logits_entries[0]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>ecf49a2 -
Add margin-based decision analysis, interventional counterfactuals, and run comparison (v3.0)
gary-boon +1: Claude Opus 4.6Phase 1: Per-token margin computation with stability classification (stable/moderate/boundary/fragile), layer-wise margin tracking via logit lens, commitment layer detection, flip event detection, causal margin contribution decomposition per layer ((W_U[winner] - W_U[runner-up]) · residual_ℓ), and hidden state caching for intervention reuse. Phase 2: POST /analyze/intervention endpoint supporting mask_system, mask_user_span, mask_generated (real forward-pass re-evaluation with attention_mask), temperature_sweep (cached logits), layer_ablation, head_ablation, and expert_mask. Returns margin shift, stability change, and winner change with full diagnostics. Phase 3: POST /analyze/compare endpoint for per-token margin diffs between cached runs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>121a2d9
-
Add deep inspection: data-driven pattern classification, attention/MLP tracking, logit lens
gary-boon +1: Claude Opus 4.6- Replace hardcoded position-based head/layer pattern classification with data-driven detection (sink, previous-token, local, induction, positional, semantic) - Layer patterns computed as confidence-weighted majority vote of head patterns - Add forward hooks for attention and MLP output norms per layer - Add logit lens computation at sampled layers (every n_layers//5) - Compute per-head sink weight, local weight, and induction weight metrics Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>54d9b6e
-
Change default model to Devstral and optimise attention extraction
gary-boon +1: Claude Opus 4.6Switch the default model fallback from codegen-350m to devstral-small, matching the instruction-tuned model used for PhD research. Also includes batched tensor operations for attention extraction (reduces GPU→CPU sync points from ~4000 to ~40 per token). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>d8d197a
-
Fix MistralTokenizer loading and token section boundaries
gary-boon +1: Claude Opus 4.5- Load MistralTokenizer in switch_model() for correct Tekken encoding - Fix token section boundary to include [/SYSTEMPROMPT] in system section - Add attention row endpoint for overlay visualization Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>76b614d
-
Fix MistralTokenizer not loaded during model switch
gary-boon +1: Claude Opus 4.5The switch_model endpoint was not creating the MistralTokenizer, causing special tokens to be decoded incorrectly when switching from CodeGen to Devstral. Also adds attention overlay feature changes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>2bdf299
-
feat: add top-k/top-p sampling and detailed logits/probability tracking
gary-boon +1: Claude Opus 4.5- Add top_k and top_p sampling parameters to research attention endpoints - Capture raw logits before temperature scaling for visualization - Track greedy token (argmax) separately from sampled token - Add raw probabilities (T=1) for comparison in UI - Track cumulative probability and rank for each token alternative - Add sampling metadata (temperature, top_k, top_p, was_greedy) - Handle selected token outliers when not in top-N display Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>2c6343b
-
fix: Improve token alternatives numerical stability and temperature control
gary-boon +1: Claude Opus 4.5- Use log_softmax instead of softmax for numerically stable probability computation at low temperatures (fixes underflow showing 0% for alternatives) - Use topk on logits instead of probs to correctly identify top tokens regardless of temperature (fixes showing special tokens like <unk>) - Remove backend temperature override - now controlled by frontend UI - Use correct tokenizer (MistralTokenizer) for Devstral model when decoding alternative token texts in SSE endpoint 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>bc1f0e0
-
Store matrices as numpy arrays instead of Python lists
gary-boon +1: Claude Opus 4.5Memory optimization for large models like Devstral: - Store attention/QKV matrices as numpy arrays (~4 bytes/float) - Previously used Python lists (~28 bytes/float = 7x overhead) - Convert to lists only when frontend requests via /matrix endpoint This reduces cache memory from ~25GB to ~3.5GB for: 40 layers × 32 heads × 32 steps × ~90KB per head 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>f2e89c2 -
Add per-step memory cleanup for large model support
gary-boon +1: Claude Opus 4.5- Delete outputs, logits, probs tensors after each generation step - Run garbage collection every 8 steps for large models - Clear MPS cache on Apple Silicon to release GPU memory This prevents memory accumulation during generation with models like Devstral that have 40 layers × 32 heads, which previously caused RAM exhaustion on longer token sequences (32+). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>a94eb19 -
Fix RAM exhaustion for large token generation
gary-boon +1: Claude Opus 4.5Add memory management to MatrixCache: - Track request IDs for each cache entry - Clear old request cache entries before starting new analysis - Force garbage collection after clearing cache - Clear GPU/MPS cache on Apple Silicon This prevents memory accumulation when running multiple analyses, particularly for large models like Devstral with 40 layers × 32 heads per step. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>959074d -
feat: add auto_complete parameter for token generation
gary-boon +1: Claude Opus 4.5- Add auto_complete parameter to streaming and non-streaming endpoints - Cap max_tokens at 128 when auto_complete is enabled - Existing EOS early-stop logic handles completion detection 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>bb689ce -
fix: add QKV extraction support for Mistral/Devstral architecture
gary-boon +1: Claude Opus 4.5- Add make_separate_proj_hook for models with separate q_proj, k_proj, v_proj - Handle GQA (Grouped Query Attention) by expanding K/V heads to match Q heads - Support both CodeGen (combined qkv_proj) and Mistral (separate projections) - Update both streaming and non-streaming endpoints 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>d1d37a8 -
feat: implement lazy-loading for attention matrices
gary-boon +1: Claude Opus 4.5- Add MatrixCache class with 60-min TTL for storing attention/QKV matrices - Modify response builder to cache matrices instead of including in payload - Add new endpoint /analyze/research/attention/matrix for on-demand retrieval - Include requestId in responses for cache lookup - Reduces initial response from 400MB+ to ~500KB-1MB 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>929ba88
-
Add avg_entropy calculation for attention heads
gary-boon +1: Claude Opus 4.5- Compute normalized attention entropy averaged over query positions - Normalize by log(k_i) where k_i = number of keys each position can attend to - Average over latter half of positions for more stable signal - Return both entropy (last-token) and avg_entropy fields for each head 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>66a46b6
-
Revert QKV visualization fixes - need better approach for data streaming
gary-boon +1: Claude Opus 4.5Reverts commits: - a79cb83 Add safety checks for missing QKV keys - decb5ab Limit QKV matrices to top 5 heads per layer - 9056859 Fix QKV matrix extraction for Mistral/Devstral architecture - 4ec134b Fix QKV visualization for Mistral/Devstral architecture The QKV fixes caused response sizes to explode, causing 504 timeouts. Need to implement a better approach (lazy loading) before re-enabling. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>d0b7e29 -
Add safety checks for missing QKV keys
gary-boon +1: Claude Opus 4.5If a hook fails silently, the layer's qkv_captures dict might exist but be missing the 'q', 'k', or 'v' keys. This would cause a KeyError. Now we check for each key's existence before accessing it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>a79cb83 -
Limit QKV matrices to top 5 heads per layer to reduce response size
gary-boon +1: Claude Opus 4.5The QKV visualization fix caused response sizes to explode because we were sending Q/K/V matrices for all 32 heads × 40 layers = 1280 sets of matrices. For a 50 token sequence with head_dim=128, that's ~25 million floats per token step, causing 504 Gateway Timeouts in streaming. Now only the top 5 heads (by attention weight) per layer retain their QKV matrices. This reduces the QKV data by ~85% while still providing meaningful visualization data for the most important attention patterns. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>decb5ab -
Fix QKV matrix extraction for Mistral/Devstral architecture
gary-boon +1: Claude Opus 4.5- Use explicit head_dim from model config (Mistral uses 128, not hidden_size/num_heads=160) - Handle both tuple and tensor outputs from projection hooks - Properly reshape Q/K/V tensors to [seq_len, num_heads, head_dim] - Support GQA by expanding K/V heads (8) to match Q heads (32) - Add warning logs for debugging hook failures This fixes the "too many indices for tensor of dimension 2" and shape mismatch errors when extracting QKV matrices from Devstral. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>9056859 -
Fix QKV visualization for Mistral/Devstral architecture
gary-boon +1: Claude Opus 4.5- Add separate Q, K, V hook makers for Mistral-style architectures that use separate projections (q_proj, k_proj, v_proj) - Update layer iteration to use adapter's _get_layers() method for architecture-agnostic layer access - Maintain backwards compatibility with CodeGen's combined QKV projection - Apply same fix to both regular and streaming endpoints This fixes the QKV visualization which was silently failing for Devstral because the hook registration only supported CodeGen's transformer.h structure. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>4ec134b
-
Fix: Import time module at top level for SSE events
gary-boon +1: Claude Opus 4.5The sse_event() helper function uses time.time() but time was only imported inside the event_generator function, causing NameError. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>15a862b -
Add SSE streaming endpoint for real-time analysis progress
gary-boon +1: Claude Opus 4.5Add /analyze/research/attention/stream endpoint that streams progress events during attention analysis via Server-Sent Events (SSE). Progress events are emitted at each stage: - tokenizing: After prompt tokenization, with token count - generating: Per-token, with step index and total steps - extracting: Per-layer (on last token), with layer/head counts - serializing: During response building, with size estimate - complete: When analysis finishes, with timing and size metadata - result: Final payload with complete analysis data Each event includes: - type, stage, totalStages, progress, stageProgress, detail - metadata object with stage-specific info (counts, sizes, timing) Benefits: - Frontend gets real progress updates instead of fake animation - Users see exactly what's happening: "Processing layer 15/40" - Response size displayed during serialization - No more "stuck at 95%" - progress reflects actual work 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Author…172a186 -
feat: Include token metadata in analysis response
gary-boon +1: Claude Opus 4.5Extend promptTokens and generatedTokens with full metadata: - bpe_pieces: BPE breakdown of each token - is_special: Whether token is a special token (eos/bos/pad/unk) - is_multi_split: Whether token is part of a multi-split identifier - num_pieces: Number of BPE pieces This eliminates the need for per-token /token/metadata API calls, reducing backend requests and enabling instant hover tooltips. Also removes debug logging from /token/metadata endpoint. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>ee0f6c9 -
feat: Implement tier-based model filtering by device type
gary-boon +1: Claude Opus 4.5Add min_device field to model configs to specify minimum hardware requirement: - CodeGen 350M: min_device="cpu" (available to all users) - Code Llama 7B: min_device="gpu" (GPU users only) - Devstral Small 24B: min_device="gpu" (GPU users only) Update /models endpoint to filter models based on backend hardware: - GPU backends return all 3 models - CPU backends only return CodeGen 350M This ensures users only see models their backend can actually serve, preventing confusion and failed requests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>6bf9f5c -
Fix: Add attn_implementation="eager" to model switch function
gary-boon +1: Claude Opus 4.5When switching models via /models/switch endpoint, the model was being loaded without attn_implementation="eager", causing Devstral/Mistral models to return None for attention weights. This broke the Research Attention Analyzer with error: "TypeError: object of type 'NoneType' has no len()" Added trust_remote_code=True and attn_implementation="eager" to match the initial model loading configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>f94a7ae
-
Add tokenSections boundaries and update system prompt
gary-boon +1: Claude Opus 4.5- Return tokenSections in research endpoint response with boundaries for system prompt, user prompt, and output sections - Estimate system prompt boundary for Devstral using MistralTokenizer - Update default system prompt to handle both code completion and instruction-style prompts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>c6f4cc5 -
Fix: Handle MistralCommonTokenizer pad_token setter
gary-boon +1: Claude Opus 4.5MistralCommonTokenizer (used by Devstral) doesn't allow setting pad_token property. Wrap in try/except to handle this gracefully. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>e20ccaf -
Integrate mistral-common for correct Devstral tokenization
gary-boon +1: Claude Opus 4.5Root cause: Devstral's Tekken tokenizer is incompatible with HuggingFace's standard tokenization. When [INST]/[/INST] are formatted as text and tokenized with HF, the model receives corrupted tokens -> garbage output. Solution: - Add mistral-common>=1.5.0 dependency - Create MistralTokenizerWrapper using MistralTokenizer.from_hf_hub() - Use encode_chat_completion() for correct Tekken token encoding - Relax numpy/pydantic version constraints for compatibility The mistral-common library produces correct token sequences by encoding chat messages directly to token IDs, bypassing text-based formatting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>ed06dcb -
Remove mistral_common to fix dependency conflict
gary-boon +1: Claude Opus 4.5- Remove mistral_common from requirements (conflicts with numpy/pydantic) - Simplify prompt_formatter to use manual Mistral format - Format: [INST] {system}\n\n{user} [/INST] (no <s>, tokenizer adds BOS) - Keep recommended_temperature=0.15 for Devstral 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>3d9d9ee -
Use mistral_common for proper Devstral prompt formatting
gary-boon +1: Claude Opus 4.5- Add mistral_common>=1.5.0 dependency - Update prompt_formatter to use MistralTokenizer for Devstral - Add recommended_temperature to model configs (0.15 for Devstral) - Remove hardcoded <s> from manual format (tokenizer adds BOS) The mistral_common library provides the correct chat template encoding that Devstral expects, which should fix the garbage token output. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>3e80769 -
Add system prompt support for instruction-tuned models
gary-boon +1: Claude Opus 4.5- Add prompt_style and system_prompt fields to ModelConfig - Create prompt_formatter.py service for unified prompt handling - Update research endpoint to use formatter with proper system prompts - Devstral now receives proper system + user message format This fixes the garbage token output from Devstral by properly formatting prompts for instruction-tuned models. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>2860768 -
fix: Simpler prompt format and temperature=0 for Devstral
gary-boon +1: Claude Opus 4.5- Remove newlines from instruction (they become <SPECIAL_*> tokens) - Add explicit <s> BOS token to manual format - Use temperature=0 for fully deterministic output - Clearer instruction: "Continue this Python code:" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>76020ee -
fix: Sanitize JSON response for NaN/Inf float values
gary-boon +1: Claude Opus 4.5Add sanitize_for_json() helper that recursively traverses response and replaces NaN/Infinity values with null. Fixes JSON serialization errors when attention/entropy calculations produce invalid floats. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>99f6209 -
fix: Check chat_template is set before using apply_chat_template
gary-boon +1: Claude Opus 4.5The tokenizer has the apply_chat_template method but no template configured. Now we check tokenizer.chat_template is not None before attempting to use it, falling back to manual [INST] format. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>474927d -
fix: Add chat template support for Devstral instruct model
gary-boon +1: Claude Opus 4.5- Add uses_chat_template field to model configs - Apply tokenizer.apply_chat_template() for instruct models - Fallback to manual [INST] format if needed - Lower temperature to 0.15 for instruct models (better code output) Fixes garbage token generation when using Devstral without proper instruction formatting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>8d85da8
-
fix: Convert bfloat16 to float32 for numpy compatibility
gary-boon +1: Claude Opus 4.5NumPy doesn't support bfloat16 directly. Add .float() conversion before .numpy() calls in attention/activation extraction. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>cb6f39c