Spaces:
Paused
Paused
Commit History
-
Backend: ephemeral flag on analyze stream — leave no cache trace
gary-boon +1: Claude Opus 4.7 (1M context)The What-if Rewrite & rerun re-uses /analyze/research/attention/stream to regenerate text with an edited prompt. That endpoint unconditionally called matrix_cache.clear_old_requests(new_id) on every invocation, evicting the original traced run's matrices and hidden states — so opening the Mechanism / Attention lenses afterwards surfaced "Matrix data not found. Cache may have expired." Adds an opt-in ephemeral=true field on the stream request body. When true, the handler skips every cache touch: - clear_old_requests (keep prior runs intact) - set_request_metadata - set_step_query_info - hidden_state_cache.store_step / store_input_ids - matrix_cache.store (per-layer, per-head) Default behaviour (no flag) is byte-identical to before, so every existing caller — the main "Generate trace" path, the non-streaming sibling, every other endpoint — is untouched. Only the What-if regen opts in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>88879ce -
Backend: add mask_positions intervention for What-if "Test this token"
gary-boon +1: Claude Opus 4.7 (1M context)The frontend's What-if lens "Test this token" mode posts intervention_type=mask_positions with a list of arbitrary prompt positions to mask, but the backend only ever implemented the three contiguous-region masks (mask_system / mask_user_span / mask_generated). Requests hit the "Unknown intervention type" fallthrough. Adds a mask_positions branch to /analyze/intervention: - params.positions: list[int] (clamped to valid range, deduped) - params.selected_token_id: int (optional; defaults to original step's winner — the token whose logit delta we track) - params.top_k: int (default 8, clamped 1–50) Zeroes those positions in the attention mask, re-runs the forward pass, returns the standard InterventionResponse with a rich details dict matching the frontend's AblationResult interface: mask_type, mask_positions_count, ablated_positions, seq_len, prompt_len, selected_token_id / selected_token, selected_logit_original / _ablated / _delta, and top_k_after_ablation with per-en…f578686
-
Merge pull request #6 from gary-boon/feature/rq1-ffn-intermediate
gary-boonUnverified signatureBackend: emit per-token embedding norms, top-K SwiGLU neurons, unembedding cosine + Devstral tokenizer fallbacka1b2ad2 -
Backend: unembedding cosine emission + Devstral tokenizer fallback
Two changes that accumulated together in the working tree. 1. Unembedding cosine decomposition emitted per top-K alternative For Step 8's "How this token won the vote" disclosure, the backend now decomposes each alternative's logit into the geometric factors: logit_v = cos(W_E[v], h_final) × ||W_E[v]|| × ||h_final|| Cosine similarity and the unembedding row norm (||W_E[v]||) are appended to every entry in the logits payload — both in the regular and outlier paths, in both the non-SSE and SSE generation loops. Without this, the SSE endpoint (which the frontend uses) returned no cosine data while the non-SSE endpoint did, leading to a silent "cosine not available" state in the lens. Logging on cosine prep upgraded to info-level on success and warning-level on failure so future regressions are visible. 2. Tokenizer fallback for Devstral on tokenizers >= 0.20 Devstral-Small-2507 ships only tekken.json. AutoTokenizer with newer tokenizers is …3a10cf3
-
Emit per-token embedding L2 norms for RQ1 Step 3
Adds a `_compute_embedding_norms` helper that bounds memory by unique token IDs (`torch.unique` + `weight.index_select`) so long contexts don't allocate `[1, seq_len, hidden_size]` × 2 unnecessarily. Both analyse endpoints now return `embeddingNorms: List[float]` covering the prompt + generated tokens, used by the new code-first Mechanism lens to surface real per-token embedding magnitudes instead of synthetic illustrative values. Also extends MatrixCache with `get_aggregate_attention_matrix()` (mean / max pooling across heads) so the Step 4 attention view can fetch the aggregate matrix from a single endpoint instead of synthesising it on the client. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>336085f -
Emit per-layer top-K SwiGLU intermediate activations for RQ1 Step 5
Adds a forward hook on Mistral/LLaMA-style FFN modules that captures the top-K (=32) hidden neurons of `silu(W_gate · x) ⊙ (W_up · x)` at the predicting (last) position for each generation step. Stored on each layer entry as `ffn_top_neurons: { k, intermediate_size, neurons: [{idx, value}] }`. Memory-bounded: top-32 of 14,336 ≈ 256 bytes per layer per step, versus the 57KB-per-layer-per-step the full intermediate vector would require. The top-K is what the RQ1 frontend's "Behind the SwiGLU" disclosure needs to render an honest per-neuron view rather than synthesising activations or showing only summary stats. Only registers on layers exposing both `gate_proj` and `up_proj` (SwiGLU); other architectures fall back to the existing `gate_stats` / `mlp_output_norm` / `ffn_contribution` summary fields. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>019492b
-
Merge: F2 backend — emit effective_probability for sampling-effective margins (#5)
gary-boonUnverified signatureF2 backend: emit effective_probability for sampling-effective margins (re-target)e11c277 -
Merge: F1 backend — expose query_position and query_token_id (#4)
gary-boonUnverified signatureF1 backend: expose query_position and query_token_id (re-target)778d7c8 -
Merge: F3a — derive aggregate-row num_heads from request metadata (#1)
gary-boonUnverified signatureFix: derive aggregate-row num_heads from request metadata78c7200 -
Emit effective_probability and sampling eligibility per alternative
gary-boon`alternatives[].probability` is the post-temperature, PRE-filter probability — the model's preference. The sampler draws from `probs_filtered`, the post-top-k/top-p renormalised distribution. With filtering active, the panel was reporting margins and per-token probabilities that didn't reflect the actual decision. For each alternative now emit: - `effective_probability`: post-filter renormalised probability (what the sampler actually drew from). 0 if filtered out. Equals `probability` when no filter is active. - `is_eligible`: convenience flag (`effective_probability > 0`). Sampling metadata gains: - `is_filtered`: bool, true if top_k or top_p constrained the set. - `eligible_count`: count of tokens in the eligible set. The frontend can now show one distribution when defaults are in effect, or two (model preference vs sampling-effective) when filtering is active. Same change applied to both SSE and non-SSE analysis paths.6f43bd6 -
Expose query_position and query_token_id in /attention/row response
gary-boonThe cached attention[-1] is the row of the *predicting* query position (the position whose logits selected the step-s token), not the predicted token's self-attention. The frontend was labelling this row as "attention influences on this token", which implies a self-attention relationship the data does not represent. Record (query_position, query_token_id) per generation step on the MatrixCache during analysis (both SSE and non-SSE paths). Surface them in the /attention/row response so the Why this token? panel can show explicit provenance ("Attention used to predict this token, from query position N — `<token>`") instead of the misleading framing. This is a documentation/honesty fix, not a data fix — the underlying attention values are unchanged. It supports the project's stated "grounded hypotheses, not causal claims" framing.d2d962d -
Derive aggregate-row num_heads from request metadata, not live model
gary-boonThe /attention/row aggregate endpoint was reading num_attention_heads from manager.model.config at request time. After a model switch, that returns the *new* model's head count even when serving rows from the *previous* model's cached attention. The per-head row data is correct (stored under request_id keys) but the aggregation averaged the wrong number of heads, silently producing wrong values. Record per-request metadata (num_heads, num_layers, model_id) at analysis time on the MatrixCache. get_aggregate_row derives num_heads from this metadata; the explicit argument is retained as an optional override for tests. The endpoint no longer touches manager.model when serving aggregate rows. clear_request and clear_old_requests drop metadata alongside the per-head cache entries.3506846
-
Fix empty text and incorrect is_special for Mistral control tokens
gary-boon +1: Claude Opus 4.6 (1M context)mistral-common's decode_token() returns "" for chat-template tokens (<s>, [INST], [/INST], [SYSTEM_PROMPT], [/SYSTEM_PROMPT], tool markers). Fall back to the HF tokenizer so every token arrives with a printable string form. Also widen special_token_ids from {eos, bos, pad, unk} to include all control-token IDs from mistral-common's Tekkenizer, fixing is_special for chat-template delimiters. Applied to both streaming and non-streaming endpoints, plus the logit-candidate decode helper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b5e4add
-
Fix score-all classification in vectorised path
gary-boon +1: Claude Opus 4.6 (1M context)The vectorised GPU path (used for most layers) still had the old cascade with "semantic"/"positional" catch-alls. Updated to match the score-all-then-rank system with behaviour types + code cues. Also fixed self reference error — code runs in a standalone async function, not a class method. Use dict-based cache instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>e375e45
-
Merge branch 'main' into feature/score-all-classification
gary-boonbfdde66 -
Pin PyTorch <2.11.0 to fix CUDA driver incompatibility on HF Spaces
gary-boon +1: Claude Opus 4.6 (1M context)PyTorch 2.11.0 (released 2026-03-23) requires a newer CUDA driver than HuggingFace Spaces GPU infrastructure provides (driver 12090). This caused CUDA init to fail, forcing Devstral to load on CPU and be filtered from the model selector. PyTorch 2.10.x (the version deployed successfully on 2026-02-25) works with the current HF Spaces driver. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>8b9ad4b -
Revert "Fix model selector not showing loaded GPU model on CPU-detected hardware"
gary-boonThis reverts commit 6c5265e6b7f804836967492ff5fdb2dd3c69a236.ff8be9d -
Fix model selector not showing loaded GPU model on CPU-detected hardware
gary-boon +1: Claude Opus 4.6 (1M context)The /models endpoint filtered models by hardware capability, hiding GPU-only models (Devstral) when CUDA init failed (e.g. driver too old). But the model was already loaded and running on CPU successfully. Now always includes the currently loaded model in the available list, regardless of hardware detection. If it's running, it's available. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>6c5265e -
Refactor head classification from cascade to score-all-then-rank
gary-boon +1: Claude Opus 4.6 (1M context)Replace priority-ordered first-match cascade with two-dimension system: Behaviour type (attention geometry) — all scored simultaneously: - attention_sink, previous_token, local, induction, focused, diffuse - Primary = highest qualifying score (with min thresholds), secondary = runner-up - "focused" and "diffuse" replace weak "positional"/"semantic" catch-alls Code cue (separate dimension — code token relevance): - delimiter_sensitive: attention to (){}[]:;, - keyword_sensitive: attention to language keywords (def, return, if, etc.) - pattern_reuse: derived from induction signal (repeated spans) - Each scored by proportion of attention mass on target token class - Evidence text included (e.g. "34% attention on delimiter sensitive tokens") Per-head output now includes: - pattern (primary behaviour type + confidence) - secondary_behaviour (type + score) - code_cue (type + score + evidence text) - secondary_cue (type + score) Token texts decoded and cached per generation step for code-…10baadf -
Add FFN contribution metrics and gate activation stats to research endpoint
gary-boon +1: Claude Opus 4.6 (1M context)- Compute attn_contribution/ffn_contribution ratios from existing attention and MLP output norm hooks (no new forward passes needed) - Add gate activation hook for SwiGLU models (LLaMA/Mistral) capturing sparsity, mean, and max of gate values per layer per token - Expose ffnType and intermediateSize in modelInfo response - Clear gate_activation_stats each generation step alongside existing dicts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>82349c1
-
Add tuned lens as supplementary projection mode for logit lens
gary-boon +1: Claude Opus 4.6Introduces per-layer affine probes (trained to minimise KL divergence with the final layer) that correct for subspace mismatch in early transformer layers. Positioned alongside the existing raw logit lens with a frontend toggle, allowing developers to compare projections and assess whether observed commitment patterns are robust to projection choice. New: backend/tuned_lens.py (runtime), scripts/train_tuned_lens.py (training) Modified: model_service.py (load at startup, tuned lens computation, tuned commitment summary, modelInfo.tunedLensAvailable, health endpoint) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>6f48db0 -
Rename 'counterfactual' terminology to 'input ablation' in docstrings
gary-boon +1: Claude Opus 4.6Aligns with extension terminology update — these are ablation studies, not true counterfactuals. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>0d76811 -
Fix margin winner_token in margin_data and extract _classify_stability helper
gary-boon +1: Claude Opus 4.6The Decision Stability display was showing the sampled token as the winner rather than the logit-top token. When temperature changed the pick (e.g. "(number" was logit winner but "(n" was sampled), both sides showed the same token. Now includes winner_token in margin_data from logits_entries[0]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>ecf49a2 -
Add margin-based decision analysis, interventional counterfactuals, and run comparison (v3.0)
gary-boon +1: Claude Opus 4.6Phase 1: Per-token margin computation with stability classification (stable/moderate/boundary/fragile), layer-wise margin tracking via logit lens, commitment layer detection, flip event detection, causal margin contribution decomposition per layer ((W_U[winner] - W_U[runner-up]) · residual_ℓ), and hidden state caching for intervention reuse. Phase 2: POST /analyze/intervention endpoint supporting mask_system, mask_user_span, mask_generated (real forward-pass re-evaluation with attention_mask), temperature_sweep (cached logits), layer_ablation, head_ablation, and expert_mask. Returns margin shift, stability change, and winner change with full diagnostics. Phase 3: POST /analyze/compare endpoint for per-token margin diffs between cached runs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>121a2d9
-
Add deep inspection: data-driven pattern classification, attention/MLP tracking, logit lens
gary-boon +1: Claude Opus 4.6- Replace hardcoded position-based head/layer pattern classification with data-driven detection (sink, previous-token, local, induction, positional, semantic) - Layer patterns computed as confidence-weighted majority vote of head patterns - Add forward hooks for attention and MLP output norms per layer - Add logit lens computation at sampled layers (every n_layers//5) - Compute per-head sink weight, local weight, and induction weight metrics Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>54d9b6e
-
Change default model to Devstral and optimise attention extraction
gary-boon +1: Claude Opus 4.6Switch the default model fallback from codegen-350m to devstral-small, matching the instruction-tuned model used for PhD research. Also includes batched tensor operations for attention extraction (reduces GPU→CPU sync points from ~4000 to ~40 per token). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>d8d197a
-
Fix PyTorch dynamo cache directory issue in containers
gary-boon +1: Claude Opus 4.5- Set USER env var to avoid getpass.getuser() failure - Set TORCHINDUCTOR_CACHE_DIR to bypass username lookup - Set TORCH_HOME for torch cache - Create cache directories with proper permissions Fixes: KeyError: 'getpwuid(): uid not found: 1000' Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>9978aec
-
Fix MistralTokenizer loading and token section boundaries
gary-boon +1: Claude Opus 4.5- Load MistralTokenizer in switch_model() for correct Tekken encoding - Fix token section boundary to include [/SYSTEMPROMPT] in system section - Add attention row endpoint for overlay visualization Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>76b614d
-
Fix MistralTokenizer not loaded during model switch
gary-boon +1: Claude Opus 4.5The switch_model endpoint was not creating the MistralTokenizer, causing special tokens to be decoded incorrectly when switching from CodeGen to Devstral. Also adds attention overlay feature changes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>2bdf299
-
feat: add top-k/top-p sampling and detailed logits/probability tracking
gary-boon +1: Claude Opus 4.5- Add top_k and top_p sampling parameters to research attention endpoints - Capture raw logits before temperature scaling for visualization - Track greedy token (argmax) separately from sampled token - Add raw probabilities (T=1) for comparison in UI - Track cumulative probability and rank for each token alternative - Add sampling metadata (temperature, top_k, top_p, was_greedy) - Handle selected token outliers when not in top-N display Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>2c6343b
-
fix: Improve token alternatives numerical stability and temperature control
gary-boon +1: Claude Opus 4.5- Use log_softmax instead of softmax for numerically stable probability computation at low temperatures (fixes underflow showing 0% for alternatives) - Use topk on logits instead of probs to correctly identify top tokens regardless of temperature (fixes showing special tokens like <unk>) - Remove backend temperature override - now controlled by frontend UI - Use correct tokenizer (MistralTokenizer) for Devstral model when decoding alternative token texts in SSE endpoint 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>bc1f0e0
-
Store matrices as numpy arrays instead of Python lists
gary-boon +1: Claude Opus 4.5Memory optimization for large models like Devstral: - Store attention/QKV matrices as numpy arrays (~4 bytes/float) - Previously used Python lists (~28 bytes/float = 7x overhead) - Convert to lists only when frontend requests via /matrix endpoint This reduces cache memory from ~25GB to ~3.5GB for: 40 layers × 32 heads × 32 steps × ~90KB per head 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>f2e89c2 -
Add per-step memory cleanup for large model support
gary-boon +1: Claude Opus 4.5- Delete outputs, logits, probs tensors after each generation step - Run garbage collection every 8 steps for large models - Clear MPS cache on Apple Silicon to release GPU memory This prevents memory accumulation during generation with models like Devstral that have 40 layers × 32 heads, which previously caused RAM exhaustion on longer token sequences (32+). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>a94eb19 -
Fix RAM exhaustion for large token generation
gary-boon +1: Claude Opus 4.5Add memory management to MatrixCache: - Track request IDs for each cache entry - Clear old request cache entries before starting new analysis - Force garbage collection after clearing cache - Clear GPU/MPS cache on Apple Silicon This prevents memory accumulation when running multiple analyses, particularly for large models like Devstral with 40 layers × 32 heads per step. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>959074d -
feat: add auto_complete parameter for token generation
gary-boon +1: Claude Opus 4.5- Add auto_complete parameter to streaming and non-streaming endpoints - Cap max_tokens at 128 when auto_complete is enabled - Existing EOS early-stop logic handles completion detection 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>bb689ce -
fix: add QKV extraction support for Mistral/Devstral architecture
gary-boon +1: Claude Opus 4.5- Add make_separate_proj_hook for models with separate q_proj, k_proj, v_proj - Handle GQA (Grouped Query Attention) by expanding K/V heads to match Q heads - Support both CodeGen (combined qkv_proj) and Mistral (separate projections) - Update both streaming and non-streaming endpoints 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>d1d37a8 -
feat: implement lazy-loading for attention matrices
gary-boon +1: Claude Opus 4.5- Add MatrixCache class with 60-min TTL for storing attention/QKV matrices - Modify response builder to cache matrices instead of including in payload - Add new endpoint /analyze/research/attention/matrix for on-demand retrieval - Include requestId in responses for cache lookup - Reduces initial response from 400MB+ to ~500KB-1MB 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>929ba88
-
Add avg_entropy calculation for attention heads
gary-boon +1: Claude Opus 4.5- Compute normalized attention entropy averaged over query positions - Normalize by log(k_i) where k_i = number of keys each position can attend to - Average over latter half of positions for more stable signal - Return both entropy (last-token) and avg_entropy fields for each head 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>66a46b6
-
Revert QKV visualization fixes - need better approach for data streaming
gary-boon +1: Claude Opus 4.5Reverts commits: - a79cb83 Add safety checks for missing QKV keys - decb5ab Limit QKV matrices to top 5 heads per layer - 9056859 Fix QKV matrix extraction for Mistral/Devstral architecture - 4ec134b Fix QKV visualization for Mistral/Devstral architecture The QKV fixes caused response sizes to explode, causing 504 timeouts. Need to implement a better approach (lazy loading) before re-enabling. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>d0b7e29 -
Add safety checks for missing QKV keys
gary-boon +1: Claude Opus 4.5If a hook fails silently, the layer's qkv_captures dict might exist but be missing the 'q', 'k', or 'v' keys. This would cause a KeyError. Now we check for each key's existence before accessing it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>a79cb83 -
Limit QKV matrices to top 5 heads per layer to reduce response size
gary-boon +1: Claude Opus 4.5The QKV visualization fix caused response sizes to explode because we were sending Q/K/V matrices for all 32 heads × 40 layers = 1280 sets of matrices. For a 50 token sequence with head_dim=128, that's ~25 million floats per token step, causing 504 Gateway Timeouts in streaming. Now only the top 5 heads (by attention weight) per layer retain their QKV matrices. This reduces the QKV data by ~85% while still providing meaningful visualization data for the most important attention patterns. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>decb5ab -
Fix QKV matrix extraction for Mistral/Devstral architecture
gary-boon +1: Claude Opus 4.5- Use explicit head_dim from model config (Mistral uses 128, not hidden_size/num_heads=160) - Handle both tuple and tensor outputs from projection hooks - Properly reshape Q/K/V tensors to [seq_len, num_heads, head_dim] - Support GQA by expanding K/V heads (8) to match Q heads (32) - Add warning logs for debugging hook failures This fixes the "too many indices for tensor of dimension 2" and shape mismatch errors when extracting QKV matrices from Devstral. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>9056859 -
Fix QKV visualization for Mistral/Devstral architecture
gary-boon +1: Claude Opus 4.5- Add separate Q, K, V hook makers for Mistral-style architectures that use separate projections (q_proj, k_proj, v_proj) - Update layer iteration to use adapter's _get_layers() method for architecture-agnostic layer access - Maintain backwards compatibility with CodeGen's combined QKV projection - Apply same fix to both regular and streaming endpoints This fixes the QKV visualization which was silently failing for Devstral because the hook registration only supported CodeGen's transformer.h structure. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>4ec134b
-
Add future considerations doc for response size optimization
gary-boon +1: Claude Opus 4.5Document the scalability challenge with research attention analysis: - 8 tokens currently produces ~357MB response - Full coding task (50+ tokens) could be 2-5GB+ - Outlines potential solutions: binary format, selective detail, streaming, compression, server-side analysis This is tracked for future work - current implementation prioritizes completeness for PhD research. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>3e67ea2 -
Fix: Import time module at top level for SSE events
gary-boon +1: Claude Opus 4.5The sse_event() helper function uses time.time() but time was only imported inside the event_generator function, causing NameError. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>15a862b -
Add SSE streaming endpoint for real-time analysis progress
gary-boon +1: Claude Opus 4.5Add /analyze/research/attention/stream endpoint that streams progress events during attention analysis via Server-Sent Events (SSE). Progress events are emitted at each stage: - tokenizing: After prompt tokenization, with token count - generating: Per-token, with step index and total steps - extracting: Per-layer (on last token), with layer/head counts - serializing: During response building, with size estimate - complete: When analysis finishes, with timing and size metadata - result: Final payload with complete analysis data Each event includes: - type, stage, totalStages, progress, stageProgress, detail - metadata object with stage-specific info (counts, sizes, timing) Benefits: - Frontend gets real progress updates instead of fake animation - Users see exactly what's happening: "Processing layer 15/40" - Response size displayed during serialization - No more "stuck at 95%" - progress reflects actual work 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Author…172a186 -
feat: Include token metadata in analysis response
gary-boon +1: Claude Opus 4.5Extend promptTokens and generatedTokens with full metadata: - bpe_pieces: BPE breakdown of each token - is_special: Whether token is a special token (eos/bos/pad/unk) - is_multi_split: Whether token is part of a multi-split identifier - num_pieces: Number of BPE pieces This eliminates the need for per-token /token/metadata API calls, reducing backend requests and enabling instant hover tooltips. Also removes debug logging from /token/metadata endpoint. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>ee0f6c9 -
feat: Implement tier-based model filtering by device type
gary-boon +1: Claude Opus 4.5Add min_device field to model configs to specify minimum hardware requirement: - CodeGen 350M: min_device="cpu" (available to all users) - Code Llama 7B: min_device="gpu" (GPU users only) - Devstral Small 24B: min_device="gpu" (GPU users only) Update /models endpoint to filter models based on backend hardware: - GPU backends return all 3 models - CPU backends only return CodeGen 350M This ensures users only see models their backend can actually serve, preventing confusion and failed requests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>6bf9f5c -
Fix: Add attn_implementation="eager" to model switch function
gary-boon +1: Claude Opus 4.5When switching models via /models/switch endpoint, the model was being loaded without attn_implementation="eager", causing Devstral/Mistral models to return None for attention weights. This broke the Research Attention Analyzer with error: "TypeError: object of type 'NoneType' has no len()" Added trust_remote_code=True and attn_implementation="eager" to match the initial model loading configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>f94a7ae
-
Add Phase 5: Performance optimizations to phased plan
gary-boon +1: Claude Opus 4.5Document future performance improvements: - Lazy loading attention data (recommended approach) - Server-Sent Events for progress streaming - Client-side progress estimation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>383a328