Spaces:
Sleeping
Sleeping
Commit History
-
style: apply ruff formatting to structured_summarizer
ming +1: ClaudeAuto-formatted by ruff during pre-commit hook. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>bd7d2c1 -
docs: add comprehensive V4 API documentation and optimize inference with SDPA
ming +1: ClaudeAdd detailed V4 structured summarization API documentation to CLAUDE.md covering: - Architecture overview with GPU optimization (CUDA/MPS/CPU) - NDJSON streaming format (43% faster time-to-first-token) - Style-based prompts (executive, skimmer, eli5) - Performance benchmarks (1.5B vs 3B models) - Quantization hierarchy and deployment strategies Optimize model loading with SDPA attention implementation: - Add attn_implementation="sdpa" for faster inference - Graceful fallback to default attention if SDPA unsupported - Apply to all loading paths (4-bit, FP16, FP32, MPS, CUDA) - Reduce memory usage and improve token generation speed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>0072188
-
chore: code formatting improvements and update gitignore
ming- Format structured_summarizer.py with proper line breaks - Add test output files to .gitignore - Update V4_TESTING_LEARNINGS.mddb3b809 -
Add V4 local server setup with MPS optimization for Android testing
ming +1: Claude- Optimize V4 model for Apple Silicon MPS GPU (4x faster than CPU) - Fix MPS detection and BFloat16 incompatibility - Add comprehensive local server management guide - Add Android integration documentation with connection details - Add startup script for easy server management Performance improvements: - CPU (before): 2+ minutes (timeout) - MPS (after): 32 seconds for complete summary - Inference speed: 2.7 tokens/second on M4 MacBook Pro 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>45b6536
-
Remove outlines library and all related code
ming- Remove outlines dependency from requirements.txt (NumPy compatibility issue) - Remove outlines import validation from Dockerfile - Remove /stream-json endpoint from V4 API (outlines-based) - Remove all outlines-related code from structured_summarizer.py - Remove outlines import tests - Delete test_v4_live.py (outlines integration tests) - Remove all stream-json endpoint tests from test_v4_api.py This resolves the Hugging Face build failure caused by outlines==0.0.44 incompatibility with newer NumPy versions (ModuleNotFoundError: numpy.lib.function_base). V4 API now only supports /stream and /stream-ndjson endpoints using NDJSON patch-based structured summarization (no external dependencies).d25a17f -
Migrate to Ruff for linting/formatting and add comprehensive import tests
ming- Replace black, isort, and flake8 with ruff (Rust-based, 10-100x faster) - Add ruff.toml configuration with appropriate rules and ignores - Update pre-commit hook to run ruff check --fix and ruff format before tests - Update CLAUDE.md documentation with new ruff commands - Fix code style issues (nested if statements, formatting) - Add comprehensive import tests (test_imports.py) to catch import errors early - Fix config.py to ignore extra environment variables (prevents build failures) - Format all code with ruff (18 files reformatted) - Auto-fix 32 linting issues - Fix remaining test linting issues (B017, SIM117, B023) Benefits: - Single tool instead of three (black/isort/flake8) - Faster pre-commit hooks - Better CI/CD performance - Automatic import validation before deployment29ed661
-
Fix Outlines API usage for V4 JSON streaming endpoint
ming- Update to use outlines.generate.json() API pattern - Fix 'takes 1 positional argument but 2 were given' error - Add improved test script with detailed error reportingb47201f
-
Fix Outlines API usage - handle different calling patterns
ming- Try json_schema(schema) then call with model - Fallback to json_schema(model, schema) - Handle different generator patterns (.stream() vs direct call) - Fixes 'takes 1 positional argument but 2 were given' error4c4036e -
Fix Outlines import - use generator instead of generate
ming- Change from outlines.generate to outlines.generator - Try multiple patterns based on available attributes - Add detailed logging for each pattern attempt9452571 -
Change Outlines debug log to info level for visibility
ming5fa0ba2 -
Add detailed Outlines API exploration and logging
ming- Try multiple import patterns with detailed logging - Log available attributes to help diagnose API structure - More robust detection of generate.json functionf099d0c -
Fix Outlines import - use correct API for installed version
ming- Try multiple import patterns to handle different Outlines versions - Use outlines.generate.json directly instead of outlines.generate.json wrapper - Fixes 'cannot import name generate' errore8ab865 -
Improve Outlines import error handling and logging
ming- Capture and log actual import errors for debugging - Log success when Outlines imports correctly - Catch all exceptions, not just ImportError33cd483 -
Improve error messaging for Outlines unavailability
ming734e281 -
Fix Python 3.10 requirement and torch_dtype deprecation
ming- Update Dockerfile to Python 3.10 (Outlines requires 3.10+) - Replace deprecated torch_dtype with dtype parameter - Fixes TypeError with Outlines library on Python 3.96b859f2 -
Add Outlines JSON streaming endpoint for V4 structured summarization
ming- Add outlines>=0.0.34 dependency - Add StructuredSummary Pydantic schema - Wrap Qwen model with Outlines Transformers wrapper - Implement summarize_structured_stream_json() method - Add /api/v4/scrape-and-summarize/stream-json endpoint - Stream JSON tokens token-by-token via SSE - Add Outlines warmup in warm_up_model() - Graceful degradation when Outlines not available441f66b
-
Fix JSON parsing errors in V4 NDJSON stream
mingIssue: - Model generates invalid JSON with unescaped quotes (e.g., "TVNZ's") - JSON parsing fails with 'Expecting : delimiter' error - Code tried to use undefined 'patch' variable after parse failure Fixes: - Initialize patch = None before try block - Added JSON repair logic with two strategies: 1. Extract first complete JSON object from incomplete lines 2. Attempt to escape unescaped quotes in value fields - Improved error logging to show more context (150 chars) - Added CRITICAL JSON FORMATTING RULES to prompt: - Emphasize proper quote escaping (\" for quotes in strings) - Remind model to test JSON validity - Warn against unescaped quotes in values This handles malformed JSON gracefully while improving model output quality.85dcd04 -
Implement Option 3: Use FP16 for 2-3x faster inference
mingSpeed optimization: - Added v4_use_fp16_for_speed config option - When enabled, uses FP16 instead of 4-bit quantization - FP16 is 2-3x faster than 4-bit NF4 quantization - Enabled by default in Dockerfile for maximum speed Memory trade-off: - FP16 uses ~2-3GB GPU memory (vs ~1GB for 4-bit) - Still fits comfortably on T4 GPU (16GB total) Expected results: - Generation time: 24.9s → 8-12s (2-3x speedup) - Same output quality - Faster token generation (~20-30 tokens/sec vs ~6 tokens/sec) This completes the speed optimization plan (Option 1 + 2 + 3)7fff563 -
Optimize V4 generation speed: greedy decoding + reduced max_tokens
mingSpeed optimizations: - Switched from sampling to greedy decoding (2-3x faster) - Changed do_sample=True to do_sample=False - Removed temperature and top_p parameters (not needed for greedy) - Reduced max_tokens from 512 to 256 (model only needs ~150 tokens) Expected results: - Generation time: 24.5s → 8-12s (2-3x speedup) - Deterministic output (better for structured JSON) - Lower memory usage (smaller token limit) Trade-offs: - Slightly less diverse output (but fine for structured JSON) - Still using 4-bit quantization (memory efficient)fd2a8c1 -
Fix buffer parsing and strengthen brevity constraints
mingBuffer parsing fix: - Parse remaining buffer content after streamer loop ends - Fixes issue where {"op": "done"} was left unparsed - Now properly detects completion signal from model Brevity improvements: - Reduced title limit: 8-12 words → 6-10 words MAX - Reduced key points: 10-15 words → 8-12 words MAX - Added explicit examples of SHORT titles - Added CRITICAL BREVITY RULES section with caps warnings - Emphasized 'BE SHORT!' throughout prompt Expected results: - Shorter, punchier titles (6-10 words) - More concise key points - Proper detection of completion signald112a13 -
Optimize V4 output verbosity and generation speed
mingBrevity improvements: - Added explicit length constraints in system prompt - Title: 8-12 words max (was unlimited) - Main summary: 2-3 sentences max - Key points: 3-5 items, each 10-15 words - Category: 1-2 words only Speed improvements: - Switched from greedy decoding to sampling (5-10x faster) - Changed do_sample=False to do_sample=True - Set temperature=0.3 (low but fast) - Added top_p=0.9 for quality Token optimization: - Reduced default max_tokens from 1024 to 512 - More concise outputs complete within token limit Expected results: - Generation time: 49s → 5-10s - Titles: 20+ words → 8-12 words - Complete output with all required fields17499f7 -
Fix bitsandbytes UID error with getpass patch
ming- Added getpass.getuser() mock in structured_summarizer.py - Mock catches KeyError and returns USER env var or 'user' - Added HOME=/tmp and USER=user to Dockerfile - Fixes 'getpwuid(): uid not found: 1000' error in HF Spaces - Allows bitsandbytes 4-bit NF4 quantization to initializedd29a6d -
Switch V4 to GPU INT4 quantization with Qwen-1.5B
ming- Upgraded transformers to 4.44+ and accelerate to 0.33+ - Added bitsandbytes for 4-bit NF4 quantization on GPU - Removed torchao dependency (causes HF Spaces errors) - Added TRANSFORMERS_NO_TORCHAO=1 to prevent import errors - Updated to use Qwen/Qwen2.5-1.5B-Instruct model - GPU: 4-bit NF4 quantization via bitsandbytes - CPU fallback: FP32 + dynamic INT8 quantizationa36f560 -
debug: Add comprehensive logging to diagnose 4-token issue
mingPROBLEM: Model generates only 4 tokens and produces no valid NDJSON patches. Need to see what's actually being generated before filtering. DIAGNOSTIC LOGGING ADDED: 1. Prompt Logging: - Full prompt length - First 500 chars (shows chat template opening) - Last 200 chars (shows generation prompt start) - Verifies apply_chat_template() output is correct 2. Generation Config Logging: - max_new_tokens value - EOS token ID - Pad token ID - Confirms generation settings 3. Raw Token Logging: - Every token chunk as it arrives (debug level) - Shows exact model output before any processing 4. Raw Line Logging: - Every complete line before heuristic filter - Shows what gets filtered and why 5. Buffer State Logging: - Unparsed buffer contents after generation - Detects partial/incomplete lines This will reveal: - Is the chat template correct? - Is the model hitting EOS immediately? - Is the model generating prose instead of JSON? - Are we filtering o…df75294
-
feat: Guarantee complete V4 NDJSON summaries with fallback
mingPROBLEM: - Model often stops early (e.g., 76 tokens) without completing all fields - Missing: title, main_summary, read_time_min (often null) - Client receives incomplete structured summaries SOLUTION: 1. Tightened system prompt: - Explicit ordering: title → main_summary → category → sentiment → read_time_min → key_points - Hard rule: NEVER emit {"op":"done"} until all fields are set - Requires at least 5 key_points before completion 2. Added server-side fallback (_fallback_fill_missing_fields): - read_time_min: Estimated from word count (200 words/min) - main_summary: Derived from first 3 key points - title: Derived from main_summary (first ~14 words) - No external dependencies, uses article content 3. Emit synthetic patches for missing fields: - Maintains NDJSON protocol consistency - Client still sees all updates as patch events - Transparent logging shows which fields were filled 4. Comprehensive logging: - 📝 Model set: tracks each scalar fiel…b321440 -
fix: Use Qwen chat template and harden NDJSON parsing
mingSTEP 1: Use Qwen2.5 native chat template - Replace manual Phi-3 format with tokenizer.apply_chat_template() - Ensures correct special tokens for Qwen models - Adds explicit NDJSON-only reminder in user message - Critical fix: previous Phi-3 format was incompatible with Qwen STEP 2: Harden NDJSON parsing - Add heuristic check: skip lines not starting with '{' or missing 'op' - Prevents parsing C++ code, prose, or random hallucinations - Reduces log spam from invalid JSON parsing attempts - Cleaner error messages for debugging This should fix the model hallucination issue where Qwen2.5-1.5B was generating C++ code instead of NDJSON patches.bf21a65
-
feat: Change device_map to auto for V4 model
ming- Switch from device_map='cpu' to device_map='auto' - Allows model to automatically use GPU if available - Falls back to CPU if no GPU present - Better flexibility for different deployment environmentsd0701b0 -
feat: Add V4 NDJSON patch-based structured summarization
ming- Refactor StructuredSummarizer with new NDJSON streaming protocol - Add summarize_structured_stream_ndjson() method for patch-based streaming - Implement state management with _empty_state() and _apply_patch() - Update system prompt to instruct model to output NDJSON patches - Add new /api/v4/scrape-and-summarize/stream-ndjson endpoint - Support incremental state updates via delta/state events - Use deterministic decoding (greedy) for consistent results - Maintain backwards compatibility with existing stream endpoint Event structure: - delta: JSON patch object (set/append/done operations) - state: Current accumulated state - done: Completion flag - tokens_used: Token count - latency_ms: Final latency metric Test suite: - test_v4_ndjson_mock.py: Protocol logic validation (PASSED ✅) - test_v4_ndjson_http.py: HTTP endpoint test - test_v4_ndjson_url.py: URL scraping test (PASSED ✅) - test_v4_ndjson.py: Direct service test Documentation: - NDJSON_REFACTOR_SUMMARY.md:…93c9664