Spaces:
Running
Running
Commit History
-
style: apply ruff formatting to structured_summarizer
ming +1: ClaudeAuto-formatted by ruff during pre-commit hook. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>bd7d2c1 -
docs: add comprehensive V4 API documentation and optimize inference with SDPA
ming +1: ClaudeAdd detailed V4 structured summarization API documentation to CLAUDE.md covering: - Architecture overview with GPU optimization (CUDA/MPS/CPU) - NDJSON streaming format (43% faster time-to-first-token) - Style-based prompts (executive, skimmer, eli5) - Performance benchmarks (1.5B vs 3B models) - Quantization hierarchy and deployment strategies Optimize model loading with SDPA attention implementation: - Add attn_implementation="sdpa" for faster inference - Graceful fallback to default attention if SDPA unsupported - Apply to all loading paths (4-bit, FP16, FP32, MPS, CUDA) - Reduce memory usage and improve token generation speed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>0072188
-
chore: code formatting improvements and update gitignore
ming- Format structured_summarizer.py with proper line breaks - Add test output files to .gitignore - Update V4_TESTING_LEARNINGS.mddb3b809 -
Add V4 local server setup with MPS optimization for Android testing
ming +1: Claude- Optimize V4 model for Apple Silicon MPS GPU (4x faster than CPU) - Fix MPS detection and BFloat16 incompatibility - Add comprehensive local server management guide - Add Android integration documentation with connection details - Add startup script for easy server management Performance improvements: - CPU (before): 2+ minutes (timeout) - MPS (after): 32 seconds for complete summary - Inference speed: 2.7 tokens/second on M4 MacBook Pro 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>45b6536
-
Apply ruff formatting to Loguru migration files
ming +1: ClaudeAutomatic code formatting applied by ruff during pre-commit hook. These are cosmetic changes only (code style, no functional changes). Files formatted: - app/core/logging.py - tests/test_logging.py - tests/test_loguru_context.py - tests/test_loguru_integration.py - tests/test_loguru_setup.py 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>75fe59b -
Migrate logging from stdlib to Loguru for structured logging
ming +1: ClaudeThis migration replaces Python's standard logging module with Loguru, providing structured logging with automatic request ID propagation, environment-aware formatting, and improved developer experience. ## Changes ### Core Implementation - app/core/logging.py: Complete rewrite (224 lines) - Environment-aware formatting (JSON for prod, colored text for dev) - Context variables for automatic request ID propagation - Custom JSON serializer with structured fields - Backward-compatible get_logger() and RequestLogger APIs - app/core/middleware.py: - Integrated context var (request_id_var) for request ID injection - Automatic propagation eliminates manual parameter passing - app/core/config.py: - Added LOG_FORMAT setting (json/text/auto) with validator - Auto-detection based on HF_SPACE_ROOT_PATH environment - requirements.txt: - Added loguru>=0.7.0 dependency ### Testing (44 tests, 100% passing) - tests/test_loguru_context.py: Context variable propagation (7 tests) - …7ab470d -
Remove outlines library and all related code
ming- Remove outlines dependency from requirements.txt (NumPy compatibility issue) - Remove outlines import validation from Dockerfile - Remove /stream-json endpoint from V4 API (outlines-based) - Remove all outlines-related code from structured_summarizer.py - Remove outlines import tests - Delete test_v4_live.py (outlines integration tests) - Remove all stream-json endpoint tests from test_v4_api.py This resolves the Hugging Face build failure caused by outlines==0.0.44 incompatibility with newer NumPy versions (ModuleNotFoundError: numpy.lib.function_base). V4 API now only supports /stream and /stream-ndjson endpoints using NDJSON patch-based structured summarization (no external dependencies).d25a17f -
Migrate to Ruff for linting/formatting and add comprehensive import tests
ming- Replace black, isort, and flake8 with ruff (Rust-based, 10-100x faster) - Add ruff.toml configuration with appropriate rules and ignores - Update pre-commit hook to run ruff check --fix and ruff format before tests - Update CLAUDE.md documentation with new ruff commands - Fix code style issues (nested if statements, formatting) - Add comprehensive import tests (test_imports.py) to catch import errors early - Fix config.py to ignore extra environment variables (prevents build failures) - Format all code with ruff (18 files reformatted) - Auto-fix 32 linting issues - Fix remaining test linting issues (B017, SIM117, B023) Benefits: - Single tool instead of three (black/isort/flake8) - Faster pre-commit hooks - Better CI/CD performance - Automatic import validation before deployment29ed661
-
Fix Outlines API usage for V4 JSON streaming endpoint
ming- Update to use outlines.generate.json() API pattern - Fix 'takes 1 positional argument but 2 were given' error - Add improved test script with detailed error reportingb47201f
-
Fix Outlines API usage - handle different calling patterns
ming- Try json_schema(schema) then call with model - Fallback to json_schema(model, schema) - Handle different generator patterns (.stream() vs direct call) - Fixes 'takes 1 positional argument but 2 were given' error4c4036e -
Fix Outlines import - use generator instead of generate
ming- Change from outlines.generate to outlines.generator - Try multiple patterns based on available attributes - Add detailed logging for each pattern attempt9452571 -
Change Outlines debug log to info level for visibility
ming5fa0ba2 -
Add detailed Outlines API exploration and logging
ming- Try multiple import patterns with detailed logging - Log available attributes to help diagnose API structure - More robust detection of generate.json functionf099d0c -
Fix Outlines import - use correct API for installed version
ming- Try multiple import patterns to handle different Outlines versions - Use outlines.generate.json directly instead of outlines.generate.json wrapper - Fixes 'cannot import name generate' errore8ab865 -
Improve Outlines import error handling and logging
ming- Capture and log actual import errors for debugging - Log success when Outlines imports correctly - Catch all exceptions, not just ImportError33cd483 -
Improve error messaging for Outlines unavailability
ming734e281 -
Fix Python 3.10 requirement and torch_dtype deprecation
ming- Update Dockerfile to Python 3.10 (Outlines requires 3.10+) - Replace deprecated torch_dtype with dtype parameter - Fixes TypeError with Outlines library on Python 3.96b859f2 -
Add Outlines JSON streaming endpoint for V4 structured summarization
ming- Add outlines>=0.0.34 dependency - Add StructuredSummary Pydantic schema - Wrap Qwen model with Outlines Transformers wrapper - Implement summarize_structured_stream_json() method - Add /api/v4/scrape-and-summarize/stream-json endpoint - Stream JSON tokens token-by-token via SSE - Add Outlines warmup in warm_up_model() - Graceful degradation when Outlines not available441f66b
-
Fix JSON parsing errors in V4 NDJSON stream
mingIssue: - Model generates invalid JSON with unescaped quotes (e.g., "TVNZ's") - JSON parsing fails with 'Expecting : delimiter' error - Code tried to use undefined 'patch' variable after parse failure Fixes: - Initialize patch = None before try block - Added JSON repair logic with two strategies: 1. Extract first complete JSON object from incomplete lines 2. Attempt to escape unescaped quotes in value fields - Improved error logging to show more context (150 chars) - Added CRITICAL JSON FORMATTING RULES to prompt: - Emphasize proper quote escaping (\" for quotes in strings) - Remind model to test JSON validity - Warn against unescaped quotes in values This handles malformed JSON gracefully while improving model output quality.85dcd04 -
fix: Move inputs to model device in _single_chunk_summarize to fix CPU/GPU device mismatch
mingcfe8d29 -
Fix device placement error in V2 BART model
mingIssue: - RuntimeError: tensors on CPU vs CUDA mismatch - BatchEncoding objects don't have .to() method - Inputs need to be moved tensor-by-tensor to model device Fixes: - Fixed _generate_test() to move inputs properly - Fixed summarize_text_stream() to move inputs to model device - Fixed warm_up_model() test input device placement - Added isinstance check for torch.Tensor before moving This ensures all input tensors are on the same device as the model, even when using device_map='auto' for multi-device models.12a2e7c -
Implement Option 3: Use FP16 for 2-3x faster inference
mingSpeed optimization: - Added v4_use_fp16_for_speed config option - When enabled, uses FP16 instead of 4-bit quantization - FP16 is 2-3x faster than 4-bit NF4 quantization - Enabled by default in Dockerfile for maximum speed Memory trade-off: - FP16 uses ~2-3GB GPU memory (vs ~1GB for 4-bit) - Still fits comfortably on T4 GPU (16GB total) Expected results: - Generation time: 24.9s → 8-12s (2-3x speedup) - Same output quality - Faster token generation (~20-30 tokens/sec vs ~6 tokens/sec) This completes the speed optimization plan (Option 1 + 2 + 3)7fff563 -
Optimize V4 generation speed: greedy decoding + reduced max_tokens
mingSpeed optimizations: - Switched from sampling to greedy decoding (2-3x faster) - Changed do_sample=True to do_sample=False - Removed temperature and top_p parameters (not needed for greedy) - Reduced max_tokens from 512 to 256 (model only needs ~150 tokens) Expected results: - Generation time: 24.5s → 8-12s (2-3x speedup) - Deterministic output (better for structured JSON) - Lower memory usage (smaller token limit) Trade-offs: - Slightly less diverse output (but fine for structured JSON) - Still using 4-bit quantization (memory efficient)fd2a8c1 -
Fix buffer parsing and strengthen brevity constraints
mingBuffer parsing fix: - Parse remaining buffer content after streamer loop ends - Fixes issue where {"op": "done"} was left unparsed - Now properly detects completion signal from model Brevity improvements: - Reduced title limit: 8-12 words → 6-10 words MAX - Reduced key points: 10-15 words → 8-12 words MAX - Added explicit examples of SHORT titles - Added CRITICAL BREVITY RULES section with caps warnings - Emphasized 'BE SHORT!' throughout prompt Expected results: - Shorter, punchier titles (6-10 words) - More concise key points - Proper detection of completion signald112a13 -
Optimize V4 output verbosity and generation speed
mingBrevity improvements: - Added explicit length constraints in system prompt - Title: 8-12 words max (was unlimited) - Main summary: 2-3 sentences max - Key points: 3-5 items, each 10-15 words - Category: 1-2 words only Speed improvements: - Switched from greedy decoding to sampling (5-10x faster) - Changed do_sample=False to do_sample=True - Set temperature=0.3 (low but fast) - Added top_p=0.9 for quality Token optimization: - Reduced default max_tokens from 1024 to 512 - More concise outputs complete within token limit Expected results: - Generation time: 49s → 5-10s - Titles: 20+ words → 8-12 words - Complete output with all required fields17499f7 -
Fix bitsandbytes UID error with getpass patch
ming- Added getpass.getuser() mock in structured_summarizer.py - Mock catches KeyError and returns USER env var or 'user' - Added HOME=/tmp and USER=user to Dockerfile - Fixes 'getpwuid(): uid not found: 1000' error in HF Spaces - Allows bitsandbytes 4-bit NF4 quantization to initializedd29a6d -
Switch V4 to GPU INT4 quantization with Qwen-1.5B
ming- Upgraded transformers to 4.44+ and accelerate to 0.33+ - Added bitsandbytes for 4-bit NF4 quantization on GPU - Removed torchao dependency (causes HF Spaces errors) - Added TRANSFORMERS_NO_TORCHAO=1 to prevent import errors - Updated to use Qwen/Qwen2.5-1.5B-Instruct model - GPU: 4-bit NF4 quantization via bitsandbytes - CPU fallback: FP32 + dynamic INT8 quantizationa36f560 -
debug: Add comprehensive logging to diagnose 4-token issue
mingPROBLEM: Model generates only 4 tokens and produces no valid NDJSON patches. Need to see what's actually being generated before filtering. DIAGNOSTIC LOGGING ADDED: 1. Prompt Logging: - Full prompt length - First 500 chars (shows chat template opening) - Last 200 chars (shows generation prompt start) - Verifies apply_chat_template() output is correct 2. Generation Config Logging: - max_new_tokens value - EOS token ID - Pad token ID - Confirms generation settings 3. Raw Token Logging: - Every token chunk as it arrives (debug level) - Shows exact model output before any processing 4. Raw Line Logging: - Every complete line before heuristic filter - Shows what gets filtered and why 5. Buffer State Logging: - Unparsed buffer contents after generation - Detects partial/incomplete lines This will reveal: - Is the chat template correct? - Is the model hitting EOS immediately? - Is the model generating prose instead of JSON? - Are we filtering o…df75294
-
feat: Guarantee complete V4 NDJSON summaries with fallback
mingPROBLEM: - Model often stops early (e.g., 76 tokens) without completing all fields - Missing: title, main_summary, read_time_min (often null) - Client receives incomplete structured summaries SOLUTION: 1. Tightened system prompt: - Explicit ordering: title → main_summary → category → sentiment → read_time_min → key_points - Hard rule: NEVER emit {"op":"done"} until all fields are set - Requires at least 5 key_points before completion 2. Added server-side fallback (_fallback_fill_missing_fields): - read_time_min: Estimated from word count (200 words/min) - main_summary: Derived from first 3 key points - title: Derived from main_summary (first ~14 words) - No external dependencies, uses article content 3. Emit synthetic patches for missing fields: - Maintains NDJSON protocol consistency - Client still sees all updates as patch events - Transparent logging shows which fields were filled 4. Comprehensive logging: - 📝 Model set: tracks each scalar fiel…b321440 -
fix: Use Qwen chat template and harden NDJSON parsing
mingSTEP 1: Use Qwen2.5 native chat template - Replace manual Phi-3 format with tokenizer.apply_chat_template() - Ensures correct special tokens for Qwen models - Adds explicit NDJSON-only reminder in user message - Critical fix: previous Phi-3 format was incompatible with Qwen STEP 2: Harden NDJSON parsing - Add heuristic check: skip lines not starting with '{' or missing 'op' - Prevents parsing C++ code, prose, or random hallucinations - Reduces log spam from invalid JSON parsing attempts - Cleaner error messages for debugging This should fix the model hallucination issue where Qwen2.5-1.5B was generating C++ code instead of NDJSON patches.bf21a65
-
feat: Switch V4 to Qwen2.5-1.5B for HF memory compatibility
ming- Replace Phi-3-mini (3.8B) with Qwen2.5-1.5B-Instruct - Phi-3-mini still exceeded 16GB limit even with V2 disabled - Qwen2.5-1.5B: ~3GB RAM (comfortable within 16GB) - 3x larger than 0.5B version (much better quality) - Good structured output and NDJSON generation capability - Memory: V4: 3GB + system: 2GB = ~5GB (plenty of headroom)fe47248 -
perf: Disable V2 warmup to save memory for V4
ming- Disable V2 HuggingFace warmup by default (enable_v2_warmup=False) - Frees up ~1.2GB RAM for V4 Phi-3-mini model - V2 endpoints still work (model loads on first request) - Required to fit Phi-3-mini within HF 16GB limit - V2: 0GB + V4 peak: 15GB + system: 2GB = ~17GB (within limit)1b76b21 -
feat: Change device_map to auto for V4 model
ming- Switch from device_map='cpu' to device_map='auto' - Allows model to automatically use GPU if available - Falls back to CPU if no GPU present - Better flexibility for different deployment environmentsd0701b0 -
feat: Switch V4 model to Phi-3-mini for better structured output
ming- Replace Qwen/Qwen2.5-0.5B-Instruct with microsoft/Phi-3-mini-4k-instruct - Phi-3-mini (3.8B params) has much better instruction following capability - Expected improvements: - Valid NDJSON patch generation - Better structured JSON output - More reliable field completion - Trade-off: Slower generation (~20-30s vs ~10s) but higher quality - RAM usage: ~7-8GB (quantized) vs ~1-2GB for Qwen2.5-0.5B7019b66 -
feat: Add V4 NDJSON patch-based structured summarization
ming- Refactor StructuredSummarizer with new NDJSON streaming protocol - Add summarize_structured_stream_ndjson() method for patch-based streaming - Implement state management with _empty_state() and _apply_patch() - Update system prompt to instruct model to output NDJSON patches - Add new /api/v4/scrape-and-summarize/stream-ndjson endpoint - Support incremental state updates via delta/state events - Use deterministic decoding (greedy) for consistent results - Maintain backwards compatibility with existing stream endpoint Event structure: - delta: JSON patch object (set/append/done operations) - state: Current accumulated state - done: Completion flag - tokens_used: Token count - latency_ms: Final latency metric Test suite: - test_v4_ndjson_mock.py: Protocol logic validation (PASSED ✅) - test_v4_ndjson_http.py: HTTP endpoint test - test_v4_ndjson_url.py: URL scraping test (PASSED ✅) - test_v4_ndjson.py: Direct service test Documentation: - NDJSON_REFACTOR_SUMMARY.md:…93c9664
-
Revert adaptive token logic, restore client-controlled max_tokens
ming +1: Claude- Restore client max_tokens control in V3 API (was being ignored) - Remove min_length parameter from HF streaming service - Re-enable temperature and top_p sampling parameters - Simplify token generation: min_new_tokens = max(20, min(50, max_tokens//4)) - Set neutral length_penalty = 1.0 to avoid length bias - Reduce recursive chunk tokens from 200 to 80 for tighter summaries - Remove adaptive token calculation tests and parameter validation tests - Update default max_tokens back to 256 in schema This simplifies the summarization logic by removing server-side token overrides and allowing the client (Android app) to control summary length directly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>6a1e8a3 -
fix: Backend ignores client max_tokens to verify Android app hypothesis
ming +1: ClaudeThis is a diagnostic commit to verify that the Android app's max_tokens=256 is causing early stopping, NOT a backend issue. Changes Made: 1. **Remove Client max_tokens Constraint** (app/api/v3/scrape_summarize.py:121-123) - BEFORE: min(text_length // 3, payload.max_tokens, 1024) - AFTER: min(text_length // 3, 1024) - Backend now IGNORES client's max_tokens value - Always uses adaptive calculation for quality 2. **Remove Incompatible Generation Parameters** (app/services/hf_streaming_summarizer.py) - Removed temperature, top_p from gen_kwargs (lines ~375, ~670) - Removed length_penalty from gen_kwargs (lines ~391) - These parameters are invalid with do_sample=False (greedy decoding) - Eliminates transformers warning: "generation flags are not valid and may be ignored: ['temperature', 'top_p', 'length_penalty']" 3. **Update Tests** (tests/test_v3_api.py) - test_adaptive_tokens_medium_article: Now expects 600-700 tokens (not 450-512) - test_user_max_…80ea70f -
fix: CRITICAL - Override model config defaults causing early stopping
ming +1: ClaudeThis commit fixes the ROOT CAUSE of early stopping issues in V3 summaries. The distilbart-cnn-6-6 model configuration has defaults that were OVERRIDING our min_new_tokens settings and causing premature summary termination. Critical Model Config Defaults (Previously Unaddressed): - forced_eos_token_id: 2 (forced EOS token emission) - early_stopping: true (stops at first "complete" sequence) - max_length: 142 (model's trained default for news summaries) These config values were taking precedence over our min_new_tokens parameter, causing summaries to stop at ~100-150 tokens even when min_new_tokens=200. Changes Made: 1. Added forced_eos_token_id=None to BOTH generation locations - Lines 398-400 (main summarization) - Lines 683-685 (_single_chunk_summarize) - Disables model config's forced EOS behavior 2. Added forced_bos_token_id=None for consistency - Prevents any forced BOS token injection 3. Added early_stopping=False explicitly - Ensures min_new_tokens is respecte…6c96c54 -
fix: Improve V3 summary completeness with enhanced token allocation
ming +1: ClaudeThis commit addresses remaining early stopping issues by fixing three cascading problems that caused summaries to truncate prematurely, especially for long articles. Critical Changes: 1. Increased recursive chunk token limit: 80 → 200 tokens per chunk - Location: app/services/hf_streaming_summarizer.py:480 - Formula: min(max_new_tokens // 2, 200) - Impact: 2.5x more tokens for long article chunks (>1500 chars) 2. Raised min_new_tokens floor: 50 → 200 tokens - Location: app/services/hf_streaming_summarizer.py:389-391, 651-652 - Formula: max(50, min(max_new_tokens // 2, 200)) - Impact: Prevents early stopping at 50 tokens when 512 allocated 3. More aggressive adaptive formula: text_length // 4 → // 3 - Location: app/api/v3/scrape_summarize.py:121 - Impact: 6000-char articles get ~2000 tokens (before 1024 cap) Expected Results: - Short articles (<1500 chars): Better min quality guarantee (50 → 200 tokens) - Medium articles (2000 chars): More complete summaries …6b2de93 -
fix: V3 API mid-sentence cutoff with adaptive token calculation
ming +1: ClaudeThis commit fixes the issue where V3 API summaries were cutting off mid-sentence by implementing adaptive token allocation and improving generation parameters. Changes: - Increase default max_tokens from 256 to 512 (app/api/v3/schemas.py) - Add adaptive token calculation based on input length (app/api/v3/scrape_summarize.py) - Formula: min(max(text_length // 4, 300), user_max, 1024) - Calculate min_length as 60% of max to encourage complete thoughts - Update HF service to accept min_length parameter (app/services/hf_streaming_summarizer.py) - Increase length_penalty from 1.0 to 1.2 to favor complete sentences - Add 10 new tests for adaptive tokens and summary completeness Results: - Short articles (~500 chars): 300-400 tokens - Medium articles (~1500 chars): 500-700 tokens - Long articles (~3000+ chars): 800-1024 tokens - All V3 tests passing (16/16) - 89% coverage for V3-specific code 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <norep…5e83010
-
Fix V3 API to support both URL and text input
ming +1: ClaudeThe V3 endpoint was causing 422 errors when receiving plain text from the Android app because it only accepted URLs. This update adds intelligent input detection to handle both modes: - URL mode: Scrapes article from URL then summarizes - Text mode: Summarizes provided text directly (no scraping) Changes: - Updated schema to make url and text optional (exactly one required) - Added model validator to ensure mutual exclusivity - Enhanced endpoint to detect and route based on input type - Updated metadata to include input_type field - Added comprehensive documentation and examples This fixes the 422 Unprocessable Entity errors and provides a single unified endpoint for the Android app. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>f724bab
-
feat: Implement V3 Web Scraping + Summarization API
ming✨ New Features: - Add V3 API endpoint: POST /api/v3/scrape-and-summarize/stream - Backend web scraping with trafilatura (95%+ success rate) - In-memory TTL-based caching (1 hour default, configurable) - User-agent rotation to avoid anti-scraping measures - Metadata extraction (title, author, date, site_name) - SSRF protection (blocks localhost and private IPs) - Streaming SSE response with metadata + content chunks 📦 New Components: - ArticleScraperService: High-quality article extraction - SimpleCache: In-memory cache with TTL and max size - V3 Router: Complete API implementation with validation - V3 Schemas: Request/response models with security validators 🧪 Testing: - 30 new tests (100% passing) - Cache tests: TTL, expiration, thread safety - Scraper tests: Success, timeouts, validation - API tests: Security (SSRF), error handling, streaming format 📝 Documentation: - Updated CLAUDE.md with V3 details - Updated README.md with V3 usage examples - Added V3_SCRAPING_IMPLEMENTATION…2ed2bd7
-
Update v2 API and HF streaming summarizer improvements
mingfc9914e -
Improve V2 summarization: adaptive tokens, recursive summarization, better defaults
ming- Add temperature and top_p parameters to SummarizeRequest schema (defaults: 0.3, 0.9) - Implement adaptive max_new_tokens logic: 60-100 for short texts, proportional for long texts - Add recursive summarization for texts >1500 chars with chunking and summary-of-summaries - Fix generation parameters to prevent rambling (reduce min_new_tokens, neutral length_penalty) - Update default prompt to be more concise - Add comprehensive unit tests for all improvements (40 tests) - Fix V2 API to generate concise summaries instead of rambling output9884884 -
Comment out asyncio.sleep for better streaming performance
ming698636a
-
Enhance HF V2 summaries: increase token limits, improve length parameters
ming- Increase default max_new_tokens from 64-128 to 256 minimum - Add min_new_tokens floor (96-192) to prevent premature stopping - Set length_penalty to 1.1 to encourage longer outputs - Expand encoder max length to 2048 tokens (from 512/1024) - Add repetition controls (no_repeat_ngram_size=3, repetition_penalty=1.05) - Add chunking helper function for very long inputs - Add defensive checks for torch availability in tests52f6c42 -
Improve V2 summarization prompts for more detailed output
ming- Updated default prompt in HF streaming service to request comprehensive summaries - Enhanced prompt asks for main arguments, key findings, important details, and examples - Applied consistent improved prompts across V1 and V2 API endpoints - Optimized for sshleifer/distilbart-cnn-6-6 model to produce more detailed summariesde8b12b -
Add model ID logging to HF streaming service
ming- Log model ID during initialization - Log model ID when processing text - Log model ID when summarization completes - Improves debugging and model trackinge186c67 -
Fix HF streaming beam expansion issue
ming- Pin num_beams=1, num_beam_groups=1, num_return_sequences=1 - Add safety loop to reassert values if upstream re-injects them - Remove diversity_penalty and num_return_sequences_per_prompt - Prevents ValueError: TextStreamer only supports batch size 1df1ab37 -
Enforce singleton batch for HF streaming (fix TextStreamer batch size 1 error)
ming- Replace existing batch normalization with hardened _to_singleton_batch() function - Add final assertions and logging to crash early if batch size != 1 - Remove num_beams parameter to avoid multi-sequence behavior - Ensure all tensor fields (input_ids, attention_mask, etc.) are normalized to batch size 1 - Add comprehensive error handling with clear logging for debugging843f837