Spaces:
Paused
Paused
Commit History
-
Replace dual 8B with single 30B-A3B FP8 vision model
Simplify pipeline architecture: - Vision: Qwen3-VL-30B-A3B-Thinking-FP8 (~30-35GB) replaces dual 8B - Embedding: 2B model (2048-dim) replaces 8B - Reranker: 2B model replaces 8B - Total VRAM: ~38-43GB (was ~68GB), 45GB+ headroom on 4xL4 Key changes: - vLLM with FP8 quantization (built-in, no autoawq needed) - Proper Qwen3-VL chat template formatting via processor - Removed dual-model Thinking→Instruct pipeline - Single model handles analysis + structured JSON output Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>706520f -
Reduce thinking model max_new_tokens to fix slow inference
Root cause: max_new_tokens=32768 caused ~27 min inference time, appearing as a "hang". Reduced to 8192 for ~7 min thinking stage. The device mismatch warning is expected behavior - transformers handles routing internally for device_map="auto" models. Also includes ruff auto-fixes for unused imports. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>0699c5f
-
Fix multi-GPU compatibility issues (6 locations)
1. Meta tensor error in reranker (BLOCKING): - Use .clone() to force weight materialization before copying - Create Linear layer with correct device/dtype from lm_head 2. Remove all .to(device) calls that break distributed models: - scripts/qwen3_vl/qwen3_vl_embedding.py:388 - scripts/qwen3_vl/qwen3_vl_reranker.py:109, 374 - models/real.py:259 - rag/vectorstore.py:130 - rag/retriever.py:149 With device_map="auto", transformers handles device routing internally. Manual .to(device) calls cause device mismatches on multi-GPU setups. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>d1901ae -
Fix multi-GPU support in vendored Qwen3-VL scripts
The original scripts used .to(device) which moves the entire model to a single GPU. With vision model already on GPU 0 (~22GB), this caused OOM when loading embedding model. Changes: - Remove .to(device) calls in both Qwen3VLEmbedder and Qwen3VLReranker - Add device_map="auto" to from_pretrained() for multi-GPU distribution - Update device references to use model's distributed device This allows the embedding and reranker models to be distributed across available GPUs alongside the vision model. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>c4bfdfa -
Fix embedding/reranker loading with official Qwen3-VL classes
Root cause: AutoModel.from_pretrained() loads base transformer instead of specialized embedding/reranking variants. Changes: - Vendor official scripts from QwenLM/Qwen3-VL-Embedding repo - Replace AutoModel with Qwen3VLEmbedder for embedding model - Replace AutoModel with Qwen3VLReranker for reranker model - Update embed()/rerank() methods to use official process() API The official loaders handle: - Proper last-token pooling and L2 normalization (embedding) - Yes/no binary scoring from LM head weights (reranker) This eliminates the fallback L2 norm heuristic scoring that was producing "less accurate" results. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>455c786