Gemma Fanout
🚀
1
Generate 10 query fanouts with Gemma 3 270M
Dan Petrovic PRO
dejanseo
AI & ML interests
DEJAN - The AI SEO agency.
[DEJAN](https://dejan.ai/)
Recent Activity
liked a Space 3 days ago
dejanseo/gemma-fanout updated a Space 4 days ago
dejanseo/gemma-fanout published a Space 4 days ago
dejanseo/gemma-fanoutOrganizations
liked a
Space 3 days ago
updated a
Space 4 days ago
published a
Space 4 days ago
liked a
model 4 days ago
updated a
model 4 days ago
published a
model 4 days ago
posted an update 7 days ago
Post
93
10 Search Subqueries in 200 Microseconds: 1-Bit Consistency Fanout
We built Fanout Diffusion, an ultra-low-latency model that expands a search query into 10 diverse subquery vectors in a single forward pass.
Inspired by the continuous retrieval framework in R4T (arXiv:2603.06397), our goal was to make query expansion fast enough for production search without running language models.
How it works:
• Direct Embedding Space: Takes a query embedded via google/embeddinggemma-300m (768-dim) and predicts 10 distinct subquery vectors simultaneously.
• 1-Step Consistency Denoiser: Generates all 10 slots analytically in a single pass with zero ODE integration loops.
• 1-Bit Hardware Tensor Cores: Quantized ternary weights (-1, 0, +1) running on Ampere/Ada sub-byte PTX instructions with INT4 outer projections.
Key numbers on an RTX 4090:
• Latency: 0.200 ms (ONNX) / 0.329 ms (native C++ CUDA graph)
• Throughput: 417,035 queries/second in batch mode
• Model Size: 1.64 MB (PyTorch QAT) / 32.1 MB (ONNX graph)
• Quality: 0.683 prompt alignment across 540k search queries
Interactive Space:
dejanseo/fanout-diffusion
Model weights, ONNX graph, and C++ engine:
dejanseo/fanout-diffusion
We built Fanout Diffusion, an ultra-low-latency model that expands a search query into 10 diverse subquery vectors in a single forward pass.
Inspired by the continuous retrieval framework in R4T (arXiv:2603.06397), our goal was to make query expansion fast enough for production search without running language models.
How it works:
• Direct Embedding Space: Takes a query embedded via google/embeddinggemma-300m (768-dim) and predicts 10 distinct subquery vectors simultaneously.
• 1-Step Consistency Denoiser: Generates all 10 slots analytically in a single pass with zero ODE integration loops.
• 1-Bit Hardware Tensor Cores: Quantized ternary weights (-1, 0, +1) running on Ampere/Ada sub-byte PTX instructions with INT4 outer projections.
Key numbers on an RTX 4090:
• Latency: 0.200 ms (ONNX) / 0.329 ms (native C++ CUDA graph)
• Throughput: 417,035 queries/second in batch mode
• Model Size: 1.64 MB (PyTorch QAT) / 32.1 MB (ONNX graph)
• Quality: 0.683 prompt alignment across 540k search queries
Interactive Space:
dejanseo/fanout-diffusion
Model weights, ONNX graph, and C++ engine:
dejanseo/fanout-diffusion
- 1 reply
updated a
Space 7 days ago
updated a
model 7 days ago
liked a
Space 7 days ago
published a
Space 7 days ago
published a
model 7 days ago
liked a
model 9 days ago
updated a
Space 12 days ago
upvoted a paper 12 days ago
updated 3
Spaces 24 days ago
replied to their post about 1 month ago
I would be delighted if that was the case.
replied to their post about 1 month ago
