Dipankar Sarkar PRO
dipankarsarkar
AI & ML interests
Building the AI-native stack. Agents as infrastructure, safety as architecture, performance as plumbing. I publish the receipts: papers, datasets, demos.
Recent Activity
repliedto ManniX-ITA's post 30 minutes ago
# opencoti-llamafile 0.10.3-c5 — settled admission for multi-agent serving
New cut of the opencoti single-file inference engine (llamafile 0.10.3 / llama.cpp + 87 additive patches). Zero-dependency APE: one executable for Linux, Windows, macOS & BSD.
What's new vs c4:
**PolyKV fan-out — pool from a live session.** `POST /polykv/pools` gains `from_session`/`from_slot`: the shared prefix is snapshotted server-side from the session's cached KV — no tokens resent, token-exact. Ephemeral pools auto-release when orchestrators die mid-round.
**PolyKV P7 — settled admission.** Spawning agents faster than the tps signal settles was oversubscribing pools. Now: a per-pool settle window paces admits just enough for a reliable reading; warming sessions no longer bias the mean; the post-admit forecast uses the measured per-admit drop; idle gaps (agents mid-tool-call) no longer read as free capacity; `guarantee_min_sessions` means a new/nested pool always gets its first agent — capacity checks can never deadlock an orchestrator; the enforced gate applies to new sessions only, with per-request `overcommit`. Benchmark (multi-agent courier, floor 15 tok/s): time-under-floor −52%, deep sub-floor −83%, delivery p50 −43%, 100% task score.
**Zero-conf GPU sharing.** Instances on one GPU discover each other over shared memory — no ports, no config — and split compute by `--gpu-share-weight`. Measured (3090): weights 2:1 → 71.7/36.2 tok/s; holds at `--parallel 4` and under MTP. Idle peers cost nothing (solo = full speed), crashes age out in 3 s; `GET /gpu/peers` shows live shares + busy %.
From c5 every release ships per-platform side-load DSOs: `dso/<ver>/` with Linux x86_64 + sbsa `.so` and a Windows `.dll`.
https://huggingface.co/ManniX-ITA/opencoti-llamafile repliedto SeaWolf-AI's post about 10 hours ago
📱 POCKET — a 35-billion-parameter model that runs on your iPhone, and on your PC with no GPU
We're releasing POCKET, VIDRAFT's flagship Darwin-36B-Opus compressed for on-device use. No fork, no CUDA, no cloud — it runs on stock llama.cpp. It's a sparse Mixture-of-Experts model (256 experts, only 8 active per token), so the file can be large while the work per token stays small. That's what lets a 35B model run on a phone, and generate fast on a CPU with no graphics card.
Measured (POCKET-35B IQ1_M vs Bonsai-27B Q1_0):
• CPU generate (Xeon, 16 threads): 27.0 vs 10.1 tok/s → 2.69× faster
• GPU generate (H100): 197 vs 89 tok/s → 2.22× faster
• GPU prompt processing (H100): 753 vs 1816 → 0.41× (Bonsai wins this one — MoE prefill wakes every expert, so sparsity stops helping there. We say so.)
• Quality (HellaSwag, 400 q): 61.0% vs 60.0% → a tie (confidence intervals overlap)
On a real consumer laptop — MacBook M3 Pro (18 GB) — POCKET wins every axis, prompt processing included:
• Metal generate: 25.4 vs 12.8 → 1.99×
• CPU generate: 13.8 vs 4.4 → 3.13×
• Metal prompt: 240.7 vs 73.4 → 3.28×
One more quiet fact: the same-size, quality-oriented rival Ternary-Bonsai-27B (7.2 GB) fails to load in upstream llama.cpp at all — it needs the PrismML fork. POCKET runs on the tools you already have: LM Studio, Ollama, PocketPal, MLX.
📖 Full story (tech, measurements, recipes): https://huggingface.co/blog/FINAL-Bench/pocket
Models:
📦 POCKET-35B-GGUF (PC / server, no GPU): https://huggingface.co/FINAL-Bench/POCKET-35B-GGUF
🇰🇷 POCKET-KR-GGUF (Android): https://huggingface.co/FINAL-Bench/POCKET-KR-GGUF
🍎 POCKET-KR-MLX (iPhone / Mac): https://huggingface.co/FINAL-Bench/POCKET-KR-MLX
🌍 POCKET-EN-GGUF (English phone / PC): https://huggingface.co/FINAL-Bench/POCKET-EN-GGUF
🖥️ Live demo (answering on a CPU, no GPU): https://huggingface.co/spaces/FINAL-Bench/POCKET-35B-CPU
📚 Collection: https://huggingface.co/collections/FINAL-Bench/pocket-models-6a618ee5d23eafb7e185a5c6
repliedto SeaWolf-AI's post about 12 hours ago
📱 POCKET — a 35-billion-parameter model that runs on your iPhone, and on your PC with no GPU
We're releasing POCKET, VIDRAFT's flagship Darwin-36B-Opus compressed for on-device use. No fork, no CUDA, no cloud — it runs on stock llama.cpp. It's a sparse Mixture-of-Experts model (256 experts, only 8 active per token), so the file can be large while the work per token stays small. That's what lets a 35B model run on a phone, and generate fast on a CPU with no graphics card.
Measured (POCKET-35B IQ1_M vs Bonsai-27B Q1_0):
• CPU generate (Xeon, 16 threads): 27.0 vs 10.1 tok/s → 2.69× faster
• GPU generate (H100): 197 vs 89 tok/s → 2.22× faster
• GPU prompt processing (H100): 753 vs 1816 → 0.41× (Bonsai wins this one — MoE prefill wakes every expert, so sparsity stops helping there. We say so.)
• Quality (HellaSwag, 400 q): 61.0% vs 60.0% → a tie (confidence intervals overlap)
On a real consumer laptop — MacBook M3 Pro (18 GB) — POCKET wins every axis, prompt processing included:
• Metal generate: 25.4 vs 12.8 → 1.99×
• CPU generate: 13.8 vs 4.4 → 3.13×
• Metal prompt: 240.7 vs 73.4 → 3.28×
One more quiet fact: the same-size, quality-oriented rival Ternary-Bonsai-27B (7.2 GB) fails to load in upstream llama.cpp at all — it needs the PrismML fork. POCKET runs on the tools you already have: LM Studio, Ollama, PocketPal, MLX.
📖 Full story (tech, measurements, recipes): https://huggingface.co/blog/FINAL-Bench/pocket
Models:
📦 POCKET-35B-GGUF (PC / server, no GPU): https://huggingface.co/FINAL-Bench/POCKET-35B-GGUF
🇰🇷 POCKET-KR-GGUF (Android): https://huggingface.co/FINAL-Bench/POCKET-KR-GGUF
🍎 POCKET-KR-MLX (iPhone / Mac): https://huggingface.co/FINAL-Bench/POCKET-KR-MLX
🌍 POCKET-EN-GGUF (English phone / PC): https://huggingface.co/FINAL-Bench/POCKET-EN-GGUF
🖥️ Live demo (answering on a CPU, no GPU): https://huggingface.co/spaces/FINAL-Bench/POCKET-35B-CPU
📚 Collection: https://huggingface.co/collections/FINAL-Bench/pocket-models-6a618ee5d23eafb7e185a5c6
Organizations
harness self-test (own repo, will be closed)
1
#4 opened 7 days ago
by dipankarsarkar
self-test PR (endpoint confirmation)
1
#3 opened 7 days ago
by dipankarsarkar
endpoint confirmation (self-test, will be closed)
1
#2 opened 10 days ago
by dipankarsarkar
endpoint confirmation (self-test, will be closed)
1
#2 opened 10 days ago
by dipankarsarkar