Download README.md from Bosalt/README: direct link, hf CLI and curl.
- Browser
- Download file 29.2 kB
-
https://huggingface.co/spaces/Bosalt/README/resolve/main/README.md
- Command line
-
hf download hf://spaces/Bosalt/README/README.md
-
curl -L -o README.md https://huggingface.co/spaces/Bosalt/README/resolve/main/README.md
title: Basalt v0.11.0
emoji: ⚡
colorFrom: blue
colorTo: gray
sdk: static
pinned: false
basalt — your Mac is the AI lab now.
Serve, train, quantize, compose and dissect open LLMs, neural speech, OCR and image models on Apple Silicon. One Swift 6.4 binary speaks the OpenAI API, the Anthropic API and MCP, and a native macOS 27 Liquid Glass app sits on top. No Python at runtime. No cloud unless you ask for it.
Runs open-weight models on Apple Silicon. Logit-verified here: Gemma 4, Qwen3.8 and LFM2.5 (the table below); measured and packaged in huggingface.co/goodmai: Ornith, Bonsai, Qwen3.6-35B-A3B, DeepSeek-OCR-2, OpenJev and Laya (see the models table); gpt-oss and Llama are MLX-compatible. Logos are trademarks of their owners, shown only to indicate compatibility. Anthropic's models are closed-weight, so they are not in the hive; the Anthropic API is supported so that Claude Code can talk to a local model.
Five things in one repo: a local inference server your tools already know how to talk to, LoRA distillation and post-quantization tuning, a docker-compose for multi-model systems, a live X-ray microscope for what happens inside the model, and a native Mac app that puts it all one keystroke away.
Checked against reference logits on every change to the inference path: GoldenLogitsTests replays recorded prompts and fails when a greedy token or a logit moves. Per-family reference logits for qwen3, qwen3_5, gemma4, lfm2 and nemotron_h, dense and MoE, are in Fixtures/ReferenceLogits.
| Golden-tested model | Class | Fixture |
|---|---|---|
mlx-community/gemma-4-26B-A4B-it-qat-4bit |
sliding-window + full attention, MoE | golden |
lukaskremla/Qwen3.8-27B-3bit-MLX-TextOnly |
hybrid linear/full attention, dense | golden |
LiquidAI/LFM2.5-24B-A2B-MLX-4bit |
hybrid conv + attention, MoE | golden |
One-click launch from huggingface.co/goodmai (42+ verified models)
Every model card under goodmai carries a Run with Basalt button (goodmai-run-with-basalt.static.hf.space) that opens Basalt.app via basalt://run?model=... or prints the exact basalt serve / basalt compose CLI command:
| Model / Stack | Role & footprint on 24 GB Mac | Launch |
|---|---|---|
goodmai/gemma-4-12B-it-qat-4bit |
12B QAT daily-driver chat & vision (~7.6 GB) | |
goodmai/gemma-4-26B-A4B-it-qat-4bit |
26B MoE (A4B), golden-verified (~14.5 GB) |
|
goodmai/Qwen3.8-27B-QTIP-2.5bpw-MLX |
27B QTIP 2.5 bpw trellis (10.0 GB, 16k PPL 4.621) | |
goodmai/Qwen3.8-27B-GSQ-RCO-IQ3_S-Basalt-Native |
27B GSQ-RCO native i-quant (11.8 GB, PPL 4.17, 300/300 System One) | |
goodmai/Qwen3.6-35B-A3B-8bit |
35B MoE in 37.7 GB streamed from SSD at 8.77 tok/s, 0 swap | |
goodmai/Ternary-Bonsai-2-27B-MTPLX-Optimized-Speed |
27B Hadamard ternary in 7.4 GB, 27.1 tok/s with --mtp |
|
goodmai/LFM2.5-24B-A2B-MLX-4bit |
24B hybrid conv + attention MoE, golden-verified (~12.6 GB) | |
Zyphra/ZONOS3-Mini-hf |
44.1 kHz neural TTS across 43 languages (2.24 GB, RTF 0.23–0.26) | |
goodmai/DeepSeek-OCR-2-8bit |
Vision-encoder OCR with official 768 tiling (/v1/pipeline) |
|
goodmai/OpenJev-Flash-9B-MLX-4bit |
9B local System One decider (98.9% acc, −77% LLM tokens) | |
24gb-voice-zonos |
Recommended 24 GB Compose stack (Chat + ZONOS3 + Kokoro ANE) |
Inference: 40B MoE at 67 tok/s, 27B in 7.4 GB, or 37.7 GB MoE with zero swap — behind the APIs you already use
- Claude Code on your own GPU in one line.
ANTHROPIC_BASE_URL=http://localhost:8080 ANTHROPIC_AUTH_TOKEN=local claude— your real API key is never touched. OpenAI SDKs, OpenCode, Continue, Cline and aider connect over the OpenAI API, Claude Desktop and Cursor over MCP (with per-tool and per-group--mcp-enable/--mcp-disable, #488). - Fast on a laptop. On an M4 Pro with 24 GB: Recursive
Flint-40B-A2.5B(oQ3e) decodes at 67.02 tok/s in 17.61 GB (#522); Qwen3-4B decodes at 78–89 tok/s; Ternary Bonsai 2 27B fits in 7.4 GB and runs at 20.9–21.5 tok/s, 27.1 with its MTP head and 31–38 with prompt lookup on code (#239, #263);.ggufGSQ-RCO /i-quantcheckpoints execute directly on custom Metal kernels (#433, #515). - Models larger than RAM, zero swap.
Qwen3.6-35B-A3B-8bit(37.7 GB) decodes at 8.77 tok/s on a 24 GB Mac with 0 swap via batched POSIX AIO andLIO_NOWAITnext-layer SSD expert prefetch (--expert-streaming, #477, #478, #516). - Speculative decoding that stays exact. Prompt lookup gives 1.84–2.66× on code edits (Bonsai 2 27B, #239) and 3.45–3.62× (Gemma 4 12B, #265), with greedy output unchanged.
- The next turn costs only its new tokens. A prefix-tree prompt cache, even for hybrid models: on Ornith-1.5-9B with a 12.6k-token agent prompt, the second turn takes about 1 s instead of about 36 s (#82).
- Long contexts in less memory. KV quantization with exact zones (
4|8|k8v4, plus experimentalturbo3|turbo4|turbo8v4andpolar4RoPE-pair keys, #472, #473): a 16k prompt peaks at 1.03 GB instead of 3.11 GB on Qwen3-4B (#159). - Neural TTS, full-duplex speech & native MLX image synthesis.
Zyphra/ZONOS3-Mini-hfsynthesizes44.1 kHzspeech across 43 languages and 27 voices in 2.24 GB at 3.8–4.4× real time (RTF 0.228–0.263, #527) alongsideKokoro-82M(ANE+MLX),Qwen3-TTS,Fun-CosyVoice3andChatterboxon/v1/audio/speech(#506, #513), whileFLUX.2/3-klein,Z-Image-Turbo,Iris-3B,SANA-1.5andStep1X-Editpower/v1/images/*(#519). - A whole local stack. Tool calls, JSON-schema, regex and grammar output (XGrammar),
Ripple-65BandNemotron-3.5-Lightning(#522),DeepSeek-OCR-2(#476),/v1/embeddings,/v1/rerank, LoRA/PHM/OFT adapters picked per request, and ordered LAN-before-cloud fallback tiers (#480).
Details: REST API · integrations · commands and flags · models · benchmarks
Training & Quantization: distill from a frontier model, recover sub-3-bit quality
- Distill a frontier model into a LoRA, PHM or OFT adapter.
basalt models distillsends your prompts to a teacher behind an OpenAI- or Anthropic-format API, caps the spend with--budget-usd, can keep only verified answers, and checkpoints so an interrupted run resumes (#393).--on-policytrains on the student's own samples with per-token reverse KL from a basalt teacher, andbasalt compose trainstreams Hugging Facenemotron:*corpora intolora, hypercomplexphm/Compacter(#501) or GPU-pure Cayley / matrix-expoft(#508, #511) adapters. - Post-quantization PV-Tuning.
basalt models convert --experimental-pv-tunealternates continuous scale/bias updates with discrete integer-code refinement on Apple Silicon, cutting 2–3-bit quantization perplexity loss by 8.7–29% in under 3 minutes (#486, #515). - Bitshift trellis quantization (QTIP).
basalt models convert --experimental-qtip --bits 2.5fitsQwen3.8-27Bin 10.00 GB and beats 3-bit MLX affine & PV-Tuning on 16k perplexity (4.621 vs 4.702 / 4.652), while 2.0 bpw (8.48 GB) cuts 2-bit perplexity by 72.1% (5.345 vs 19.146, #494, #515). - Train while you serve. The MCP
distilltool is time-sliced: the model is held one optimizer step at a time, soservekeeps answering. - Self-training that learns nothing unverified.
train_expertscollects answers to checkable tasks, keeps only the verified ones, trains a LoRA and reports held-out accuracy before and after. - A quality gate before anything sticks. A gated adapter stays only if KL and perplexity on held-out prompts stay within limits; a failed gate undoes it bit for bit.
- JEPA and conversion.
basalt models train-jepatrains a JEPA adapter and projector on multi-view data;basalt models convertimports.ggufi-quants, merges a LoRA, requantizes or exports HF safetensors.
Details: training commands · self-training, distillation and the quality gate
Compose: one YAML file — or 84 hardware-calibrated profiles — for a whole model system
- docker-compose, for models. Local and cloud models, how each stays in memory (
resident,on-demandwithidle_unload#481,sequential), and a perceive → decide → answer →speak/imaginepipeline — OCR, ASR, a System One + Belnap epistemic cascade, the chat model,ZONOS3-Mini44.1 kHz voice (#527) and MLX image synthesis (#519) — in one file. - 84 built-in profiles for every Mac (
16 GBto256 GB). Seven unified-memory tiers (16gb,24gb,32-36gb,48gb,64gb,128gb,256gb, plus32gb,36gband236gbaliases) × 12 workload archetypes (daily-chat,coding-agent,voice-zonos,multimodal-desk,reasoning-lab,document-ocr,voice-duplex,creative-studio,rag-knowledge,hybrid-cloud,moe-frontier,research-xray) launch in one command (basalt compose up 24gb-voice-zonos, #528). - Refuses what will not fit.
basalt compose planadds up the bytes (including unmerged Jev LoRA adapters #462), sizing models you have not downloaded from the Hugging Face API, and refuses a system that does not fit before anything loads.compose pullfetches and verifies every model ahead of time;compose benchruns declarative benchmark suites and exports 10 leaderboard submission bundles (#505). - Change it live.
compose diffshows what would change;compose upon a running system reloads only the models, experimental flags or MCP toggles that changed (#489) and swaps pipelines between requests.compose psshows state and spend,compose downstops it. - Local judges and cloud budgets.
OpenJev-Flash-9B-4bitfits beside Gemma 4 12B in 15.3 GB on a 24 GB Mac, reaching 98.9% triage accuracy while cutting downstream LLM tokens by 77% (#465), and Laya answers on the Neural Engine in ~11 ms. Answer routes send hard queries to a cloud model while its daily budget lasts: 100% accuracy at $0.035 per 1000 requests, p95 4.8 s vs 10.0 s local only (#418). - Private by default. Keys come only from environment variables, and text read from your documents or audio never goes to a cloud model.
- Google's Gemini Decision 1 as a decision stage.
cloud:gemini/gemini-decision-1answers yes/no, choice and score questions over text, images, audio, video and PDFs in one pass, in front of a local decider (#449), andbasalt models calibrate-decidefits local readout temperatures against it (#504).
Details: compose reference and measurements · pipeline presets and memory modes · cloud decisions with Gemini
Research: open the model up while it runs
- Model X-ray, Belnap epistemic gating and live telemetry.
model_describe,model_tensors,model_tensor,kv_inspect,trace_runandtelemetry_queryexpose layer anatomy, Marchenko–Pastur spectra, the prompt-cache tree, per-head logit lens, four-valued Belnap(support, refutation, needs_tool)evidence states on theSIXTEEN_3trilattice (#500, #507), complex MVN phasor probes (#509), Fisher–Rao geodesic drift (#503), and OpenMetrics at/api/v1/metrics(#458, #498). - Reversible experiments. With
serve --research-tools: steering and head or expert ablation (intervene), weight deltas, live re-packing of a layer to 2–8-bit, mxfp4, nf4 or ternary, andexperiment_comparewith KL and perplexity. Every experiment is an undoable overlay with a TTL and a journal; model files on disk are never written. - Find the circuit behind a behaviour.
trace_tokensandactivation_contrastrank heads and experts by Cohen's d and AUROC, then a conditional ablation fires only on the tokens that trigger it (#405).steer_searchandsteer_committurn contrastive directions into gated adapters. - The model studies itself.
introspecthands the X-ray tools to the served model and records its hypotheses, tests and outcomes. - Papers and hybrid compute in the engine. CacheBlend (arXiv:2405.16444), EPIC chunk heads (arXiv:2410.15332), L2-norm KV eviction (arXiv:2406.11430), DuoAttention, Token Recycling, Walsh–Hadamard value rotation, and Neural Engine System One / embedding / OCR towers (#512, #516) — each opt-in, with its measured trade-offs next to its flag.
- Benchmarks that run the model's code. A five-rung ladder ending in an Earth-to-Moon mission executes the model's own code against unseen inputs, and ARC-AGI runs under official pass@2 rules.
Details: research tools · benchmarks · serve flags
Basalt for Mac: macOS 27 Liquid Glass, one keystroke or one click away
- A native SwiftUI app on a supervised engine. Built for macOS 27 Liquid Glass (#479) in English and Russian (#456): 257 ms warm window, 0 idle wakeups, 7.4 ms p95 per streamed token in a 2,000-message transcript, and 40 ms
⌥Spacequick-ask overhead (#463). - Compose Profiles (
⌥⌘C) with automatic RAM recommendation. Detects your Mac's unified memory (16 / 24 / 32–36 / 48 / 64 / 128 / 256 GB), highlights the 4 primary recommended profiles for your machine at the top, lets you inspect YAML and switch any of all 84 profiles live (#528). - Full-duplex voice & 2K Image Studio. Hands-free streaming conversation backed by
SpeechAnalyzerandZONOS3-Mini/Kokoro-82M(#514, #527), plus an interactive 2K RGBA Image Canvas with live TAEF previews and a Before/After slider (#519). - One-click
basalt://launch from Hugging Face. Clicking Run with Basalt on anygoodmaimodel card opensBasalt.appviabasalt://run?model=...and switches the supervised engine to that model or pipeline role. - Ask from anywhere & drop in files.
⌥Spaceopens a floating panel over any app without an Accessibility permission; PDF, image and audio attachments run throughperceive → decide → answerwith full stage receipts, alongside Shortcuts, Spotlight, Monitor and notarized.dmg+ Homebrew cask packaging (#457).
Details: Basalt for Mac
Quick start
Requirements: macOS 27+, Xcode 27.0+ / Swift 6.4 with the Metal toolchain (xcodebuild -downloadComponent MetalToolchain), Apple Silicon M1–M4, 2–38 GB of disk depending on the model. Full steps, building from source and the ./basalt-claude launcher are in docs/install.md.
brew tap goodmai/basalt https://github.com/goodmai/basalt
brew trust goodmai/basalt
brew install goodmai/basalt/basalt # builds CLI & server from source
basalt onboard # picks a model that fits your Mac and downloads it
basalt serve --model goodmai/gemma-4-12B-it-qat-4bit --rest
# in another terminal: Claude Code, backed by your own GPU
ANTHROPIC_BASE_URL=http://localhost:8080 ANTHROPIC_AUTH_TOKEN=local claude
Stream a 37.7 GB MoE model on a 24 GB Mac, or run a full multi-model compose system:
basalt serve --model goodmai/Qwen3.6-35B-A3B-8bit --expert-streaming --rest
basalt compose plan 24gb-voice-zonos
basalt compose up 24gb-voice-zonos # 84 profiles across 16–256 GB Macs
The Mac app, from a checkout (registers basalt:// 1-click links automatically on first open):
./scripts/bundle_app.swift # release build, Metal kernels, then .build/Basalt.app
open .build/Basalt.app
Local 4B–12B models are not a drop-in replacement for a frontier model — they are for private, offline, zero-cost agentic work and for hybrid local/cloud systems. basalt fit tells you what your machine can actually run. The model table lists what broke too.
Star it if you want more MLX-native tooling in Swift, and open an issue with your Mac and your tok/s — PRs adding a row to the benchmark table are welcome.
Documentation
| Topic | Doc |
|---|---|
Install, requirements, build from source, ./basalt-claude |
docs/install.md |
CLI commands, serve flags, prompt cache, CacheBlend, PV-Tuning |
docs/commands.md |
| REST API: security, model management, OpenAI, Anthropic, images, SSE, OpenMetrics, Swagger | docs/rest-api.md |
| Claude Code, OpenCode, OpenAI SDKs, MCP for Claude Desktop and Cursor | docs/integrations.md |
Verified models, goodmai Hub stack and downloading |
docs/models.md |
| Features at a glance, benchmark ladder, ARC-AGI, provenance | docs/benchmarks.md |
| Pipeline presets and memory modes | docs/pipelines.md |
| Cloud decisions: Gemini Decision 1, multimodal media, the calibration teacher | docs/cloud-decisions.md |
basalt compose reference and measurements |
docs/compose.md |
Basalt for Mac & basalt:// deep links |
docs/mac-app.md |
| Model X-ray, telemetry, experiments, self-training, distillation | docs/research-tools.md |
| Project structure, architecture notes, scripts, dependencies | docs/architecture.md |
| Architecture patterns and modules (Russian) | docs/ARCHITECTURE_PATTERNS_AND_MODULES.md |
| Cloud: OpenRouter setup, cost optimization, model comparison | docs/cloud/OPENROUTER_SETUP.md · docs/cloud/COST_OPTIMIZATION.md · docs/cloud/MODEL_COMPARISON.md |
| Working on basalt with AI agents | docs/AI_SETUP.md · docs/AGENT_OPERATIONS.md |
| Roadmap, privacy policy | docs/ROADMAP.md · docs/PRIVACY_POLICY.md |
| Research notes on agent CLIs, checklists, specs (Russian) | docs/features.md · docs/specs/FEATURES_CHECKLIST.md · docs/specs/AUDIT_CHECKLIST.md · docs/specs/EMOJI_AS_CODE.md |
| Contributing, changelog | CONTRIBUTING.md · CHANGELOG.md |
License
MIT © 2026 goodmai
Model weights are not covered by this license — each model on HuggingFace carries its own terms.