README / README.md
goodmai's picture
Escape parentheses in HF Models badge URL
63800ad verified
|
Raw History Blame Contribute Delete
29.2 kB
metadata
title: Basalt v0.11.0
emoji: ⚡
colorFrom: blue
colorTo: gray
sdk: static
pinned: false

basalt logo

basalt — your Mac is the AI lab now.

Serve, train, quantize, compose and dissect open LLMs, neural speech, OCR and image models on Apple Silicon. One Swift 6.4 binary speaks the OpenAI API, the Anthropic API and MCP, and a native macOS 27 Liquid Glass app sits on top. No Python at runtime. No cloud unless you ask for it.

English · Русский

Release v0.11.0 CI Release Security Audit License: MIT Models on HF Run with Basalt

Platform Swift 6.4 Homebrew Ladder ARC-AGI

Open models basalt runs: Qwen, Gemma, Ornith, gpt-oss, DeepSeek, Llama

Runs open-weight models on Apple Silicon. Logit-verified here: Gemma 4, Qwen3.8 and LFM2.5 (the table below); measured and packaged in huggingface.co/goodmai: Ornith, Bonsai, Qwen3.6-35B-A3B, DeepSeek-OCR-2, OpenJev and Laya (see the models table); gpt-oss and Llama are MLX-compatible. Logos are trademarks of their owners, shown only to indicate compatibility. Anthropic's models are closed-weight, so they are not in the hive; the Anthropic API is supported so that Claude Code can talk to a local model.

Five things in one repo: a local inference server your tools already know how to talk to, LoRA distillation and post-quantization tuning, a docker-compose for multi-model systems, a live X-ray microscope for what happens inside the model, and a native Mac app that puts it all one keystroke away.

Checked against reference logits on every change to the inference path: GoldenLogitsTests replays recorded prompts and fails when a greedy token or a logit moves. Per-family reference logits for qwen3, qwen3_5, gemma4, lfm2 and nemotron_h, dense and MoE, are in Fixtures/ReferenceLogits.

Golden-tested model Class Fixture
mlx-community/gemma-4-26B-A4B-it-qat-4bit sliding-window + full attention, MoE golden
lukaskremla/Qwen3.8-27B-3bit-MLX-TextOnly hybrid linear/full attention, dense golden
LiquidAI/LFM2.5-24B-A2B-MLX-4bit hybrid conv + attention, MoE golden

One-click launch from huggingface.co/goodmai (42+ verified models)

Every model card under goodmai carries a Run with Basalt button (goodmai-run-with-basalt.static.hf.space) that opens Basalt.app via basalt://run?model=... or prints the exact basalt serve / basalt compose CLI command:

Model / Stack Role & footprint on 24 GB Mac Launch
goodmai/gemma-4-12B-it-qat-4bit 12B QAT daily-driver chat & vision (~7.6 GB) Run with Basalt
goodmai/gemma-4-26B-A4B-it-qat-4bit 26B MoE (A4B), golden-verified (~14.5 GB) Run with Basalt
goodmai/Qwen3.8-27B-QTIP-2.5bpw-MLX 27B QTIP 2.5 bpw trellis (10.0 GB, 16k PPL 4.621) Run with Basalt
goodmai/Qwen3.8-27B-GSQ-RCO-IQ3_S-Basalt-Native 27B GSQ-RCO native i-quant (11.8 GB, PPL 4.17, 300/300 System One) Run with Basalt
goodmai/Qwen3.6-35B-A3B-8bit 35B MoE in 37.7 GB streamed from SSD at 8.77 tok/s, 0 swap Run with Basalt
goodmai/Ternary-Bonsai-2-27B-MTPLX-Optimized-Speed 27B Hadamard ternary in 7.4 GB, 27.1 tok/s with --mtp Run with Basalt
goodmai/LFM2.5-24B-A2B-MLX-4bit 24B hybrid conv + attention MoE, golden-verified (~12.6 GB) Run with Basalt
Zyphra/ZONOS3-Mini-hf 44.1 kHz neural TTS across 43 languages (2.24 GB, RTF 0.23–0.26) Run with Basalt
goodmai/DeepSeek-OCR-2-8bit Vision-encoder OCR with official 768 tiling (/v1/pipeline) Run with Basalt
goodmai/OpenJev-Flash-9B-MLX-4bit 9B local System One decider (98.9% acc, −77% LLM tokens) Run with Basalt
24gb-voice-zonos Recommended 24 GB Compose stack (Chat + ZONOS3 + Kokoro ANE) Run with Basalt

Inference: 40B MoE at 67 tok/s, 27B in 7.4 GB, or 37.7 GB MoE with zero swap — behind the APIs you already use

  • Claude Code on your own GPU in one line. ANTHROPIC_BASE_URL=http://localhost:8080 ANTHROPIC_AUTH_TOKEN=local claude — your real API key is never touched. OpenAI SDKs, OpenCode, Continue, Cline and aider connect over the OpenAI API, Claude Desktop and Cursor over MCP (with per-tool and per-group --mcp-enable / --mcp-disable, #488).
  • Fast on a laptop. On an M4 Pro with 24 GB: Recursive Flint-40B-A2.5B (oQ3e) decodes at 67.02 tok/s in 17.61 GB (#522); Qwen3-4B decodes at 78–89 tok/s; Ternary Bonsai 2 27B fits in 7.4 GB and runs at 20.9–21.5 tok/s, 27.1 with its MTP head and 31–38 with prompt lookup on code (#239, #263); .gguf GSQ-RCO / i-quant checkpoints execute directly on custom Metal kernels (#433, #515).
  • Models larger than RAM, zero swap. Qwen3.6-35B-A3B-8bit (37.7 GB) decodes at 8.77 tok/s on a 24 GB Mac with 0 swap via batched POSIX AIO and LIO_NOWAIT next-layer SSD expert prefetch (--expert-streaming, #477, #478, #516).
  • Speculative decoding that stays exact. Prompt lookup gives 1.84–2.66× on code edits (Bonsai 2 27B, #239) and 3.45–3.62× (Gemma 4 12B, #265), with greedy output unchanged.
  • The next turn costs only its new tokens. A prefix-tree prompt cache, even for hybrid models: on Ornith-1.5-9B with a 12.6k-token agent prompt, the second turn takes about 1 s instead of about 36 s (#82).
  • Long contexts in less memory. KV quantization with exact zones (4|8|k8v4, plus experimental turbo3|turbo4|turbo8v4 and polar4 RoPE-pair keys, #472, #473): a 16k prompt peaks at 1.03 GB instead of 3.11 GB on Qwen3-4B (#159).
  • Neural TTS, full-duplex speech & native MLX image synthesis. Zyphra/ZONOS3-Mini-hf synthesizes 44.1 kHz speech across 43 languages and 27 voices in 2.24 GB at 3.8–4.4× real time (RTF 0.228–0.263, #527) alongside Kokoro-82M (ANE+MLX), Qwen3-TTS, Fun-CosyVoice3 and Chatterbox on /v1/audio/speech (#506, #513), while FLUX.2/3-klein, Z-Image-Turbo, Iris-3B, SANA-1.5 and Step1X-Edit power /v1/images/* (#519).
  • A whole local stack. Tool calls, JSON-schema, regex and grammar output (XGrammar), Ripple-65B and Nemotron-3.5-Lightning (#522), DeepSeek-OCR-2 (#476), /v1/embeddings, /v1/rerank, LoRA/PHM/OFT adapters picked per request, and ordered LAN-before-cloud fallback tiers (#480).

Details: REST API · integrations · commands and flags · models · benchmarks

Training & Quantization: distill from a frontier model, recover sub-3-bit quality

  • Distill a frontier model into a LoRA, PHM or OFT adapter. basalt models distill sends your prompts to a teacher behind an OpenAI- or Anthropic-format API, caps the spend with --budget-usd, can keep only verified answers, and checkpoints so an interrupted run resumes (#393). --on-policy trains on the student's own samples with per-token reverse KL from a basalt teacher, and basalt compose train streams Hugging Face nemotron:* corpora into lora, hypercomplex phm / Compacter (#501) or GPU-pure Cayley / matrix-exp oft (#508, #511) adapters.
  • Post-quantization PV-Tuning. basalt models convert --experimental-pv-tune alternates continuous scale/bias updates with discrete integer-code refinement on Apple Silicon, cutting 2–3-bit quantization perplexity loss by 8.7–29% in under 3 minutes (#486, #515).
  • Bitshift trellis quantization (QTIP). basalt models convert --experimental-qtip --bits 2.5 fits Qwen3.8-27B in 10.00 GB and beats 3-bit MLX affine & PV-Tuning on 16k perplexity (4.621 vs 4.702 / 4.652), while 2.0 bpw (8.48 GB) cuts 2-bit perplexity by 72.1% (5.345 vs 19.146, #494, #515).
  • Train while you serve. The MCP distill tool is time-sliced: the model is held one optimizer step at a time, so serve keeps answering.
  • Self-training that learns nothing unverified. train_experts collects answers to checkable tasks, keeps only the verified ones, trains a LoRA and reports held-out accuracy before and after.
  • A quality gate before anything sticks. A gated adapter stays only if KL and perplexity on held-out prompts stay within limits; a failed gate undoes it bit for bit.
  • JEPA and conversion. basalt models train-jepa trains a JEPA adapter and projector on multi-view data; basalt models convert imports .gguf i-quants, merges a LoRA, requantizes or exports HF safetensors.

Details: training commands · self-training, distillation and the quality gate

Compose: one YAML file — or 84 hardware-calibrated profiles — for a whole model system

  • docker-compose, for models. Local and cloud models, how each stays in memory (resident, on-demand with idle_unload #481, sequential), and a perceive → decide → answer → speak / imagine pipeline — OCR, ASR, a System One + Belnap epistemic cascade, the chat model, ZONOS3-Mini 44.1 kHz voice (#527) and MLX image synthesis (#519) — in one file.
  • 84 built-in profiles for every Mac (16 GB to 256 GB). Seven unified-memory tiers (16gb, 24gb, 32-36gb, 48gb, 64gb, 128gb, 256gb, plus 32gb, 36gb and 236gb aliases) × 12 workload archetypes (daily-chat, coding-agent, voice-zonos, multimodal-desk, reasoning-lab, document-ocr, voice-duplex, creative-studio, rag-knowledge, hybrid-cloud, moe-frontier, research-xray) launch in one command (basalt compose up 24gb-voice-zonos, #528).
  • Refuses what will not fit. basalt compose plan adds up the bytes (including unmerged Jev LoRA adapters #462), sizing models you have not downloaded from the Hugging Face API, and refuses a system that does not fit before anything loads. compose pull fetches and verifies every model ahead of time; compose bench runs declarative benchmark suites and exports 10 leaderboard submission bundles (#505).
  • Change it live. compose diff shows what would change; compose up on a running system reloads only the models, experimental flags or MCP toggles that changed (#489) and swaps pipelines between requests. compose ps shows state and spend, compose down stops it.
  • Local judges and cloud budgets. OpenJev-Flash-9B-4bit fits beside Gemma 4 12B in 15.3 GB on a 24 GB Mac, reaching 98.9% triage accuracy while cutting downstream LLM tokens by 77% (#465), and Laya answers on the Neural Engine in ~11 ms. Answer routes send hard queries to a cloud model while its daily budget lasts: 100% accuracy at $0.035 per 1000 requests, p95 4.8 s vs 10.0 s local only (#418).
  • Private by default. Keys come only from environment variables, and text read from your documents or audio never goes to a cloud model.
  • Google's Gemini Decision 1 as a decision stage. cloud:gemini/gemini-decision-1 answers yes/no, choice and score questions over text, images, audio, video and PDFs in one pass, in front of a local decider (#449), and basalt models calibrate-decide fits local readout temperatures against it (#504).

Details: compose reference and measurements · pipeline presets and memory modes · cloud decisions with Gemini

Research: open the model up while it runs

  • Model X-ray, Belnap epistemic gating and live telemetry. model_describe, model_tensors, model_tensor, kv_inspect, trace_run and telemetry_query expose layer anatomy, Marchenko–Pastur spectra, the prompt-cache tree, per-head logit lens, four-valued Belnap (support, refutation, needs_tool) evidence states on the SIXTEEN_3 trilattice (#500, #507), complex MVN phasor probes (#509), Fisher–Rao geodesic drift (#503), and OpenMetrics at /api/v1/metrics (#458, #498).
  • Reversible experiments. With serve --research-tools: steering and head or expert ablation (intervene), weight deltas, live re-packing of a layer to 2–8-bit, mxfp4, nf4 or ternary, and experiment_compare with KL and perplexity. Every experiment is an undoable overlay with a TTL and a journal; model files on disk are never written.
  • Find the circuit behind a behaviour. trace_tokens and activation_contrast rank heads and experts by Cohen's d and AUROC, then a conditional ablation fires only on the tokens that trigger it (#405). steer_search and steer_commit turn contrastive directions into gated adapters.
  • The model studies itself. introspect hands the X-ray tools to the served model and records its hypotheses, tests and outcomes.
  • Papers and hybrid compute in the engine. CacheBlend (arXiv:2405.16444), EPIC chunk heads (arXiv:2410.15332), L2-norm KV eviction (arXiv:2406.11430), DuoAttention, Token Recycling, Walsh–Hadamard value rotation, and Neural Engine System One / embedding / OCR towers (#512, #516) — each opt-in, with its measured trade-offs next to its flag.
  • Benchmarks that run the model's code. A five-rung ladder ending in an Earth-to-Moon mission executes the model's own code against unseen inputs, and ARC-AGI runs under official pass@2 rules.

Details: research tools · benchmarks · serve flags

Basalt for Mac: macOS 27 Liquid Glass, one keystroke or one click away

  • A native SwiftUI app on a supervised engine. Built for macOS 27 Liquid Glass (#479) in English and Russian (#456): 257 ms warm window, 0 idle wakeups, 7.4 ms p95 per streamed token in a 2,000-message transcript, and 40 ms ⌥Space quick-ask overhead (#463).
  • Compose Profiles (⌥⌘C) with automatic RAM recommendation. Detects your Mac's unified memory (16 / 24 / 32–36 / 48 / 64 / 128 / 256 GB), highlights the 4 primary recommended profiles for your machine at the top, lets you inspect YAML and switch any of all 84 profiles live (#528).
  • Full-duplex voice & 2K Image Studio. Hands-free streaming conversation backed by SpeechAnalyzer and ZONOS3-Mini / Kokoro-82M (#514, #527), plus an interactive 2K RGBA Image Canvas with live TAEF previews and a Before/After slider (#519).
  • One-click basalt:// launch from Hugging Face. Clicking Run with Basalt on any goodmai model card opens Basalt.app via basalt://run?model=... and switches the supervised engine to that model or pipeline role.
  • Ask from anywhere & drop in files. ⌥Space opens a floating panel over any app without an Accessibility permission; PDF, image and audio attachments run through perceive → decide → answer with full stage receipts, alongside Shortcuts, Spotlight, Monitor and notarized .dmg + Homebrew cask packaging (#457).

Details: Basalt for Mac


Quick start

Requirements: macOS 27+, Xcode 27.0+ / Swift 6.4 with the Metal toolchain (xcodebuild -downloadComponent MetalToolchain), Apple Silicon M1–M4, 2–38 GB of disk depending on the model. Full steps, building from source and the ./basalt-claude launcher are in docs/install.md.

brew tap goodmai/basalt https://github.com/goodmai/basalt
brew trust goodmai/basalt
brew install goodmai/basalt/basalt          # builds CLI & server from source

basalt onboard                              # picks a model that fits your Mac and downloads it
basalt serve --model goodmai/gemma-4-12B-it-qat-4bit --rest

# in another terminal: Claude Code, backed by your own GPU
ANTHROPIC_BASE_URL=http://localhost:8080 ANTHROPIC_AUTH_TOKEN=local claude

Stream a 37.7 GB MoE model on a 24 GB Mac, or run a full multi-model compose system:

basalt serve --model goodmai/Qwen3.6-35B-A3B-8bit --expert-streaming --rest

basalt compose plan 24gb-voice-zonos
basalt compose up 24gb-voice-zonos          # 84 profiles across 16–256 GB Macs

The Mac app, from a checkout (registers basalt:// 1-click links automatically on first open):

./scripts/bundle_app.swift                  # release build, Metal kernels, then .build/Basalt.app
open .build/Basalt.app

Local 4B–12B models are not a drop-in replacement for a frontier model — they are for private, offline, zero-cost agentic work and for hybrid local/cloud systems. basalt fit tells you what your machine can actually run. The model table lists what broke too.

Star it if you want more MLX-native tooling in Swift, and open an issue with your Mac and your tok/s — PRs adding a row to the benchmark table are welcome.


Documentation

Topic Doc
Install, requirements, build from source, ./basalt-claude docs/install.md
CLI commands, serve flags, prompt cache, CacheBlend, PV-Tuning docs/commands.md
REST API: security, model management, OpenAI, Anthropic, images, SSE, OpenMetrics, Swagger docs/rest-api.md
Claude Code, OpenCode, OpenAI SDKs, MCP for Claude Desktop and Cursor docs/integrations.md
Verified models, goodmai Hub stack and downloading docs/models.md
Features at a glance, benchmark ladder, ARC-AGI, provenance docs/benchmarks.md
Pipeline presets and memory modes docs/pipelines.md
Cloud decisions: Gemini Decision 1, multimodal media, the calibration teacher docs/cloud-decisions.md
basalt compose reference and measurements docs/compose.md
Basalt for Mac & basalt:// deep links docs/mac-app.md
Model X-ray, telemetry, experiments, self-training, distillation docs/research-tools.md
Project structure, architecture notes, scripts, dependencies docs/architecture.md
Architecture patterns and modules (Russian) docs/ARCHITECTURE_PATTERNS_AND_MODULES.md
Cloud: OpenRouter setup, cost optimization, model comparison docs/cloud/OPENROUTER_SETUP.md · docs/cloud/COST_OPTIMIZATION.md · docs/cloud/MODEL_COMPARISON.md
Working on basalt with AI agents docs/AI_SETUP.md · docs/AGENT_OPERATIONS.md
Roadmap, privacy policy docs/ROADMAP.md · docs/PRIVACY_POLICY.md
Research notes on agent CLIs, checklists, specs (Russian) docs/features.md · docs/specs/FEATURES_CHECKLIST.md · docs/specs/AUDIT_CHECKLIST.md · docs/specs/EMOJI_AS_CODE.md
Contributing, changelog CONTRIBUTING.md · CHANGELOG.md

License

MIT © 2026 goodmai

Model weights are not covered by this license — each model on HuggingFace carries its own terms.