Spaces:
Running
Running
| # Architecture | |
| ## One engine, many hosts | |
| ``` | |
| crates/microquack-core crates/microquack-llm | |
| (canonical: all sound (the droid mind: tiny | |
| behavior) transformer inference) | |
| \ / | | |
| crates/microquack-wasm trained in llm/ | |
| / (browser binding) (PyTorch -> binary weights) | |
| crates/microquack-cli | |
| (render / validate / fit / schema) | |
| | | |
| apps/web (TypeScript: UI, canvas, WebAudio, 3D stage only) | |
| | | |
| microduck robot (native crates, 48 kHz PCM) <- planned consumer | |
| ``` | |
| `microquack-core` contains everything that affects the sound: score types and | |
| parsing, pitch curves, envelopes, oscillators and partials, ring modulation, | |
| noise, the seeded PRNG, phrase grammar, the canonical semantic emotion | |
| registry and voice-mode shaping, contour output for motion, and rendering — | |
| both full-phrase (`render_score`) and incremental | |
| blocks (`Engine::render_into`, which `render_score` itself uses, so offline | |
| and streaming output are identical). It contains no WebAudio, WASM APIs, | |
| device access, filesystem loading, browser code, or Python-specific code. | |
| ## The score contract | |
| A score is a JSON list of syllables with snake_case fields (`f0_start`, | |
| `amp_points`, ...), defined once as Serde types in `microquack-core`. | |
| `schema/score.schema.json` is generated from those types (`pnpm run schema`); | |
| nothing else defines the format. `schema/emotions.json` and | |
| `apps/web/src/emotions.generated.ts` are likewise generated from the Rust | |
| registry with `pnpm run emotions`. Python uses the JSON file for teacher-data | |
| generation. Scores use seconds and hertz, so they are sample-rate | |
| independent; the renderer owns all rate-specific behavior. The TypeScript app | |
| treats scores as opaque JSON strings — it never reimplements or redeclares the | |
| format. | |
| ## The chat and the droid mind | |
| The visitor reads and writes English; the robot answers in droid speak with | |
| an English gloss. `crates/microquack-llm` runs a ~1.3M-parameter | |
| character-level transformer (trained from scratch in `llm/` on a synthetic | |
| space-adventure corpus) with a KV cache and greedy decoding, compiled into | |
| the same WASM module as the synth. Its output is `(emotion, reply)`; the | |
| emotion drives phrase generation and the body-language gestures. Rust/ | |
| PyTorch parity is enforced by `cargo test -p microquack-llm` against the | |
| committed model (`apps/web/public/assets/llm/droidmind.bin`) and test | |
| vectors. The current model artifact supports its original eight emotions. | |
| Rust validates that list against the canonical 73-state registry, then maps | |
| each semantic state to one of 12 broad voice modes and four shared affect | |
| values. The expanded 1,000-row teacher set is review data only; it does not | |
| replace the production weights. Between conversations the robot utters an | |
| idle thought every 10–15 seconds. | |
| ## The web app | |
| `apps/web` owns UI, canvas, and WebAudio only: | |
| - `src/wasm/` — generated wasm-bindgen output (`pnpm run build:wasm`). | |
| - `robot.ts` — assembles the real microduck from its mesh library | |
| (`microduck.glb`) and kinematic tree (`kinematics.json`, MuJoCo | |
| convention), exposes head joints and a synthetic jaw hinge. | |
| - `scene.ts` — the procedural night-plain world (three moons, stars, rocks) | |
| and the speech-driven articulation. | |
| - `mind.ts` / `droidtext.ts` / `gestures.ts` — the chat brain wrapper, | |
| droid-speak captions, and generic body language driven by the Rust voice | |
| mode and affect values. TypeScript does not map the 73 semantic states. | |
| - Short phrases render whole (`render_score` → `AudioBuffer`). Continuous | |
| synthesis, when it arrives, must run the `Engine` in an AudioWorklet — | |
| WASM alone does not prevent main-thread stalls. | |
| ## Research | |
| Fitting is a CLI subcommand (`microquack fit`): it analyzes a real WAV | |
| (pitch track, envelope, harmonics, noisiness) and emits a breakpoint score, | |
| rendering every candidate through the same engine that ships. `research/` | |
| holds the fitted corpus (`scores/fits.json`), pregenerated phrase scores, | |
| and the frozen conformance fixtures. | |
| ## Conformance gate | |
| `crates/microquack-core/tests/conformance.rs` renders all 90 frozen scores | |
| (79 fitted + 6 phrases + 5 presets) against `research/fixtures/`: exact | |
| sample counts, RMS/peak within 1e-6, and 21 full waveforms within 1e-4 per | |
| sample. Streaming must equal offline rendering exactly, and phrase generation | |
| must be seed-deterministic. Any DSP change that breaks these is a behavior | |
| change and requires refreezing fixtures deliberately. | |
| ## Device integration (planned) | |
| The Microduck repository consumes `microquack-core` at an exact version | |
| (`microquack-core = "=x.y.z"`, or a pinned git revision until published) — | |
| never vendored source. The device adapter feeds `Engine::render_into` into | |
| its existing 48 kHz PCM path and reads `ControlFrame` values to animate the | |
| physical beak and body. The 79 fitted scores must be revalidated at 48 kHz | |
| before the robot ships them. | |
| ## Release model | |
| One git tag should produce, all reporting the same version and commit: | |
| - the `microquack-core` crate (device), | |
| - the WASM/npm package (website), | |
| - the CLI binary (render, validate, fit, schema, emotions), | |
| - the score corpus and generated schemas. | |
| ## Roadmap | |
| See [VISION.md](VISION.md): preset conversations, the droid-speak language | |
| model, procedural full-body motion, and browser-side locomotion policies from | |
| `microduck_rl` via ONNX Runtime Web. | |