docs/ARCHITECTURE.md

Architecture

One engine, many hosts

   crates/microquack-core        crates/microquack-llm
   (canonical: all sound          (the droid mind: tiny
    behavior)                      transformer inference)
        \                          /            |
         crates/microquack-wasm            trained in llm/
        /      (browser binding)           (PyTorch -> binary weights)
  crates/microquack-cli
  (render / validate / fit / schema)
           |
      apps/web (TypeScript: UI, canvas, WebAudio, 3D stage only)
           |
  microduck robot (native crates, 48 kHz PCM)   <- planned consumer

microquack-core contains everything that affects the sound: score types and parsing, pitch curves, envelopes, oscillators and partials, ring modulation, noise, the seeded PRNG, phrase grammar, the canonical semantic emotion registry and voice-mode shaping, contour output for motion, and rendering — both full-phrase (render_score) and incremental blocks (Engine::render_into, which render_score itself uses, so offline and streaming output are identical). It contains no WebAudio, WASM APIs, device access, filesystem loading, browser code, or Python-specific code.

The score contract

A score is a JSON list of syllables with snake_case fields (f0_start, amp_points, ...), defined once as Serde types in microquack-core. schema/score.schema.json is generated from those types (pnpm run schema); nothing else defines the format. schema/emotions.json and apps/web/src/emotions.generated.ts are likewise generated from the Rust registry with pnpm run emotions. Python uses the JSON file for teacher-data generation. Scores use seconds and hertz, so they are sample-rate independent; the renderer owns all rate-specific behavior. The TypeScript app treats scores as opaque JSON strings — it never reimplements or redeclares the format.

The chat and the droid mind

The visitor reads and writes English; the robot answers in droid speak with an English gloss. crates/microquack-llm runs a ~1.3M-parameter character-level transformer (trained from scratch in llm/ on a synthetic space-adventure corpus) with a KV cache and greedy decoding, compiled into the same WASM module as the synth. Its output is (emotion, reply); the emotion drives phrase generation and the body-language gestures. Rust/ PyTorch parity is enforced by cargo test -p microquack-llm against the committed model (apps/web/public/assets/llm/droidmind.bin) and test vectors. The current model artifact supports its original eight emotions. Rust validates that list against the canonical 73-state registry, then maps each semantic state to one of 12 broad voice modes and four shared affect values. The expanded 1,000-row teacher set is review data only; it does not replace the production weights. Between conversations the robot utters an idle thought every 10–15 seconds.

The web app

apps/web owns UI, canvas, and WebAudio only:

• src/wasm/ — generated wasm-bindgen output (pnpm run build:wasm).
• robot.ts — assembles the real microduck from its mesh library (microduck.glb) and kinematic tree (kinematics.json, MuJoCo convention), exposes head joints and a synthetic jaw hinge.
• scene.ts — the procedural night-plain world (three moons, stars, rocks) and the speech-driven articulation.
• mind.ts / droidtext.ts / gestures.ts — the chat brain wrapper, droid-speak captions, and generic body language driven by the Rust voice mode and affect values. TypeScript does not map the 73 semantic states.
• Short phrases render whole (render_score → AudioBuffer). Continuous synthesis, when it arrives, must run the Engine in an AudioWorklet — WASM alone does not prevent main-thread stalls.

Research

Fitting is a CLI subcommand (microquack fit): it analyzes a real WAV (pitch track, envelope, harmonics, noisiness) and emits a breakpoint score, rendering every candidate through the same engine that ships. research/ holds the fitted corpus (scores/fits.json), pregenerated phrase scores, and the frozen conformance fixtures.

Conformance gate

crates/microquack-core/tests/conformance.rs renders all 90 frozen scores (79 fitted + 6 phrases + 5 presets) against research/fixtures/: exact sample counts, RMS/peak within 1e-6, and 21 full waveforms within 1e-4 per sample. Streaming must equal offline rendering exactly, and phrase generation must be seed-deterministic. Any DSP change that breaks these is a behavior change and requires refreezing fixtures deliberately.

Device integration (planned)

The Microduck repository consumes microquack-core at an exact version (microquack-core = "=x.y.z", or a pinned git revision until published) — never vendored source. The device adapter feeds Engine::render_into into its existing 48 kHz PCM path and reads ControlFrame values to animate the physical beak and body. The 79 fitted scores must be revalidated at 48 kHz before the robot ships them.

Release model

One git tag should produce, all reporting the same version and commit:

• the microquack-core crate (device),
• the WASM/npm package (website),
• the CLI binary (render, validate, fit, schema, emotions),
• the score corpus and generated schemas.

Roadmap

See VISION.md: preset conversations, the droid-speak language model, procedural full-body motion, and browser-side locomotion policies from microduck_rl via ONNX Runtime Web.