Keys Ghost Machine v0.1.0: a hands-free audio/visual agent, powered by TensorFold
Meet Boo: talk to it from your phone, show it things through your camera, and let it hand real work to a tool-using agent, with no keyboard and no terminal. It runs entirely on your own hardware, on a DGX Spark or an Apple silicon Mac, and you reach it from anywhere on your private Tailscale network.
▶ 46-second demo: https://github.com/drowzeys/keys-ghost-machine-audio-visual-agent/blob/main/docs/media/promo-45s.mp4
What it does
- Speech in, speech out: Parakeet v3 hears you, Nemotron 3.5 Bolt answers, Drama 3 speaks, starting about half a second after you stop talking on a Mac.
- Sees through your camera: Cosmos3-Edge describes what's in front of you, reads text, and tells your left from your right on a selfie camera; the hand tracker counts fingers exactly.
- Live facts in seconds: questions about the president, the weather or the news are looked up on the web in the same turn.
- Real work in the background: "ask Hermes…" hands a task to a tool-using agent, which asks you out loud before running anything that changes your machine.
- Remembers: camera captions every 1 s to 15 min while the camera is on, kept 1 day to 1 week.
- Yours to shape: name it, pick one of 14 voices (Vesper is the default), and turn on the Futuristic effect.
Measured (TensorFold 1.1.4)
| Apple silicon Mac | DGX Spark | |
|---|---|---|
| Voice to voice | 0.47–0.60 s | 0.55–0.77 s |
| Camera answer (first token) | 79 ms | 103 ms |
| Voice starts (Vesper) | 122 ms | 121 ms |
Requirements
- Mac: Apple silicon with 96 GB unified memory minimum (128 GB+ recommended).
- DGX Spark: one Spark (128 GB), about 64 GB free.
- Disk: about 50 GB for the models.
Get it
- Code (MIT): https://github.com/drowzeys/keys-ghost-machine-audio-visual-agent
- Prebuilt Mac and DGX Spark builds of TensorFold 1.1.4: https://github.com/drowzeys/keys-ghost-machine-audio-visual-agent/releases/tag/v0.1.0
- Docker image (DGX Spark):
docker pull ghcr.io/drowzeys/keys-ghost-machine-audio-visual-agent:0.1.0 - 14 voices + model card: https://huggingface.co/drowzeys/keys-ghost-machine-audio-visual-agent
- TensorFold source branches: https://github.com/drowzeys/TensorFold
Built on the work of many people: NVIDIA (Bolt, Cosmos3-Edge, Parakeet), Fish Audio (Drama 3) and plotarmordev's NVFP4 build, Ash Hart (TensorFold), tonyd2wild (Eyes of Kai), Nous Research (Hermes), the qwen-audio-agent design, Silero, MediaPipe, Three.js and more. The full list is in the Credits section of the README. These are unofficial community builds of TensorFold, not official releases.
Questions and bug reports: GitHub Issues.