| --- |
| title: Live Code Camera Engine |
| emoji: 🎥 |
| colorFrom: blue |
| colorTo: purple |
| sdk: docker |
| pinned: false |
| --- |
| |
| # Live Code Camera Engine |
|
|
| A live **camera + microphone → evidence compression → multimodal LLM → code stream** engine. |
|
|
| This is the upgraded version of the click-to-capture prototype. It continuously samples the camera, tracks evidence density, collects speech transcript text, and periodically asks a multimodal LLM to generate or revise code based on what the camera and audio are exposing. |
|
|
| ## Core primitive |
|
|
| ```text |
| iPhone / laptop camera |
| → live frame stream |
| → evidence compressor |
| → rolling observation buffer |
| → LLM code synthesizer |
| → live code updates |
| → receipt chain |
| ``` |
|
|
| ## What "continuity" means here |
|
|
| The practical version is: |
|
|
| ```text |
| iPhone Safari / laptop browser |
| → HTTPS Hugging Face Space |
| → FastAPI WebSocket endpoint |
| → multimodal LLM backend |
| ``` |
|
|
| If your iPhone and laptop are near each other, the shared continuity is not magic hardware continuity; it is a shared live endpoint. You open the same Space URL from either device, grant camera/mic permissions, and the stream becomes an evidence source. |
|
|
| ## Run locally |
|
|
| ```bash |
| cd live-code-camera-engine |
| python3 -m venv .venv |
| source .venv/bin/activate |
| pip install -r backend/requirements.txt |
| |
| export PROVIDER=ollama |
| export OLLAMA_MODEL=llava |
| uvicorn backend.main:app --host 0.0.0.0 --port 8000 |
| ``` |
|
|
| Open: |
|
|
| ```text |
| http://localhost:8000 |
| ``` |
|
|
| For iPhone on same Wi-Fi, use your laptop LAN IP: |
|
|
| ```text |
| http://YOUR_LAPTOP_IP:8000 |
| ``` |
|
|
| Camera/mic access on mobile is most reliable over HTTPS, except localhost. For remote iPhone use, deploy to Hugging Face Spaces or use an HTTPS tunnel. |
|
|
| ## Deploy to Hugging Face Spaces |
|
|
| Create a Docker Space and upload this folder. Required files are already included: |
|
|
| ```text |
| Dockerfile |
| requirements.txt |
| backend/main.py |
| frontend/index.html |
| ``` |
|
|
| Set secrets/environment variables in the Space: |
|
|
| ```text |
| PROVIDER=ollama or openai |
| OPENAI_API_KEY=... |
| OPENAI_MODEL=<current multimodal model> |
| ``` |
|
|
| For local Ollama you normally run the backend locally, not inside Hugging Face, unless you provide a reachable Ollama endpoint. |
|
|
| ## Modes |
|
|
| - `continuous_code`: generate code as the camera moves. |
| - `debug_visible_error`: point camera at terminal/browser error and generate a fix. |
| - `derive_pipeline`: infer a pipeline from scene + speech. |
| - `soft_qr_xlc`: decode/encode QR-like exoatomic carriers. |
| - `research_prism`: convert video/audio observations into code + research hypotheses. |
|
|
| ## Receipts |
|
|
| Every synthesis has: |
|
|
| ```text |
| session_id |
| frame hashes |
| transcript hash |
| observation entropy |
| frame-delta score |
| prompt hash |
| provider |
| timestamp |
| ``` |
|
|
| This makes the stream auditable without storing raw frames by default. |
|
|
| ## Safety boundaries |
|
|
| - The app does not execute generated code automatically. |
| - It does not secretly record; the browser asks for permissions. |
| - It does not store raw camera frames unless you add persistence. |
| - It should not be used for covert surveillance, credential theft, malware, or unauthorized automation. |
| - Multi-speaker separation is represented as an extension point. True speaker diarization needs an ASR/diarization model such as Whisper-family ASR plus a diarization stack. |
|
|
