Spaces:
Running
Running
Commit History
-
fix: take sqlite off the bucket - it is the mount, not the tuning Running
A 106 MB webui.db restored from a copy verified good on another machine corrupted itself AGAIN within minutes of load, with WAL and mmap already disabled. That settles it: the HF bucket is a FUSE object-storage mount with no coherency guarantee, and the failing statement was always the timer poll scanning the chat table -- roughly 100 MB, once per second. sqlite cannot be tuned into safety on that mount; it has to leave it. - DATA_DIR moves to container-local disk (/webui-data). - uploads/ and vector_db/ are symlinked back to the bucket. Those are large write-once blobs, which is the one thing object storage is good at, so they stay persistent AND a restart no longer re-extracts 1.2 GB. - restore() skips the asset tar when the directories already have content. - ARCHIVE_INTERVAL 3600 -> 300. COST, STATED PLAINLY: the database is now ephemeral. A container restart loses writes since the last archive, so the ceiling on loss is five minutes instead of zero. That is a rea…df99538 -
fix: SQLite pragmas were wrong for a bucket mount - THIS is what corrupts it
Found the mechanism. /data is object storage over FUSE, and open-webui ships defaults tuned for a real disk: DATABASE_ENABLE_SQLITE_WAL True DATABASE_SQLITE_PRAGMA_MMAP_SIZE 268435456 (256 MB memory-mapped I/O) DATABASE_SQLITE_PRAGMA_SYNCHRONOUS NORMAL WAL requires a coherent -shm shared-memory file; mmap requires coherent page mapping. A FUSE/object-storage mount guarantees NEITHER. That is the mechanism behind every today: the database corrupted itself within minutes of load, repeatedly, on files that had just passed integrity_check -- including one restored 20 minutes earlier and one salvage that passed integrity_check, the smoke read, core-tables AND populated(). Everything added today cleans up AFTER that. This stops it happening: DATABASE_ENABLE_SQLITE_WAL=false journal_mode=DELETE, no -shm DATABASE_SQLITE_PRAGMA_MMAP_SIZE=0 no memory-mapped I/O DATABASE_SQLITE_PRAGMA_SYNCHRONOUS=FULL every commit durable before it returns Slo…56a4ed1 -
fix: stop the automation scheduler so the corruption stops and login works
Tommy cannot log in. The signin endpoint is healthy -- a dummy credential gets a clean 400 -- but the restored database is being corrupted AGAIN at runtime: 259 scheduler errors and 368 'disk image is malformed' between 20:26 and 20:31, still climbing. Open WebUI answers a FAILED USER LOOKUP with the same message as a wrong password ('The email or password provided is incorrect'), so page-level damage under the user/auth tables presents exactly as bad credentials. The 122 restored automations start firing the moment the app is up and corrupt the database within minutes on this bucket mount -- which locks him out of the very UI he would use to turn them off. He cannot delete them manually if he cannot get in. THOX_DISABLE_AUTOMATIONS=1 writes automations.enable=false straight to the config table at boot, before the app starts. Written to the DB rather than passed as an env var because these are PersistentConfig values: an environment variable seeds only a FIRST boot and this Space alw…9b8856c -
feat: deliberate fresh start - do not restore the automation-storm history
Tommy's call: the restored history carried ~122 scheduled automations firing on the order of a thousand times a minute, so it is not worth having back. The blank database is now the INTENDED state rather than damage for the gate to repair, and the recovery machinery is told so explicitly instead of being weakened. THOX_FRESH_START carries a token (set in the Dockerfile). The reset runs only when the marker on the bucket does not already hold that exact token, so it is one-shot: a redeploy cannot silently wipe the database again. Change the token to request another reset; clear the variable to disable the mechanism. While the marker matches, an unpopulated database is served as-is instead of triggering a restore. WITHOUT the token every guard added today still applies unchanged -- a blank database still takes the recovery path. The outgoing database is RENAMED to webui.db.prefresh.<timestamp>, never deleted. "Do not restore it" is not the same as "destroy it", and that distinction is…7d71358 -
fix: a fully-migrated EMPTY database still is not our database - gate on populated()
servable() was not enough, and the gap is subtle. After the blank 4096-byte database was served, ALEMBIC BUILT THE ENTIRE SCHEMA INTO IT from revision zero -- config, user, auth, chat, alembic_version all present, all empty. The file now passes integrity_check, passes every readability probe, and passes the core-tables check, while containing none of Tommy's data. A presence test would have served that blank site forever, and each fix so far would have looked correct while doing nothing. populated(db) = servable(db) AND the `user` table is non-empty. open-webui cannot be used without an account, so zero users means the database is either brand new or has lost everything; on this Space the Dataset is the source of truth, so both cases want a restore rather than a blank site. - the boot's existing-database branch and its post-salvage check both gate on `state populated` now. - backup_db() gates on populated() too, so a blank-but-complete database can never overwrite a good …f27ddf4 -
fix: an EMPTY database is structurally perfect - stop serving and archiving one
The Space came up SERVING NOTHING. Boot logged: [thox-boot] existing database on the bucket (4096 bytes) - no seed A blank sqlite file is 4096 bytes, passes integrity_check, and has no tables, so the gate cleared it, skipped recovery, and alembic built a fresh schema from revision zero -- while 417 chats sat untouched in the Dataset. `integrity` tolerates absent tables ON PURPOSE (a first boot has none), so it can never be the check that guards a REPLACEMENT. - the existing-database branch now requires `state servable`, not `state integrity`, so an empty or half-built file takes the recovery path instead of silently replacing the site with a blank one. And the part that would have made this permanent: - backup_db() now REFUSES to archive unless the live database is servable, the snapshot has at least one chat, and it is not >50% smaller than the copy already in the Dataset (override: ARCHIVE_ALLOW_SHRINK=1). The archive is the last copy; it was ~55 minu…af23802 -
fix: a salvage missing core tables is not a recovery - require servable()
Second RUNTIME_ERROR, same root shape as the first. The logical salvage returned a structurally PERFECT database -- integrity_check ok, every readability probe ok -- and the app died on: sqlite3.OperationalError: no such table: config The salvage had dropped tables while reporting success, and the gate waved it through. integrity() tolerates ABSENT tables ON PURPOSE: a fresh database has none and alembic is about to create them. That tolerance is right for a first boot and wrong for validating a REPLACEMENT, so absence is now judged separately, and only where a replacement is being validated. - servable(db) = integrity(db) AND the tables open-webui cannot start without are present (config, user, auth, chat, alembic_version). - the boot requires `state servable` on the file it installed, not just `state integrity`, before it will keep a salvage. - _logical_salvage now reports how many TABLES it rebuilt and names any it could not, so a shortfall like this is visi…8e7ce69 -
fix: verify a salvage before trusting it; stop the avatar hook racing alembic
The 19:33 boot took the Space to RUNTIME_ERROR. Three defects, in the order they bit: 1. THE SALVAGE WAS TRUSTED WITHOUT BEING READ BACK. `state recover` reported success and prepare opened the file, but alembic then died on `PRAGMA main.table_info("alembic_version")` with `database disk image is malformed`. A salvage we cannot read back is not a recovery. The boot now runs `state integrity` on the file it actually installed and falls back to the Dataset when it does not hold up, clearing the failed attempt so a half-good file can be neither served nor re-quarantined. 2. THE REAPER ATE THE QUARANTINE IT HAD JUST MADE. The glob webui.db.corrupt.* also matches the -wal/-shm sqlite creates beside a quarantine when the salvage opens it, and those carry a FRESHER mtime -- so `ls -1t | tail -n +3` kept two sidecars and deleted the 98 MB quarantine they belonged to, destroying the only copy of the newest writes. Sidecars are now removed outright and never co…70f532d -
fix: gate on integrity_check + a smoke read, not quick_check alone
Boot logged `existing database on the bucket - no seed` and six minutes later the running app was failing every ~11s with `database disk image is malformed` on the automations query. The gate was clearing a database the application cannot read, which is worse than no gate: it routes around the recovery path. integrity() now requires two things: 1. PRAGMA integrity_check rather than quick_check. quick_check skips per-page validation and index/table consistency checks, so it is a strict subset. The performance rationale in the original comment does not survive measurement -- on the real 106.4 MB database the two pragmas came out at 0.12s and 0.13s. Honest caveat: 14 random page-corruption trials did NOT produce a case where quick_check passed and integrity_check failed, so this is chosen for being a superset at equal cost, not as proof of which pragma missed the live fault. 2. A smoke read of the tables open-webui actually polls (chat, automation…d5ea7f6 -
fix: reconcile a stamped-but-unapplied migration; stop racing alembic's DDL
The previous commit fixed the malformed-JSON loop (0 occurrences since), and the Space now boots clean with no corruption and no restore. But that rebuild pulled open-webui 0.11.0 -> 0.11.1, because the Dockerfile pins `:main`, and 0.11.1 is the upstream rework that filters the timer poll on an indexed `chat.timer_at`. The app then failed once a second with `no such column: chat.timer_at` -- 391 scheduler errors in seven minutes. Same traceback storm, different cause. Alembic claimed to have added it: Will assume non-transactional DDL. Running upgrade 6d09d1bf1f23 -> d4c1a8e37b62, add chat timer_at and ... A STAMPED REVISION IS NOT PROOF THE DDL LANDED. Alembic runs non-transactionally on sqlite and never re-runs a stamped revision, so the database was left permanently one column short of what the code requires. The likely mechanism is this repo's own checkpoint loop: a SECOND process issuing wal_checkpoint(TRUNCATE) against a bucket-mounted database while alembic was mid-m…c221bf4 -
fix: repair invalid chat.meta JSON, the actual cause of the restart loop
"DATABASE IS CORRUPT" was the last link in the chain, not the first, which is why restoring 106 MB from the Dataset never fixed it. open-webui's scheduler polls due timers once a SECOND with an unindexed scan over every chat row: WHERE JSON_EXTRACT(chat.meta,'$."internal"') IS 1 AND ... sqlite raises `malformed JSON` if a SINGLE scanned value is not valid JSON, so one poisoned chat.meta broke the whole scheduler every second, each time printing a ~100-line traceback. That write+log pressure kept the WAL permanently hot, so wal_checkpoint(TRUNCATE) could never fold it -- the log said so verbatim, `checkpoint busy (wal=483 pages)`. /data is a bucket mount and is not write-through, so the next restart raced that unflushed WAL and landed a torn file, which boot correctly judged corrupt and replaced with the hourly snapshot -- carrying the same poisoned row. Round again. Guards, in the order they now run: - `state prepare` repairs invalid JSON and sets journal_mode=WAL before …8d8da73
-
fix(audio): point TTS at the THOX speech service
TommyVoice call mode still hung after the previous fix. Switching TTS to "" moved it from one broken path to another: open-webui's browser fallback polls speechSynthesis.getVoices() every 100ms in CallOverlay.svelte and resolves only once the list is non-empty, with NO timeout and NO fallback. iOS Safari leaves that list empty until voiceschanged fires and blocks speak() outside a user gesture, so the promise never resolves and the UI sits on "Tap to interrupt" forever -- the same symptom, a different cause. Both server-free options are therefore dead here: `transformers` 500s because cmu-arctic-xvectors is a dataset script that datasets v4 refuses to load, and "" hangs on the platform the user is actually holding. So TTS now points at thox-voice-stt's new /v1/audio/speech (Piper, CPU, RTF ~0.05). open-webui transcodes the audio and plays it through an <audio> element, which never touches getVoices(). Verified live end to end: mic->text 2.6s, text->answer 11.3s, answer->speech 0.7s, 14.6…6b41ebe -
fix(audio): disable the broken local Transformers TTS
TommyVoice CALL MODE hung forever on "Tap to interrupt". STT was fine, the model answered fine; TTS returned 500 and the UI sat waiting for speech that would never arrive. Nothing on screen pointed at TTS, which is what made it hard to see. Root cause is upstream and permanent on this image: `_tts_transformers` loads CMU Arctic speaker embeddings with `load_dataset(...)`, that is a SCRIPT-based dataset, and `datasets` v4 removed script support: RuntimeError: Dataset scripts are no longer supported, but found cmu-arctic-xvectors.py Not a cold cache and not slow CPU -- it can never succeed. So seed_stt.py now sets the engine to "" when it finds "transformers", which makes CallOverlay.svelte fall back to the browser's own speechSynthesis. That is the better answer anyway on a free CPU Space: instant, no model download, and a voice the device already has. Only "transformers" is touched. openai/elevenlabs/azure/mistral are working server engines and an operator who chos…9de169b -
fix: recover from a corrupt database on boot instead of 503ing
Tommywebui.thox.ai went down with `database disk image is malformed`. The boot guard was `[ -f "$DATA_DIR/webui.db" ]`, which treats a CORRUPT database as "already seeded" and hands it straight to open-webui. Startup then fails and the Space 503s with no path back, because the restore branch only ran when the file was ABSENT. Presence is not health. The bucket mount is not write-through -- the checkpoint loop exists precisely because writes sit in webui.db-wal inside the container -- so a restart racing the WAL flush can leave a torn main database. Any restart can therefore trigger this, including one caused by a docs-only push, which is how it happened. Now: PRAGMA quick_check on boot, and on failure quarantine the file, drop the WAL/shm sidecars, and restore from the archive Dataset. - quick_check, not integrity_check: same verdict for structural damage without walking every page of a ~30 MB file on every boot. - Quarantine, never delete. The archive is hourly, so the corrupt file ma…7d018d9 -
feat(audio): point STT at the THOX faster-whisper Space
TommyAdds scripts/seed_stt.py, run from the boot script after seed_connections.py. Writes the config table directly for the same reason connections do: AUDIO_STT_* are PersistentConfig, so env vars seed only a FIRST boot and this Space always has a database on the bucket. Env would be ignored. Two things verified against a copy of the live database rather than assumed: - The base URL is a BASE. routers/audio.py builds f'{api_base_url}/audio/transcriptions', so the value must end at /v1. The value already stored here was " https://api.openai.com/v1/audio/transcriptions" -- full path AND a leading space -- which would have doubled the path. It never mattered because the engine was empty (local Whisper). - The config table is key/value with dotted keys (343 rows), matching seed_connections.py's accessors. The seeder does NOT overwrite unconditionally. An admin who picks a different STT provider in Settings stores it in this same table, and a seeder that rewrote it every boot woul…b0446a6 -
fix: periodic WAL checkpoint - bucket mount is not write-through, restarts raced the flush
Verified signaturea113afb -
fix: periodic WAL checkpoint - bucket mount is not write-through, restarts raced the flush
Verified signature94a02de -
feat: one-way bucket->Dataset archive (pure sink; never restores, never seeds)
Verified signature56cc9ce -
feat: one-way bucket->Dataset archive (pure sink; never restores, never seeds)
Verified signature10652a5 -
feat: move state onto the mounted /data bucket; retire Dataset backup loop
Verified signatureaee1503 -
fix: normalise line endings to LF (CRLF broke the bash entrypoint)
Verified signature07eb496 -
fix: normalise line endings to LF (CRLF broke the bash entrypoint)
Verified signature26b3655 -
fix: pin WEBUI_URL to the custom domain (start.sh overwrites it from SPACE_HOST)
Verified signaturec61bbfd -
fix: structural scrub of connection-key arrays (denylist missed xai-/tho- prefixes)
Verified signature90f9e1b -
fix: scrub credentials from webui.db before it leaves the Space
Verified signature93964c7 -
THOX Open WebUI: branded image + Dataset-backed state + connection seeding
Verified signature39365b3