Spaces:
Running
Running
Commit History
-
fix: take sqlite off the bucket - it is the mount, not the tuning Running
A 106 MB webui.db restored from a copy verified good on another machine corrupted itself AGAIN within minutes of load, with WAL and mmap already disabled. That settles it: the HF bucket is a FUSE object-storage mount with no coherency guarantee, and the failing statement was always the timer poll scanning the chat table -- roughly 100 MB, once per second. sqlite cannot be tuned into safety on that mount; it has to leave it. - DATA_DIR moves to container-local disk (/webui-data). - uploads/ and vector_db/ are symlinked back to the bucket. Those are large write-once blobs, which is the one thing object storage is good at, so they stay persistent AND a restart no longer re-extracts 1.2 GB. - restore() skips the asset tar when the directories already have content. - ARCHIVE_INTERVAL 3600 -> 300. COST, STATED PLAINLY: the database is now ephemeral. A container restart loses writes since the last archive, so the ceiling on loss is five minutes instead of zero. That is a rea…df99538 -
fix: SQLite pragmas were wrong for a bucket mount - THIS is what corrupts it
Found the mechanism. /data is object storage over FUSE, and open-webui ships defaults tuned for a real disk: DATABASE_ENABLE_SQLITE_WAL True DATABASE_SQLITE_PRAGMA_MMAP_SIZE 268435456 (256 MB memory-mapped I/O) DATABASE_SQLITE_PRAGMA_SYNCHRONOUS NORMAL WAL requires a coherent -shm shared-memory file; mmap requires coherent page mapping. A FUSE/object-storage mount guarantees NEITHER. That is the mechanism behind every today: the database corrupted itself within minutes of load, repeatedly, on files that had just passed integrity_check -- including one restored 20 minutes earlier and one salvage that passed integrity_check, the smoke read, core-tables AND populated(). Everything added today cleans up AFTER that. This stops it happening: DATABASE_ENABLE_SQLITE_WAL=false journal_mode=DELETE, no -shm DATABASE_SQLITE_PRAGMA_MMAP_SIZE=0 no memory-mapped I/O DATABASE_SQLITE_PRAGMA_SYNCHRONOUS=FULL every commit durable before it returns Slo…56a4ed1 -
fix: stop the automation scheduler so the corruption stops and login works
Tommy cannot log in. The signin endpoint is healthy -- a dummy credential gets a clean 400 -- but the restored database is being corrupted AGAIN at runtime: 259 scheduler errors and 368 'disk image is malformed' between 20:26 and 20:31, still climbing. Open WebUI answers a FAILED USER LOOKUP with the same message as a wrong password ('The email or password provided is incorrect'), so page-level damage under the user/auth tables presents exactly as bad credentials. The 122 restored automations start firing the moment the app is up and corrupt the database within minutes on this bucket mount -- which locks him out of the very UI he would use to turn them off. He cannot delete them manually if he cannot get in. THOX_DISABLE_AUTOMATIONS=1 writes automations.enable=false straight to the config table at boot, before the app starts. Written to the DB rather than passed as an env var because these are PersistentConfig values: an environment variable seeds only a FIRST boot and this Space alw…9b8856c -
fix: a fully-migrated EMPTY database still is not our database - gate on populated()
servable() was not enough, and the gap is subtle. After the blank 4096-byte database was served, ALEMBIC BUILT THE ENTIRE SCHEMA INTO IT from revision zero -- config, user, auth, chat, alembic_version all present, all empty. The file now passes integrity_check, passes every readability probe, and passes the core-tables check, while containing none of Tommy's data. A presence test would have served that blank site forever, and each fix so far would have looked correct while doing nothing. populated(db) = servable(db) AND the `user` table is non-empty. open-webui cannot be used without an account, so zero users means the database is either brand new or has lost everything; on this Space the Dataset is the source of truth, so both cases want a restore rather than a blank site. - the boot's existing-database branch and its post-salvage check both gate on `state populated` now. - backup_db() gates on populated() too, so a blank-but-complete database can never overwrite a good …f27ddf4 -
fix: an EMPTY database is structurally perfect - stop serving and archiving one
The Space came up SERVING NOTHING. Boot logged: [thox-boot] existing database on the bucket (4096 bytes) - no seed A blank sqlite file is 4096 bytes, passes integrity_check, and has no tables, so the gate cleared it, skipped recovery, and alembic built a fresh schema from revision zero -- while 417 chats sat untouched in the Dataset. `integrity` tolerates absent tables ON PURPOSE (a first boot has none), so it can never be the check that guards a REPLACEMENT. - the existing-database branch now requires `state servable`, not `state integrity`, so an empty or half-built file takes the recovery path instead of silently replacing the site with a blank one. And the part that would have made this permanent: - backup_db() now REFUSES to archive unless the live database is servable, the snapshot has at least one chat, and it is not >50% smaller than the copy already in the Dataset (override: ARCHIVE_ALLOW_SHRINK=1). The archive is the last copy; it was ~55 minu…af23802 -
fix: a salvage missing core tables is not a recovery - require servable()
Second RUNTIME_ERROR, same root shape as the first. The logical salvage returned a structurally PERFECT database -- integrity_check ok, every readability probe ok -- and the app died on: sqlite3.OperationalError: no such table: config The salvage had dropped tables while reporting success, and the gate waved it through. integrity() tolerates ABSENT tables ON PURPOSE: a fresh database has none and alembic is about to create them. That tolerance is right for a first boot and wrong for validating a REPLACEMENT, so absence is now judged separately, and only where a replacement is being validated. - servable(db) = integrity(db) AND the tables open-webui cannot start without are present (config, user, auth, chat, alembic_version). - the boot requires `state servable` on the file it installed, not just `state integrity`, before it will keep a salvage. - _logical_salvage now reports how many TABLES it rebuilt and names any it could not, so a shortfall like this is visi…8e7ce69 -
fix: verify a salvage before trusting it; stop the avatar hook racing alembic
The 19:33 boot took the Space to RUNTIME_ERROR. Three defects, in the order they bit: 1. THE SALVAGE WAS TRUSTED WITHOUT BEING READ BACK. `state recover` reported success and prepare opened the file, but alembic then died on `PRAGMA main.table_info("alembic_version")` with `database disk image is malformed`. A salvage we cannot read back is not a recovery. The boot now runs `state integrity` on the file it actually installed and falls back to the Dataset when it does not hold up, clearing the failed attempt so a half-good file can be neither served nor re-quarantined. 2. THE REAPER ATE THE QUARANTINE IT HAD JUST MADE. The glob webui.db.corrupt.* also matches the -wal/-shm sqlite creates beside a quarantine when the salvage opens it, and those carry a FRESHER mtime -- so `ls -1t | tail -n +3` kept two sidecars and deleted the 98 MB quarantine they belonged to, destroying the only copy of the newest writes. Sidecars are now removed outright and never co…70f532d -
fix: gate on integrity_check + a smoke read, not quick_check alone
Boot logged `existing database on the bucket - no seed` and six minutes later the running app was failing every ~11s with `database disk image is malformed` on the automations query. The gate was clearing a database the application cannot read, which is worse than no gate: it routes around the recovery path. integrity() now requires two things: 1. PRAGMA integrity_check rather than quick_check. quick_check skips per-page validation and index/table consistency checks, so it is a strict subset. The performance rationale in the original comment does not survive measurement -- on the real 106.4 MB database the two pragmas came out at 0.12s and 0.13s. Honest caveat: 14 random page-corruption trials did NOT produce a case where quick_check passed and integrity_check failed, so this is chosen for being a superset at equal cost, not as proof of which pragma missed the live fault. 2. A smoke read of the tables open-webui actually polls (chat, automation…d5ea7f6 -
fix: reconcile a stamped-but-unapplied migration; stop racing alembic's DDL
The previous commit fixed the malformed-JSON loop (0 occurrences since), and the Space now boots clean with no corruption and no restore. But that rebuild pulled open-webui 0.11.0 -> 0.11.1, because the Dockerfile pins `:main`, and 0.11.1 is the upstream rework that filters the timer poll on an indexed `chat.timer_at`. The app then failed once a second with `no such column: chat.timer_at` -- 391 scheduler errors in seven minutes. Same traceback storm, different cause. Alembic claimed to have added it: Will assume non-transactional DDL. Running upgrade 6d09d1bf1f23 -> d4c1a8e37b62, add chat timer_at and ... A STAMPED REVISION IS NOT PROOF THE DDL LANDED. Alembic runs non-transactionally on sqlite and never re-runs a stamped revision, so the database was left permanently one column short of what the code requires. The likely mechanism is this repo's own checkpoint loop: a SECOND process issuing wal_checkpoint(TRUNCATE) against a bucket-mounted database while alembic was mid-m…c221bf4 -
fix: repair invalid chat.meta JSON, the actual cause of the restart loop
"DATABASE IS CORRUPT" was the last link in the chain, not the first, which is why restoring 106 MB from the Dataset never fixed it. open-webui's scheduler polls due timers once a SECOND with an unindexed scan over every chat row: WHERE JSON_EXTRACT(chat.meta,'$."internal"') IS 1 AND ... sqlite raises `malformed JSON` if a SINGLE scanned value is not valid JSON, so one poisoned chat.meta broke the whole scheduler every second, each time printing a ~100-line traceback. That write+log pressure kept the WAL permanently hot, so wal_checkpoint(TRUNCATE) could never fold it -- the log said so verbatim, `checkpoint busy (wal=483 pages)`. /data is a bucket mount and is not write-through, so the next restart raced that unflushed WAL and landed a torn file, which boot correctly judged corrupt and replaced with the hourly snapshot -- carrying the same poisoned row. Round again. Guards, in the order they now run: - `state prepare` repairs invalid JSON and sets journal_mode=WAL before …8d8da73
-
fix: periodic WAL checkpoint - bucket mount is not write-through, restarts raced the flush
Verified signature94a02de -
feat: one-way bucket->Dataset archive (pure sink; never restores, never seeds)
Verified signature10652a5 -
fix: normalise line endings to LF (CRLF broke the bash entrypoint)
Verified signature07eb496 -
fix: structural scrub of connection-key arrays (denylist missed xai-/tho- prefixes)
Verified signature90f9e1b -
fix: scrub credentials from webui.db before it leaves the Space
Verified signature93964c7 -
THOX Open WebUI: branded image + Dataset-backed state + connection seeding
Verified signature39365b3