Spaces:
Running on CPU Upgrade
Running on CPU Upgrade
Download platform/harness/datastore.py from fsanyoto/runloopable: direct link, hf CLI and curl.
- Browser
- Download file 237 kB
-
https://huggingface.co/spaces/fsanyoto/runloopable/resolve/main/platform/harness/datastore.py
- Command line
-
hf download hf://spaces/fsanyoto/runloopable/platform/harness/datastore.py
-
curl -L -o datastore.py https://huggingface.co/spaces/fsanyoto/runloopable/resolve/main/platform/harness/datastore.py
237 kB
"""harness/store.py β the tenant data store (OM-1, 2026-07-11).
Incremental Odoo β DuckDB mirror: the local analytical store that saved views/dashboards (OM-3)
and the AI Analyst's tools (OM-4) query at warehouse speed, instead of hammering live XML-RPC
per question.
Plan: AutoWiki/research/omni-adoption.md Part IV (OM-1) + [[odoo-open-source]] BUILD-NOW #1.
Design rules:
- The store is a RAW mirror (unscoped). Business scope (wholesale teams, GIFTWARE exclusion,
confirmed-only) is applied by the SEMANTIC layer at query time (OM-2) β one source of truth.
- Sync is CHECKPOINTED + BOUNDED + RESUMABLE (loop-library discipline): backfill paginates by
id; live mode advances a write_date cursor; every run is bounded by max_batches and safe to
kill/re-run (upserts are idempotent).
- validate() compares the store against LIVE Odoo (raw fidelity: counts + monthly sums to the
cent) β the store never validates itself. Odoo hard-deletes (unlink) don't move write_date;
the count checks are the drift detector for those.
- Client data: the .duckdb file lives under data/store/ (git-ignored). READ-ONLY on Odoo.
"""
import datetime as dt
import hashlib
import json
import math
import os
import pickle
import re
import sys
import time
import tempfile
from collections import OrderedDict, deque
from contextlib import contextmanager, nullcontext
from pathlib import Path
import duckdb
import core.odoo as O
from core import request_timing as _request_timing
_STORE_DIR = Path(__file__).resolve().parents[1] / "data" / "store"
# β MUTABLE ON PURPOSE, and read LATE by every function below (Python resolves a module global at
# CALL time), which is how `provision_tenant.py` already points a fresh instance at its own file.
# EXIT-4a (X7) formalises that: `AIOS_DUCKDB_PATH` lets a container name the file without an edit,
# `path_for()` owns the per-tenant naming convention, and `use_path()` is the SAFE way to switch β
# see its docstring for why a bare assignment is not.
# The DEFAULT is unchanged: <app root>/data/store/royal.duckdb.
DB_PATH = Path(os.environ.get("AIOS_DUCKDB_PATH") or (_STORE_DIR / "royal.duckdb"))
_LEGACY_PIN = None
# βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
# β DEBT D-10 (wave 24) β THE CONNECTOR PAUSE REACHES THE MEASURE MIRROR.
#
# WHAT THE BUG WAS. Pausing a tenant's Odoo connector froze the CUSTOMER POOL (DEBT-2 built that:
# `routes_customers._pool_for` serves the pause-time snapshot) and nothing else. Every measure
# column and condition in the product is answered from THIS store, and this store kept pulling
# from Odoo the entire time a connector was paused β 12 sync passes at boot and another every
# 1800 s, hundreds of `search_read`s each. The Settings copy said so out loud ("measures not
# already computed may still reach the source until their cutover") and that sentence was the
# debt row.
#
# β WHY A PROBE AND NOT AN IMPORT. The pause flag lives in the tenant's store bucket, which is
# `harness.runtime`'s to read β and `runtime` already imports THIS module (`_ds.DB_PATH`), so an
# import back at module initialization would be a cycle. The harness installs the pause probe
# at its own import. The frozen-config check below imports the sibling lazily at call time.
#
# β IT FAILS TOWARDS "NOT PAUSED", deliberately and consistently with the shipped decision one
# layer up (`routes_keychain.odoo_paused`: "an unreachable flags bucket must never freeze a live
# surface"). The alternative β assume paused when the answer cannot be resolved β turns any
# transient store glitch into a mirror that silently stops advancing, which is the failure mode
# nobody notices for a week. Two copies of that policy would be worse than one; this is the same
# one.
_paused_probe = None
# Same tenant bucket as the API's `odoo_relational.CONFIG_KEY`; no upward API import is allowed.
_ODOO_CONFIG_KEY = "odoo_connector_config"
def set_paused_probe(fn):
"""Install the callable that answers 'is the connector paused for the tenant whose store this
process has open?'. `harness.runtime` installs the real one; pass None to remove it."""
global _paused_probe
_paused_probe = fn
def source_paused():
"""True when this mirror's tenant has paused or disconnected Odoo. Never raises."""
try:
# Resolve afresh from the current mirror path, using the runtime's one tenant resolver.
# Credential deletion changes the resolved pause id; the durable freeze does not depend
# on that id and must still stop sync after deletion, path switches and process restarts.
from harness import runtime
key = runtime.tenant_for_store_path(DB_PATH)
if key:
rt = runtime.get_runtime(key)
if str(getattr(rt, "key", "") or "") == key:
cfg = rt.get(_ODOO_CONFIG_KEY, fresh=True)
if isinstance(cfg, dict) and cfg.get("frozen"):
return True
except Exception:
pass
# The lazy import also installs the ordinary pause probe for standalone sync workers.
try:
return bool(_paused_probe and _paused_probe())
except Exception:
return False
#: What a sync entry point answers instead of reaching Odoo. A PHASE WORD, not an exception and
#: not a silent zero-row success: `status()` keeps whatever the last real pass wrote, so the
#: mirror still reports the age it genuinely has ([[no-unverifiable-aggregates]] β a paused
#: mirror that reported itself as freshly synced would be the unverifiable claim).
PAUSED_PHASE = "paused"
def _paused_notice(what, log):
# Printed as well as logged, and that is deliberate: `api/main.py` passes a swallowing
# `log=lambda *a, **k: None`, so the log-only version of this line would never be seen by
# the one caller that runs it unattended every 30 minutes.
msg = (f"[datastore] {what} skipped - the tenant's Odoo connector is PAUSED; the mirror is "
f"serving its pause-time state. Resume it under Settings > Connectors.")
try:
log(msg)
except Exception:
pass
print(msg)
def path_for(tenant_key):
"""The canonical DuckDB file for a tenant slug. `royal-imports` keeps the historical
`royal.duckdb` name so tenant #0's existing 175 MB file is not orphaned by a rename.
File-per-tenant IS the isolation model (C1c): DuckDB has no row-level security, so the only
boundary that holds is the operating system's β one file, one tenant.
"""
key = str(tenant_key or "").strip().lower()
# AIOS_DUCKDB_STORE_DIR moves every tenant's file, so the sync worker never opens the web's.
store_dir = Path(os.environ.get("AIOS_DUCKDB_STORE_DIR") or _STORE_DIR)
if key in ("", "royal-imports"):
return DB_PATH if key == "" else Path(
os.environ.get("AIOS_DUCKDB_PATH") or (store_dir / "royal.duckdb"))
safe = "".join(c if (c.isalnum() or c in "-_") else "_" for c in key)[:60]
return store_dir / f"{safe}.duckdb"
def apply_limits(con):
"""Bound each instance's memory, execution threads and local spill space."""
web = os.environ.get("AIOS_PROCESS_ROLE") == "web"
memory = str(os.environ.get("AIOS_DUCKDB_MEMORY") or (
os.environ.get("AIOS_WEB_DUCKDB_MEMORY") or "6GB" if web else "4GB")).strip()
threads = str(os.environ.get("AIOS_DUCKDB_THREADS") or (
os.environ.get("AIOS_WEB_DUCKDB_THREADS") or "2" if web else "1")).strip()
spill = str(os.environ.get("AIOS_DUCKDB_MAX_TEMP_SIZE") or "4GB").strip()
temp = Path(os.environ.get("AIOS_DUCKDB_TEMP_DIR") or (
Path(tempfile.gettempdir()) / f"aios-duckdb-{os.getpid()}"))
rows = con.execute("PRAGMA database_list").fetchall()
identity = next((str(row[2]) for row in rows if row[2]), str(id(con)))
temp = temp / hashlib.sha256(identity.encode()).hexdigest()[:16]
temp.mkdir(parents=True, exist_ok=True, mode=0o700)
if not re.fullmatch(r"\d+(\.\d+)?\s*[KMGT]i?B", memory, re.IGNORECASE):
raise ValueError("Invalid AIOS_DUCKDB_MEMORY")
if not threads.isdigit() or int(threads) < 1:
raise ValueError("Invalid AIOS_DUCKDB_THREADS")
if not re.fullmatch(r"\d+(\.\d+)?\s*[KMGT]i?B", spill, re.IGNORECASE):
raise ValueError("Invalid AIOS_DUCKDB_MAX_TEMP_SIZE")
if memory and re.fullmatch(r"\d+(\.\d+)?\s*[KMGT]i?B", memory, re.IGNORECASE):
con.execute(f"SET memory_limit='{memory}'")
if threads.isdigit() and int(threads) > 0:
con.execute(f"SET threads={int(threads)}")
# DuckDB refuses any temp_directory SET, the same path included, once the instance has spilled.
if con.execute("SELECT current_setting('temp_directory')").fetchone()[0] != str(temp):
try:
con.execute("SET temp_directory='" + str(temp).replace("'", "''") + "'")
except duckdb.NotImplementedException as exc:
if "Cannot switch temporary directory" not in str(exc):
raise
con.execute(f"SET max_temp_directory_size='{spill}'")
return con
def bounded_connection(con):
"""Apply instance budgets and request deadlines to a temporary analytical connection."""
try:
return _TimedCursor(apply_limits(con))
except BaseException:
try:
con.close()
except Exception:
pass
raise
def use_path(path):
"""Repoint the store at `path`, CLOSING the process singleton first.
β A bare `datastore.DB_PATH = β¦` is not enough and is the more dangerous half of this
operation. `_instance()` caches ONE open connection for the life of the process, so after a
plain reassignment every read keeps being served from the PREVIOUSLY opened file β a silent
cross-tenant read, returning real rows that belong to somebody else. Closing the connection
and clearing the readiness memo (which is also per-file) is what makes the switch honest.
β AND CLOSING THE CONNECTION IS STILL NOT ENOUGH ON ITS OWN, because `ro_con()` caches a
cursor in a `threading.local`: this function can only clear the CALLING thread's, and
`ro_con`'s liveness probe (`SELECT 1`) SUCCEEDS on a cursor whose underlying connection was
closed out from under it in some cases β so another thread would keep answering from the old
file with no error to notice. The generation counter below is what closes that: `ro_con`
compares the generation its cursor was opened under and reopens when it has moved, which is
a check no individual thread can forget to do.
Returns the new path. Callers: `provision_tenant.py`, and any single-tenant worker process.
NOT a per-request operation β one process serves one analytical store at a time, which is why
`harness.runtime` hands out the PATH rather than switching the global underneath a request.
"""
global DB_PATH, _LEGACY_PIN
# Before the drain: an idle cursor from the old store must refuse, never rebind to the new one.
_STORE_EPOCH[0] += 1
with _swap_instance():
_RO_TLS.__dict__.pop("cur", None)
_READY["ok"] = False
_GENERATION[0] += 1 # every other thread's cached cursor is now stale
_VERSION.update(stamp=None, file=None, version=None, checked=0.0)
DB_PATH = Path(path)
_LEGACY_PIN = DB_PATH.resolve()
_invalidate_counts()
return DB_PATH
# Entity specs: Odoo model β store table. m2o fields land as <name>_id + <name>_name.
# 'archivable' entities are pulled with active in [True, False] (the re-SKU/merge rule).
ENTITIES = {
"sale_order": {
"model": "sale.order", "archivable": False,
"fields": ["name", "date_order", "commitment_date", "delivery_status", "partner_id",
"team_id", "user_id", "pricelist_id", "state", "amount_untaxed", "invoice_status",
"write_date"],
# Existing orders do not receive a new write_date when our schema grows. Without this
# declared, resumable pass newly added columns remain NULL on every old order.
"backfill_fields": ["commitment_date", "delivery_status", "pricelist_id", "user_id"],
"backfill_before_delta": True,
# Partner merges can reassign old orders without advancing write_date.
"refresh_fields": ["partner_id"],
},
"sale_order_line": {
"model": "sale.order.line", "archivable": False,
"fields": ["order_id", "order_partner_id", "product_id", "product_uom_qty",
"product_uom", "price_subtotal", "margin", "purchase_price", "write_date"],
"backfill_fields": ["product_uom"],
},
"res_partner": {
"model": "res.partner", "archivable": True,
# `agent` (bool, labelled "Creditor/Agent") is THE discriminator between a real sales
# AGENT and an internal SALESPERSON who merely carries commission lines β the commission
# table itself cannot tell them apart. `salesman_as_agent` ("convert salesman into agent")
# marks agents who came from staff; the owner ruling on whether those count as agents is
# OUTSTANDING, so both flags are synced and neither reading is baked in.
# β WAVE 29 (W29-T54, amendment A4). `customer_rank` is the ONLY predicate that
# distinguishes "a partner Odoo has TRANSACTED with" (today's population, 2,465) from
# "a customer" (3,614 with `rank>0 active`) β a 1,149-partner drop, the same join-drop
# class as item 22's SKU pool. It cannot be derived from the document tables, because
# the missing partners are precisely the ones with no document.
# β It is synced HERE rather than read from live Odoo because `read_customers` also runs
# at BOOT off a hydrated snapshot, where Odoo may be unreachable β sourcing it live would
# make the population "whatever source answered this time", changing size with the network.
# β WAVE 30 (session E's ask, W30-T33). `street`/`street2`/`zip` are the POSTAL half of
# an address the mirror has never carried: it holds `city`, `state_id` and `country_id`
# and stops there, so `odoo_relational.read_customers` β which reads only the mirror β
# cannot serve a full address at all, however the column is declared.
# β DECLARING THEM DOES NOT FILL THEM, and the difference is a whole sync: an existing
# mirror ALTERs the columns in on the next `sync_all()` and leaves them NULL until the
# backfill below has walked every partner (a partner's `write_date` did not move because
# WE changed the schema). Every projection degrades a missing/NULL column to blank rather
# than raising, so declaring early is safe β it is just not yet true.
# ββ WAVE 33 (W33-T48, owner item 13) β THE CONTACT/IDENTITY HALF, MEASURED BEFORE IT WAS
# ADDED. The owner: *"I could have sworn we have more relevant Fields under 'Odoo vendors'
# etc."* He was right, and the census
# (`AutoWiki/waves/wave33/odoo-field-census.md`, live read-only, 2026-08-14) says by how
# much: `res.partner` declares 198 fields, **77 are populated on all 3,621 customers and 76
# on all 658 vendors**, and this mirror carried THIRTEEN. Every one of the six added here
# was measured non-empty on the live population first β none is added on the strength of
# "Odoo probably has that", which is how a schema grows columns nobody can fill.
# β THE MIRROR IS THE CEILING, NOT THE GRID. Every reader in `odoo_relational.py` reads
# THIS file, so a column added to a `*_fields()` contract with no entry here renders as a
# blank column forever β which is D-165's shape exactly, and the reason the widening starts
# on this line rather than on the grid.
"fields": ["name", "team_id", "city", "state_id", "country_id", "active", "category_id",
"agent", "salesman_as_agent", "customer_rank",
"street", "street2", "zip",
"email", "phone", "mobile", "website", "vat", "ref",
"property_payment_term_id", "property_product_pricelist", "create_date",
"partner_latitude", "partner_longitude",
"write_date"],
# `backfill_fields` carries it too, so mirrors that already exist pick it up on the next
# sync_all() instead of needing a reseed.
# β D-165 IS ABOUT THIS LIST, AND THE CENSUS SETTLED WHICH HALF IS WRONG: `street`,
# `street2` and `zip` are `store=True` on `res.partner` and POPULATED in live Odoo β they
# are declared here and still NULL in the mirror purely because a backfill pass has not
# walked every partner (changing OUR schema does not move a partner's `write_date`, so an
# incremental sync never revisits them). Declaring is not filling; the fix is a run.
"backfill_fields": ["agent", "salesman_as_agent", # added 2026-07-28
"customer_rank", # added 2026-08-11 (W29-T54)
"street", "street2", "zip", # added 2026-08-12 (W30-T33)
"email", "phone", "mobile", # added 2026-08-14 (W33-T48)
"website", "vat", "ref", "category_id",
"property_payment_term_id", "property_product_pricelist", "create_date",
"partner_latitude", "partner_longitude", "category_source_ids"],
# Rank drift behind the write_date cursor requires recurring observation;
# schema backfill alone cannot maintain the customer population predicate.
"refresh_fields": ["customer_rank"],
# m2m fields land as <base>_id (the FIRST linked id β the Customers-module convention:
# a customer's assigned agent is agent_ids[0], ~one per customer, MECE) + <field> as a
# JSON list of ALL ids. Declared explicitly: a 2-element m2m read would otherwise be
# indistinguishable from an m2o (id, name) pair in _flatten.
"m2m": ["agent_ids"],
# Named many2many fields store source names despite their singular `_id` suffix.
"named_m2m": {"category_id": "res.partner.category"},
"raw_m2m": {"category_source_ids": "category_id"},
},
"product_product": {
"model": "product.product", "archivable": True,
# ββ W41-T26 (owner instruction 28) β `barcode` IS the UPC. MEASURED on live Odoo
# 2026-08-24 before it was declared, per this file's own standing rule that a column is
# added on evidence and not on "Odoo probably has that": of 5,874 ACTIVE products, 3,877
# carry a real barcode and 1,997 carry none. It is a `store=True` char column on
# `product.product`, so the mirror can hold it.
"fields": ["default_code", "name", "categ_id", "type", "standard_price", "active",
"barcode", "uom_id", "create_date", "product_tmpl_id", "volume",
"display_name", "product_tag_ids", "incoming_qty", "free_qty",
"virtual_available", "write_date"],
# Computed or template-related in Odoo 16: they change with no product.product write_date.
"refresh_fields": ["display_name", "product_tag_ids", "incoming_qty", "free_qty",
"virtual_available"],
# β WITHOUT THIS THE COLUMN ALTERS IN AND STAYS NULL FOREVER β the same trap
# `account_move.invoice_origin` and the `res_partner` address block are both commented
# for. WE moved the schema; no product's `write_date` moved, so an incremental
# `sync_all()` never revisits a single existing row and the column is indistinguishable
# from "this product genuinely has no barcode".
# β DECLARING IS NOT FILLING. This entry only makes the next backfill pass pick the
# column up; until that run happens the mirror's `barcode` is NULL for all 5,874 rows.
# W41-T26 could not run it (a sync writes the tenant store, which the ticket forbids).
# β AND FILLING IS NOT READING β the third link, stated so nobody reads a green backfill
# as a shipped column. `aios-web/api/odoo_relational.py::read_products` builds a FIXED
# select list (id, default_code, name, categ_name, type, standard_price, active) and does
# NOT select `barcode`, so even a fully backfilled mirror surfaces nothing. THE GRID'S UPC
# DOES NOT COME FROM HERE: it comes live from `modules/products.py::catalogue()`. This
# column exists so the mirror is not permanently blind to a field the source has, which is
# D-165's shape; widening `read_products` is a separate ticket with its own fence.
"backfill_fields": ["barcode", "uom_id", "create_date", "product_tmpl_id", "volume"],
},
"account_move": {
"model": "account.move", "archivable": False,
# β `invoice_origin` added 2026-08-09 (wave 28, DEBT D-88) β the ORDER NAME an invoice was
# raised from ("S12345"), which is the only single stored key that joins an invoice back
# to its order.
# β WITHOUT IT THAT JOIN IS TWO HOPS, and the second one is expensive: invoice ->
# account_move_line -> `sale_line_ids` (the m2m BU bridge) -> sale_order_line ->
# sale_order, across 963,783 move lines. Wave 27 wanted an order->invoice preset link and
# had no single column to derive it from, so the link was not built at all.
# β IT IS A NAME, NOT AN ID, and Odoo writes free text into it β a manual invoice can hold
# anything, and a merged one can hold several origins space-separated. So it is a JOIN
# HINT, and whatever consumes it matches against the order names that actually exist
# rather than trusting the string. The m2m route above stays the authority for BU
# attribution; this does not replace it.
"fields": ["name", "move_type", "state", "invoice_date", "invoice_date_due",
"partner_id", "amount_untaxed_signed", "amount_residual_signed",
"payment_state", "invoice_origin", "write_date"],
# Existing rows never re-sync on their own (WE moved the schema; their write_date did
# not), so without this the column ALTERs in and stays NULL forever β indistinguishable
# from "this invoice genuinely has no origin". Declared, re-entrant, checkpointed.
"backfill_fields": ["invoice_origin"], # added 2026-08-09
},
"account_payment": {
"model": "account.payment", "archivable": False, "optional": True,
"fields": ["date", "state", "payment_type", "partner_type", "partner_id",
"is_internal_transfer", "amount_company_currency_signed", "write_date"],
},
"account_move_line": {
"model": "account.move.line", "archivable": False,
# display_type/move_type/parent_state/price_subtotal/product_id added 2026-07-28 so the
# invoice-LINE grain is queryable (topic `invoice_lines`).
# β display_type values here are 'product', 'cogs', 'payment_term', 'line_note',
# 'line_section'. Commission rows attach to 'cogs' lines as well as 'product' ones
# (96,936 of 147,160 in 2025). Filtering to 'product' is a GRAIN guard: measured
# 2026-07-29, omitting it does not move revenue (non-product lines carry
# price_subtotal = 0) but it inflates commission ROW COUNTS ~80% and would corrupt any
# count-of-lines measure.
"fields": ["move_id", "account_id", "partner_id", "date", "debit", "credit",
"balance", "display_type", "move_type", "parent_state", "price_subtotal",
"product_id", "analytic_distribution", "write_date", "amount_residual", "date_maturity", "reconciled"],
"backfill_fields": ["display_type", "move_type", "parent_state", "price_subtotal",
"product_id", "analytic_distribution", "amount_residual", "date_maturity", "reconciled"],
"backfill_before_delta": True,
# the BU bridge: account.move carries NO business unit (every invoice sits on team 1 β
# see [[invoice-bu-attribution]]), so Fisch/Royal on the invoice basis is only reachable
# through the originating sale order. sale_line_id (the FIRST linked sale line) is
# lossless here: ZERO 2025 invoice lines span more than one team (measured).
"m2m": ["sale_line_ids"],
},
"account_invoice_line_agent": {
# The OCA sale-commission module: one row per (invoice line Γ agent). This is the SECOND
# agent source β per-line commission attribution β and it names a different population
# than res_partner.agent_ids (the customer-master book). See topic `invoice_lines`.
# β TWO similarly-named FKs, do not confuse them:
# object_id -> account_move_line (THE grain; join on this)
# invoice_id -> account_move (the parent document)
"model": "account.invoice.line.agent", "archivable": False,
"fields": ["agent_id", "commission_id", "amount", "invoice_id", "object_id",
"invoice_date", "settled", "write_date"],
},
"account_account": {
# NOTE: no `active` field on account.account in this Odoo version (learned 2026-07-11 β
# a ('active','in',...) domain 500s); treat as non-archivable.
"model": "account.account", "archivable": False,
# non_trade feeds the line-grain Overdue (ar.AGED_LINE_DOMAIN). Not a grid column, so unaudited.
"fields": ["code", "name", "account_type", "non_trade", "write_date"],
"backfill_fields": ["non_trade"],
},
# ββ W37-T13 (owner: stock moved IN and OUT per SKU) β MEASURED FIRST in
# `proto/P1-stock-moves.md`, and every clause below is a finding rather than a guess.
#
# β THE TWO LOCATION COLUMNS ARE BOTH LOAD-BEARING AND THIS IS THE TICKET'S HEADLINE TRAP.
# A move is an IN when it ARRIVES somewhere internal from somewhere that is not, and an OUT
# when it LEAVES somewhere internal for somewhere that is not. `internal -> internal` is
# **58% of all moves** and is NEITHER β a one-sided domain counts every one of them as BOTH,
# which roughly doubles both columns while looking entirely plausible.
#
# β `picking_code` IS NOT THE DISCRIMINATOR, and the reason is worth the line: it is
# `store=False`, so a `read_group` on it fails hard, AND `('picking_code','=',False)` returns
# 0 while 5,800 pickingless done moves exist β **the count that would have warned you also
# reads 0**. `picking_id` is stored and is carried instead, for provenance only.
#
# β `quantity_done` IS THE AGGREGATE, not `product_uom_qty`: it ties to
# `stock.move.line.qty_done` to the cent over the full window, and it is what actually moved.
# β NOT ARCHIVABLE β `stock.move` has no `active` field; a move is history and is never
# archived. Stated so the next reader does not "fix" it by adding `active in [True, False]`.
# β SIZE: 236k rows over 1,095 days (P1), about a quarter of `account_move_line`'s 963,783.
"stock_landed_cost": {
"model": "stock.landed.cost", "archivable": False, "optional": True,
"fields": ["name", "vendor_bill_id", "write_date"],
"m2m": ["picking_ids"],
},
"stock_valuation_adjustment_lines": {
"model": "stock.valuation.adjustment.lines", "archivable": False, "optional": True,
"fields": ["cost_id", "product_id", "move_id", "quantity", "former_cost",
"additional_landed_cost", "final_cost", "write_date"],
},
"account_bank_statement_line": {
"model": "account.bank.statement.line", "archivable": False, "optional": True,
"fields": ["move_id", "statement_id", "currency_id", "write_date"],
},
"stock_picking": {
"model": "stock.picking", "archivable": False, "optional": True,
"fields": ["name", "origin", "sale_id", "partner_id", "picking_type_code", "state",
"scheduled_date", "date_done", "write_date"],
"backfill_fields": ["name", "origin", "sale_id", "partner_id", "picking_type_code",
"state", "scheduled_date", "date_done"],
"backfill_required": True,
},
"stock_move": {
# β `optional` β see `ready()`. A NEW entity must never gate the readiness of surfaces
# that do not read it, or adding one takes the product down until it has synced.
"model": "stock.move", "archivable": False, "optional": True,
"fields": ["name", "product_id", "product_uom", "quantity_done", "date", "state",
"location_id", "location_dest_id", "picking_id", "sale_line_id", "write_date"],
"backfill_fields": ["name", "product_uom", "sale_line_id"],
},
"uom_uom": {
"model": "uom.uom", "archivable": True, "optional": True,
"fields": ["factor", "category_id", "active", "write_date"],
},
# β THE USAGE LOOKUP, WITHOUT WHICH THE ROW ABOVE CANNOT BE READ. `location_id` syncs as
# `location_id` + `location_id_name` (the m2o convention) β a NAME, never a usage β so the
# two-sided domain has nothing to test until this table exists. It is tiny (a few hundred
# rows) and it is the whole difference between a metric and a column of doubled numbers.
# β `usage` is a selection: `internal` | `supplier` | `customer` | `inventory` | `production`
# | `transit` | `view`. β Adjustments (`inventory`) and scrap are 30% of IN / 20% of OUT and
# the two obvious exclusions are NOT equivalent β the 5,320-unit gap is all
# `Virtual Locations/Scrap` β so the metric that consumes this CHOOSES deliberately and says
# which; that choice lives in `model/metrics/stock.yml`, not here.
"stock_location": {
"model": "stock.location", "archivable": True, "optional": True,
"fields": ["name", "complete_name", "usage", "active", "parent_path", "write_date"],
"backfill_fields": ["parent_path"],
},
"stock_valuation_layer": {
"model": "stock.valuation.layer", "archivable": False, "optional": True,
"fields": ["product_id", "quantity", "value", "create_date", "stock_move_id", "write_date"],
},
"stock_quant": {
"model": "stock.quant", "archivable": False, "optional": True,
"fields": ["product_id", "location_id", "quantity", "reserved_quantity", "write_date"],
},
"purchase_order": {
"model": "purchase.order", "archivable": False, "optional": True,
"fields": ["name", "partner_id", "date_approve", "date_order", "state",
"amount_untaxed", "amount_total", "currency_id", "write_date"],
},
"purchase_order_line": {
"model": "purchase.order.line", "archivable": False, "optional": True,
"fields": ["order_id", "sale_line_id", "product_id", "product_uom", "product_uom_qty", "product_qty",
"qty_received", "price_unit", "price_subtotal", "date_planned", "write_date"],
"backfill_fields": ["sale_line_id"],
"backfill_required": True,
},
}
from core import connector_spec as _connector_spec
_ODOO_MIRROR_STORE = None
ODOO_MIRROR_STATUS = {}
_ODOO_DYNAMIC_ROSTER = "odoo_dynamic_models"
_ODOO_DYNAMIC_TABLE = re.compile(r"odoo_dynamic_[0-9a-f]{64}")
def odoo_dynamic_model_identity(tenant_key, connection_id, model, table_key):
"""Resolve a dynamic database to its durable tenant, connection and source identity."""
tenant, cid, native, key = map(str, (tenant_key or "", connection_id or "", model or "", table_key or ""))
base = _connector_spec.odoo_model_table_key(native)
if (not re.fullmatch(r"[a-z0-9][a-z0-9_-]{0,59}", tenant)
or not cid or len(cid) > 128 or native != native.strip().lower()
or (cid != "legacy-env" and (not re.fullmatch(r"[a-z0-9_]+", cid)
or "__" in cid or cid != cid.strip("_")))
or key not in (base, base + "__" + cid) or len(key) > 60
or not re.fullmatch(r"[a-z0-9_]+", key)):
raise ValueError("Invalid dynamic Odoo database identity")
source = next((name for name, spec in ENTITIES.items()
if spec["model"] == native and not spec.get("metadata_only")), None)
if source is None:
token = hashlib.sha256(json.dumps([tenant, cid, native], separators=(",", ":")).encode()).hexdigest()
source = "odoo_dynamic_" + token
return {"tenant": tenant, "connection_id": cid, "model": native, "table_key": key, "source": source}
def register_odoo_dynamic_model(rt, con, *, connection_id, model, table_key):
"""Register an enabled model inside its private Odoo publication candidate without populating rows."""
candidate = getattr(_SYNC_TLS, "candidate", None)
if (candidate is None or candidate["tenant"] != rt.key or candidate["connector"] != "odoo"
or candidate["con"] is not con or candidate["path"] != Path(rt.datastore_path).resolve()):
raise ValueError("Dynamic Odoo registration requires the tenant's publication candidate")
identity = odoo_dynamic_model_identity(rt.key, connection_id, model, table_key)
con.execute(f"CREATE TABLE IF NOT EXISTS {_ODOO_DYNAMIC_ROSTER} "
"(tenant VARCHAR, connection_id VARCHAR, model VARCHAR, table_key VARCHAR PRIMARY KEY, "
"source VARCHAR, UNIQUE(connection_id, model))")
values = list(identity.values())
prior = con.execute(f"SELECT tenant, connection_id, model, table_key, source FROM {_ODOO_DYNAMIC_ROSTER} "
"WHERE table_key=? OR (connection_id=? AND model=?)",
[table_key, connection_id, model]).fetchall()
if prior and prior != [tuple(values)]:
raise ValueError("Dynamic Odoo database identity cannot replace another binding")
# Snapshot restore recreates this roster with CTAS, which does not retain constraints.
# The publication lock serializes registration; the binding check above still rejects reuse.
con.execute(f"INSERT INTO {_ODOO_DYNAMIC_ROSTER} SELECT ?,?,?,?,? WHERE NOT EXISTS "
f"(SELECT 1 FROM {_ODOO_DYNAMIC_ROSTER} WHERE table_key=?)", [*values, table_key])
return identity
def _validate_odoo_dynamic_roster(con, tables, *, tenant_key=None, paths=None):
"""Require every dynamic table and roster entry to name the same saved population."""
dynamic = {name for name in tables if _ODOO_DYNAMIC_TABLE.fullmatch(name)}
if _ODOO_DYNAMIC_ROSTER not in tables:
if dynamic:
raise ValueError("Dynamic Odoo tables have no durable roster")
return
source = "read_parquet(?)" if paths is not None else _ODOO_DYNAMIC_ROSTER
params = [list(map(str, paths[_ODOO_DYNAMIC_ROSTER]))] if paths is not None else []
registered, keys, bindings, tenants = set(), set(), set(), set()
con.execute(f"SELECT tenant, connection_id, model, table_key, source FROM {source}", params)
while batch := con.fetchmany(256):
for tenant, cid, model, key, table in batch:
identity = odoo_dynamic_model_identity(tenant, cid, model, key)
if (table != identity["source"] or table not in tables
or (tenant_key is not None and tenant != tenant_key)
or key in keys or (cid, model) in bindings):
raise ValueError("Dynamic Odoo roster disagrees with its saved population")
registered.add(table)
keys.add(key)
bindings.add((cid, model))
tenants.add(tenant)
if len(tenants) > 1 or dynamic - registered:
raise ValueError("Dynamic Odoo roster omits a table or mixes tenants")
def _mirror_connection(tenant_key, rt):
"""Open one tenant's file without rebinding the process used by live readers."""
if rt.key != tenant_key:
raise ValueError("Odoo mirror runtime belongs to another tenant")
path = tenant_handle(tenant_key, rt.datastore_path).path
candidate = _candidate_connection(tenant_key, path)
if candidate is not None:
return candidate, path
if path.resolve() == DB_PATH.resolve():
rt.assert_datastore_matches()
return connect(), path
path.parent.mkdir(parents=True, exist_ok=True)
con = apply_limits(duckdb.connect(str(path)))
_ensure_sync_state(con)
return con, path
def _candidate_connection(tenant_key=None, path=None):
candidate = getattr(_SYNC_TLS, "candidate", None)
if candidate is None:
return None
if tenant_key is not None and candidate["tenant"] is None:
tenant_handle(tenant_key, candidate["path"])
if ((tenant_key is not None and candidate["tenant"] is not None and tenant_key != candidate["tenant"])
or (path is not None and Path(path).resolve() != candidate["path"])):
raise ValueError("Sync candidate belongs to another tenant")
return candidate["con"].cursor()
def _publication(con, source):
exists = con.execute("SELECT 1 FROM information_schema.tables "
"WHERE table_name='_mirror_publications'").fetchone()
if not exists:
return {}
row = con.execute("SELECT manifest FROM _mirror_publications WHERE source=?", [source]).fetchone()
return json.loads(row[0]) if row else {}
def _write_publication(con, source, manifest):
con.execute("CREATE TABLE IF NOT EXISTS _mirror_publications "
"(source VARCHAR PRIMARY KEY, manifest VARCHAR)")
con.execute("INSERT OR REPLACE INTO _mirror_publications VALUES (?,?)",
[source, json.dumps(manifest, sort_keys=True)])
def snapshot_status(tenant_key, source, *, rt=None):
"""Read the publication committed with this tenant's serving rows, without creating a mirror."""
source = _connector_spec.CONNECTOR_SPEC_KEYS.get(source, source)
key = str(tenant_key or "").strip().lower()
if not key or (rt is not None and rt.key != key):
raise ValueError("Snapshot status requires the requested tenant")
path = Path(rt.datastore_path) if rt is not None else path_for(key)
if rt is not None and path.resolve() != path_for(key).resolve():
raise ValueError("Snapshot status requires the tenant's canonical mirror")
result = {"snapshot_id": "unknown", "started_at": "unknown", "completed_at": "unknown",
"state": "unknown", "rows": {}}
if rt is not None:
from harness import mirror_store
saved = mirror_store.publication_status(source, key, rt)
result.update(saved)
if not (serving_file(path)[0] if versioned_mode() else path).exists():
return result
con = tenant_handle(key, path).ro_cursor()
try:
manifest = _publication(con, source)
if manifest:
for field in ("snapshot_id", "started_at", "completed_at", "state"):
result[field] = manifest.get(field) or "unknown"
result["rows"] = {name: meta["rows"] for name, meta in manifest.get("tables", {}).items()}
if manifest.get("validation"):
result.setdefault("validation", manifest["validation"])
if result.get("hold"):
result["state"] = "held"
return result
finally:
con.close()
def snapshot_id(tenant_key, source) -> str:
"""Return the last committed source identity, or unknown before its first publication."""
return snapshot_status(tenant_key, source)["snapshot_id"]
SNAPSHOT_CHECK_DEFAULTS = {"row_drop_limit": 0.02, "balance_tolerance": 0.01,
"closed_period_tolerance": 0.01}
def snapshot_check_policy(config):
"""Validate tenant publication thresholds, recorded exceptions and explicit closing dates."""
if not isinstance(config, dict):
raise ValueError("Snapshot checks must be an object.")
if set(config) - {*SNAPSHOT_CHECK_DEFAULTS, "ledger_exceptions", "closed_through"}:
raise ValueError("Unknown snapshot check setting.")
policy = {**SNAPSHOT_CHECK_DEFAULTS, **config}
for key in SNAPSHOT_CHECK_DEFAULTS:
value = policy[key]
if (isinstance(value, bool) or not isinstance(value, (int, float))
or not math.isfinite(value) or value < 0 or (key == "row_drop_limit" and value >= 1)):
raise ValueError("Snapshot check thresholds must be finite and nonnegative. Row drop must be below 1.")
exceptions = policy.setdefault("ledger_exceptions", [])
seen = set()
if not isinstance(exceptions, list):
raise ValueError("Ledger exceptions must be a list.")
for entry in exceptions:
if (not isinstance(entry, dict) or set(entry) != {"company_id", "balance", "reason"}
or type(entry["company_id"]) is not int or entry["company_id"] <= 0
or entry["company_id"] in seen
or isinstance(entry["balance"], bool) or not isinstance(entry["balance"], (int, float))
or not math.isfinite(entry["balance"])
or not isinstance(entry["reason"], str) or not entry["reason"].strip()):
raise ValueError("Each ledger exception needs a unique company, a finite balance and a reason.")
seen.add(entry["company_id"])
closed = policy.setdefault("closed_through", {})
if not isinstance(closed, dict):
raise ValueError("Closing dates must name their company.")
for company, date in closed.items():
if (not re.fullmatch(r"[1-9][0-9]*", str(company)) or not isinstance(date, str)
or not re.fullmatch(r"\d{4}-\d{2}-\d{2}", date)):
raise ValueError("Closing dates need a company id and an ISO date.")
dt.date.fromisoformat(date)
return policy
def _confirmed_row_drop(rt, published, diffs, previous_pass):
"""Accept only the preceding clean, completed drop observation against this publication."""
if not isinstance(previous_pass, dict) or not diffs:
return False
publication = _publication(published, "odoo")
identity = publication.get("snapshot_id")
if (publication.get("tenant") != rt.key or not re.fullmatch(r"[0-9a-f]{32}", str(identity))
or previous_pass.get("tenant") != rt.key or previous_pass.get("connector") != "odoo"
or previous_pass.get("state") != "held" or not previous_pass.get("completed_at")
or not re.fullmatch(r"[0-9a-f]{32}", str(previous_pass.get("pass_id", "")))
or previous_pass.get("previous_snapshot_id") != identity):
return False
validation = previous_pass.get("validation")
checks = validation.get("checks") if isinstance(validation, dict) else None
if not isinstance(checks, list) or any(not isinstance(row, dict) for row in checks):
return False
failed = [row for row in checks if row.get("state") == "failed"]
if len(failed) != 1 or failed[0].get("check") != "row_count_drop":
return False
def counts(rows):
if not isinstance(rows, list) or not rows:
return None
result = {}
for row in rows:
if (not isinstance(row, dict) or not isinstance(row.get("table"), str)
or row["table"] in result or type(row.get("previous")) is not int
or type(row.get("candidate")) is not int
or not 0 <= row["candidate"] < row["previous"]):
return None
result[row["table"]] = (row["previous"], row["candidate"])
return result
prior_counts = counts(failed[0].get("diffs"))
return prior_counts is not None and prior_counts == counts(diffs)
def check_odoo_snapshot(rt, candidate, published, row_counts, *, previous_pass=None):
"""Check a complete private observation against the previous publication and tenant policy."""
from decimal import Decimal
cfg = rt.get(_ODOO_CONFIG_KEY, fresh=True) or {}
policy = snapshot_check_policy(cfg.get("snapshot_checks", {}))
checks = []
def record(name, diffs=(), **details):
# Odoo ledger warnings stay warnings. An unconfirmed drop in our copy holds publication.
found = "failed" if name == "row_count_drop" else "warning"
checks.append({"check": name, "state": found if diffs else "passed",
"diffs": list(diffs), **details})
def columns(con, table):
return {row[0] for row in con.execute(
"SELECT column_name FROM information_schema.columns WHERE table_name=?", [table]).fetchall()}
# A whole-table replace is a complete count-checked read of Odoo, so its drop is real, not a lost copy.
guarded_tables = (*ENTITIES, *sorted(name for name in _odoo_mirror_store()._tables(published)
if _ODOO_DYNAMIC_TABLE.fullmatch(name)))
previous_counts = {name: published.execute(f'SELECT count(*) FROM "{name}"').fetchone()[0]
for name in guarded_tables if columns(published, name)
and (name not in REPLACE_EACH_PASS or REPLACE_EACH_PASS[name].get("scope") == "internal")}
record("row_count_drop", [
{"table": name, "previous": count, "candidate": row_counts.get(name, 0),
"removed": count - row_counts.get(name, 0),
"drop_share": (count - row_counts.get(name, 0)) / count,
"limit": policy["row_drop_limit"]}
for name, count in previous_counts.items() if count
and Decimal(count - row_counts.get(name, 0)) / Decimal(count) > Decimal(str(policy["row_drop_limit"]))])
drop = checks[-1]
if _confirmed_row_drop(rt, published, drop["diffs"], previous_pass):
drop.update(state="warning", confirmed_by=previous_pass["pass_id"])
ledger = columns(candidate, "account_move_line")
required = {"account_id", "company_id", "balance", "date", "parent_state", "display_type"}
if not ledger and not columns(published, "account_move_line"):
for name in ("ledger_balance", "unmapped_accounts", "closed_period_balances"):
record(name, state="not_applicable", reason="No ledger in this source selection.")
return {"thresholds": policy, "checks": checks}
if required - ledger:
for name in ("ledger_balance", "unmapped_accounts", "closed_period_balances"):
record(name, [{"missing_columns": sorted(required - ledger)}])
return {"thresholds": policy, "checks": checks}
where = "parent_state='posted' AND coalesce(display_type, '') NOT IN ('line_note','line_section')"
invalid = candidate.execute(f"SELECT count(*) FROM account_move_line WHERE {where} AND "
"(balance IS NULL OR NOT isfinite(balance) OR company_id IS NULL "
"OR TRY_CAST(date AS DATE) IS NULL)").fetchone()[0]
totals = candidate.execute(f"SELECT company_id, sum(CAST(balance AS DECIMAL(38,6))) "
f"FROM account_move_line WHERE {where} AND isfinite(balance) GROUP BY company_id").fetchall()
exceptions = {entry["company_id"]: entry for entry in policy["ledger_exceptions"]}
balance_diffs, accepted = [], []
for company, value in totals:
balanced = abs(value) <= Decimal(str(policy["balance_tolerance"]))
expected = Decimal(0) if balanced else Decimal(str(exceptions.get(company, {}).get("balance", 0)))
delta = value - expected
item = {"company_id": company, "balance": float(value), "expected": float(expected),
"difference": float(delta)}
if abs(delta) > Decimal(str(policy["balance_tolerance"])):
balance_diffs.append(item)
elif not balanced and company in exceptions:
accepted.append({**item, "reason": exceptions[company]["reason"]})
move_cols = ledger & {"move_id", "move_name"}
if "move_id" in move_cols:
# Name the posted moves whose own lines do not sum to zero: that is the cause in Odoo.
named = ", any_value(move_name)" if "move_name" in move_cols else ""
for item in balance_diffs:
moves = candidate.execute(
f"SELECT move_id, sum(CAST(balance AS DECIMAL(38,6))) AS total{named} FROM account_move_line "
f"WHERE {where} AND isfinite(balance) AND company_id=? AND move_id IS NOT NULL "
"GROUP BY move_id HAVING abs(sum(CAST(balance AS DECIMAL(38,6)))) > ? ORDER BY abs(total) DESC, move_id",
[item["company_id"], Decimal(str(policy["balance_tolerance"]))]).fetchall()
item["unbalanced_moves"] = len(moves)
item["moves"] = [{"move_id": row[0], **({"move_name": row[2]} if named else {}),
"balance": float(row[1])} for row in moves[:10]]
if invalid:
balance_diffs.append({"invalid_ledger_lines": invalid})
record("ledger_balance", balance_diffs, exceptions=accepted)
account_cols = columns(candidate, "account_account")
if {"id", "code", "account_type"} <= account_cols:
missing = candidate.execute(
f"SELECT l.account_id, count(*) FROM account_move_line l LEFT JOIN account_account a "
f"ON a.id=l.account_id WHERE {where} AND (a.id IS NULL OR trim(coalesce(a.code,''))='' "
"OR trim(coalesce(a.account_type,''))='') GROUP BY l.account_id ORDER BY l.account_id").fetchall()
else:
missing = candidate.execute(f"SELECT account_id, count(*) FROM account_move_line WHERE {where} "
"GROUP BY account_id ORDER BY account_id").fetchall()
record("unmapped_accounts", [{"account_id": account, "lines": count} for account, count in missing])
closed = {int(company): date for company, date in policy["closed_through"].items()}
for con in (published, candidate):
available = columns(con, "res_company")
fields = [name for name in ("fiscalyear_lock_date", "period_lock_date") if name in available]
if "id" in available and fields:
rows = con.execute("SELECT id," + ",".join(f"TRY_CAST({name} AS DATE)" for name in fields)
+ " FROM res_company").fetchall()
for company, *dates in rows:
for date in dates:
if date is not None:
closed[company] = max(closed.get(company, ""), date.isoformat())
if not closed:
record("closed_period_balances", state="not_checked", reason="No recorded closing date.", closed_through={})
elif not columns(published, "account_move_line"):
record("closed_period_balances", state="not_checked", reason="No published ledger baseline.", closed_through=closed)
elif required - columns(published, "account_move_line"):
record("closed_period_balances", [{"missing_baseline_columns": sorted(
required - columns(published, "account_move_line"))}], closed_through=closed)
else:
def balances(con, company, date):
return {(account, period): value for account, period, value in con.execute(
f"SELECT account_id, substr(CAST(date AS VARCHAR),1,7), sum(CAST(balance AS DECIMAL(38,6))) "
f"FROM account_move_line WHERE {where} AND company_id=? AND TRY_CAST(date AS DATE)<=? "
"AND isfinite(balance) GROUP BY 1,2", [company, date]).fetchall()}
changed = []
for company, date in sorted(closed.items()):
before, after = balances(published, company, date), balances(candidate, company, date)
for account, period in sorted(before.keys() | after.keys(), key=str):
old, new = before.get((account, period), Decimal(0)), after.get((account, period), Decimal(0))
if abs(new - old) > Decimal(str(policy["closed_period_tolerance"])):
changed.append({"company_id": company, "account_id": account, "period": period,
"previous": float(old), "candidate": float(new), "difference": float(new - old)})
record("closed_period_balances", changed, closed_through=closed)
return {"thresholds": policy, "checks": checks}
def _odoo_mirror_store():
"""Add Odoo marker and refusal policy to the shared checkpoint engine."""
global _ODOO_MIRROR_STORE
if _ODOO_MIRROR_STORE is None:
from harness import mirror_store
class OdooMirrorStore(mirror_store.MirrorStore):
def _valid_table(self, name):
return (name in {"odoo_connections", "odoo_record_ids", _ODOO_DYNAMIC_ROSTER}
or isinstance(name, str) and bool(_ODOO_DYNAMIC_TABLE.fullmatch(name))
or super()._valid_table(name))
def _tables(self, con):
tables = super()._tables(con)
_validate_odoo_dynamic_roster(con, tables)
return tables
def _connect(self, tenant_key, rt):
self._mirror_cache_key(tenant_key, rt)
return _mirror_connection(tenant_key, rt)
def _owns_state(self, entity):
table = entity.split(".", 1)[0]
return table in self.spec.tables or bool(_ODOO_DYNAMIC_TABLE.fullmatch(table))
def _install_snapshot(self, con, manifest, rows, paths, only=None):
if (any(not self._valid_table(name) for name in paths)
or any(row[0].split(".", 1)[0] not in paths for row in rows)):
raise ValueError("Odoo mirror install names an unrelated or missing table")
_validate_odoo_dynamic_roster(con, set(paths), tenant_key=manifest.get("tenant"), paths=paths)
return super()._install_snapshot(con, manifest, rows, paths, only=only)
def _restore_complete(self, tables, rows, con=None):
"""Require live sync entities and installed receipts for their required column backfills."""
entities = set(tables) & set(self.spec.tables)
return (con is not None and super()._restore_complete(entities, rows)
and all(row[1] == "live" for row in rows if row[0] in entities)
and all(_required_columns_ready(con, key, ENTITIES.get(key) or {})
for key in entities))
def _validated_snapshot(self, tenant_key, rt, con, raw, tmp, **kwargs):
# Odoo never advertises a partial restore or replaces it with a seed.
snapshot = super()._validated_snapshot(tenant_key, rt, con, raw, tmp,
allow_partial=False)
_manifest, rows, paths, _total, _refused = snapshot
if any(row[0].split(".", 1)[0] not in paths for row in rows):
raise ValueError("Odoo mirror state names a missing table")
_validate_odoo_dynamic_roster(con, set(paths), tenant_key=tenant_key, paths=paths)
return snapshot
def _refused_may_publish(self, tenant_key, rt, con, cache):
return False
_ODOO_MIRROR_STORE = OdooMirrorStore("odoo", _connector_spec.mirror_for("odoo"))
return _ODOO_MIRROR_STORE
def _mirror_measurement(started):
"""Record elapsed time and the process RSS high-water mark, including native DuckDB memory."""
try:
import resource
peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
peak = int(peak if sys.platform == "darwin" else peak * 1024)
except ImportError:
peak = None
return {"seconds": round(time.monotonic() - started, 3), "process_peak_rss_bytes": peak,
"database_node_limit_bytes": 2 * 1024 ** 3}
def odoo_mirror_restored(tenant_key, rt):
"""Check this process's successful manifest restore for the exact tenant and mirror path."""
store = _odoo_mirror_store()
cache_key = store._mirror_cache_key(tenant_key, rt)
with store.lock:
result = store.cache.get(cache_key, {})
return result.get("state") in ("restored", "local") and bool(result.get("manifest"))
def restore_odoo_mirror(tenant_key, rt, log=print):
"""Restore only this tenant's saved Odoo tables and preserve a refusal until restart."""
if versioned_mode() and getattr(_SYNC_TLS, "candidate", None) is None:
# Readers hold the serving version read-only, so the restore writes the next version.
try:
with versioned_write(tenant_key) as handle:
result = _restore_odoo_mirror(tenant_key, rt, log)
handle["publish"] = result.get("state") != "refused"
except MirrorVersionRefused as exc:
# The restore itself worked. Its version failed verify, so the old version serves.
log(str(exc))
result = {**result, "problems": [*result.get("problems", []), str(exc)]}
return result
return _restore_odoo_mirror(tenant_key, rt, log)
def _restore_odoo_mirror(tenant_key, rt, log):
started = time.monotonic()
result = _odoo_mirror_store().restore_mirror(tenant_key, rt, log=log)
status = ODOO_MIRROR_STATUS.setdefault(str(Path(rt.datastore_path).resolve()), {})
status.update(state=result["state"], problems=list(result["problems"]))
status.setdefault("restore", _mirror_measurement(started))
log(f"Odoo mirror restore {tenant_key}: {result['state']} {status['restore']}")
return result
def persist_odoo_mirror(tenant_key, rt, changed_tables, log=print):
"""Save local Odoo observations through the shared seam without contacting Odoo."""
if getattr(_SYNC_TLS, "candidate", None) is not None:
return {} # The enclosing source pass saves and measures after completion and validation.
started = time.monotonic()
result = restore_odoo_mirror(tenant_key, rt, log=lambda *_: None)
if result["state"] == "refused":
raise RuntimeError("Odoo saved mirror was refused; repair its store access and restart before syncing")
report = {"problems": []}
con, _path = _mirror_connection(tenant_key, rt)
try:
con.execute("BEGIN TRANSACTION")
try:
_odoo_mirror_store().persist_mirror(tenant_key, rt, con, changed_tables,
report=report, log=log)
except BaseException:
con.execute("ROLLBACK")
raise
else:
con.execute("COMMIT")
finally:
con.close()
status = ODOO_MIRROR_STATUS[str(Path(rt.datastore_path).resolve())]
status["save"] = _mirror_measurement(started)
status["problems"] = report["problems"]
# Resident sync suppresses row logs; checkpoint resource evidence must remain visible.
print(f"[datastore] Odoo mirror save {tenant_key}: {status['save']}")
if report["problems"]:
raise RuntimeError("; ".join(report["problems"]))
return status["save"]
def _resident_odoo_runtime():
from harness import runtime
key = runtime.tenant_for_store_path(DB_PATH)
if not key:
return None # A bespoke scratch file has no durable tenant destination.
rt = runtime.get_runtime(key)
if rt.key != key or Path(rt.datastore_path).resolve() != DB_PATH.resolve():
raise RuntimeError("Odoo checkpoint requires the resident tenant's mirror")
rt.assert_datastore_matches()
return rt
def _odoo_restore_refused(path=None):
return ODOO_MIRROR_STATUS.get(str(Path(path or DB_PATH).resolve()), {}).get("state") == "refused"
BATCH = 2000
# Fields that are genuinely BOOLEAN. This list is load-bearing in TWO places: _cols types the
# column, and _flatten must NOT collapse a real False into NULL. Odoo returns False both for
# "empty" and for "boolean false", so a bool field missing from here silently becomes NULL β and
# for res_partner.agent that would erase the ONE flag separating an AGENT from a SALESPERSON
# (see [[invoice-line-agent-commission]]): every row would read "not an agent" indistinguishably
# from "unknown".
BOOL_FIELDS = ("active", "agent", "salesman_as_agent", "settled", "commission_free",
"is_internal_transfer", "non_trade")
# β WAVE 29 β fields that are genuinely INTEGER COUNTS, and this list exists because its absence
# shipped a silent defect within minutes of `customer_rank` being added to the res_partner spec.
# `_cols` types anything it does not recognise as VARCHAR, so `customer_rank` landed as text, the
# mirror populated correctly, every row looked right β and `odoo_relational`'s own predicate
# `customer_rank > 0` died with `Binder Error: Cannot compare values of type VARCHAR and type
# INTEGER_LITERAL`. β THE COLUMN WAS PRESENT AND THE DATA WAS CORRECT; only its TYPE was wrong,
# which is why "is the field in the mirror?" answered yes and the feature still could not run.
# β A new numeric field belongs HERE, exactly as a new boolean belongs in BOOL_FIELDS above.
INT_FIELDS = ("customer_rank", "supplier_rank")
import threading as _thr
_RO_TLS = _thr.local()
_SYNC_TLS = _thr.local()
_READY = {"ok": False}
_INSTANCE = {"con": None}
_INSTANCE_LOCK = _thr.Lock()
#: Bumped by `use_path()` whenever the store FILE changes. Every per-thread cursor is stamped with
#: the generation it was opened under, so a thread that never called `use_path` still cannot keep
#: reading the previous tenant's file β see `ro_con`. A plain "is the cursor alive?" probe cannot
#: answer that question, which is why this counter exists rather than more probing.
_GENERATION = [0]
#: Bumped after every mirror snapshot install (`MirrorStore._install_snapshot`). D-2670: the
#: Sheet change token caches its table scan per generation, so an install must move this.
_MIRROR_INSTALLS = [0]
_MIRROR_WRITES = [0]
_ACTIVE_WRITES = [0]
_COUNT_CACHE = OrderedDict()
_COUNT_CACHE_MAX = 256
_COUNT_CACHE_LOCK = _thr.RLock()
_WRITE_HISTORY = OrderedDict()
_WRITE_HISTORY_TENANTS = 16
_WRITE_HISTORY_EVENTS = 32
_WRITE_HISTORY_STACK = 6
def _remember_source_write(handle):
"""Retain bounded accounting evidence without retaining frames, SQL or arguments."""
if handle is None:
return
try:
frames, frame = [], sys._getframe(1)
try:
for _ in range(_WRITE_HISTORY_STACK):
if frame is None:
break
symbol = frame.f_code.co_name
basename = frame.f_code.co_filename.replace("\\", "/").rsplit("/", 1)[-1]
frames.append({"symbol": symbol if re.fullmatch(r"[\w.<>]{1,64}", symbol, re.ASCII)
else "<unavailable>",
"file": basename if re.fullmatch(r"[\w.-]{1,64}\.py", basename, re.ASCII)
else "<unavailable>", "line": frame.f_lineno})
frame = frame.f_back
finally:
del frame
event = {"counter": handle.writes, "pid": os.getpid(), "thread": _thr.get_ident(),
"time_ns": time.time_ns(), "classification": "source_write", "stack": frames}
key = _weakref.ref(handle)
history = _WRITE_HISTORY.get(key)
if history is None:
history = {"events": deque(maxlen=_WRITE_HISTORY_EVENTS), "lost_through": handle.writes - 1}
_WRITE_HISTORY[key] = history
_WRITE_HISTORY.move_to_end(key)
while len(_WRITE_HISTORY) > _WRITE_HISTORY_TENANTS:
_WRITE_HISTORY.popitem(last=False)
if len(history["events"]) == _WRITE_HISTORY_EVENTS:
history["lost_through"] = history["events"][0]["counter"]
history["events"].append(event)
except BaseException:
pass
def mirror_write_evidence(cur, before, after):
"""Report only retained events inside this handle's exact compared counter interval."""
result = {"status": "unavailable", "complete": False, "events": []}
if type(before) is not int or type(after) is not int or after < before:
return result
handle = getattr(cur, "_mirror_handle", None)
if handle is None:
return result # Legacy counters combine tenants and cannot attribute a tenant's writes.
if before == after:
return {"status": "not_moved", "complete": False, "events": []}
with _COUNT_CACHE_LOCK:
history = _WRITE_HISTORY.get(_weakref.ref(handle))
if history is None:
return result
events = [dict(event, stack=[dict(frame) for frame in event["stack"]])
for event in history["events"] if before < event["counter"] <= after]
overflow = before < history["lost_through"]
complete = not overflow and len(events) == after - before
return {"status": "overflow" if overflow else "retained" if complete else "unavailable",
"complete": complete, "events": events}
def _invalidate_counts(path=None, *, source_write=True):
"""Clear count results for all writes; only source writes move the mirror identity."""
with _COUNT_CACHE_LOCK:
_COUNT_CACHE.clear()
if source_write:
_MIRROR_WRITES[0] += 1
handle = _HANDLES_LIVE.get(str(Path(path or DB_PATH).resolve()))
if handle is not None:
handle.writes += 1
_remember_source_write(handle)
def note_mirror_install(path=None, *, cur=None):
"""Mark that a mirror install committed or rolled back, so cached change tokens recompute."""
if cur is not None:
bound = getattr(cur, "_mirror_handle", None)
if bound is not None:
path = bound.path
else:
marks = mirror_file_marks(cur)
candidate = getattr(_SYNC_TLS, "candidate", None)
path = candidate["path"] if candidate is not None else (marks[0] if marks else path)
_MIRROR_INSTALLS[0] += 1
with _COUNT_CACHE_LOCK:
bound = _HANDLES_LIVE.get(str(Path(path or DB_PATH).resolve()))
if bound is not None:
bound.installs += 1
_invalidate_counts(path)
def mirror_generation(cur=None):
"""The install generation of the serving mirror: file swaps and snapshot installs both move it."""
handle = getattr(cur, "_mirror_handle", None)
if handle is not None:
return (str(handle.path), _pointer_stamp(handle.path) if versioned_mode() else None,
handle.installs)
return (str(DB_PATH), _GENERATION[0], _MIRROR_INSTALLS[0])
def mirror_writes(cur=None):
"""Return permanent-source writes for this tenant, or the legacy single-store token."""
handle = getattr(cur, "_mirror_handle", None)
return handle.writes if handle is not None else _MIRROR_WRITES[0]
def mirror_active_writes(cur=None):
"""Return active writes for the cursor's tenant without another tenant's writer state."""
handle = getattr(cur, "_mirror_handle", None)
return handle.active_writes if handle is not None else _ACTIVE_WRITES[0]
def mark_ready():
"""In-process signal that the store is fully synced (every entity live). The sync writer calls
this after an all-live pass so a reader never has to (re)discover usability by touching the
file. Belt-and-suspenders alongside the shared-instance model below."""
_READY["ok"] = True
def _instance():
"""THE one process-wide read-write DuckDB connection. Readers take .cursor() off it and the
sync writer uses it too β DuckDB serves concurrent reads + a single writer from ONE instance
via MVCC, so a read NEVER contends with a sync pass for the file lock.
This is the fix for the 'connection error unless I refresh' bug (2026-07-16): an independent
read-only `duckdb.connect()` FAILS while the writer holds the lock ('Can't open a connection
to same database file with a different configuration than existing connections'); a cursor on
the shared instance does not. Openers must ensure the file is fully present first β ensure_seed
writes atomically (os.replace) and every read path guards on DB_PATH.exists() β so the instance
never binds a partial or absent file. (Trade-off: the app now holds the write lock for its whole
life, so a SEPARATE process β e.g. mcp_server.py β cannot open the same store file concurrently;
in production only the app runs, and that was already true during any sync pass.)"""
if versioned_mode():
_follow_version()
con = _INSTANCE["con"]
if con is not None:
return con
opened = False
with _INSTANCE_LOCK:
_wait_for_swap_locked()
if _INSTANCE["con"] is None:
DB_PATH.parent.mkdir(parents=True, exist_ok=True)
if versioned_mode():
stamp = _pointer_stamp()
physical, version = serving_file()
_INSTANCE["con"] = apply_limits(duckdb.connect(str(physical), read_only=True))
_VERSION.update(stamp=stamp, file=str(physical), version=version,
checked=time.monotonic())
opened = True
else:
_INSTANCE["con"] = apply_limits(duckdb.connect(str(DB_PATH)))
_own(_INSTANCE["con"])
con = _INSTANCE["con"]
if opened:
_publish_serving()
return con
# ----------------------------------------------------------------------------- read drain (F47)
# A swap never closes an instance under a cursor that is mid-read. Each `_TimedCursor` taken off a
# shared instance registers with that instance's `_Owner`. A swap marks the owner not live, waits
# (bounded) until none of its cursors is busy, then closes it. An idle cursor whose instance was
# swapped out rebinds to the serving instance on its next statement, within the same store only.
#: The longest a swap waits for open reads before it closes their instance anyway.
DRAIN_SECONDS = 30.0
#: A result nobody fetched from for this long counts as abandoned, not mid-read.
READ_IDLE_SECONDS = 2.0
_READS = _thr.Condition()
_OWNERS = {} # id(instance) -> _Owner, for every shared instance this module opened
#: Bumped by `use_path` only. A cursor opened under another epoch never rebinds (tenant switch).
_STORE_EPOCH = [0]
_SWAP = {"active": False, "thread": None}
_SWAP_COND = _thr.Condition(_INSTANCE_LOCK)
_SWAP_LOCK = _thr.RLock()
class _Owner:
"""One shared DuckDB instance and the cursors reading it."""
__slots__ = ("live", "cursors", "__weakref__")
def __init__(self):
import weakref
self.live = True
self.cursors = weakref.WeakSet()
def _own(con):
owner = _Owner()
_OWNERS[id(con)] = owner
return owner
def _drain_seconds():
try:
return max(0.0, float(os.environ.get("AIOS_MIRROR_DRAIN_SECONDS") or DRAIN_SECONDS))
except ValueError:
return DRAIN_SECONDS
def open_reads(con=None):
"""How many cursors on `con` (default: the serving instance) are mid-read right now."""
owner = _OWNERS.get(id(con if con is not None else _INSTANCE.get("con")))
if owner is None:
return 0
now = time.monotonic()
with _READS:
return sum(1 for cur in list(owner.cursors) if cur._busy(now))
def _drain(owner, seconds, ignore_thread=None):
"""Wait until no cursor on `owner` is mid-read, at most `seconds`. True when it drained.
`ignore_thread` skips a pending result left on the swapping thread: it cannot be mid-read.
"""
if owner is None:
return True
deadline = time.monotonic() + max(0.0, seconds)
with _READS:
while True:
now = time.monotonic()
if not any(cur._busy(now, ignore_thread) for cur in list(owner.cursors)):
return True
if now >= deadline:
return False
_READS.wait(min(deadline - now, 0.05))
def _close_instance(con):
owner = _OWNERS.pop(id(con), None)
if owner is not None:
owner.live = False
try:
con.close()
except Exception: # noqa: BLE001
pass
def _wait_for_swap_locked():
"""Hold a new reader until a running swap installs the next file. Call under _INSTANCE_LOCK."""
deadline = time.monotonic() + _drain_seconds() + 5.0
while _SWAP["active"] and _SWAP["thread"] != _thr.get_ident():
left = deadline - time.monotonic()
if left <= 0:
break
_SWAP_COND.wait(left)
@contextmanager
def _swap_instance():
"""Detach the serving instance, drain its open reads, then run the body under _INSTANCE_LOCK
after closing it. New readers wait for the body (they would reopen the old file otherwise:
DuckDB hands back the cached instance for a path that is still open)."""
with _SWAP_LOCK:
with _INSTANCE_LOCK:
old = _INSTANCE.get("con")
owner = _OWNERS.get(id(old)) if old is not None else None
_INSTANCE["con"] = None
_SWAP.update(active=True, thread=_thr.get_ident())
if owner is not None:
owner.live = False
try:
if not _drain(owner, _drain_seconds(), ignore_thread=_thr.get_ident()):
print(f"[datastore] mirror swap closed {open_reads(old)} read(s) still open after "
f"{_drain_seconds():.0f} s", flush=True)
with _INSTANCE_LOCK:
if old is not None:
_close_instance(old)
yield
finally:
with _INSTANCE_LOCK:
_SWAP.update(active=False, thread=None)
_SWAP_COND.notify_all()
def _open_cursor():
"""(raw cursor, owner) on the serving instance. Retries once if a swap closed it meanwhile."""
for attempt in range(2):
con = _instance()
owner = _OWNERS.get(id(con))
try:
return con.cursor(), owner
except Exception: # noqa: BLE001
if attempt or owner is None or owner.live:
raise
raise RuntimeError("unreachable")
# ----------------------------------------------------------------------------- mirror versions
# Several uvicorn workers serve one mirror: DuckDB allows one read-write process OR many read-only
# processes per file, never both. In versioned mode every web process opens the serving version
# read-only, and a write (hand-off install, boot restore) copies it to the next version, writes
# there, checkpoints, and moves the pointer `<DB_PATH>.current`. The logical DB_PATH never changes,
# so every path-equality guard keeps holding. Readers follow the pointer on their next read.
VERSIONS_ENV = "AIOS_MIRROR_VERSIONS"
#: How often a reader stats the pointer. A stat is microseconds, the throttle keeps hot loops flat.
VERSION_CHECK_SECONDS = 0.25
#: Versions kept on disk: the serving one and the one before it.
VERSIONS_KEPT = 2
_VERSION = {"stamp": None, "file": None, "version": None, "checked": 0.0}
_VERSION_WRITE_LOCK = _thr.RLock()
#: Instances dropped by a version switch: (version, connection). Their open cursors finish on the
#: old file. `_reap_retired` closes each once its cursors are done, or past the drain bound.
_RETIRED = deque()
_RETIRED_AT = {} # id(retired instance) -> when it was retired
#: The last version verify: {version, ok, problems, at}. A refused version never serves.
VERSION_VERIFY = {"last": None}
class MirrorVersionRefused(RuntimeError):
"""A new mirror version failed its verify, so the previous version keeps serving."""
def versioned_mode():
"""True for a web process launched with AIOS_MIRROR_VERSIONS=1 (several uvicorn workers).
Web only: its sole mirror writes are the Odoo boot restore and hand-off installs, both versioned.
"""
# A literal name, so the portability gate resolves the read (VERSIONS_ENV names the same var).
return os.environ.get("AIOS_MIRROR_VERSIONS") == "1" and os.environ.get("AIOS_PROCESS_ROLE") == "web"
def _pointer_path(path=None):
base = Path(path or DB_PATH)
return base.with_name(base.name + ".current")
def _version_file(version, path=None):
base = Path(path or DB_PATH)
return base.with_name(f"{base.stem}.v{int(version)}{base.suffix}")
def _pointer_stamp(path=None):
try:
st = os.stat(_pointer_path(path))
except OSError:
return None
return (st.st_ino, st.st_mtime_ns, st.st_size)
def serving_file(path=None):
"""(physical file, version) the pointer names. The logical path and 0 when no version exists."""
base = Path(path or DB_PATH)
try:
with open(_pointer_path(base), encoding="utf-8") as handle:
pointer = json.load(handle)
candidate = base.with_name(str(pointer["file"]))
if candidate.name == Path(str(pointer["file"])).name and candidate.exists():
return candidate, int(pointer.get("version") or 0)
except (OSError, ValueError, KeyError, TypeError):
pass
return base, 0
def serving_version():
"""{version, file} this process reads, for readiness and the primary's state file."""
return {"version": _VERSION["version"], "file": _VERSION["file"]}
def _follow_version(force=False):
"""Drop the instance when the pointer moved, so the next read opens the new version."""
now = time.monotonic()
if not force and now - _VERSION["checked"] < VERSION_CHECK_SECONDS:
return False
_VERSION["checked"] = now
if _RETIRED:
_reap_retired()
stamp = _pointer_stamp()
if stamp == _VERSION["stamp"] or _INSTANCE["con"] is None:
return False
with _INSTANCE_LOCK:
old = _INSTANCE["con"]
if old is None or stamp == _VERSION["stamp"]:
return False
# Dropped, not closed: an in-flight cursor finishes on the old version.
version = _VERSION["version"]
owner = _OWNERS.get(id(old))
if owner is not None:
owner.live = False # an idle cursor rebinds to the new version on its next statement
_RETIRED.append((version, old))
_INSTANCE["con"] = None
_VERSION["stamp"] = stamp
_RO_TLS.__dict__.pop("cur", None)
_READY["ok"] = False
_GENERATION[0] += 1 # every thread's cached cursor reopens on the new version
_invalidate_counts()
_RETIRED_AT[id(old)] = time.monotonic()
_reap_retired()
return True
def _reap_retired():
"""Close each retired instance whose cursors finished, or that passed the drain bound.
Readers call this through `_follow_version`, so no thread of its own is needed.
"""
now = time.monotonic()
closed = False
for version, con in list(_RETIRED):
if open_reads(con) and now - _RETIRED_AT.get(id(con), now) < _drain_seconds():
continue
with _INSTANCE_LOCK:
try:
_RETIRED.remove((version, con))
except ValueError:
continue
_RETIRED_AT.pop(id(con), None)
_close_instance(con)
closed = True
if closed:
try:
_publish_serving()
_sweep_versions()
except OSError as exc:
print(f"[datastore] mirror version sweep failed ({type(exc).__name__})", flush=True)
def _serving_dir(path=None):
base = Path(path or DB_PATH)
return base.with_name(base.name + ".serving")
def _held_versions():
held = {version for version, _con in list(_RETIRED) if version is not None}
if _INSTANCE.get("con") is not None and _VERSION["version"] is not None:
held.add(int(_VERSION["version"]))
return sorted(held)
def _publish_serving(path=None):
"""Write this worker's open versions to `<DB_PATH>.serving/<pid>.json` for the sweep."""
if not versioned_mode():
return
folder = _serving_dir(path)
folder.mkdir(parents=True, exist_ok=True)
target = folder / f"{os.getpid()}.json"
tmp = folder / f"{os.getpid()}.json.tmp"
with open(tmp, "w", encoding="utf-8", newline="") as handle:
json.dump({"pid": os.getpid(), "versions": _held_versions(), "at": time.time()}, handle)
os.replace(tmp, target)
def _pid_alive(pid):
if pid == os.getpid():
return True
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
except (PermissionError, OSError):
return True
return True
def workers_holding(path=None):
"""{version: [pid, ...]} every live worker still reads. Dead workers' markers are removed."""
held = {}
folder = _serving_dir(path)
if not folder.is_dir():
return held
for marker in folder.glob("*.json"):
try:
data = json.loads(marker.read_text(encoding="utf-8"))
pid = int(data["pid"])
except (OSError, ValueError, KeyError, TypeError):
continue
if not _pid_alive(pid):
marker.unlink(missing_ok=True)
continue
for version in data.get("versions") or ():
held.setdefault(int(version), []).append(pid)
return held
def _sweep_versions(path=None):
"""Unlink versions older than the kept window that no live worker still reads."""
_physical, serving = serving_file(path)
held = workers_holding(path)
removed = []
for old, files in _existing_versions(path).items():
if old <= serving - VERSIONS_KEPT and old not in held:
for item in files:
item.unlink(missing_ok=True)
removed.append(old)
return sorted(removed)
@contextmanager
def _file_lock(path):
try:
import fcntl
except ImportError: # not Linux: one process only
yield
return
path.parent.mkdir(parents=True, exist_ok=True)
fd = os.open(str(path), os.O_RDWR | os.O_CREAT, 0o600)
try:
fcntl.flock(fd, fcntl.LOCK_EX)
yield
finally:
try:
fcntl.flock(fd, fcntl.LOCK_UN)
finally:
os.close(fd)
def _existing_versions(path=None):
base = Path(path or DB_PATH)
rx = re.compile(re.escape(base.stem) + r"\.v(\d+)" + re.escape(base.suffix) + r"(\.wal)?$")
found = {}
if base.parent.is_dir():
for item in base.parent.iterdir():
match = rx.fullmatch(item.name)
if match:
found.setdefault(int(match.group(1)), []).append(item)
return found
def _publish_version(version, new, path=None):
base = Path(path or DB_PATH)
pointer = _pointer_path(base)
tmp = pointer.with_name(pointer.name + f".{os.getpid()}.tmp")
with open(tmp, "w", encoding="utf-8", newline="") as handle:
json.dump({"version": int(version), "file": new.name, "at": time.time(), "pid": os.getpid()}, handle)
os.replace(tmp, pointer)
if not base.exists():
# Readiness and seed checks ask for the logical file. It never receives a write.
try:
os.link(new, base)
except OSError:
import shutil
shutil.copyfile(new, base)
if base.resolve() == DB_PATH.resolve():
_publish_serving()
# An old version goes only once no live worker reads it (each worker writes its marker).
_sweep_versions(base)
def _version_facts(con):
"""{tables, phases, publications} of one mirror file, the baseline a new version must keep."""
tables = {row[0] for row in con.execute(
"SELECT table_name FROM information_schema.tables WHERE table_schema = 'main'").fetchall()}
phases = (dict(con.execute("SELECT entity, phase FROM _sync_state").fetchall())
if "_sync_state" in tables else {})
publications = ({row[0] for row in con.execute("SELECT source FROM _mirror_publications").fetchall()}
if "_mirror_publications" in tables else set())
return {"tables": tables, "phases": phases, "publications": publications}
def verify_version(path, baseline=None, expect_tables=()):
"""Problems that stop a new version serving. An empty list means it may serve.
It must open read-only, read `_sync_state`, keep each required entity that was live live and
readable, keep each publication, and hold every table in `expect_tables`.
"""
try:
con = duckdb.connect(str(path), read_only=True)
except Exception as exc: # noqa: BLE001
return [f"does not open read-only ({type(exc).__name__}: {str(exc)[:120]})"]
problems = []
try:
try:
facts = _version_facts(con)
except Exception as exc: # noqa: BLE001
return [f"metadata unreadable ({type(exc).__name__}: {str(exc)[:120]})"]
if "_sync_state" not in facts["tables"]:
problems.append("_sync_state is missing")
base = baseline or {}
needed = set(expect_tables)
for key, spec in ENTITIES.items():
if spec.get("optional") or (base.get("phases") or {}).get(key) != "live":
continue
if facts["phases"].get(key) != "live":
problems.append(f"{key} phase went from live to {facts['phases'].get(key)}")
if key in (base.get("tables") or ()):
needed.add(key)
for table in sorted(needed):
if table not in facts["tables"]:
problems.append(f"table {table} is missing")
continue
try:
con.execute(f'SELECT 1 FROM "{table}" LIMIT 1').fetchall()
except Exception as exc: # noqa: BLE001
problems.append(f"table {table} is unreadable ({type(exc).__name__})")
for source in sorted(base.get("publications") or ()):
if source not in facts["publications"]:
problems.append(f"publication {source} is missing")
finally:
con.close()
return problems
@contextmanager
def versioned_write(tenant_key=None):
"""Write the next mirror version, then publish it. Yields {publish, version, path} or None.
No tenant key keeps the legacy pin. A named tenant always uses its configured path.
Outside versioned mode it yields None and the caller writes in place. Inside, every
`connect()` and `_mirror_connection` on this thread lands in the new version. Set
`handle["publish"] = False` to discard an unchanged version. Before the pointer moves the new
version passes `verify_version` (add tables to `handle["expect_tables"]`); a refused version
is discarded, the old one keeps serving, and `MirrorVersionRefused` is raised.
"""
if getattr(_SYNC_TLS, "candidate", None) is not None:
raise RuntimeError("A source publication is already active on this thread")
key = str(tenant_key).strip().lower() if tenant_key is not None else None
base = tenant_handle(key).path if key is not None else DB_PATH.resolve()
if not versioned_mode():
if key is None:
yield None
return
con = connect(key)
try:
_SYNC_TLS.candidate = {"tenant": key, "connector": "mirror-version", "path": base, "con": con}
yield None
finally:
_SYNC_TLS.candidate = None
con.close()
return
import shutil
published = refused = None
pointer = _pointer_path(base)
with _VERSION_WRITE_LOCK, _file_lock(pointer.with_name(pointer.name + ".lock")):
source, version = serving_file(base)
number = max([version, *_existing_versions(base).keys()]) + 1
new = _version_file(number, base)
for stale in (new, new.with_name(new.name + ".wal")):
stale.unlink(missing_ok=True)
if source.exists():
tmp = new.with_name(new.name + f".{os.getpid()}.tmp")
shutil.copyfile(source, tmp)
os.replace(tmp, new)
wal = source.with_name(source.name + ".wal")
if wal.exists():
shutil.copyfile(wal, new.with_name(new.name + ".wal"))
con = apply_limits(duckdb.connect(str(new)))
handle = {"publish": True, "version": number, "path": new, "expect_tables": []}
ok = False
baseline = None
try:
_ensure_sync_state(con)
baseline = _version_facts(con)
_SYNC_TLS.candidate = {"tenant": key, "connector": "mirror-version",
"path": base.resolve(), "con": con}
yield handle
ok = True
finally:
_SYNC_TLS.candidate = None
try:
if ok and handle["publish"]:
con.execute("CHECKPOINT")
finally:
con.close()
if ok and handle["publish"]:
refused = verify_version(new, baseline, handle.get("expect_tables") or ())
VERSION_VERIFY["last"] = {"version": number, "ok": not refused,
"problems": list(refused), "at": time.time()}
if ok and handle["publish"] and not refused:
_publish_version(number, new, base)
published = number
else:
for item in (new, new.with_name(new.name + ".wal")):
item.unlink(missing_ok=True)
if refused:
message = f"mirror version {number} refused, version {version} keeps serving: " + "; ".join(refused)
print(f"[datastore] {message}", flush=True)
raise MirrorVersionRefused(message)
if published is not None:
note_mirror_install(base)
if base.resolve() == DB_PATH.resolve():
_follow_version(force=True)
def _temporary_write(cur, statements):
"""Prove a single write affects only a local relation; unfamiliar cases stay source writes."""
if len(statements) != 1:
return False # A batch may create/drop a shadow before a later persistent write.
try:
sql = statements[0].query
encoded = sql.encode("utf-8")
tokens = duckdb.tokenize(sql)
words = [encoded[start:tokens[i + 1][0] if i + 1 < len(tokens) else len(encoded)]
.decode("utf-8").strip() for i, (start, _kind) in enumerate(tokens)]
upper = [word.upper() for word in words]
create = 3 if upper[:3] == ["CREATE", "OR", "REPLACE"] else 1
if (upper[:1] == ["CREATE"] and len(upper) > create + 1
and upper[create] in {"TEMP", "TEMPORARY"}
and upper[create + 1] in {"TABLE", "VIEW"}):
return True
kind = "table"
if upper[:2] in (["INSERT", "INTO"], ["DELETE", "FROM"]):
index = 2
elif upper[:1] == ["UPDATE"]:
index = 1
elif upper[:2] in (["DROP", "TABLE"], ["DROP", "VIEW"]):
kind, index = upper[1].lower(), 2
if upper[index:index + 2] == ["IF", "EXISTS"]:
index += 2
else:
return False
names = []
fold = str.maketrans("ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")
while index < len(words):
word = words[index]
if re.fullmatch(r'"(?:[^"]|"")+"', word):
names.append(word[1:-1].replace('""', '"').translate(fold))
elif _IDENT_OK.fullmatch(word):
names.append(word.translate(fold))
else:
return False
index += 1
if index >= len(words) or words[index] != ".":
break
index += 1
if not names or len(names) > 3:
return False
if upper[0] == "DROP" and words[index:] not in ([], [";"]):
return False # In particular, CASCADE may affect persistent dependants.
if len(names) == 3:
if names[:2] != ["temp", "main"]:
return False
elif len(names) == 2 and names[0] == "temp":
if cur.execute("SELECT EXISTS (SELECT 1 FROM duckdb_schemas() "
"WHERE lower(schema_name)='temp' AND NOT internal)").fetchone()[0]:
return False
else:
# Prove the default DuckDB search order, including TEMP shadows of main tables.
# Custom search paths or ambiguous two-part catalog names remain conservative.
if len(names) == 2 and names[0] != "main":
return False
schema, search, ambiguous = cur.execute(
"SELECT current_schema(), current_setting('search_path'), "
"EXISTS (SELECT 1 FROM duckdb_databases() WHERE lower(database_name)='main')"
).fetchone()
if schema != "main" or search or (len(names) == 2 and ambiguous):
return False
catalogue, column = (("duckdb_views", "view_name") if kind == "view"
else ("duckdb_tables", "table_name"))
return cur.execute(
f"SELECT count(*)=1 FROM {catalogue}() WHERE temporary AND schema_name='main' "
f"AND translate({column}, 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz')=?",
[names[-1]]).fetchone()[0]
except Exception as exc:
_request_timing.raise_if_resource_error(exc)
return False
class _TimedCursor:
"""Keep cursor chaining and thread-local identity while timing execute and result conversion."""
_FETCH = {"fetchone", "fetchall", "fetchmany", "fetchdf", "fetch_df", "df", "fetchnumpy",
"fetch_arrow_table", "fetch_record_batch", "arrow", "pl", "fetch_df_chunk"}
#: A fetch that returns the whole remaining result: the read is over after it.
_TERMINAL = {"fetchall", "fetchdf", "fetch_df", "df", "fetchnumpy", "fetch_arrow_table", "arrow", "pl"}
def __init__(self, cursor, owner=None, factory=None, mirror_handle=None):
self._cursor = cursor
self._statement = ""
self._statement_ms = 0.0
self._request = None
self._in_transaction = False
# Drain state (F47): calls in progress, an unfetched result, and who read it last.
self._owner = owner
self._factory = factory
self._mirror_handle = mirror_handle
self._mirror_path = mirror_handle.path if mirror_handle is not None else DB_PATH.resolve()
self._closed = False
self._epoch = _STORE_EPOCH[0]
self._calls = 0
self._pending = False
self._last = 0.0
self._thread = None
self._read_request = None
if owner is not None:
with _READS:
owner.cursors.add(self)
def _busy(self, now, ignore_thread=None):
"""True while a call runs, or a result waits to be fetched by a live request."""
if self._calls:
return True
if not self._pending or (ignore_thread is not None and self._thread == ignore_thread):
return False
if getattr(self._read_request, "closed", False):
return False # the request that read it has finished
return now - self._last < READ_IDLE_SECONDS
def _rebind(self):
"""Move an idle cursor off a swapped-out instance onto the serving one (same store only)."""
if self._mirror_handle is None and self._epoch != _STORE_EPOCH[0]:
raise RuntimeError("The data store moved to another file. Retry the request.")
if self._in_transaction:
raise RuntimeError("The data store was swapped during a transaction. Retry the request.")
raw, owner = self._factory()
with _READS:
if self._owner is not None:
self._owner.cursors.discard(self)
self._cursor, self._owner = raw, owner
if owner is not None:
owner.cursors.add(self)
def _settle(self, name, result, failed):
with _READS:
self._calls -= 1
self._last = time.monotonic()
if failed or name in self._TERMINAL:
self._pending = False
elif name == "fetchone":
self._pending = result is not None
elif name in {"fetchmany", "fetch_df_chunk"}:
try:
self._pending = len(result) > 0
except TypeError:
self._pending = False
_READS.notify_all()
def _call(self, name, *args, **kwargs):
acc = _request_timing.CURRENT.get()
execute = name in {"execute", "executemany"}
if execute and self._mirror_handle is not None:
self._mirror_handle.follow_version()
owner = self._owner
if execute and owner is not None and not owner.live and self._factory is not None:
self._rebind()
with _READS:
self._calls += 1
self._thread = _thr.get_ident()
self._read_request = acc
result, failed = None, True
try:
result = self._run(name, acc, execute, args, kwargs)
failed = False
return result
finally:
self._settle(name, result, failed)
def _run(self, name, acc, execute, args, kwargs):
writes = False
source_write = True
bound_writer = None
transaction = None
if execute:
statement = args[0] if args else kwargs.get("query", "<statement>")
self._statement = _request_timing.sql_text(statement) if acc is not None else ""
self._statement_ms = 0.0
self._request = acc
try:
statements = self._cursor.extract_statements(statement)
writes = any(s.type.name not in {"SELECT", "TRANSACTION"} for s in statements)
if writes:
source_write = not _temporary_write(self._cursor, statements)
for parsed in statements:
if parsed.type.name == "TRANSACTION":
transaction = not re.match(r"\s*(COMMIT|ROLLBACK|END)\b", parsed.query, re.I)
except Exception as exc:
_request_timing.raise_if_resource_error(exc)
# Unclassified statements cannot establish a reusable read snapshot.
writes, source_write, self._in_transaction = True, True, True
if transaction is not None:
# A failed statement batch may leave its BEGIN active.
self._in_transaction = True
if writes:
with _COUNT_CACHE_LOCK:
_ACTIVE_WRITES[0] += 1
bound_writer = self._mirror_handle or _HANDLES_LIVE.get(str(self._mirror_path))
if bound_writer is not None:
bound_writer.active_writes += 1
_invalidate_counts(self._mirror_path, source_write=source_write)
started = time.perf_counter()
try:
with _request_timing.thread_scope(), _request_timing.duckdb_operation(self._cursor):
result = getattr(self._cursor, name)(*args, **kwargs)
if transaction is not None:
self._in_transaction = transaction
if execute:
with _READS:
# A row result now waits to be fetched. A write or transaction leaves none.
self._pending = not writes and transaction is None
return self if result is self._cursor else result
finally:
if writes:
with _COUNT_CACHE_LOCK:
_ACTIVE_WRITES[0] -= 1
if bound_writer is not None:
bound_writer.active_writes -= 1
_invalidate_counts(self._mirror_path, source_write=source_write)
if acc is not None and self._request is acc:
elapsed = (time.perf_counter() - started) * 1000
self._statement_ms += elapsed
_request_timing.record_duckdb(self._statement, elapsed, count=int(execute),
statement_ms=self._statement_ms)
def execute(self, *args, **kwargs):
return self._call("execute", *args, **kwargs)
def executemany(self, *args, **kwargs):
return self._call("executemany", *args, **kwargs)
def cursor(self, *args, **kwargs):
if self._mirror_handle is not None:
_assert_tenant_path(self._mirror_handle.tenant_key, self._mirror_handle.path)
child = _TimedCursor(self._cursor.cursor(*args, **kwargs), self._owner, self._factory,
self._mirror_handle)
# A DuckDB child cursor dies when its parent closes, and __del__ closes a collected parent.
child._parent = self
return child
def close(self):
if self._closed:
return
self._closed = True
try:
return self._cursor.close()
finally:
self._release()
def _release(self):
"""Unregister a reader only after its underlying close or context exit finishes."""
with _READS:
self._pending = False
if self._owner is not None:
self._owner.cursors.discard(self)
_READS.notify_all()
if self._mirror_handle is not None:
with _HANDLES_LOCK:
self._mirror_handle._reap()
_trim_handles()
def __getattr__(self, name):
if name in self._FETCH:
return lambda *args, **kwargs: self._call(name, *args, **kwargs)
return getattr(self._cursor, name)
def __enter__(self):
self._cursor.__enter__()
return self
def __exit__(self, *args):
if self._closed:
return
self._closed = True
try:
exit_cursor = getattr(self._cursor, "__exit__", None)
if exit_cursor is not None:
return exit_cursor(*args)
self._cursor.close()
finally:
self._release()
def __iter__(self):
return self
def __next__(self):
row = self.fetchone()
if row is None:
raise StopIteration
return row
def __del__(self):
if getattr(self, "_mirror_handle", None) is not None:
try:
if self._closed:
return
self._closed = True
with _READS:
self._pending = False
if self._owner is not None:
self._owner.cursors.discard(self)
_READS.notify_all()
self._cursor.close()
# Collection can run while a weak-set reader holds _READS.
if _HANDLES_LOCK.acquire(blocking=False):
try:
self._mirror_handle._reap()
_trim_handles()
finally:
_HANDLES_LOCK.release()
except Exception:
pass
import weakref as _weakref
_HANDLES_LOCK = _thr.RLock()
_HANDLES = OrderedDict()
_HANDLES_LIVE = _weakref.WeakValueDictionary()
MAX_IDLE_HANDLES = max(0, int(os.environ.get("AIOS_MAX_IDLE_MIRRORS") or 8))
class TenantMirrorHandle:
"""Own one tenant's mirror connections without changing the legacy writer path."""
def __init__(self, path, tenant_key):
self.path, self.tenant_key = Path(path).resolve(), tenant_key
self.writes, self.active_writes, self.installs = 0, 0, 0
self._con, self._physical, self._stamp = None, None, None
self._retired = []
def _held(self, con):
owner = _OWNERS.get(id(con))
with _READS:
return owner is not None and bool(list(owner.cursors))
def held(self):
with _READS:
return any(getattr(cur, "_mirror_handle", None) is self
for owner in list(_OWNERS.values()) for cur in list(owner.cursors))
def _publish(self):
if not versioned_mode() or self.path == DB_PATH.resolve():
return
versions = {version for version, _con in self._retired}
if self._con is not None:
versions.add(self._physical[1])
folder = _serving_dir(self.path)
folder.mkdir(parents=True, exist_ok=True)
target = folder / f"{os.getpid()}.json"
tmp = target.with_suffix(".json.tmp")
tmp.write_text(json.dumps({"pid": os.getpid(), "versions": sorted(versions),
"at": time.time()}), encoding="utf-8")
os.replace(tmp, target)
def _reap(self):
for version, con in list(self._retired):
if not self._held(con):
self._retired.remove((version, con))
_close_instance(con)
self._publish()
def _close_idle(self):
if self.held():
return False
self._reap()
if self._con is not None:
_close_instance(self._con)
self._con = None
self._publish()
return True
def follow_version(self):
if not versioned_mode():
return
if self.path == DB_PATH.resolve():
with _HANDLES_LOCK:
_follow_version()
elif self._con is not None and _pointer_stamp(self.path) != self._stamp:
raw, _owner = self._open_cursor()
raw.close()
def _open_cursor(self):
_assert_tenant_path(self.tenant_key, self.path)
with _HANDLES_LOCK:
return self._open_cursor_locked()
def _open_cursor_locked(self):
physical = serving_file(self.path) if versioned_mode() else (self.path, 0)
if not physical[0].is_file():
raise RuntimeError("The tenant mirror has not been attached yet.")
# Reuse the writer's instance when this is the explicitly pinned legacy store.
if self.path == DB_PATH.resolve():
return _open_cursor()
stamp = _pointer_stamp(self.path) if versioned_mode() else None
if self._con is not None and (physical != self._physical or stamp != self._stamp):
_OWNERS[id(self._con)].live = False
self._retired.append((self._physical[1], self._con))
self._con = None
self._reap()
if self._con is None:
self._con = apply_limits(duckdb.connect(str(physical[0]), read_only=versioned_mode()))
self._physical, self._stamp = physical, stamp
_own(self._con)
self._publish()
return self._con.cursor(), _OWNERS[id(self._con)]
def ro_cursor(self):
_assert_tenant_path(self.tenant_key, self.path)
with _HANDLES_LOCK:
cur = _TimedCursor(*self._open_cursor_locked(), factory=self._open_cursor, mirror_handle=self)
_HANDLES[str(self.path)] = self
_HANDLES.move_to_end(str(self.path))
_trim_handles()
return cur
def ro_con(self):
if not self.ready():
raise RuntimeError("The tenant data store is completing its first sync. Try again shortly.")
return self.ro_cursor()
def ready(self):
return ready(handle=self)
def entity_live(self, key):
return entity_live(key, handle=self)
def _trim_handles():
"""Bound idle handles while every borrowed cursor keeps its own handle open."""
idle = [key for key, handle in _HANDLES.items() if not handle.held()]
for key in idle[:max(0, len(idle) - MAX_IDLE_HANDLES)]:
if _HANDLES[key]._close_idle():
_HANDLES.pop(key)
def _assert_tenant_path(key, target):
"""Require forward ownership evidence even for configured canonical paths."""
canonical = path_for(key).resolve()
safe = "".join(c if c.isalnum() or c in "-_" else "_" for c in key)[:60]
from harness.runtime import DatastoreMismatch, TenantRuntime
owner, registry_ok = TenantRuntime._store_owner(target)
if not registry_ok or owner != key:
# Unregistered injective canonical keys still need proof that no registered tenant shares their path.
if not registry_ok or owner or target != canonical or safe != key:
raise DatastoreMismatch("The mirror path's tenant ownership could not be verified.")
def tenant_handle(tenant_key, path=None):
"""Resolve a tenant-bound handle by canonical path and refuse path aliases across tenants."""
key = str(tenant_key or "").strip().lower()
target = Path(path or path_for(key)).resolve()
if not key:
raise ValueError("A tenant mirror requires a tenant key")
_assert_tenant_path(key, target)
with _HANDLES_LOCK:
handle = _HANDLES_LIVE.get(str(target))
if handle is not None and handle.tenant_key != key:
from harness.runtime import DatastoreMismatch
raise DatastoreMismatch("The mirror handle belongs to another tenant.")
if handle is None:
handle = TenantMirrorHandle(target, key)
_HANDLES_LIVE[str(target)] = handle
_HANDLES[str(target)] = handle
_HANDLES.move_to_end(str(target))
_trim_handles()
return handle
def ro_cursor():
"""A read cursor on the shared instance β the contention-free way to read the store."""
return _TimedCursor(*_open_cursor(), factory=_open_cursor)
def ready(*, handle=None):
"""True once EVERY entity has completed its backfill (phase 'live'). Gate all store READS on
this: a mid-backfill store would return PARTIAL totals silently β worse than an error. Positive
results cache for the process (backfill never regresses). Reads via a shared-instance cursor,
so it is never mis-reported as 'not ready' just because a background sync pass is mid-flight."""
path = handle.path if handle is not None else DB_PATH
if _odoo_restore_refused(path):
return False
if handle is None and _READY["ok"]:
return True
if not (serving_file(path)[0] if versioned_mode() else path).exists():
return False
try:
cur = handle.ro_cursor() if handle is not None else ro_cursor()
try:
phases = {r[0]: r[1] for r in
cur.execute("SELECT entity, phase FROM _sync_state").fetchall()}
finally:
cur.close()
except Exception as exc:
_request_timing.raise_if_resource_error(exc)
return False
# ββ OPTIONAL ENTITIES DO NOT GATE READINESS, AND WITHOUT THIS CLAUSE ADDING ONE IS AN
# OUTAGE. `ready()` gates EVERY store read β the Analyst, every measure column, every
# read-through grid β so a new entity in `ENTITIES` makes a mirror that has never synced it
# report NOT READY and refuses all of them with "the data cache is still warming up". A Space
# hydrates from a SNAPSHOT taken before the entity existed, so this would fire on every
# deploy of a wave that adds one: a whole-product outage from a schema line, with the app
# RUNNING and every gate green. MEASURED on this box the moment `stock_move` was declared
# (W37-T13): `ready()` flipped to False against an otherwise complete mirror.
# β THE GUARANTEE IS NOT WEAKENED, IT IS SCOPED. `ready()` still means "every entity the
# EXISTING surfaces read has finished its backfill". An optional entity's own consumers must
# check ITS phase before promising a column from it β which is what
# `semantic._source_ready` does, refusing the offer with a cause rather than serving blanks.
_required = [k for k, v in ENTITIES.items() if not v.get("optional")]
if all(phases.get(k) == "live" for k in _required):
if handle is None:
_READY["ok"] = True
return True
return False
def _required_columns_ready(cur, key, spec):
"""Check the declared column migration for entities that require complete projections."""
if not spec.get("backfill_required"):
return True
return _columns_backfilled(cur, key, spec)
def _columns_backfilled(cur, key, spec):
"""Require completion of each declared field, including completed legacy cohorts."""
states = _column_backfill_states(cur, key)
return all(states.get(field, (None,))[0] == "done"
for field in spec.get("backfill_fields", ()) if field != "write_date")
def _column_backfill_states(cur, key):
"""Expand legacy cohorts while giving each field's own checkpoint precedence."""
prefix = _connection_state_key(f"{key}.cols.")
prior = cur.execute("SELECT entity, phase, last_id, rows FROM _sync_state "
"WHERE starts_with(entity, ?)", [prefix]).fetchall()
states, individual = {}, {}
for name, phase, last, count in prior:
fields = name[len(prefix):].split(",")
point = (phase, last or 0, count or 0)
if len(fields) == 1:
individual[fields[0]] = point
for field in fields:
previous = states.get(field)
if previous is None or (phase == "done", point[1:]) > (previous[0] == "done", previous[1:]):
states[field] = point
states.update(individual)
return states
ODOO_FILL_STATUS_KEY = "odoo_column_fill_v1"
def column_fill_progress(con, key, spec):
"""Read the slowest unfinished field's durable progress without reading the source."""
states = _column_backfill_states(con, key)
pending = {field: states.get(field, ("running", 0, 0))
for field in spec.get("backfill_fields", ())
if field != "write_date" and states.get(field, (None,))[0] != "done"}
for field in spec.get("m2m") or ():
point = con.execute("SELECT phase, last_id, rows FROM _sync_state WHERE entity=?",
[_connection_state_key(f"{key}.m2m.{field}")]).fetchone()
if not point or point[0] != "done":
pending[field] = (point[0], point[1] or 0, point[2] or 0) if point else ("running", 0, 0)
if not pending:
return None
total = con.execute(f"SELECT count(*) FROM {key} WHERE {_connection_where()}").fetchone()[0]
last = min(point[1] for point in pending.values())
done = min(point[2] for point in pending.values())
return {"entity": key, "state": "filling", "fields": sorted(pending),
"rowsDone": min(done, total), "rowsTotal": total, "lastId": last}
def _record_column_fill(rt, con, key, spec):
"""Save each connection's field progress for cards in other processes."""
progress = column_fill_progress(con, key, spec)
if progress is None and (spec.get("m2m") or any(
field != "write_date" for field in spec.get("backfill_fields", ()))):
total = con.execute(f"SELECT count(*) FROM {key} WHERE {_connection_where()}").fetchone()[0]
progress = {"entity": key, "state": "done", "rowsDone": total, "rowsTotal": total}
binding = getattr(_SYNC_TLS, "connection", None)
connection = binding["id"] if binding else "default"
def update(raw):
raw = dict(raw) if isinstance(raw, dict) else {}
entities = dict(raw.get("entities") or {})
connections = dict(entities.get(key) or {})
if progress is None:
connections.pop(connection, None)
else:
connections[connection] = progress
if connections:
entities[key] = connections
else:
entities.pop(key, None)
raw["entities"] = entities
return raw
rt.update(ODOO_FILL_STATUS_KEY, update, flush="sync")
def _checkpoint_column_fields(con, key, fields, phase, last, count):
"""Save field progress and settle obsolete cohort markers once their fields finish."""
stamp = time.strftime("%Y-%m-%d %H:%M:%S")
if fields:
con.executemany("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
[[_connection_state_key(f"{key}.cols.{field}"), phase, last, None, count, stamp] for field in fields])
if phase == "done":
states = _column_backfill_states(con, key)
prefix = _connection_state_key(f"{key}.cols.")
prior = con.execute("SELECT entity FROM _sync_state WHERE starts_with(entity, ?) "
"AND phase != 'done'", [prefix]).fetchall()
for (name,) in prior:
cohort = name[len(prefix):].split(",")
if all(states[field][0] == "done" for field in cohort):
con.execute("UPDATE _sync_state SET phase='done', updated_at=? WHERE entity=?",
[stamp, name])
def entity_live(key, *, handle=None):
"""Has THIS entity finished its backfill? The per-entity half of `ready()`.
Exists because `ready()` is deliberately blind to optional entities (see above), so a caller
that depends on one has to ask about it specifically instead of inferring it from a global
flag that was never about them.
"""
path = handle.path if handle is not None else DB_PATH
if _odoo_restore_refused(path) or not (serving_file(path)[0] if versioned_mode() else path).exists():
return False
try:
cur = handle.ro_cursor() if handle is not None else ro_cursor()
try:
row = cur.execute("SELECT phase FROM _sync_state WHERE entity = ?", [key]).fetchone()
columns_ready = _required_columns_ready(cur, key, ENTITIES.get(key) or {})
finally:
cur.close()
except Exception as exc: # noqa: BLE001
_request_timing.raise_if_resource_error(exc)
return False
return bool(row) and row[0] == "live" and columns_ready
def ro_con():
"""Per-thread cached read CURSOR on the shared instance for page/module store reads (the
open cost dominates repeated small queries β the Expenses retrofit measured 7.7sβ1.0s). No
lock contention with the sync writer; a cursor that has gone bad is reopened once. REFUSES
until ready() β callers either fall back to live reads (modules) or relay a readable message
(Analyst).
β THE GENERATION CHECK IS A CORRECTNESS GUARD, NOT AN OPTIMISATION (EXIT-4a). The liveness
probe below asks "does this cursor still work?", which is a different question from "is this
cursor still pointing at the file we are supposed to be reading?". After `use_path()` switches
tenants, a cursor cached in ANOTHER thread's `threading.local` can still answer `SELECT 1` β
and would then serve that thread real, plausible rows from the PREVIOUS tenant's file, with no
exception anywhere. `use_path` bumps the generation; a cursor opened under an older one is
discarded here, which is the only place every thread is guaranteed to pass through."""
if _odoo_restore_refused():
raise RuntimeError("The saved Odoo mirror could not be restored. Repair store access and restart.")
if versioned_mode():
_follow_version()
cur = getattr(_RO_TLS, "cur", None)
if cur is not None and getattr(_RO_TLS, "gen", None) != _GENERATION[0]:
try:
cur.close()
except Exception:
pass
cur, _RO_TLS.cur = None, None # the store moved underneath this thread
if cur is not None:
try:
cur.execute("SELECT 1").fetchall() # cheap liveness probe, fetched so it is not a pending read
return cur
except Exception as exc:
_request_timing.raise_if_resource_error(exc)
_RO_TLS.cur = None # stale β reopen below
if not ready():
raise RuntimeError("the tenant data store is completing its FIRST sync on this "
"deployment; dashboards work meanwhile (live reads); retry "
"store queries in a few minutes")
cur = ro_cursor()
_RO_TLS.cur = cur
_RO_TLS.gen = _GENERATION[0] # stamp WHICH file this cursor belongs to
return cur
# βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
# β WAVE 30 / OWNER RULING R6 + R7 β THE WINDOW. Read this before touching `window()`.
#
# R6, verbatim: *"there is no cap in how many data from the API source (as long as its from a
# connected source like Odoo) that can be pulled into the app."* R7 says how: a connected grid
# reads THROUGH this mirror instead of copying rows into the one `user_tables` document, which
# `MAX_ROWS = 60_000` bounds. The mirror is already uncapped β 971,034 GL lines live in this file
# today β so "no cap" is not a bigger number, it is a different mechanism: serve a SLICE and count
# the whole.
#
# ββ `total` IS A `SELECT count(*)` OVER THE SAME PREDICATE, NEVER `len(rows)`. A window whose
# count is its own length is a fabricated aggregate that reads as authoritative β the class this
# repo has paid for twice ([[no-unverifiable-aggregates]], [[one-question-two-normalizers]]: a
# display predicate and a fold predicate answering one question differently). The two statements
# below are built from ONE `where` + ONE `params` tuple for exactly that reason; they cannot drift
# apart without deleting a line.
#
# β AND THE PREDICATE PUSHES DOWN. `where` is compiled by `harness/filter_sql.py` β the same
# compiler the client's filter engine is held in step with β so a filter matching rows outside the
# loaded window still COUNTS them. A caller that filters the returned list instead has silently
# asked "how many of the 200 rows in memory match" about a 971,034-row table.
#
# β `order_by` IS REQUIRED IN PRACTICE AND DEFAULTED HERE. Two pages of an UNORDERED window are
# not guaranteed to partition the table: DuckDB may legally return a row on page 1 and again on
# page 2, and the user sees a duplicate with no error anywhere. The default is the physical
# rowid-ish `1` only when a caller genuinely has no key; every real caller passes one.
#
# β TRUST BOUNDARY. `table`, `select`, `where` and `order_by` are SQL we author (a spec row, or
# `filter_sql`'s output over a whitelisted column map). Every VALUE is bound through `params`.
# `table` is additionally shape-checked below β not because a caller is hostile, but because a
# typo'd identifier interpolated into two statements is worth one cheap assertion.
#: The most rows ONE request may carry back. Not a cap on the data β every row is reachable by
#: paging and `total` always tells the truth about how many there are β but a bound on the memory
#: a single response can cost. R6's second sentence applies: a caller that asks for more is CLAMPED
#: and the clamp is REPORTED (`routes_odoo_tables` turns it into a `limits` entry), never silent.
WINDOW_MAX = 5_000
#: What a caller gets when it names no window at all. Matches `semantic.store_rows`' own default so
#: the two windowed readers in this codebase do not disagree about what "a page" means.
WINDOW_DEFAULT = 200
_IDENT_OK = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
#: A scalar subquery inside SQL text: `window(page_first=True)` defers it until the page is chosen.
_SUBQUERY = re.compile(r"\(\s*SELECT\b", re.IGNORECASE)
_ORDINAL_ORDER = re.compile(
r"(?:\(\s*)*[+-]?\d[\d_]*(?:\s*\))*\s*(?:ASC|DESC)?\s*(?:NULLS\s+(?:FIRST|LAST))?", re.I)
_ALL_ORDER = re.compile(r"ALL\s*(?:ASC|DESC)?\s*(?:NULLS\s+(?:FIRST|LAST))?", re.I)
def _sql_terms(sql):
"""Split trusted SQL at unquoted, top-level commas, or decline unfamiliar syntax."""
depth, quote, start, index = 0, None, 0, 0
terms = []
while index < len(sql):
char = sql[index]
if quote:
if char == "\\":
return None
if char == quote:
if index + 1 < len(sql) and sql[index + 1] == quote:
index += 2
continue
quote = None
elif char in "'\"":
quote = char
elif (not (char.isalnum() or char.isspace() or char in "_.,()+-*/%=<>!|:&~^")
or sql[index:index + 2] in ("--", "/*", "*/", "->", "=>", ":=")):
return None
elif char == "(":
depth += 1
elif char == ")":
depth -= 1
if depth < 0:
return None
elif char == "," and depth == 0:
terms.append(sql[start:index].strip())
start = index + 1
index += 1
if quote or depth:
return None
terms.append(sql[start:].strip())
return terms if all(terms) else None
def _sql_tokens(sql):
"""Use DuckDB's lexer for identifier quoting and expression comparisons."""
tokens = duckdb.tokenize(sql)
encoded = sql.encode("utf-8")
result = [(encoded[start:tokens[i + 1][0] if i + 1 < len(tokens) else len(encoded)].decode("utf-8").strip(), kind)
for i, (start, kind) in enumerate(tokens)]
if any(text.upper() in ("COLUMNS", "EXCLUDE", "REPLACE", "USING") for text, _kind in result):
raise ValueError("Unrecognized page ordering syntax")
return result
def _order_needs_projection(order, projection=""):
"""Keep positional, ALL, alias-dependent and unfamiliar orders after projection."""
terms = _sql_terms(order)
if terms is None or any(_ORDINAL_ORDER.fullmatch(t) or _ALL_ORDER.fullmatch(t) for t in terms):
return True
selected = _sql_terms(projection) if projection else []
if selected is None:
return True
try:
fold = str.maketrans("ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")
def normalized(tokens):
return [(text[1:-1].replace('""', '"').translate(fold) if text.startswith('"')
else text if text.startswith("'") else text.translate(fold), kind) for text, kind in tokens]
ordered = normalized(_sql_tokens(order))
aliases = {}
for term in selected:
tokens = _sql_tokens(term)
depth, top = 0, []
for i, (text, _kind) in enumerate(tokens):
if text == ")":
depth -= 1
if depth == 0:
top.append(i)
if text == "(":
depth += 1
if not top or any(tokens[i][0] in ("*", ":", ":=") for i in top):
return True
last = top[-1]
if len(top) >= 2 and tokens[top[-2]][0].upper() == "AS":
expression = tokens[:top[-2]]
elif (last > 0 and tokens[last][1] == duckdb.token_type.identifier
and tokens[last - 1][0] != "."):
expression = tokens[:last]
else:
continue
alias = normalized([tokens[last]])[0][0]
if tokens[last][0].startswith("'"):
alias = tokens[last][0][1:-1].replace("''", "'").translate(fold)
aliases.setdefault(alias, []).append(normalized(expression))
for alias, expressions in aliases.items():
covered = set()
# A repeated source expression is safe, including generated casts and passthrough aliases.
if all(expr == expressions[0] for expr in expressions):
expr = expressions[0]
for i in range(len(ordered) - len(expr) + 1):
if expr and ordered[i:i + len(expr)] == expr:
covered.update(range(i, i + len(expr)))
for i, token in enumerate(ordered):
if (token[0] == alias and token[1] in (duckdb.token_type.identifier, duckdb.token_type.keyword)
and i not in covered
and (i == 0 or ordered[i - 1][0] != ".")
and (i + 1 == len(ordered) or ordered[i + 1][0] not in (".", "("))):
return True
except (duckdb.Error, AttributeError, ValueError, UnicodeError):
return True
return False
def _physical_page_order(order, page_key):
"""Make physical pages total without changing ALL or unfamiliar SQL ordering semantics."""
terms = _sql_terms(order)
if terms is None or any(_ALL_ORDER.fullmatch(term) for term in terms):
return order
try:
_sql_tokens(order)
except (duckdb.Error, AttributeError, ValueError):
return order
return f"{order}, {page_key} ASC"
class CursorPagingUnsupported(ValueError):
"""The original SQL requires projection semantics that cannot move behind a key page."""
def _cursor_order(order):
"""Split trusted row ordering into expressions, directions and null positions."""
terms = _sql_terms(order)
if not terms or any(_ALL_ORDER.fullmatch(term) for term in terms):
raise CursorPagingUnsupported("cursor paging requires explicit row sort keys")
out = []
for term in terms:
match = re.fullmatch(r"(.*?)(?:\s+(ASC|DESC))?(?:\s+NULLS\s+(FIRST|LAST))?",
term.strip(), flags=re.I)
expr, direction, nulls = match.groups()
if not expr:
raise ValueError("cursor paging requires a row sort expression")
out.append((expr, (direction or "ASC").upper(), (nulls or "LAST").upper()))
return out
def _cursor_predicate(terms, values):
"""Seek strictly past a complete tuple, including nulls and the physical tiebreak."""
if len(values) != len(terms) or any(v is not None and not isinstance(v, str) for v in values):
raise ValueError("cursor sort keys do not match this window")
branches, equal, bindings = [], [], []
for (expr, direction, nulls), value in zip(terms, values):
ref = f"({expr})"
if value is None:
greater, args = (f"{ref} IS NOT NULL", []) if nulls == "FIRST" else ("FALSE", [])
else:
op = ">" if direction == "ASC" else "<"
greater, args = f"{ref} {op} cast_to_type(?, {ref})", [value]
if nulls == "LAST":
greater = f"({greater} OR {ref} IS NULL)"
branches.append((" AND ".join([*(e[0] for e in equal), greater])))
bindings.extend([*(v for e in equal for v in e[1]), *args])
equal.append((f"{ref} IS NOT DISTINCT FROM cast_to_type(?, {ref})", [value]))
return " OR ".join(f"({branch})" for branch in branches), bindings
def _cursor_window(cur, name, proj, pred, params, order, page_key, offset, limit, cursor):
"""Select one keyset page and its lookahead before evaluating the payload projection."""
terms = _cursor_order(order)
if cursor is not None:
if (not isinstance(cursor, dict) or cursor.get("order") != order
or not isinstance(cursor.get("values"), list)
or type(cursor.get("offset")) is not int or cursor["offset"] < 0):
raise ValueError("cursor does not match this window")
seek, bindings = _cursor_predicate(terms, cursor["values"])
pred = f"({pred}) AND ({seek})" if pred else seek
params = (*params, *bindings)
offset = cursor["offset"]
tail = f" WHERE {pred}" if pred else ""
jump = f" OFFSET {offset}" if cursor is None and offset else ""
key_columns = ", ".join(f'{expr} AS "_seek_{i}"' for i, (expr, _, _) in enumerate(terms))
key_order = ", ".join(f'"_seek_{i}" {direction} NULLS {nulls}'
for i, (_, direction, nulls) in enumerate(terms))
boundary = ", ".join(f'CAST(({expr}) AS VARCHAR)' for expr, _, _ in terms)
got = cur.execute(
f'WITH _mirror_keys AS MATERIALIZED (SELECT {page_key} AS "_page_key", {key_columns} '
f'FROM {name}{tail} ORDER BY {order} LIMIT {limit + 1}{jump}), '
f'_mirror_page AS MATERIALIZED (SELECT * FROM _mirror_keys ORDER BY {key_order} LIMIT {limit}) '
f'SELECT {proj}, {boundary}, (SELECT count(*) > {limit} FROM _mirror_keys) '
f'FROM {name} SEMI JOIN _mirror_page ON {page_key} = _mirror_page."_page_key" '
f'ORDER BY {order}', params).fetchall()
width = len(terms) + 1
columns = [d[0] for d in (cur.description or [])][:-width]
position = ({"order": order, "values": list(got[-1][-width:-1]),
"offset": offset + len(got)} if got and got[-1][-1] else None)
return [tuple(row[:-width]) for row in got], columns, position, offset
def _page_is_row_local(projection, order, cur):
"""Prove outer expressions scalar with DuckDB's parser and bounded catalogue lookups."""
metadata, verified, expanding, bound = {}, set(), set(), set()
def parse(expressions, ordering=""):
sql = "SELECT " + expressions + (" ORDER BY " + ordering if ordering else "")
parsed = json.loads(cur.execute("SELECT json_serialize_sql(?)", [sql]).fetchone()[0])
if parsed.get("error") or len(parsed["statements"]) != 1:
raise ValueError("Unrecognized page expression")
node = parsed["statements"][0]["node"]
if (node["type"] != "SELECT_NODE" or node["from_table"]["type"] != "EMPTY"
or node["cte_map"]["map"] or node["group_expressions"] or node["group_sets"]
or node["aggregate_handling"] != "STANDARD_HANDLING"
or any(node[key] is not None for key in ("where_clause", "having", "sample", "qualify"))
or any(m["type"] != "ORDER_MODIFIER" for m in node["modifiers"])):
raise ValueError("Projection changes the outer population")
return [node["select_list"], node["modifiers"]]
def aggregate_is_local(value, scope):
columns = set()
def collect(node):
if isinstance(node, list):
return all(collect(v) for v in node)
if not isinstance(node, dict):
return True
kind = node.get("class")
if kind == "COLUMN_REF":
columns.add(tuple(node["column_names"]))
elif kind is not None and kind not in {
"CONSTANT", "CAST", "CASE", "COMPARISON", "CONJUNCTION",
"OPERATOR", "FUNCTION", "PARAMETER"}:
return False
return all(collect(v) for v in node.values())
if not collect(value):
return False
if not columns:
return True
if scope is None or scope[1] or scope[0]["type"] != "SELECT_NODE":
return False
source = scope[0]["from_table"]
if (source["type"] != "BASE_TABLE" or source.get("column_name_alias")
or source.get("at_clause") or source.get("sample")):
return False
alias = source.get("alias") or source["table_name"]
fold = str.maketrans("ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz")
if any(len(c) != 2 or c[0].translate(fold) != alias.translate(fold) for c in columns):
return False
quote = lambda name: '"' + name.replace('"', '""') + '"'
table = '.'.join(quote(source[key]) for key in ("catalog_name", "schema_name", "table_name")
if source.get(key))
selected = ','.join('.'.join(map(quote, c)) for c in sorted(columns))
sql = f"DESCRIBE SELECT {selected} FROM {table} AS {quote(alias)}"
if sql not in bound:
# A matching alias alone cannot prove that a missing column will not bind outward.
cur.execute(sql).fetchall()
bound.add(sql)
return True
def references(value, required, nested, scalar=True, scope=None):
if isinstance(value, list):
return all(references(v, required, nested, scalar, scope) for v in value)
if not isinstance(value, dict):
return True
if "cte_map" in value:
scope = (value, bool(value["cte_map"]["map"]) or bool(scope and scope[1]))
kind = value.get("class")
if scalar and kind == "SUBQUERY":
# Nested aggregates must bind locally even inside a syntactic scalar subquery.
return (value["subquery_type"] == "SCALAR" and value.get("child") is None
and references(value["subquery"], required, nested, False, scope))
if scalar and kind is not None:
if kind not in {"COLUMN_REF", "CONSTANT", "CAST", "CASE", "COMPARISON",
"CONJUNCTION", "OPERATOR", "FUNCTION", "PARAMETER"}:
return False
if kind == "OPERATOR" and value["type"] not in {
"OPERATOR_COALESCE", "OPERATOR_NOT", "OPERATOR_IS_NULL",
"OPERATOR_IS_NOT_NULL", "COMPARE_IN", "COMPARE_NOT_IN"}:
return False
if "function_name" in value:
if value.get("catalog") or value.get("schema", "") not in ("", "main"):
return False
if scalar and (value.get("distinct") or value.get("filter") is not None
or value.get("export_state") or value.get("order_bys", {}).get("orders")):
return False
name = value["function_name"].lower()
required[name] = required.get(name, False) or scalar
if not scalar:
nested.append((name, value, scope))
return all(references(v, required, nested, scalar, scope) for v in value.values())
def prove(expressions, ordering=""):
required, nested = {}, []
if not references(parse(expressions, ordering), required, nested):
return False
missing = sorted(set(required) - metadata.keys())
if missing:
rows = cur.execute(
"SELECT lower(function_name), function_type, internal, macro_definition "
"FROM duckdb_functions() WHERE lower(function_name) = ANY(?)", [missing]).fetchall()
metadata.update((name, []) for name in missing)
for name, function_type, internal, definition in rows:
metadata[name].append((function_type, internal, definition))
for name, scalar in required.items():
if (name, scalar) in verified:
continue
if not metadata[name] or name in expanding or len(expanding) >= 16:
return False
for function_type, internal, definition in metadata[name]:
if function_type == "macro":
# Only engine-owned macros with independently scalar bodies are admitted.
if not internal or not definition:
return False
expanding.add(name)
try:
if not prove(definition):
return False
finally:
expanding.remove(name)
elif function_type not in ({"scalar"} if scalar else {"scalar", "aggregate", "table"}):
return False
verified.add((name, scalar))
for name, value, scope in nested:
if (value.get("class") != "WINDOW"
and any(kind == "aggregate" for kind, _internal, _definition in metadata[name])
and not aggregate_is_local(value, scope)):
return False
return True
try:
return prove(projection, order)
except Exception as exc:
# A parser/catalogue refusal never authorizes moving the original projection.
_request_timing.raise_if_resource_error(exc)
return False
def mirror_file_marks(cur):
"""Identify the cursor's actual file and WAL, or decline an unnamed mirror."""
row = cur.execute("SELECT path FROM duckdb_databases() "
"WHERE database_name = current_database()").fetchone()
path = row[0] if row else None
if not path or path == ":memory:":
return None
marks = []
for name in (path, path + ".wal"):
try:
stat = os.stat(name)
marks.append((stat.st_dev, stat.st_ino, stat.st_size, stat.st_mtime_ns, stat.st_ctime_ns))
except OSError:
marks.append(None)
return (str(Path(path).resolve()), tuple(marks)) if marks[0] is not None else None
def _count_revision(cur):
"""Cache only managed, autocommit reads of one persistent mirror without local relations."""
if not isinstance(cur, _TimedCursor) or cur._in_transaction or mirror_active_writes(cur):
return None
generation = mirror_generation(cur), mirror_writes(cur)
marks = mirror_file_marks(cur)
if marks is None:
return None
unsafe = cur.execute(
"SELECT (SELECT count(*) FROM duckdb_databases() WHERE NOT internal) <> 1 "
"OR EXISTS (SELECT 1 FROM duckdb_tables() WHERE temporary AND NOT internal) "
"OR EXISTS (SELECT 1 FROM duckdb_views() WHERE temporary AND NOT internal)").fetchone()[0]
return None if unsafe or generation != (mirror_generation(cur), mirror_writes(cur)) else (marks, generation)
def count_rows(table=None, where=None, params=(), cur=None, from_sql=None):
"""Return the exact matching row population, reusing only an unchanged mirror revision."""
# Callers supply deterministic compiled SQL, as they do for window().
if bool(table) == bool(from_sql):
raise ValueError("datastore.count_rows: pass exactly one of `table` or `from_sql`")
name = str(from_sql or table).strip()
if table and not _IDENT_OK.fullmatch(name):
raise ValueError(f"datastore.count_rows: {table!r} is not a table identifier")
pred = str(where or "").strip()
sql = f"SELECT count(*) FROM {name}" + (f" WHERE {pred}" if pred else "")
params = tuple(params or ())
cur = cur if cur is not None else ro_con()
revision = _count_revision(cur)
key = None
if revision is not None:
try:
# SQL and typed bindings remain private, including large IN predicates.
key = revision, hashlib.sha256(pickle.dumps((sql, params), protocol=5)).digest()
except (TypeError, ValueError, AttributeError, OverflowError, pickle.PickleError):
pass
if key is not None:
with _COUNT_CACHE_LOCK:
if key in _COUNT_CACHE:
_COUNT_CACHE.move_to_end(key)
return _COUNT_CACHE[key]
total = int(cur.execute(sql, params).fetchone()[0] or 0)
if key is not None and _count_revision(cur) == revision:
with _COUNT_CACHE_LOCK:
_COUNT_CACHE[key] = total
_COUNT_CACHE.move_to_end(key)
while len(_COUNT_CACHE) > _COUNT_CACHE_MAX:
_COUNT_CACHE.popitem(last=False)
return total
def window(table=None, select=None, where=None, params=(), order_by=None,
offset=0, limit=WINDOW_DEFAULT, cur=None, from_sql=None, group_by=None,
page_first=False, page_key=None, cursor=None, with_cursor=False):
"""One page of a mirror table PLUS the true count of everything the same predicate matches.
Returns `{"rows": [tuple, ...], "columns": [name, ...], "total": int, "offset": int,
"limit": int, "clamped": bool}` β `rows` is the slice, `total` is the population.
`select` is the projection expression list ("id, name, amount_untaxed"); `where` is a WHERE
body with `?` placeholders; `params` binds them, and is used by BOTH statements.
β `table` and `from_sql` are the SAME argument wearing two trust levels, and they are separate
names on purpose. `table` is an identifier and is SHAPE-CHECKED; `from_sql` is a whole FROM
body β a parenthesised subquery with an alias β which cannot be checked at all. Two of the
eight connected grids need one (`read_customers` and `read_agents` are single statements whose
population is a SQL `UNION`), so the capability has to exist; naming it distinctly means a
reviewer can grep for every caller that hands over raw FROM SQL instead of inferring it.
ββ `group_by` (W50-T31 / C1) FOLDS THE PAGE AND DELIBERATELY DOES **NOT** FOLD THE COUNT.
`total` stays `count(*)` over the same `tail` β the ROW population the predicate matches, not
the number of groups β because that is the one number C1 calls `total_rows` and the whole
point of asking for it here is that it comes from the SAME statement, over the same `tail` and
the same `params`, as the aggregates beside it. A caller wanting "how many groups" asks for a
`count(DISTINCT ...)`, which is a projection, not a second meaning for this field.
β It defaults to None and emits nothing then, so every existing caller's two statements are
byte-identical to what they were before this argument existed.
`page_first` (a row page whose `order_by` is an expression over the table's own columns) picks
the page's rows before the projection runs, when only the projection holds a subquery.
`page_key` opts a physical table into a narrow, materialised key page before projection.
It must uniquely identify source rows, independently of the displayed record identity.
Proven row-local pages end with this physical key ascending. Fallback SQL retains its order.
`with_cursor` returns an internal continuation tuple for a signed API token.
`cursor` seeks past that tuple without OFFSET. Counts retain the complete filtered scope.
"""
if bool(table) == bool(from_sql):
raise ValueError("datastore.window: pass exactly one of `table` or `from_sql`")
if from_sql:
name = str(from_sql).strip()
else:
name = str(table or "").strip()
if not _IDENT_OK.match(name):
raise ValueError(f"datastore.window: {table!r} is not a table identifier")
proj = str(select or "").strip()
if not proj:
raise ValueError("datastore.window: a projection is required; there is no implicit *")
try:
offset = max(0, int(offset or 0))
except (TypeError, ValueError):
offset = 0
try:
limit = int(limit if limit is not None else WINDOW_DEFAULT)
except (TypeError, ValueError):
limit = WINDOW_DEFAULT
clamped = limit > WINDOW_MAX or limit < 1
limit = min(max(limit, 1), WINDOW_MAX)
params = tuple(params or ())
pred = str(where or "").strip()
tail = f" WHERE {pred}" if pred else ""
order = str(order_by or "").strip() or "1"
cur = cur if cur is not None else ro_con()
# β THE COUNT RUNS FIRST AND OVER THE SAME `tail` + `params`. Ordering it first is deliberate:
# if the projection is ever wrong, the caller fails LOUDLY on the rows query rather than
# quietly returning a good count beside a broken page.
total = count_rows(table=table, from_sql=from_sql, where=pred, params=params, cur=cur)
grp = str(group_by or "").strip()
fold = f" GROUP BY {grp}" if grp else ""
needs_projection = _order_needs_projection(order, proj) if page_first else True
row_local = (page_first and table and not grp and not needs_projection
and _page_is_row_local(proj, order, cur))
if with_cursor or cursor is not None:
if (not page_key or grp or needs_projection
or not (row_local or from_sql and _page_is_row_local(proj, order, cur))):
raise CursorPagingUnsupported("cursor paging requires a unique key and row-local SQL")
order = _physical_page_order(order, page_key)
got, cols, position, offset = _cursor_window(
cur, name, proj, pred, params, order, page_key, offset, limit, cursor)
return {"rows": got, "columns": cols, "total": total, "offset": offset,
"limit": limit, "clamped": clamped, "next_cursor": position}
if row_local and page_key:
order = _physical_page_order(order, page_key)
if row_local and page_key:
# Only keys cross the materialisation boundary; identity order still uses the full population.
got = cur.execute(
f'WITH _mirror_page AS MATERIALIZED (SELECT {page_key} AS "_page_key" '
f'FROM {name}{tail} ORDER BY {order} LIMIT {limit} OFFSET {offset}) '
f'SELECT {proj} FROM {name} SEMI JOIN _mirror_page '
f'ON {page_key} = _mirror_page."_page_key" ORDER BY {order}', params).fetchall()
elif (row_local and not page_key and _SUBQUERY.search(proj)
and not _SUBQUERY.search(f"{pred} {order}")):
# A correlated lookup in the projection joins EVERY row before the LIMIT: one 50-row page
# of the 952k-row Amazon transactions held 2.8 GB and restarted staging (hotfix sdlc71).
got = cur.execute(
f"SELECT {proj} FROM (SELECT * FROM {name}{tail} ORDER BY {order} "
f"LIMIT {int(limit)} OFFSET {int(offset)}) AS {name} ORDER BY {order}",
params).fetchall()
else:
got = cur.execute(
f"SELECT {proj} FROM {name}{tail}{fold} "
f"ORDER BY {order} LIMIT {int(limit)} OFFSET {int(offset)}",
params).fetchall()
cols = [d[0] for d in (cur.description or [])]
return {"rows": [tuple(r) for r in got], "columns": cols,
"total": total, "offset": offset, "limit": limit, "clamped": clamped}
def columns_of(table, cur=None):
"""The column names a mirror table actually has, lowercased.
β A mirror can be `ready()` and still be MISSING COLUMNS: `ready()` reads entity PHASES, while
`_ensure_columns`/`_backfill_columns` checkpoint separately. Asking before projecting is what
stops a DuckDB `BinderException: Referenced column β¦ not found` reaching a user as a bare 500
(it already cost one live, which is why `odoo_relational.columns` exists on the app side).
"""
name = str(table or "").strip()
if not _IDENT_OK.match(name):
raise ValueError(f"datastore.columns_of: {table!r} is not a table identifier")
cur = cur if cur is not None else ro_con()
try:
rows = cur.execute(f"PRAGMA table_info('{name}')").fetchall()
except Exception: # noqa: BLE001
return set()
return {str(r[1]).lower() for r in rows}
SEED_DATASET = os.environ.get("STORE_SEED_DATASET", "royal-imports/cfo-os-data")
#: The seed file inside the dataset, and the PUBLISHED manifest that describes it. The second one
#: is what makes a cut date knowable at all; see `seed_manifest` below.
SEED_PREFIX = os.environ.get('STORE_SEED_PREFIX', 'store_seed').strip('/')
if SEED_PREFIX not in {'store_seed', 'staging_store_seed'}:
raise ValueError('STORE_SEED_PREFIX must name a supported seed directory.')
if SEED_PREFIX == 'staging_store_seed' and (
not os.environ.get('STORE_SEED_DATASET') or SEED_DATASET in {
'royal-imports/cfo-os-data', 'aios-loopable/cfo-os-data'}):
raise ValueError('A staging seed requires an explicit separate staging dataset.')
SEED_FILE = SEED_PREFIX + "/royal.duckdb"
SEED_MANIFEST_FILE = SEED_PREFIX + "/manifest.json"
def seed_manifest_path(db_path=None):
"""Where the LOCAL record of "which seed is this mirror" lives: a sibling of the DuckDB file.
A sibling rather than a table inside the database, deliberately. `connect()` runs
`CREATE TABLE IF NOT EXISTS _sync_state` and nothing else, so a 180 MB seed already in the
field would never gain a new table or a new column; and the identity of a file is not a fact
about the data inside it. A sibling JSON is readable without opening DuckDB at all, which is
what lets a status route answer while the mirror is mid-sync and locked.
"""
p = Path(db_path or DB_PATH)
return p.with_name(p.name + ".seed.json")
def seed_manifest(db_path=None):
"""The recorded identity of the seed this mirror was hydrated from, or None when unrecorded.
β NONE IS THE HONEST ANSWER FOR EVERY SEED CUT BEFORE THIS EXISTED, and that is the majority
of them today. The caller reports null fields with a cause; it must never substitute the
file's mtime or the download time for the cut date. Those two say when the bytes ARRIVED,
and a freshness surface that answers "cut 2026-09-03" over data cut on 2026-08-15 is worse
than one that answers nothing: the first is believed.
"""
try:
p = seed_manifest_path(db_path)
if not p.exists():
return None
with open(p, "r", encoding="utf-8") as fh:
data = json.load(fh)
return data if isinstance(data, dict) else None
except Exception: # noqa: BLE001
return None
def _revision_from_cache_path(path):
"""The HF commit sha out of a `hf_hub_download` result path, or None.
The cache layout is `.../datasets--org--name/snapshots/<sha>/<file>`, so the sha is the
segment after `snapshots`. Parsed rather than fetched because a second network call at boot
is a second way for the seed to fail, and an unknown revision degrades to null cleanly.
ββ DO NOT `.resolve()` THIS PATH, AND THAT IS THE WHOLE FUNCTION. A snapshot directory is a
tree of SYMLINKS into `blobs/`, so `.resolve()` rewrites
`.../snapshots/<sha>/store_seed/royal.duckdb` into `.../blobs/<content-hash>` β a path with no
`snapshots` segment at all, so `parts.index` raises and every caller reads `None`.
β AND IT FAILS ONLY WHERE IT MATTERS. Windows hydrates the cache with real copies rather than
symlinks, so `.resolve()` is a no-op locally and this parsed correctly on the box it was
written on; it returned `None` on every real Linux container. MEASURED in W50-T30's two job
runs, whose `[boot] seed identity` line reported a null revision both times while the same code
answered a sha here. It is the same symlink trap `ops/submit_pg_seed_job.py` records in its own
header, where following it into `blobs/` killed a run outright.
β The raw path is tried FIRST and the resolved one only as a fallback, so a cache layout that
hands back an already-real path still answers.
"""
for candidate in (path, None):
try:
parts = Path(path).parts if candidate is not None else Path(path).resolve().parts
i = parts.index("snapshots")
rev = parts[i + 1]
if rev:
return rev
except Exception: # noqa: BLE001
continue
return None
def _write_seed_manifest(dataset, filename, revision, cut_date, content_sha=None):
"""Record what was just installed. Fail-quiet: a missing manifest degrades the status surface,
it must never abort a seed that otherwise succeeded.
β TWO IDENTITIES, NOT ONE, AND THEY FAIL INDEPENDENTLY. `revision` is the HF commit sha and is
parsed out of the local cache LAYOUT, so it is only ever as reliable as that layout.
`contentSha` is the seed's own published digest of its bytes, read from the seed's manifest, so
it depends on nothing local at all. Carrying only the first is what made the identity null on
every container until W50-T30 measured it: a surface that can name WHICH seed it is serving
must not lose that ability to a cache detail.
"""
try:
payload = {
"dataset": dataset,
"file": filename,
"revision": revision,
"contentSha": content_sha,
"cutDate": cut_date,
"recordedAt": dt.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ"),
}
with open(seed_manifest_path(), "w", encoding="utf-8") as fh:
json.dump(payload, fh, indent=2, sort_keys=True)
return payload
except Exception: # noqa: BLE001
return None
def _published_seed_manifest(tok):
"""The seed's OWN published manifest as a dict, or `{}`. One download, one parse.
Split out so the cut date and the content digest are read from a single fetch: two callers each
downloading the same cached file is two ways for a boot-time read to fail, for one artefact.
"""
try:
from huggingface_hub import hf_hub_download
m = hf_hub_download(SEED_DATASET, SEED_MANIFEST_FILE, repo_type="dataset", token=tok)
from core import blob_envelope as be
data = json.loads(be.open('royal-imports', SEED_MANIFEST_FILE, Path(m).read_bytes()))
return data if isinstance(data, dict) else {}
except Exception: # noqa: BLE001
return {}
def _first_str(data, keys):
"""The first non-blank string among `keys`, or None. Several spellings are accepted on purpose:
the publisher and the reader are different tickets in different files."""
for k in keys:
v = (data or {}).get(k)
if isinstance(v, str) and v.strip():
return v.strip()
return None
def _published_cut_date(tok, data=None):
"""The cut date the SEED ITSELF publishes, or None. Never derived from any local clock.
β A LOCAL CLOCK IS THE ONE SOURCE THIS MUST NEVER USE. `st_mtime` and the download time say
when the BYTES ARRIVED; W50-T30 measured a published seed whose newest order was 28 days older
than its own upload date, so a freshness surface reporting either would be believed and wrong,
which is worse than reporting nothing.
"""
return _first_str(_published_seed_manifest(tok) if data is None else data,
("cutDate", "cut_date", "cutAt", "cut_at"))
def _published_content_sha(tok, data=None):
"""The seed's own digest of its bytes, or None. Layout-independent, unlike the HF revision."""
return _first_str(_published_seed_manifest(tok) if data is None else data,
("sha256", "contentSha", "content_sha", "sha"))
def _seedable():
"""Is there no usable mirror to keep? True for a missing file, or for one an early reader
CREATED empty (`_instance()` makes the file) before the boot seed landed: tables none but
`_sync_state`, and that one without a row. A file holding anything else is kept."""
if not DB_PATH.exists():
return True
try:
cur = ro_cursor()
try:
names = {r[0] for r in cur.execute(
"SELECT table_name FROM information_schema.tables "
"WHERE table_schema = 'main'").fetchall()}
rows = (cur.execute("SELECT count(*) FROM _sync_state").fetchone()[0]
if "_sync_state" in names else 0)
finally:
cur.close()
except Exception: # noqa: BLE001
return False # cannot tell: keep the file, the pre-hotfix behaviour
return not (names - {"_sync_state"}) and not rows
def _install_seed(tmp):
"""Swap the downloaded seed in, closing any instance a reader opened on the old file.
ββ HOTFIX 2026-09-27 (D-3030). A reader that reached `_instance()` during the seed download
CREATED an empty royal.duckdb and kept it open; a bare `os.replace` then orphaned that handle
(Linux keeps the unlinked inode), so the whole process read an EMPTY mirror, 0 of 8 Odoo
phases, while the seed on disk had all 8 `live`. Staging's Odoo cadence is manual, so nothing
ever refilled it: every Odoo grid, measure column and Sheet cell refused "first sync".
`_swap_instance` drains open reads first (F47), then closes and swaps under `_INSTANCE_LOCK`,
so no reader can reopen the old file between the close and the swap; the close checkpoints its
WAL, and any leftover WAL is removed so it cannot be replayed onto the seed.
"""
with _swap_instance():
wal = DB_PATH.with_name(DB_PATH.name + ".wal")
if wal.exists():
wal.unlink()
os.replace(tmp, DB_PATH)
_RO_TLS.__dict__.pop("cur", None)
_READY["ok"] = False
_GENERATION[0] += 1 # every other thread's cached cursor is now stale
def _prepare_seed_mirrors(path, rt):
"""Keep seed fallback rows only for connectors without a saved tenant snapshot."""
if rt is None:
return
from harness import mirror_store
con = apply_limits(duckdb.connect(str(path)))
try:
_ensure_sync_state(con)
for connector in tuple(_connector_spec.MIRRORS):
if connector == "odoo":
continue
store = mirror_store.for_connector(connector)
if rt.download_bytes(store.prefix + "manifest.json") is None:
continue
for table in store._tables(con):
con.execute(f'DROP TABLE "{table}"')
entities = [(row[0],) for row in store._state_rows(con)]
if entities:
con.executemany("DELETE FROM _sync_state WHERE entity = ?", entities)
finally:
con.close()
def ensure_seed():
"""Fresh/ephemeral deployment (HF Space disks reset on every rebuild): hydrate the store
from the PRIVATE dataset's seed snapshot β Analyst-ready in ~a minute instead of a 20-40 min
XML-RPC backfill; the auto-sync then closes the gap via write_date cursors. Also restores
the pilot views/routines when absent. Fail-quiet: no seed/token β normal backfill."""
import shutil
rt = _resident_odoo_runtime()
if rt is not None:
if odoo_mirror_restored(rt.key, rt):
return False
result = restore_odoo_mirror(rt.key, rt)
if result["state"] != "absent":
return False
# The legacy seed contains Royal Imports data, never another tenant's data.
if rt.key != "royal-imports":
return False
if not _seedable():
return False
tok = os.environ.get("HF_TOKEN")
if not tok:
return False
tmp = DB_PATH.with_name(DB_PATH.name + '.tmp')
try:
from huggingface_hub import hf_hub_download
DB_PATH.parent.mkdir(parents=True, exist_ok=True)
p = hf_hub_download(SEED_DATASET, SEED_FILE, repo_type="dataset", token=tok)
# Atomic install: copy to a temp path then swap it in under the instance lock.
tmp = DB_PATH.with_name(DB_PATH.name + ".tmp")
from core import blob_envelope as be
tmp.write_bytes(be.open(rt.key if rt else 'royal-imports', SEED_FILE, Path(p).read_bytes()))
_prepare_seed_mirrors(tmp, rt)
_install_seed(tmp)
# β W50-T29 β RECORD WHICH SEED THIS IS, because nothing did and the status surface could
# therefore only ever say "there is a mirror". Three fields: the dataset, the commit sha of
# the exact file downloaded, and the date the DATA was cut, which comes from the seed's own
# published manifest and from nowhere else. Every one degrades to null.
_published = _published_seed_manifest(tok)
_write_seed_manifest(SEED_DATASET, SEED_FILE,
_revision_from_cache_path(p),
_published_cut_date(tok, _published),
_published_content_sha(tok, _published))
for fn in ("views.json", "routines.json"):
tgt = DB_PATH.parent / fn
if not tgt.exists():
try:
q = hf_hub_download(SEED_DATASET, f"{SEED_PREFIX}/{fn}",
repo_type="dataset", token=tok)
tgt.write_bytes(be.open(rt.key if rt else 'royal-imports',
f'{SEED_PREFIX}/{fn}', Path(q).read_bytes()))
except Exception:
pass
return True
except Exception:
return False
finally:
tmp.unlink(missing_ok=True)
def connect(tenant_key=None):
"""A cursor on the shared instance for the SYNC WRITER (and status/validate). Returned as a
cursor so callers' con.close() frees the cursor without closing the process-wide instance;
one writer cursor is active at a time (the sync thread is sequential), readers use others."""
handle = tenant_handle(tenant_key) if tenant_key is not None else None
cur = _candidate_connection(handle.tenant_key, handle.path) if handle is not None else _candidate_connection()
candidate = getattr(_SYNC_TLS, "candidate", None)
read_only = cur is None and versioned_mode()
owner = factory = None
if cur is None:
if handle is None or handle.path == DB_PATH.resolve():
cur, owner = _open_cursor()
factory = _open_cursor
elif read_only or handle.path.is_file():
cur = handle.ro_cursor()
else:
handle.path.parent.mkdir(parents=True, exist_ok=True)
cur = bounded_connection(duckdb.connect(str(handle.path)))
cur._mirror_handle, cur._mirror_path = handle, handle.path
# In versioned mode a retired instance closes once drained, so every reader cursor registers.
if ((_request_timing.CURRENT.get() is not None or (owner is not None and read_only)
or handle is not None or (candidate is not None and candidate["tenant"] is not None))
and not isinstance(cur, _TimedCursor)):
bound = (tenant_handle(candidate["tenant"], candidate["path"])
if candidate is not None and candidate["tenant"] is not None else handle)
cur = _TimedCursor(cur, owner, factory, bound)
if not read_only:
# A read-only version takes no DDL. A write there raises, loudly, by design.
_ensure_sync_state(cur)
return cur
def _ensure_sync_state(con):
con.execute("""CREATE TABLE IF NOT EXISTS _sync_state (
entity VARCHAR PRIMARY KEY, phase VARCHAR, last_id BIGINT,
cursor_wd VARCHAR, rows BIGINT, updated_at VARCHAR)""")
def _connection_state_key(key):
"""Keep legacy checkpoints and qualify other connections after the source table's first dot."""
binding = getattr(_SYNC_TLS, "connection", None)
if binding is None or binding["legacy"]:
return key
table, dot, tail = key.partition(".")
token = hashlib.sha256(binding["id"].encode()).hexdigest()
return f"{table}.connection.{token}" + (dot + tail if dot else "")
def _connection_where():
"""Restrict replacement and reconciliation to the active stable connection identity."""
binding = getattr(_SYNC_TLS, "connection", None)
return ("_connection_id = '" + binding["id"].replace("'", "''") + "'"
if binding else "TRUE")
def _connection_ids(con, model, ids):
"""Allocate durable negative mirror IDs without changing upstream cursors or legacy IDs."""
binding = getattr(_SYNC_TLS, "connection", None)
ids = sorted({int(i) for i in ids if i is not None and i is not False})
if binding is None or binding["legacy"]:
return {i: i for i in ids}
if not ids:
return {}
found = dict(con.execute("SELECT source_id, id FROM odoo_record_ids "
"WHERE connection_id=? AND model=? AND source_id IN (SELECT unnest(?::BIGINT[]))",
[binding["id"], model, ids]).fetchall())
missing = [i for i in ids if i not in found]
if missing:
last = con.execute("SELECT coalesce(min(id), 0) FROM odoo_record_ids").fetchone()[0]
rows = [(binding["id"], model, i, last - n) for n, i in enumerate(missing, 1)]
con.execute("INSERT INTO odoo_record_ids SELECT ?, ?, unnest(?::BIGINT[]), unnest(?::BIGINT[])",
[binding["id"], model, missing, [row[3] for row in rows]])
found.update((row[2], row[3]) for row in rows)
return found
def _connection_rows(rows, spec):
"""Translate row and relation identities once, after source names resolve against raw IDs."""
binding = getattr(_SYNC_TLS, "connection", None)
if binding is None:
return rows
con, model = binding["con"], spec["model"]
identities = _connection_ids(con, model, [r["id"] for r in rows])
relations = binding["relations"].get(model, {})
for row in rows:
source_id = row["id"]
row.update(id=identities[source_id], _source_id=source_id,
_connection_id=binding["id"], Store=binding["label"])
for field, (target, kind) in relations.items():
# team_id is the tenant's BU enum, not a mirrored crm.team record identity.
if field == "team_id":
continue
output_field = field if field.endswith("_id") or field not in MANY2ONE_FIELDS else field + "_id"
outputs = [output_field]
outputs += [out for out, source in (spec.get("raw_m2m") or {}).items() if source == field]
for output in outputs:
if field in (spec.get("named_m2m") or {}) and output == output_field:
continue
is_list = kind in {"many2many", "one2many"}
values = [i for row in rows for i in (json.loads(row.get(output) or "[]") if is_list
else [row.get(output)]) if i is not None]
mapped = _connection_ids(con, target, values)
for row in rows:
if output not in row:
continue
if is_list:
raw = json.loads(row[output] or "[]")
row[output] = json.dumps([mapped[i] for i in raw]) if row[output] is not None else None
first = output.removesuffix("_ids") + "_id"
if first in row:
row[first] = mapped[raw[0]] if raw else None
elif row[output] is not None:
row[output] = mapped[row[output]]
return rows
def _odoo_entries(rt):
"""Resolve every tenant Odoo credential by its stable keychain connection identity."""
from core import keychain
entries = sorted((row for row in keychain.list_entries(rt) if row.get("type") == "odoo"),
key=lambda row: (int(row.get("slot") or 1), str(row["id"])))
resolved = []
for row in entries:
cid = str(row.get("connection_id") or row["id"])
creds = keychain.odoo_creds(rt, cid)
if not creds:
raise RuntimeError("An Odoo connection has no usable tenant credential")
resolved.append({"id": cid, "label": str(row.get("label") or row["id"]), "creds": creds})
if not resolved and rt.key == "royal-imports":
resolved.append({"id": "legacy-env", "label": "Odoo 1", "creds": None})
return resolved
@contextmanager
def _odoo_connection(con, rt, entry, client_factory=None):
"""Bind a dedicated read-only transport and the saved identity namespace for one connection."""
con.execute("CREATE TABLE IF NOT EXISTS odoo_connections "
"(connection_id VARCHAR PRIMARY KEY, legacy BOOLEAN, label VARCHAR)")
con.execute("CREATE TABLE IF NOT EXISTS odoo_record_ids "
"(connection_id VARCHAR, model VARCHAR, source_id BIGINT, id BIGINT UNIQUE, "
"PRIMARY KEY(connection_id, model, source_id))")
known = dict(con.execute("SELECT connection_id, legacy FROM odoo_connections").fetchall())
if entry["id"] not in known:
# The saved binding survives a default change and disconnection.
legacy = not known
if set(known) == {"legacy-env"} and rt.key == "royal-imports":
con.execute("UPDATE odoo_connections SET connection_id=? WHERE connection_id='legacy-env'", [entry["id"]])
for table in ENTITIES:
if "_connection_id" in (columns_of(table, cur=con) or ()):
con.execute(f"UPDATE {table} SET _connection_id=? WHERE _connection_id='legacy-env'", [entry["id"]])
legacy = True
else:
con.execute("INSERT INTO odoo_connections VALUES (?,?,?)", [entry["id"], legacy, entry["label"]])
else:
legacy = known[entry["id"]]
con.execute("UPDATE odoo_connections SET label=? WHERE connection_id=?", [entry["label"], entry["id"]])
previous = getattr(_SYNC_TLS, "connection", None)
client, context = getattr(O._tlocal, "client", None), getattr(O._tlocal, "tenant_ctx", None)
try:
O._tlocal.client = (client_factory(entry["creds"]) if client_factory else O.OdooClient(creds=entry["creds"]))
O._tlocal.tenant_ctx = None
_SYNC_TLS.connection = {**entry, "legacy": legacy, "con": con, "relations": {}}
for table in ENTITIES:
cols = set(columns_of(table, cur=con) or ())
if cols:
for name, typ in (("Store", "VARCHAR"), ("_connection_id", "VARCHAR"), ("_source_id", "BIGINT")):
if name.lower() not in cols:
con.execute(f"ALTER TABLE {table} ADD COLUMN {name} {typ}")
if legacy:
con.execute(f"UPDATE {table} SET _connection_id=?, _source_id=id, Store=? WHERE _connection_id IS NULL",
[entry["id"], entry["label"]])
con.execute(f"UPDATE {table} SET Store=? WHERE _connection_id=? AND Store IS DISTINCT FROM ?",
[entry["label"], entry["id"], entry["label"]])
yield _SYNC_TLS.connection
finally:
_SYNC_TLS.connection = previous
O._tlocal.client, O._tlocal.tenant_ctx = client, context
def _connection_model(spec):
"""Read relation metadata for the active source without treating every integer as a relation."""
binding = getattr(_SYNC_TLS, "connection", None)
if binding is not None and spec["model"] not in binding["relations"]:
metadata = O.get_odoo().fields_get(spec["model"], attributes=("type", "relation"))
binding["relations"][spec["model"]] = {
field: (info["relation"], info["type"]) for field, info in metadata.items()
if info.get("type") in {"many2one", "many2many", "one2many"} and info.get("relation")}
def _connection_pending(con, key):
"""Read this connection's markers without mixing the legacy namespace with its siblings."""
prefix = _connection_state_key(key) + "."
return [(name, phase) for name, phase in con.execute(
"SELECT entity, phase FROM _sync_state WHERE starts_with(entity, ?)", [prefix]).fetchall()
if not (prefix == key + "." and name.startswith(key + ".connection."))]
def _flatten(rec, fields, m2m=(), *, named_m2m=None, analytic_names=None, named_m2m_names=None,
list_fields=()):
out = {"id": rec["id"]}
for f in m2m:
v = rec.get(f) or []
base = f.removesuffix("_ids")
out[f"{base}_id"] = v[0] if v else None
out[f] = json.dumps(list(v)) if v else None
for f in fields:
v = rec.get(f)
if f in (named_m2m or {}):
names = (named_m2m_names or {}).get(f, {})
if any(tag_id not in names or not names[tag_id] for tag_id in (v or [])):
raise RuntimeError("Source tag names could not be resolved")
out[f] = json.dumps([names[tag_id] for tag_id in v], ensure_ascii=False) if v else None
elif f == "analytic_distribution":
if v is None or v is False:
out[f] = out[f + "_details"] = None
else:
names = analytic_names or {}
out[f] = json.dumps(v, ensure_ascii=False)
out[f + "_details"] = json.dumps([
{"key": key, "percentage": share,
"accounts": [{"id": int(account_id),
"name": names.get(int(account_id)) or "Unknown analytic account"}
for account_id in key.split(",")]}
for key, share in v.items()
], ensure_ascii=False)
elif f in ID_LIST_FIELDS or f in list_fields:
out[f] = json.dumps(list(v or []))
elif isinstance(v, (list, tuple)) and len(v) == 2 and isinstance(v[0], int):
out[f"{f.removesuffix('_id')}_id"] = v[0]
out[f"{f.removesuffix('_id')}_name"] = str(v[1] or "")
elif isinstance(v, (list, tuple)):
out[f] = json.dumps(v)
elif v is False and f not in BOOL_FIELDS: # Odoo False = NULL for non-bool fields
out[f] = None
else:
out[f] = v
return out
def _flatten_records(recs, fields, m2m=(), *, named_m2m=None, raw_m2m=None, spec=None):
"""Resolve analytic and tag names once per source page, without a shared tenant cache."""
named_m2m_names = {}
for field, model in (named_m2m or {}).items():
if field not in fields:
continue
ids = sorted({tag_id for rec in recs for tag_id in (rec.get(field) or [])})
field_names = named_m2m_names.setdefault(field, {})
for start in range(0, len(ids), 1000):
page = ids[start:start + 1000]
tags = O.search_read(model, [("id", "in", page), ("active", "in", [True, False])],
["name"], limit=len(page), order="id asc")
field_names.update({tag["id"]: tag.get("name") for tag in tags})
names = {}
if "analytic_distribution" in fields:
ids = sorted({int(account_id) for rec in recs
for key in (rec.get("analytic_distribution") or {})
for account_id in key.split(",")})
for start in range(0, len(ids), 1000):
accounts = O.search_read(
"account.analytic.account",
[("id", "in", ids[start:start + 1000]), ("active", "in", [True, False])],
["name"], limit=1000, order="id asc")
names.update({account["id"]: account.get("name") for account in accounts})
binding = getattr(_SYNC_TLS, "connection", None)
relations = binding["relations"].get(spec["model"], {}) if binding and spec is not None else {}
list_fields = {field for field, (_target, kind) in relations.items() if kind in {"many2many", "one2many"}}
rows = [_flatten(rec, fields, m2m, named_m2m=named_m2m, analytic_names=names,
named_m2m_names=named_m2m_names, list_fields=list_fields) for rec in recs]
for rec, row in zip(recs, rows):
for output, source in (raw_m2m or {}).items():
if source in rec:
row[output] = json.dumps(list(rec[source] or []))
return _connection_rows(rows, spec) if spec is not None else rows
#: ββ A HARD-CODED NAME LIST DECIDES WHETHER A COLUMN IS A NUMBER, and a field missing from it
#: lands as VARCHAR β which does not fail, it fails LATER and somewhere else. MEASURED (W37-T13):
#: `stock.move.quantity_done` was absent, so 778,385 rows stored '4.0', '1.0' β¦ as TEXT, every one
#: of them cleanly castable, and the only symptom was a DuckDB
#: *"Cannot mix values of type INTEGER_LITERAL and VARCHAR in CASE expression"* thrown by a
#: FILTERED MEASURE three layers up in `semantic._measure_sql`. Nothing in the sync, the schema or
#: the row count looked wrong.
#: β SO ADDING A NUMERIC FIELD TO AN `ENTITIES` SPEC MEANS ADDING IT HERE TOO. There is no
#: inference: `_row` writes whatever Odoo returned, and Odoo returns `False` for an unset numeric,
#: so a value-sniffing default would be wrong in the other direction.
#: β AND THE TYPE IS FIXED AT CREATE. `_ensure_columns` only ADDs missing columns β it never ALTERs
#: an existing one β so a mirror that already synced the field as VARCHAR keeps it until the table
#: is dropped and re-synced. Correcting the list fixes every FRESH mirror, which is the Space's
#: case (it has never synced this entity) and was not mine (I had to ALTER my scratch copy).
NUMERIC_FIELDS = ("amount_untaxed", "amount_untaxed_signed", "amount_residual_signed",
"amount_company_currency_signed", "amount_residual",
"price_subtotal", "margin", "purchase_price", "product_uom_qty",
"standard_price", "debit", "credit", "balance", "amount",
# W37-T13 β the stock-movement quantity. See the note above.
"quantity_done", "quantity", "value", "reserved_quantity",
"amount_total", "product_qty", "qty_received", "price_unit", "factor",
"partner_latitude", "partner_longitude")
# Odoo also names this many2one without the usual `_id` suffix.
MANY2ONE_FIELDS = frozenset({"product_uom", "create_uid", "property_product_pricelist"})
# Plain x2many fields stored whole as a JSON id list, `[]` for none.
ID_LIST_FIELDS = frozenset({"product_tag_ids"})
# S72-T43 / R107: the audited Odoo fields the Data grids show. Each one is declared AND
# backfilled, because existing rows never re-sync when the schema grows. The list must equal
# `aios-web/api/odoo_relational.py::audit_mirror_additions()`; verify_odoo_relational holds both.
AUDIT_FIELD_ADDITIONS = {
"account_move": ["amount_total", "amount_total_signed", "amount_total_in_currency_signed",
"amount_untaxed", "commercial_partner_id", "commission_total", "create_date",
"date", "invoice_payment_term_id", "invoice_user_id", "is_move_sent",
"journal_id", "narration", "partner_shipping_id", "payment_reference", "ref",
"team_id"],
"account_invoice_line_agent": ["amount", "commission_id", "create_date", "settled"],
"account_move_line": ["account_root_id", "amount_currency", "create_date", "date_maturity",
"full_reconcile_id", "journal_id", "matching_number", "name",
"payment_id", "price_subtotal", "price_total", "price_unit",
"product_uom_id", "quantity", "reconciled", "ref"],
"account_account": ["create_date", "deprecated", "include_initial_balance", "internal_group",
"reconcile", "root_id"],
"sale_order": ["amount_total", "carrier_id", "commission_total", "create_date",
"effective_date", "margin", "margin_percent", "note", "online_payment_based_on",
"partner_invoice_id", "partner_shipping_id", "payment_term_id",
"picking_policy", "procurement_group_id", "recompute_delivery_price",
"require_signature", "user_id", "create_uid", "warehouse_id"],
"sale_order_line": ["commission_free", "create_date", "customer_lead", "invoice_status",
"is_delivery", "margin_percent", "name", "packaging_price", "price_reduce",
"price_reduce_taxexcl", "price_reduce_taxinc", "price_total", "price_unit",
"product_packaging_id", "product_packaging_qty", "product_uom",
"qty_delivered", "qty_delivered_method", "qty_invoiced", "qty_to_invoice",
"salesman_id", "untaxed_amount_invoiced", "untaxed_amount_to_invoice",
"untaxed_amount_to_invoice_credit_limit"],
"res_partner": ["comment", "commercial_company_name", "commercial_partner_id",
"customer_rank", "dba_id", "enable_max_invoice_age", "followup_reminder_type",
"is_company", "max_invoice_age", "online_payment", "parent_id", "sale_warn",
"supplier_rank", "team_id", "type", "tz", "x_hubspot_company_id",
"category_id"],
"stock_picking": ["antsroute_delivery_barcodes", "antsroute_delivery_comment",
"antsroute_delivery_date", "antsroute_delivery_driver",
"antsroute_delivery_status", "antsroute_delivery_vehicle_plate",
"antsroute_nb_delivery_photo", "antsroute_shipping_value", "backorder_id",
"carrier_id", "create_date", "currency_id", "date", "date_deadline",
"group_id", "is_locked", "location_dest_id", "location_id", "move_type",
"picking_type_id", "priority", "sent_to_carrier", "use_antsroute",
"user_id"],
"stock_move": ["create_date", "date_deadline", "description_picking", "fulfillment_status_id",
"group_id", "is_inventory", "origin", "partner_id", "picking_type_id",
"priority", "procure_method", "product_packaging_id", "product_qty",
"product_uom_qty", "propagate_cancel", "reference", "reservation_date",
"rule_id", "sale_line_id", "warehouse_id"],
"stock_valuation_layer": ["account_move_id", "description", "unit_cost"],
"stock_quant": ["create_date", "in_date", "inventory_date", "inventory_diff_quantity",
"package_id", "warehouse_id", "x_warehouse_id"],
"stock_location": ["active", "barcode", "company_id", "create_date", "last_inventory_date",
"location_id", "name", "warehouse_id"],
"purchase_order": ["create_date", "date_planned", "effective_date", "group_id",
"invoice_count", "invoice_status", "notes", "partner_ref",
"payment_term_id", "picking_type_id", "receipt_status", "user_id"],
"purchase_order_line": ["create_date", "name", "price_total", "product_packaging_id",
"product_packaging_qty", "qty_invoiced", "qty_received_method",
"qty_to_invoice", "total_invoiced"],
"stock_landed_cost": ["date", "state", "amount_total"],
"account_bank_statement_line": ["amount", "date", "payment_ref", "partner_id",
"journal_id", "is_reconciled"],
}
for _entity, _added in AUDIT_FIELD_ADDITIONS.items():
_spec = ENTITIES[_entity]
_new = [f for f in _added if f not in _spec["fields"]]
if _new:
_tail = ["write_date"] if "write_date" in _spec["fields"] else []
_spec["fields"] = [f for f in _spec["fields"] if f != "write_date"] + _new + _tail
_backfill = list(_spec.get("backfill_fields") or [])
_spec["backfill_fields"] = _backfill + [f for f in _added if f not in _backfill]
BOOL_FIELDS += ("deprecated", "enable_max_invoice_age", "include_initial_balance", "is_company",
"is_delivery", "is_inventory", "is_locked", "is_move_sent", "online_payment",
"propagate_cancel", "recompute_delivery_price", "reconcile", "reconciled",
"require_signature", "use_antsroute", "is_reconciled")
INT_FIELDS += ("antsroute_nb_delivery_photo", "invoice_count", "max_invoice_age")
NUMERIC_FIELDS += ("amount_currency", "amount_total_in_currency_signed", "amount_total_signed",
"antsroute_shipping_value", "commission_total", "customer_lead",
"inventory_diff_quantity", "margin_percent", "packaging_price",
"price_reduce", "price_reduce_taxexcl", "price_reduce_taxinc", "price_total",
"product_packaging_qty", "qty_delivered", "qty_invoiced", "qty_to_invoice",
"total_invoiced", "unit_cost", "untaxed_amount_invoiced",
"untaxed_amount_to_invoice", "untaxed_amount_to_invoice_credit_limit",
"former_cost", "additional_landed_cost", "final_cost")
# Live fields_get confirms these exact comodels (S72-T43). No other comodel data is
# needed to name the audited relations; archived names remain resolvable too.
ENTITIES.update({
'account_account_tag': {'model': 'account.account.tag', 'archivable': True,
'fields': ['name', 'active', 'write_date']},
'fulfillment_status': {'model': 'fulfillment.status', 'archivable': True,
'fields': ['name', 'active', 'write_date']},
})
AUDIT_M2M_ADDITIONS = {
'account_account': ['tag_ids'],
'sale_order': ['fulfillment_status_ids'],
'stock_picking': ['fulfillment_status_ids'],
}
for _entity, _added in AUDIT_M2M_ADDITIONS.items():
_spec = ENTITIES[_entity]
_spec['m2m'] = list(dict.fromkeys([*_spec.get('m2m', []), *_added]))
# Each m2m declaration uses _backfill_m2m's own durable per-field completion marker.
# Scope identifiers must be backfilled before metadata can describe existing observations.
for _entity in ("account_move", "account_move_line"):
_spec = ENTITIES[_entity]
for _field in ("company_id", "currency_id"):
if _field not in _spec["fields"]:
_spec["fields"].append(_field)
if _field not in _spec.setdefault("backfill_fields", []):
_spec["backfill_fields"].append(_field)
DATASET_LOCK_FIELDS = ("fiscalyear_lock_date", "period_lock_date", "tax_lock_date")
ENTITIES.update({
"account_bank_statement": {
"model": "account.bank.statement", "archivable": False, "optional": True,
"fields": ["date", "write_date"],
},
"res_company": {
"model": "res.company", "archivable": False, "optional": True,
"fields": [*DATASET_LOCK_FIELDS, "write_date"],
},
})
# Delivery references contain metadata only and share the tenant's publication boundary.
DELIVERY_PHOTO_FIELDS = ("id", "res_model", "res_id", "name", "mimetype", "file_size", "create_date")
DELIVERY_PHOTO_TYPES = ("image/jpeg", "image/png", "image/gif", "image/webp")
# An ID cursor disables Odoo's implicit res_field filter, so the gallery scope must be explicit.
DELIVERY_PHOTO_DOMAIN = [("res_model", "=", "stock.picking"),
("type", "=", "binary"),
("res_field", "=", False),
("mimetype", "in", list(DELIVERY_PHOTO_TYPES))]
ENTITIES["delivery_photo_metadata"] = {
"model": "ir.attachment", "archivable": False, "optional": True,
"fields": list(DELIVERY_PHOTO_FIELDS[1:]), "domain": DELIVERY_PHOTO_DOMAIN, "metadata_only": True,
}
INT_FIELDS += ("file_size",)
# The product pool's pricing and tags. Each table holds exactly Odoo's default (active_test) read,
# so it carries no `active` column and a reader needs no active filter.
ENTITIES.update({
"product_pricelist": {
"model": "product.pricelist", "archivable": False, "optional": True,
"fields": ["name"],
},
"product_pricelist_item": {
"model": "product.pricelist.item", "archivable": False, "optional": True,
"fields": ["pricelist_id", "product_id", "product_tmpl_id", "applied_on", "compute_price",
"fixed_price", "min_quantity", "date_start", "date_end", "packaging_id",
"packaging_price"],
},
"product_tag": {
"model": "product.tag", "archivable": False, "optional": True,
"fields": ["name"],
},
})
NUMERIC_FIELDS += ("volume", "incoming_qty", "free_qty", "virtual_available", "fixed_price",
"min_quantity")
#: Entities read in full every pass and swapped into the mirror in one transaction. A cursor sync
#: cannot see an Odoo delete, or a quant quantity that moved without a write_date change.
#: Scope `internal` replaces the internal-location quants only. Any other scope is the whole table.
REPLACE_EACH_PASS = {
"delivery_photo_metadata": {"domain": DELIVERY_PHOTO_DOMAIN},
"stock_quant": {"domain": [("location_id.usage", "=", "internal")], "scope": "internal"},
"product_pricelist": {"domain": []},
"product_pricelist_item": {"domain": []},
"product_tag": {"domain": []},
}
REPLACE_PAGE = 5000
REPLACE_READ_ATTEMPTS = 3
_connector_spec.register("odoo", _connector_spec.tables_for("odoo"),
mirror=_connector_spec.ConnectorMirrorSpec(tuple(ENTITIES)))
for _entity_name, _entity_spec in ENTITIES.items():
_entity_spec.update(_connector_spec.mirror_table_keys(_entity_name))
def _cols(spec):
cols = ["id BIGINT PRIMARY KEY", "Store VARCHAR", "_connection_id VARCHAR", "_source_id BIGINT"]
for f in spec["fields"]:
base = f.removesuffix("_id")
if f in (spec.get("named_m2m") or {}):
cols.append(f"{f} VARCHAR")
elif f.endswith("_id") or f in MANY2ONE_FIELDS:
cols += [f"{base}_id BIGINT", f"{base}_name VARCHAR"]
elif f in BOOL_FIELDS:
cols.append(f"{f} BOOLEAN")
elif f in INT_FIELDS:
cols.append(f"{f} BIGINT")
elif f in NUMERIC_FIELDS:
cols.append(f"{f} DOUBLE")
elif f == "analytic_distribution":
cols += [f"{f} VARCHAR", f"{f}_details VARCHAR"]
else:
cols.append(f"{f} VARCHAR")
for f in spec.get("m2m") or []:
base = f.removesuffix("_ids")
cols += [f"{base}_id BIGINT", f"{f} VARCHAR"]
cols += [f"{output} VARCHAR" for output in spec.get("raw_m2m") or {}]
return cols
def _ensure_table(con, key, spec):
con.execute(f"CREATE TABLE IF NOT EXISTS {key} ({', '.join(_cols(spec))})")
def _ensure_columns(con, key, spec):
"""Schema evolution: existing stores/seeds predate columns a newer spec introduces β
CREATE TABLE IF NOT EXISTS won't add them. Returns the newly added column names."""
have = {r[1] for r in con.execute(f"PRAGMA table_info('{key}')").fetchall()}
added = []
for c in _cols(spec):
name = c.split()[0]
if name not in have:
con.execute(f"ALTER TABLE {key} ADD COLUMN {c}")
added.append(name)
binding = getattr(_SYNC_TLS, "connection", None)
if binding and binding["legacy"]:
con.execute(f"UPDATE {key} SET _connection_id=?, _source_id=id, Store=? WHERE _connection_id IS NULL",
[binding["id"], binding["label"]])
return added
def _field_cols(f, *, named_m2m=None):
"""The store column(s) one spec field materialises into (m2o fields become _id + _name)."""
if f in (named_m2m or {}):
return [f]
if f == "analytic_distribution":
return [f, f + "_details"]
if f.endswith("_id") or f in MANY2ONE_FIELDS:
base = f.removesuffix("_id")
return [f"{base}_id", f"{base}_name"]
return [f]
# Stop before the next source page when the time budget or load window expires.
COL_BACKFILL_SECONDS = 300.0
COL_BACKFILL_WINDOW_SECONDS = 3 * 3600
COL_BACKFILL_STOP_AT = None
_fill_clock = time.monotonic
def _fill_deadline(max_seconds=None):
"""Use elapsed seconds and the optional epoch load-window end, whichever expires first."""
seconds = COL_BACKFILL_SECONDS if max_seconds is None else max_seconds
if COL_BACKFILL_STOP_AT is not None:
seconds = min(float(seconds), max(0.0, float(COL_BACKFILL_STOP_AT) - time.time()))
return _fill_clock() + max(0.0, float(seconds))
def _backfill_columns(con, key, spec, log=print, max_seconds=None, *, repeat=False):
"""Scalar/m2o schema evolution β the twin of _backfill_m2m, and for the same reason.
Each pass uses a seconds budget and resumes from its durable field markers.
_ensure_columns ALTERs a new column in, but existing rows NEVER re-sync (WE moved the schema;
their write_date didn't move), so a newly added field stays NULL forever β silently, and NULL
is indistinguishable from a legitimately empty value. That is how a store starts answering
'no rows match' instead of erroring. Re-pull just the affected fields for every row,
paginated by id, marked durably so a crash retries rather than declaring completion.
The fields are DECLARED in the spec (`backfill_fields`), never inferred from "which columns
did _ensure_columns just add" β that inference is wrong on the second run, when the ALTER has
already happened and the list comes back empty while the data is still missing. Declaring it
makes the migration re-entrant and reviewable. Leave the entry in place after it completes;
the durable marker, not the absence of the declaration, is what stops it re-running.
`repeat=True` observes declared `refresh_fields` independently of write_date.
Its separate marker resumes incomplete cycles and restarts completed ones;
it never resets a schema migration or advances the entity's delta cursor.
Returns the number of existing rows changed by a recurring refresh, so the
resident rebuild policy cannot mistake a rank-only change for a no-op.
"""
fields = [f for f in (spec.get("refresh_fields" if repeat else "backfill_fields") or [])
if f != "write_date"]
if not fields:
return 0
marker = _connection_state_key(f"{key}.{'refresh' if repeat else 'cols'}.{','.join(sorted(fields))}")
full_marker, remaining_progress = marker, []
operation = "column refresh" if repeat else "column backfill"
st = con.execute("SELECT phase, last_id, rows FROM _sync_state WHERE entity=?",
[marker]).fetchone()
if repeat and st and st[0] == "done":
st = None
if not repeat:
# Field checkpoints own progress. Joined markers remain summaries for older readers.
prefix = _connection_state_key(f"{key}.cols.")
states = _column_backfill_states(con, key)
if all(states.get(field, (None,))[0] == "done" for field in fields):
_checkpoint_column_fields(con, key, [], "done", 0, 0)
return 0
for field in fields:
phase, last_id, count = states.get(field, ("running", 0, 0))
_checkpoint_column_fields(con, key, [field], phase, last_id, count)
completed = {field for field, point in states.items() if point[0] == "done"}
fields = [field for field in fields if field not in completed]
if not fields:
last_id, count = max((point[1:] for point in states.values()), default=(0, 0))
con.execute("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
[marker, "done", last_id, None, count, time.strftime("%Y-%m-%d %H:%M:%S")])
return 0
progress = {field: states.get(field, ("running", 0, 0))[1:] for field in fields}
# Finish the furthest-ahead cohort first. A new field at zero must not rewind
# another field's running migration; each cohort keeps its own durable marker.
last_id, count = max(progress.values())
remaining_progress = [point for point in progress.values() if point[0] != last_id]
fields = [field for field in fields if progress[field][0] == last_id]
marker = prefix + ','.join(sorted(fields))
st = ("running", last_id, count)
# RESUMABLE, not restart-from-zero: these passes run to ~1M rows, and a migration that loses
# an hour of work to one dropped connection gets skipped by the next person under time
# pressure. Progress is checkpointed every page; 'done' is written only at the end, so an
# interrupted run resumes at the last committed id instead of re-pulling or, worse, declaring
# completion it never reached.
last, n = (st[1] or 0, st[2] or 0) if st else (0, 0)
log(f" {key}: {operation} {'+'.join(fields)} starting at id {last:,} ({n:,} rows)")
if last:
log(f" {key}: {operation} {'+'.join(fields)} resuming at id {last:,} ({n:,} rows already done)")
raw_m2m = spec.get("raw_m2m") or {}
requested = list(dict.fromkeys(raw_m2m.get(f, f) for f in fields))
flattened = [f for f in fields if f not in raw_m2m]
cols = [c for f in fields for c in _field_cols(f, named_m2m=spec.get("named_m2m"))]
deadline, changed = _fill_deadline(max_seconds), 0
while True:
if _fill_clock() >= deadline:
log(f" {key}: {operation} PAUSED at {n:,} rows (time bound; resumes)")
return changed
recs = O.search_read(spec["model"], _base_domain(spec) + [("id", ">", last)],
requested, limit=5000, order="id asc")
if not recs:
break
if any(field not in rec for rec in recs for field in requested):
raise RuntimeError(f"{key}: column backfill response omitted a requested field")
rows = _flatten_records(recs, flattened, named_m2m=spec.get("named_m2m"), raw_m2m=raw_m2m, spec=spec)
# One set-based write per source page also bounds million-row migration elapsed time;
# executemany paid a DuckDB transaction for each individual row.
values = ','.join('(' + ','.join('?' for _ in range(len(cols) + 1)) + ')'
for _ in rows)
assignments = ', '.join(f'{c}=src.{c}' for c in cols)
differs = ' OR '.join(f'dest.{c} IS DISTINCT FROM src.{c}' for c in cols)
next_id, next_count = recs[-1]["id"], n + len(recs)
con.execute("BEGIN TRANSACTION")
try:
updated = con.execute(
f'UPDATE {key} AS dest SET {assignments} '
f'FROM (VALUES {values}) AS src(id, {", ".join(cols)}) '
f'WHERE dest.id=src.id AND ({differs}) RETURNING dest.id',
[value for row in rows for value in [row['id']] + [row.get(c) for c in cols]]
).fetchall()
con.execute("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
[marker, "running", next_id, None, next_count, time.strftime("%Y-%m-%d %H:%M:%S")])
if not repeat:
_checkpoint_column_fields(con, key, fields, "running", next_id, next_count)
if full_marker != marker:
full_last, full_count = min([*remaining_progress, (next_id, next_count)])
con.execute("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
[full_marker, "running", full_last, None, full_count,
time.strftime("%Y-%m-%d %H:%M:%S")])
con.execute("COMMIT")
except BaseException:
con.execute("ROLLBACK")
raise
if repeat:
changed += len(updated)
last, n = next_id, next_count
log(f" {key}: {operation} {'+'.join(fields)} -> id {last:,} ({n:,} rows)")
con.execute("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
[marker, "done", last, None, n, time.strftime("%Y-%m-%d %H:%M:%S")])
if full_marker != marker:
phase = "running" if remaining_progress else "done"
full_last, full_count = min(remaining_progress) if remaining_progress else (last, n)
con.execute("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
[full_marker, phase, full_last, None, full_count,
time.strftime("%Y-%m-%d %H:%M:%S")])
if not repeat:
_checkpoint_column_fields(con, key, fields, "done", last, n)
log(f" {key}: {operation} complete -> {n:,} rows")
return changed
def _backfill_m2m(con, key, spec, log=print, max_seconds=None):
"""Commit each relation page with its cursor and resume until an empty page proves completion."""
deadline = _fill_deadline(max_seconds)
for f in spec.get("m2m") or []:
marker = _connection_state_key(f"{key}.m2m.{f}")
st = con.execute("SELECT phase, last_id, rows FROM _sync_state WHERE entity=?",
[marker]).fetchone()
if st and st[0] == "done":
continue
base = f.removesuffix("_ids")
last, count = (st[1] or 0, st[2] or 0) if st else (0, 0)
log(f" {key}: m2m {f} starting at id {last:,} ({count:,} rows)")
if last:
log(f" {key}: m2m {f} resuming at id {last:,} ({count:,} rows)")
while True:
if _fill_clock() >= deadline:
log(f" {key}: m2m {f} PAUSED at {count:,} rows (time bound; resumes)")
return
recs = O.search_read(spec["model"],
_base_domain(spec) + [(f, "!=", False), ("id", ">", last)],
[f], limit=20000, order="id asc")
if not recs:
break
if any(f not in rec for rec in recs):
raise RuntimeError(f"{key}: m2m backfill response omitted {f}")
rows = _flatten_records(recs, [], m2m=[f], spec=spec)
values = ','.join('(?,?,?)' for _ in rows)
next_id, next_count = recs[-1]["id"], count + len(recs)
con.execute("BEGIN TRANSACTION")
try:
con.execute(f"UPDATE {key} AS dest SET {base}_id=src.first_id, {f}=src.ids "
f"FROM (VALUES {values}) AS src(id, first_id, ids) WHERE dest.id=src.id",
[v for row in rows for v in (row["id"], row[f"{base}_id"], row[f])])
con.execute("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
[marker, "running", next_id, None, next_count,
time.strftime("%Y-%m-%d %H:%M:%S")])
con.execute("COMMIT")
except BaseException:
con.execute("ROLLBACK")
raise
last, count = next_id, next_count
log(f" {key}: m2m {f} -> id {last:,} ({count:,} rows)")
con.execute("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
[marker, "done", last, None, count, time.strftime("%Y-%m-%d %H:%M:%S")])
log(f" {key}: m2m backfill {f} complete -> {count:,} rows")
def _upsert(con, key, spec, recs):
if not recs:
return
rows = _flatten_records(recs, spec["fields"], spec.get("m2m") or (),
named_m2m=spec.get("named_m2m"), raw_m2m=spec.get("raw_m2m"), spec=spec)
colnames = [c.split()[0] for c in _cols(spec)]
ids = [r["id"] for r in rows]
con.execute(f"DELETE FROM {key} WHERE id IN ({','.join(map(str, ids))})")
con.executemany(
f"INSERT INTO {key} ({', '.join(colnames)}) VALUES ({', '.join('?' for _ in colnames)})",
[[r.get(c) for c in colnames] for r in rows])
def _state(con, key):
row = con.execute("SELECT phase, last_id, cursor_wd, rows FROM _sync_state WHERE entity=?",
[_connection_state_key(key)]).fetchone()
return {"phase": row[0], "last_id": row[1], "cursor_wd": row[2], "rows": row[3]} if row else None
def _save_state(con, key, phase, last_id, cursor_wd):
n = con.execute(f"SELECT count(*) FROM {key} WHERE {_connection_where()}").fetchone()[0]
con.execute("""INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)""",
[_connection_state_key(key), phase, last_id, cursor_wd, n, time.strftime("%Y-%m-%d %H:%M:%S")])
def _base_domain(spec):
return list(spec.get("domain") or ()) + ([("active", "in", [True, False])] if spec["archivable"] else [])
def sync_entity(key, max_batches=200, batch=BATCH, log=print, report=None):
"""Publish a single requested entity through the same source boundary as a full pass."""
if source_paused():
return _sync_entity(key, max_batches=max_batches, batch=batch, log=log)
candidate = getattr(_SYNC_TLS, "candidate", None)
if candidate is not None:
phase, pulled = _sync_one(key, max_batches=max_batches, batch=batch, log=log, report=report)
# A recurring refresh may report changed pages to a bespoke cursor, but a
# tenant pass cannot claim completion until those private pages finish too.
pending = _connection_pending(candidate["con"], key)
if phase == "live" and any(row[1] != "done" for row in pending if not row[0].endswith(".pass")):
phase = "backfill"
return phase, pulled
rt = _resident_odoo_runtime()
if rt is None:
return _sync_one(key, max_batches=max_batches, batch=batch, log=log, report=report)
report = sync_all(entities=[key], max_batches=max_batches, log=log, batch=batch)
return report[key]["phase"], report[key]["pulled"]
def _sync_entity(key, max_batches=200, batch=BATCH, log=print):
"""One bounded, resumable sync pass for an entity. Backfill (by id) β live (by write_date).
Returns (phase, pulled_rows). The count includes existing rows changed by
recurring scalar refreshes as well as rows pulled by the delta/backfill.
β D-10: refuses BEFORE opening a cursor or touching Odoo while the connector is paused. The
guard is here β the innermost function that reads from `O` β rather than only in `sync_all`,
so a caller that syncs ONE entity is covered by construction instead of by remembering to
add the same check. `sync_all` short-circuits too, but only to avoid eight identical notices.
"""
if source_paused():
_paused_notice(f"sync of {key}", log)
return PAUSED_PHASE, 0
spec = ENTITIES[key]
_connection_model(spec)
con = connect()
try:
_ensure_table(con, key, spec)
_ensure_columns(con, key, spec)
_backfill_columns(con, key, spec, log=log, max_seconds=COL_BACKFILL_SECONDS)
if spec.get("backfill_before_delta") and not _columns_backfilled(con, key, spec):
return "backfill", 0
columns_ready = (not spec.get("backfill_fields") or _columns_backfilled(con, key, spec))
_backfill_m2m(con, key, spec, log=log, max_seconds=COL_BACKFILL_SECONDS)
columns_ready = columns_ready and all(
con.execute("SELECT phase FROM _sync_state WHERE entity=?", [_connection_state_key(f"{key}.m2m.{field}")]).fetchone()
== ("done",) for field in spec.get("m2m") or [])
st = _state(con, key) or {"phase": "backfill", "last_id": 0, "cursor_wd": None, "rows": 0}
fields = spec["fields"] + list(spec.get("m2m") or [])
pulled, refreshed = 0, 0
if st["phase"] != "backfill":
_save_state(con, key, "catchup", st["last_id"], st["cursor_wd"])
if st["phase"] != "backfill":
refreshed = _backfill_columns(con, key, spec, log=log,
max_seconds=COL_BACKFILL_SECONDS, repeat=True)
if st["phase"] == "backfill":
last_id = st["last_id"] or 0
log(f" {key}: backfill starting at id {last_id:,} ({st['rows']:,} rows)")
for i in range(max_batches):
dom = _base_domain(spec) + [("id", ">", last_id)]
recs = O.search_read(spec["model"], dom, ["id"] + fields,
limit=batch, order="id asc")
if not recs:
# backfill complete β switch to live mode anchored at max write_date seen
wd = con.execute(f"SELECT max(write_date) FROM {key} WHERE {_connection_where()}").fetchone()[0]
_save_state(con, key, "live", last_id, wd or "1970-01-01 00:00:00")
log(f" {key}: BACKFILL COMPLETE ({st['rows'] + pulled:,} rows) -> live mode")
return ("live" if columns_ready else "backfill"), pulled
_upsert(con, key, spec, recs)
last_id = recs[-1]["id"]
pulled += len(recs)
_save_state(con, key, "backfill", last_id, None)
log(f" {key}: backfill β¦{pulled:,} rows (idβ€{last_id})")
log(f" {key}: backfill PAUSED at {pulled:,} rows this run (bounded; resumes)")
return "backfill", pulled
# live mode: write_date >= cursor (idempotent overlap; upsert dedupes). MASS-STAMP
# FALLBACK (found live 2026-07-17): a batch job can stamp >batch rows with ONE
# write_date second (an Odoo recompute stamped ~88k account_move rows '2026-07-15
# 16:35:03'; stored values carry microseconds, XML-RPC strings don't) β then the
# write_date cursor can NEVER advance and every pass re-pulls the same first page
# forever. When a full batch leaves the cursor unchanged we switch to an ID-WALK over
# write_date >= cursor (order id asc, id > tie), checkpointed as 'cursor|<id>' in
# cursor_wd so a killed walk resumes; the third component preserves its maximum date.
# On completion the cursor jumps one second past
# the stamp (safe: the walk covered every row in that second, and the stamp is in the
# past β guarded by a 2-minute recency check).
raw = st["cursor_wd"] or "1970-01-01 00:00:00"
cursor, _pipe, _tie = raw.partition("|")
_tie, _max_pipe, saved_max = _tie.partition("|")
walking, tie_id = bool(_pipe), int(_tie or 0)
walk_max = max(cursor, saved_max) if _max_pipe else cursor
if walking and not _max_pipe:
# Old checkpoints lack the walk's maximum, including at an exhausted tail.
# Only this connection's covered source IDs can supply the missing value;
# another connection's mirror IDs or unread rows must not advance the cursor.
covered_max = con.execute(
f"SELECT max(write_date) FROM {key} WHERE {_connection_where()} "
"AND coalesce(_source_id, id) <= ?", [tie_id]).fetchone()[0]
walk_max = max(walk_max, str(covered_max)) if covered_max else walk_max
delta_complete = True
for _ in range(max_batches):
if walking:
dom = _base_domain(spec) + [("write_date", ">=", cursor), ("id", ">", tie_id)]
recs = O.search_read(spec["model"], dom, ["id"] + fields,
limit=batch, order="id asc")
if not recs: # walk complete β advance past the stamp
try:
wm = dt.datetime.strptime(walk_max, "%Y-%m-%d %H:%M:%S")
if wm < dt.datetime.utcnow() - dt.timedelta(minutes=2):
cursor = (wm + dt.timedelta(seconds=1)).strftime("%Y-%m-%d %H:%M:%S")
else:
cursor = walk_max
except ValueError:
cursor = walk_max
walking = False
log(f" {key}: id-walk complete -> cursor {cursor}")
break
_upsert(con, key, spec, recs)
pulled += len(recs)
tie_id = recs[-1]["id"]
walk_max = max(walk_max, max(str(r["write_date"]) for r in recs))
_save_state(con, key, "catchup", st["last_id"], f"{cursor}|{tie_id}|{walk_max}")
continue
dom = _base_domain(spec) + [("write_date", ">=", cursor)]
recs = O.search_read(spec["model"], dom, ["id"] + fields,
limit=batch, order="write_date asc, id asc")
if not recs or (len(recs) == 1 and str(recs[0].get("write_date")) == cursor
and pulled == 0 and _already(con, key, recs[0])):
break
new_cursor = str(recs[-1]["write_date"])
if new_cursor == cursor and len(recs) == batch:
# full batch stuck on one second: enter the id-walk from id 0 (the write_date
# ordering shuffles ids within the stamp second, so the batch just pulled is
# NOT an id prefix β restart coverage by id; upserts make the overlap free)
walking, tie_id, walk_max = True, 0, cursor
_save_state(con, key, "catchup", st["last_id"], f"{cursor}|0")
log(f" {key}: mass-stamp second at {cursor} -> switching to id-walk")
continue
_upsert(con, key, spec, recs)
pulled += len(recs)
if len(recs) < batch:
cursor = new_cursor
break
cursor = new_cursor
# Checkpoint the cursor EVERY batch: a big delta (88k account_move rows after the
# 2026-07-15 recompute) means a killed pass otherwise re-pulls everything β
# upserts are idempotent, but the wasted RPC is real (learned 2026-07-17).
_save_state(con, key, "catchup", st["last_id"], cursor)
else:
delta_complete = False
_save_state(con, key, "live" if delta_complete else "catchup", st["last_id"],
f"{cursor}|{tie_id}|{walk_max}" if walking else cursor)
if pulled:
log(f" {key}: live sync +{pulled:,} rows (cursor {cursor}"
+ (f" walking id>{tie_id}" if walking else "") + ")")
pending = _connection_pending(con, key)
# Recurring refreshes have their own bounded cycle, separate from schema
# readiness. The public source wrapper also requires that cycle to finish.
columns_ready = columns_ready and all(phase == "done" for marker, phase in pending
if not marker.startswith(_connection_state_key(key + ".refresh.")))
return ("live" if columns_ready and delta_complete else "backfill"), pulled + refreshed
finally:
con.close()
def _sync_one(key, max_batches=200, batch=BATCH, log=print, report=None):
"""One entity's pass: the cursor sync, the full replace, or the cursor sync then the replace."""
_connection_model(ENTITIES[key])
plan = REPLACE_EACH_PASS.get(key)
if plan is None or source_paused():
return _sync_entity(key, max_batches=max_batches, batch=batch, log=log)
if plan.get("scope") != "internal":
return _replace_entity(key, log=log, report=report)
phase, pulled = _sync_entity(key, max_batches=max_batches, batch=batch, log=log)
replaced, changed = _replace_entity(key, log=log, report=report)
return (phase if replaced == "live" else replaced), pulled + changed
def _replace_scope(con, key, plan):
"""SQL predicate for the mirror rows a replace owns. `_replace_stage` holds the fresh read."""
if plan.get("scope") != "internal":
return _connection_where()
terms = ["id IN (SELECT id FROM _replace_stage)",
"location_id IN (SELECT location_id FROM _replace_stage WHERE location_id IS NOT NULL)"]
if {"id", "usage"} <= set(columns_of("stock_location", cur=con) or ()):
terms.append("location_id IN (SELECT id FROM stock_location WHERE usage = 'internal')")
return "(" + " OR ".join(terms) + ") AND " + _connection_where()
def _read_replace_rows(key, spec, domain):
"""Read uncapped pages again if neither surrounding count matches, with a fixed attempt bound."""
fields = spec["fields"] + list(spec.get("m2m") or [])
for _ in range(REPLACE_READ_ATTEMPTS):
expected = int(O.get_odoo().search_count(spec["model"], domain))
rows, last = [], 0
while True:
recs = O.search_read(spec["model"], domain + [("id", ">", last)], ["id"] + fields,
limit=REPLACE_PAGE, order="id asc")
if any(field not in rec for rec in recs for field in fields):
raise RuntimeError(f"{key}: full read omitted a requested field. The mirror keeps its previous rows.")
rows += _flatten_records(recs, spec["fields"], spec.get("m2m") or (),
named_m2m=spec.get("named_m2m"), raw_m2m=spec.get("raw_m2m"), spec=spec)
# Only an empty page ends a read, since Odoo may return short pages before the end.
if not recs:
break
last = recs[-1]["id"]
if len(rows) == expected:
return rows
after = int(O.get_odoo().search_count(spec["model"], domain))
if len(rows) == after:
return rows
return None
def _replace_entity(key, log=print, report=None):
"""Replace a complete read atomically, or hold a moving entity before changing rows or state."""
if source_paused():
_paused_notice(f"full read of {key}", log)
return PAUSED_PHASE, 0
spec, plan = ENTITIES[key], REPLACE_EACH_PASS[key]
rows = _read_replace_rows(key, spec, list(plan["domain"]))
if rows is None:
if report is not None:
report["cause"] = {"classifier": "odoo_entity_moving", "entity": key,
"attempts": REPLACE_READ_ATTEMPTS}
log(f" {key}: Odoo changed during all {REPLACE_READ_ATTEMPTS} full reads. Previous rows are kept.")
return "backfill", 0
con = connect()
try:
_ensure_table(con, key, spec)
_ensure_columns(con, key, spec)
colnames = [c.split()[0] for c in _cols(spec)]
cols = ", ".join(colnames)
con.execute(f"CREATE OR REPLACE TEMP TABLE _replace_stage AS SELECT {cols} FROM {key} LIMIT 0")
if rows:
con.executemany(f"INSERT INTO _replace_stage ({cols}) VALUES ({', '.join('?' for _ in colnames)})",
[[r.get(c) for c in colnames] for r in rows])
scope = _replace_scope(con, key, plan)
changed = con.execute(
f"SELECT count(DISTINCT id) FROM ((SELECT {cols} FROM _replace_stage "
f"EXCEPT SELECT {cols} FROM {key} WHERE {scope}) UNION ALL "
f"(SELECT {cols} FROM {key} WHERE {scope} EXCEPT SELECT {cols} FROM _replace_stage))"
).fetchone()[0]
if changed:
con.execute("BEGIN TRANSACTION")
try:
con.execute(f"DELETE FROM {key} WHERE {scope}")
con.execute(f"INSERT INTO {key} ({cols}) SELECT {cols} FROM _replace_stage")
except BaseException:
con.execute("ROLLBACK")
raise
con.execute("COMMIT")
if plan.get("scope") == "internal":
n = con.execute(f"SELECT count(*) FROM {key} WHERE {_connection_where()}").fetchone()[0]
con.execute("UPDATE _sync_state SET rows=? WHERE entity=?", [n, _connection_state_key(key)])
else:
_save_state(con, key, "live", max((r.get("_source_id", r["id"]) for r in rows), default=0), None)
fields = [field for field in spec.get("backfill_fields", ()) if field != "write_date"]
if fields:
_checkpoint_column_fields(con, key, fields, "done",
max((r.get("_source_id", r["id"]) for r in rows), default=0), len(rows))
log(f" {key}: full read {len(rows):,} rows, {changed:,} changed")
return "live", changed
finally:
try:
con.execute("DROP TABLE IF EXISTS _replace_stage")
finally:
con.close()
def _already(con, key, rec):
ident = _connection_ids(con, ENTITIES[key]["model"], [rec["id"]])[rec["id"]]
return bool(con.execute(f"SELECT 1 FROM {key} WHERE id=?", [ident]).fetchone())
def sync_all(entities=None, max_batches=200, log=print, after_entity=None, batch=BATCH, connection_id=None,
client_factory=None):
selected = list(ENTITIES) if entities is None else list(entities)
if ("stock_picking" in selected and "delivery_photo_metadata" in ENTITIES
and "delivery_photo_metadata" not in selected):
selected.append("delivery_photo_metadata")
if entities is not None and not selected:
return {}
# β D-10. `{}` rather than a dict of paused phases, and the caller decides why that matters:
# `api/main.py:354` loops up to 12 times until `all(phase == "live")`, and `all()` over an
# empty dict is True β so a paused store costs ONE pass instead of twelve rounds of the same
# refusal. A dict of `"paused"` phases would fail that test every time and turn the boot
# sprint into a busy-wait against a connector nobody has resumed.
if source_paused():
_paused_notice("sync", log)
return {}
rt = _resident_odoo_runtime()
entries = _odoo_entries(rt) if rt is not None else None
if entries == []:
raise RuntimeError("This tenant has no Odoo credentials. Reconnect Odoo before syncing.")
if connection_id not in (None, "default") and entries is not None and not any(e["id"] == connection_id for e in entries):
raise RuntimeError("The requested Odoo connection is no longer connected to this tenant")
if rt is not None and restore_odoo_mirror(rt.key, rt, log=log)["state"] == "refused":
raise RuntimeError("Odoo saved mirror was refused; repair store access and restart before syncing")
out = {}
from harness import mirror_store
publication = (mirror_store.sync_mirror("odoo", rt.key, rt, report=out, log=log, expected_tables=selected)
if rt is not None else nullcontext())
with publication:
candidate = getattr(_SYNC_TLS, "candidate", None)
if candidate is not None:
selected = [key for key in selected if key in candidate["requested"]]
if entries is not None and not candidate.get("resuming"):
for key in selected:
candidate["con"].execute(
"DELETE FROM _sync_state WHERE entity=? OR "
"(starts_with(entity, ?) AND ends_with(entity, '.pass'))",
[key + ".pass", key + ".connection."])
for entry in entries if entries is not None else [None]:
binding = _odoo_connection(candidate["con"], rt, entry, client_factory) if entry is not None else nullcontext()
with binding:
for key in selected:
t0 = time.time()
row = out.setdefault(key, {"phase": "live", "pulled": 0, "secs": 0})
marker = _connection_state_key(key + ".pass")
previous = candidate["con"].execute("SELECT phase FROM _sync_state WHERE entity=?", [marker]).fetchone() if entry else None
if entry and candidate.get("resuming") and previous == ("done",):
phase, pulled = "live", 0
else:
if entry:
candidate["con"].execute("DELETE FROM _sync_state WHERE entity=?", [marker])
phase, pulled = sync_entity(key, max_batches=max_batches, log=log,
**({"batch": batch} if batch != BATCH else {}),
**({"report": row} if candidate is not None else {}))
if entry and phase == "live":
candidate["con"].execute("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
[marker, "done", 0, None, 0, time.strftime("%Y-%m-%d %H:%M:%S")])
if phase != "live":
row["phase"] = phase
row["pulled"] += pulled
row["secs"] = round(row["secs"] + time.time() - t0, 1)
if rt is not None and candidate is not None:
_record_column_fill(rt, candidate["con"], key, ENTITIES[key])
if after_entity is not None:
try:
after_entity(key)
except Exception as exc: # noqa: BLE001
log(f" {key}: after-entity hook failed ({type(exc).__name__})")
if rt is not None and out:
persist_odoo_mirror(rt.key, rt, out, log=log)
# A full pass that lands every entity in live mode latches process readiness in-process β so
# readers never race the writer's lock to (re)discover the store is usable.
if entities is None and out and all(v["phase"] == "live" for v in out.values()) and ready():
mark_ready()
return out
def reconcile_deletes(entities=None, log=print):
"""Reconcile source deletions privately and publish the completed source snapshot."""
if source_paused():
_paused_notice("delete reconcile", log)
return {}
rt = _resident_odoo_runtime()
if rt is None or getattr(_SYNC_TLS, "candidate", None) is not None:
return _reconcile_deletes_candidate(entities=entities, log=log)
from harness import mirror_store
report = {}
selected = list(ENTITIES) if entities is None else list(entities)
with mirror_store.sync_mirror("odoo", rt.key, rt, report=report, log=log, expected_tables=selected):
candidate = _SYNC_TLS.candidate
selected = [key for key in selected if key in candidate["requested"]]
entries = _odoo_entries(rt)
if entries == []:
raise RuntimeError("This tenant has no Odoo credentials. Reconnect Odoo before syncing.")
result = {}
for entry in entries if entries is not None else [None]:
binding = _odoo_connection(candidate["con"], rt, entry) if entry is not None else nullcontext()
with binding:
for key, count in _reconcile_deletes_candidate(entities=selected, log=log).items():
result[key] = result.get(key, 0) + count
report.update({key: {"phase": "live"} for key in result})
return result
def _reconcile_deletes_candidate(entities=None, log=print):
"""Remove store rows hard-deleted in Odoo. unlink() doesn't move write_date, so the cursor
sync can never see a deletion β the count parity DETECTS the drift (by design); this repairs
it. Pulls the full live id set per entity in bounded id-ascending pages (id-only reads are
cheap) and deletes store ids not present live. Run from sync_runner / maintenance, not the
app's frequent passes.
β D-10: it pulls the FULL live id set per entity, so it is the single largest Odoo read in
this module and it runs unattended at boot and every fourth resync pass. Paused means paused.
"""
if source_paused():
_paused_notice("delete reconcile", log)
return {}
con = connect()
try:
out = {}
for key in (list(ENTITIES) if entities is None else entities):
spec = ENTITIES[key]
# HOTFIX 2026-09-29: DETECT FIRST, then walk. The cursor sync has already pulled every
# created row, so the mirror can only hold MORE rows than the source when something
# was deleted: equal counts prove there is nothing to purge, and the full id walk
# (1M+ ids for account_move_line) runs only for an entity whose count moved.
try:
source_n = O.get_odoo().search_count(spec["model"], _base_domain(spec))
mirror_n = con.execute(f"SELECT count(*) FROM {key} WHERE {_connection_where()}").fetchone()[0]
except Exception: # noqa: BLE001
source_n, mirror_n = None, None
if source_n is not None and int(source_n) == int(mirror_n):
out[key] = 0
continue
live_ids, last = set(), 0
while True:
recs = O.search_read(spec["model"], _base_domain(spec) + [("id", ">", last)],
["id"], limit=50000, order="id asc")
if not recs:
break
live_ids.update(_connection_ids(con, spec["model"], [r["id"] for r in recs]).values())
last = recs[-1]["id"]
store_ids = {r[0] for r in con.execute(f"SELECT id FROM {key} WHERE {_connection_where()}").fetchall()}
dead = store_ids - live_ids
if dead:
con.execute("CREATE TEMP TABLE IF NOT EXISTS _dead(id BIGINT)")
con.execute("DELETE FROM _dead")
con.executemany("INSERT INTO _dead VALUES (?)", [[i] for i in dead])
con.execute(f"DELETE FROM {key} WHERE id IN (SELECT id FROM _dead)")
con.execute("DELETE FROM _dead")
n = con.execute(f"SELECT count(*) FROM {key} WHERE {_connection_where()}").fetchone()[0]
con.execute("UPDATE _sync_state SET rows=? WHERE entity=?", [n, _connection_state_key(key)])
out[key] = len(dead)
log(f" {key}: reconciled {len(dead):,} hard-deleted rows")
if any(out.values()):
rt = _resident_odoo_runtime()
if rt is not None:
persist_odoo_mirror(rt.key, rt, [key for key, removed in out.items() if removed], log=log)
return out
finally:
con.close()
def status():
con = connect()
try:
return {r[0]: {"phase": r[1], "cursor": r[3], "rows": r[4], "updated": r[5]}
for r in con.execute("SELECT * FROM _sync_state").fetchall()}
finally:
con.close()
# ------------------------------------------------------------------ parity (store vs LIVE Odoo)
def _check(name, ours, live, tol=0.01):
gap = abs((ours or 0) - (live or 0))
return {"check": name, "ok": gap <= tol, "gap": round(gap, 4), "ours": ours, "live": live}
def validate(pre=None, months=("2026-01-01", "2026-07-01")):
"""Raw-fidelity parity: the store must mirror live Odoo β row counts per synced entity +
unscoped monetary sums over a window, to the cent. Only checks entities in LIVE phase.
β D-10 REFUSES LOUDLY HERE rather than skipping quietly, and the difference matters. Every
other paused path has a correct silent answer β serve the mirror β because the mirror is the
thing being asked for. This function's ENTIRE contract is "compare me against live Odoo", so
a paused version of it has no honest result: skipping would return an empty pass, and
proceeding would make the live call the pause forbids. An operator running a reconciliation
against a paused connector has asked a question that cannot be answered, and being told so is
the only outcome that is not a lie. (Operator-invoked only β nothing in the serving path
calls it, so this raises for a person, never inside a request.)
"""
if source_paused():
raise RuntimeError(
"the tenant's Odoo connector is PAUSED, so the store cannot be validated against "
"live Odoo - resume it under Settings > Connectors and run this again")
con = connect()
out = []
try:
live_state = status()
d0, d1 = months
for key, spec in ENTITIES.items():
if live_state.get(key, {}).get("phase") != "live":
continue
ours = con.execute(f"SELECT count(*) FROM {key}").fetchone()[0]
live = O.get_odoo().search_count(spec["model"], _base_domain(spec))
out.append(_check(f"store.{key} row count == live search_count", ours, live, tol=0))
if live_state.get("sale_order_line", {}).get("phase") == "live":
ours = con.execute(
"SELECT coalesce(sum(l.price_subtotal),0) FROM sale_order_line l "
"JOIN sale_order o ON o.id = l.order_id "
"WHERE o.date_order >= ? AND o.date_order < ?", [d0, d1]).fetchone()[0]
live = O.sum_field("sale.order.line",
[("order_id.date_order", ">=", f"{d0} 00:00:00"),
("order_id.date_order", "<", f"{d1} 00:00:00")],
"price_subtotal")
out.append(_check(f"store Ξ£ line revenue [{d0}..{d1}) == live (unscoped)", ours, live))
if live_state.get("sale_order", {}).get("phase") == "live":
ours = con.execute(
"SELECT coalesce(sum(amount_untaxed),0) FROM sale_order "
"WHERE date_order >= ? AND date_order < ?", [d0, d1]).fetchone()[0]
live = O.sum_field("sale.order",
[("date_order", ">=", f"{d0} 00:00:00"),
("date_order", "<", f"{d1} 00:00:00")], "amount_untaxed")
out.append(_check(f"store Ξ£ order amount_untaxed [{d0}..{d1}) == live (unscoped)",
ours, live))
finally:
con.close()
return out
if __name__ == "__main__":
args = sys.argv[1:]
if args and args[0] == "status":
for k, v in status().items():
print(f" {k:18s} {v['phase']:9s} rows={v['rows']:>9,} cursor={v['cursor']} ({v['updated']})")
elif args and args[0] == "reconcile":
ents = args[1].split(",") if len(args) > 1 else None
for k, n in reconcile_deletes(ents).items():
print(f" {k:18s} -{n:,} hard-deleted rows")
elif args and args[0] == "validate":
ok = True
for c in validate():
mark = "OK " if c["ok"] else "XX "
ok &= c["ok"]
print(f" {mark}{c['check'][:80]} gap={c['gap']}")
print("PARITY:", "GREEN" if ok else "RED")
sys.exit(0 if ok else 1)
else:
ents = args[0].split(",") if args else None
mb = int(args[1]) if len(args) > 1 else 200
for k, v in sync_all(ents, max_batches=mb).items():
print(f" {k:18s} {v['phase']:9s} +{v['pulled']:,} rows in {v['secs']}s")
"""harness/store.py β the tenant data store (OM-1, 2026-07-11).
Incremental Odoo β DuckDB mirror: the local analytical store that saved views/dashboards (OM-3)
and the AI Analyst's tools (OM-4) query at warehouse speed, instead of hammering live XML-RPC
per question.
Plan: AutoWiki/research/omni-adoption.md Part IV (OM-1) + [[odoo-open-source]] BUILD-NOW #1.
Design rules:
- The store is a RAW mirror (unscoped). Business scope (wholesale teams, GIFTWARE exclusion,
confirmed-only) is applied by the SEMANTIC layer at query time (OM-2) β one source of truth.
- Sync is CHECKPOINTED + BOUNDED + RESUMABLE (loop-library discipline): backfill paginates by
id; live mode advances a write_date cursor; every run is bounded by max_batches and safe to
kill/re-run (upserts are idempotent).
- validate() compares the store against LIVE Odoo (raw fidelity: counts + monthly sums to the
cent) β the store never validates itself. Odoo hard-deletes (unlink) don't move write_date;
the count checks are the drift detector for those.
- Client data: the .duckdb file lives under data/store/ (git-ignored). READ-ONLY on Odoo.
"""
import datetime as dt
import hashlib
import json
import math
import os
import pickle
import re
import sys
import time
import tempfile
from collections import OrderedDict, deque
from contextlib import contextmanager, nullcontext
from pathlib import Path
import duckdb
import core.odoo as O
from core import request_timing as _request_timing
_STORE_DIR = Path(__file__).resolve().parents[1] / "data" / "store"
# β MUTABLE ON PURPOSE, and read LATE by every function below (Python resolves a module global at
# CALL time), which is how `provision_tenant.py` already points a fresh instance at its own file.
# EXIT-4a (X7) formalises that: `AIOS_DUCKDB_PATH` lets a container name the file without an edit,
# `path_for()` owns the per-tenant naming convention, and `use_path()` is the SAFE way to switch β
# see its docstring for why a bare assignment is not.
# The DEFAULT is unchanged: <app root>/data/store/royal.duckdb.
DB_PATH = Path(os.environ.get("AIOS_DUCKDB_PATH") or (_STORE_DIR / "royal.duckdb"))
_LEGACY_PIN = None
# βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
# β DEBT D-10 (wave 24) β THE CONNECTOR PAUSE REACHES THE MEASURE MIRROR.
#
# WHAT THE BUG WAS. Pausing a tenant's Odoo connector froze the CUSTOMER POOL (DEBT-2 built that:
# `routes_customers._pool_for` serves the pause-time snapshot) and nothing else. Every measure
# column and condition in the product is answered from THIS store, and this store kept pulling
# from Odoo the entire time a connector was paused β 12 sync passes at boot and another every
# 1800 s, hundreds of `search_read`s each. The Settings copy said so out loud ("measures not
# already computed may still reach the source until their cutover") and that sentence was the
# debt row.
#
# β WHY A PROBE AND NOT AN IMPORT. The pause flag lives in the tenant's store bucket, which is
# `harness.runtime`'s to read β and `runtime` already imports THIS module (`_ds.DB_PATH`), so an
# import back at module initialization would be a cycle. The harness installs the pause probe
# at its own import. The frozen-config check below imports the sibling lazily at call time.
#
# β IT FAILS TOWARDS "NOT PAUSED", deliberately and consistently with the shipped decision one
# layer up (`routes_keychain.odoo_paused`: "an unreachable flags bucket must never freeze a live
# surface"). The alternative β assume paused when the answer cannot be resolved β turns any
# transient store glitch into a mirror that silently stops advancing, which is the failure mode
# nobody notices for a week. Two copies of that policy would be worse than one; this is the same
# one.
_paused_probe = None
# Same tenant bucket as the API's `odoo_relational.CONFIG_KEY`; no upward API import is allowed.
_ODOO_CONFIG_KEY = "odoo_connector_config"
def set_paused_probe(fn):
"""Install the callable that answers 'is the connector paused for the tenant whose store this
process has open?'. `harness.runtime` installs the real one; pass None to remove it."""
global _paused_probe
_paused_probe = fn
def source_paused():
"""True when this mirror's tenant has paused or disconnected Odoo. Never raises."""
try:
# Resolve afresh from the current mirror path, using the runtime's one tenant resolver.
# Credential deletion changes the resolved pause id; the durable freeze does not depend
# on that id and must still stop sync after deletion, path switches and process restarts.
from harness import runtime
key = runtime.tenant_for_store_path(DB_PATH)
if key:
rt = runtime.get_runtime(key)
if str(getattr(rt, "key", "") or "") == key:
cfg = rt.get(_ODOO_CONFIG_KEY, fresh=True)
if isinstance(cfg, dict) and cfg.get("frozen"):
return True
except Exception:
pass
# The lazy import also installs the ordinary pause probe for standalone sync workers.