Deploy AIOS web (React glide grid + FastAPI slice)
Browse filesThis view is limited to 50 files because it contains too many changes. Β See raw diff
- RELEASES.json +1 -1
- VERSION +1 -1
- api/ai_review.py +547 -547
- api/automation_engine.py +0 -0
- api/connectors_tt.py +568 -568
- api/deps.py +6 -1
- api/main.py +0 -0
- api/odoo_relational.py +0 -0
- api/providers.py +770 -770
- api/routes_admin.py +61 -15
- api/routes_agent_harness.py +455 -455
- api/routes_alerts.py +668 -668
- api/routes_automation.py +0 -0
- api/routes_grid.py +0 -0
- api/routes_keychain.py +0 -0
- api/routes_nav.py +7 -0
- api/routes_oauth.py +114 -114
- api/routes_publish.py +0 -0
- api/routes_query.py +11 -3
- api/routes_records.py +211 -211
- api/routes_shares.py +393 -393
- api/routes_slack.py +772 -772
- api/routes_statements.py +298 -298
- api/routes_tables.py +0 -0
- api/routes_web_agent.py +84 -84
- platform/aios_grid.py +0 -0
- platform/core/perm_scope.py +44 -0
- platform/core/perms.py +36 -1
- platform/core/registry.py +1 -1
- platform/core/script_sandbox.py +519 -519
- platform/core/shares.py +254 -254
- platform/core/store.py +0 -0
- platform/core/table_store.py +746 -746
- platform/core/user_tables.py +0 -0
- platform/harness/meta_store.py +521 -521
- platform/model/topics/odoo_accounts.yml +60 -60
- platform/model/topics/odoo_agents.yml +59 -59
- platform/model/topics/odoo_bills.yml +86 -86
- platform/model/topics/odoo_customers.yml +213 -213
- platform/model/topics/odoo_invoices.yml +98 -98
- platform/model/topics/odoo_orders.yml +87 -87
- platform/model/topics/odoo_products.yml +147 -147
- platform/model/topics/odoo_vendors.yml +103 -103
- platform/modules/collections_send.py +267 -267
- platform/modules/product_data.py +804 -804
- platform/modules/products.py +737 -737
- web/public/sample_customers.json +558 -558
- web/src/alerts/alertsModel.ts +418 -418
- web/src/assistant/AssistantPage.tsx +748 -748
- web/src/automation/AutomationBuilder.tsx +0 -0
RELEASES.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
-
"current": "
|
| 3 |
"releases": [
|
| 4 |
{
|
| 5 |
"version": "v29",
|
|
|
|
| 1 |
{
|
| 2 |
+
"current": "9445b77",
|
| 3 |
"releases": [
|
| 4 |
{
|
| 5 |
"version": "v29",
|
VERSION
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
|
|
|
|
| 1 |
+
9445b77
|
api/ai_review.py
CHANGED
|
@@ -1,547 +1,547 @@
|
|
| 1 |
-
"""AI REVIEW β let a model decide which stage a card moves to (wave 23, owner ruling R4/R14).
|
| 2 |
-
|
| 3 |
-
β WHAT THIS IS. A review stage holds a record until somebody decides where it goes next. R4 made
|
| 4 |
-
that somebody optionally a MODEL: the engine hands over the record's own values, the review's
|
| 5 |
-
prompt, and the list of stages the review is allowed to send a card to, and gets back ONE of
|
| 6 |
-
those stage labels plus a one-line reason. Every decision is written to the same `reviews` audit
|
| 7 |
-
log a human click writes to, tagged `by: "ai"` with the provider and model that made it.
|
| 8 |
-
|
| 9 |
-
β FAIL-CLOSED IN EVERY DIRECTION, and this is the whole safety story. No key configured, a
|
| 10 |
-
network failure, a slow answer, a malformed answer, or an answer naming a stage the review does
|
| 11 |
-
not offer β all return `("", {...})`, and the caller leaves the card exactly where a human would
|
| 12 |
-
have found it. The feature can be absent, broken or wrong and the worst outcome is a person doing
|
| 13 |
-
the work. Nothing here can move a card somewhere the review does not already permit.
|
| 14 |
-
|
| 15 |
-
β CHEAP FIRST (owner R14, verbatim: *"Claude is a bit too expensive"*). The ladder is
|
| 16 |
-
`groq β cerebras β openrouter β anthropic`; the first CONFIGURED provider wins, and Anthropic is
|
| 17 |
-
last rather than absent β it is the quality backstop, not the default. `AIOS_AI_REVIEW_PROVIDER`
|
| 18 |
-
pins one; `AIOS_AI_MODEL` overrides the model.
|
| 19 |
-
|
| 20 |
-
β WHY RAW HTTP RATHER THAN THE `anthropic` SDK, stated because it is a deliberate deviation from
|
| 21 |
-
the /claude-api skill's default and not an oversight. Three of the four legs are OpenAI-chat-shaped
|
| 22 |
-
endpoints with no shared SDK, so a ladder built on the SDK would be one SDK leg beside three
|
| 23 |
-
hand-rolled ones β two implementations of the same call, the seam this repo keeps closing. And
|
| 24 |
-
`aios-web/api/requirements.txt` is PINNED to what the verify battery proves ([[pin-deps-space-
|
| 25 |
-
rebuilds]]): adding a dependency there rebuilds the container, which is a real deploy risk to take
|
| 26 |
-
for one leg of an optional feature. `requests` is already a dependency and `harness/analyst.py`
|
| 27 |
-
established per-provider raw HTTP as the house pattern. Booked as a DEBT line so the integrator
|
| 28 |
-
can overturn it deliberately rather than by drift.
|
| 29 |
-
|
| 30 |
-
The Messages shape below is the current one: `x-api-key` + `anthropic-version: 2023-06-01`, and
|
| 31 |
-
`stop_reason: "refusal"` is checked BEFORE reading `content` β a refusal answers HTTP 200 with an
|
| 32 |
-
empty content list, so code that indexes `content[0]` unconditionally breaks on it.
|
| 33 |
-
"""
|
| 34 |
-
from __future__ import annotations
|
| 35 |
-
|
| 36 |
-
import json
|
| 37 |
-
import os
|
| 38 |
-
import re
|
| 39 |
-
|
| 40 |
-
import requests
|
| 41 |
-
|
| 42 |
-
#: The ladder. Order IS the policy (R14) β cheapest capable first, Anthropic last.
|
| 43 |
-
PROVIDERS = [
|
| 44 |
-
{"name": "groq", "env": "GROQ_API_KEY", "shape": "openai",
|
| 45 |
-
"url": "https://api.groq.com/openai/v1/chat/completions",
|
| 46 |
-
"model": "llama-3.3-70b-versatile"},
|
| 47 |
-
{"name": "cerebras", "env": "CEREBRAS_API_KEY", "shape": "openai",
|
| 48 |
-
"url": "https://api.cerebras.ai/v1/chat/completions",
|
| 49 |
-
"model": "gpt-oss-120b"},
|
| 50 |
-
{"name": "openrouter", "env": "OPENROUTER_API_KEY", "shape": "openai",
|
| 51 |
-
"url": "https://openrouter.ai/api/v1/chat/completions",
|
| 52 |
-
"model": "openai/gpt-4o-mini"},
|
| 53 |
-
# β haiku-class DELIBERATELY, not the default Opus tier: R14 put Anthropic on this ladder as
|
| 54 |
-
# the backstop for a one-line classification, and this is the cheapest current Claude that
|
| 55 |
-
# does it well. A bigger model here would be spending the owner's money to pick between two
|
| 56 |
-
# labels it already has in front of it.
|
| 57 |
-
{"name": "anthropic", "env": "ANTHROPIC_API_KEY", "shape": "anthropic",
|
| 58 |
-
"url": "https://api.anthropic.com/v1/messages",
|
| 59 |
-
# β AN ENV LEVER, NOT A NEW DEFAULT (W36-T35): haiku stays the choice for the reasons above,
|
| 60 |
-
# and a deployment whose side rungs are out of credit can raise the tier without a release.
|
| 61 |
-
"model": os.environ.get("AIOS_AI_REVIEW_ANTHROPIC_MODEL") or "claude-haiku-4-5"},
|
| 62 |
-
]
|
| 63 |
-
ANTHROPIC_VERSION = "2023-06-01"
|
| 64 |
-
TIMEOUT_SECONDS = float(os.environ.get("AIOS_AI_REVIEW_TIMEOUT") or 20)
|
| 65 |
-
MAX_FIELD_CHARS = 200 # per value handed to the model
|
| 66 |
-
MAX_FIELDS = 30 # columns handed to the model
|
| 67 |
-
MAX_REASON = 200
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
def ladder():
|
| 71 |
-
"""The providers that are actually usable here, in order. Empty = the feature is off."""
|
| 72 |
-
pin = (os.environ.get("AIOS_AI_REVIEW_PROVIDER") or "").strip().lower()
|
| 73 |
-
live = [p for p in PROVIDERS if (os.environ.get(p["env"]) or "").strip()]
|
| 74 |
-
if pin:
|
| 75 |
-
live = [p for p in live if p["name"] == pin]
|
| 76 |
-
return live
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
def configured():
|
| 80 |
-
return bool(ladder())
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
def _record_text(row, fields):
|
| 84 |
-
"""The record, as the model sees it. Values are truncated and the column set is bounded β
|
| 85 |
-
an automation table can carry a 32 KB JSON blob per row (C7) and a review decision does not
|
| 86 |
-
need it. Machine bookkeeping columns are dropped: a stage cell naming the stage the card is
|
| 87 |
-
sitting at would be the model reading its own question back."""
|
| 88 |
-
keys = [k for k in (fields or list((row or {}).keys()))
|
| 89 |
-
if not str(k).startswith("stage_")][:MAX_FIELDS]
|
| 90 |
-
lines = []
|
| 91 |
-
for k in keys:
|
| 92 |
-
v = (row or {}).get(k)
|
| 93 |
-
if v is None or str(v).strip() == "":
|
| 94 |
-
continue
|
| 95 |
-
lines.append(f"{k}: {str(v)[:MAX_FIELD_CHARS]}")
|
| 96 |
-
return "\n".join(lines) or "(this record has no filled-in values)"
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
def _instruction(prompt, options, label):
|
| 100 |
-
return (
|
| 101 |
-
f"You are deciding what happens to one record waiting at a review step called "
|
| 102 |
-
f"{label!r} in a workflow.\n\n"
|
| 103 |
-
f"The person who built this workflow told you: {prompt}\n\n"
|
| 104 |
-
f"Choose EXACTLY ONE of these next steps, by its exact name:\n"
|
| 105 |
-
+ "\n".join(f"- {o}" for o in options)
|
| 106 |
-
+ "\n\nAnswer with one line of JSON and nothing else:\n"
|
| 107 |
-
'{"choice": "<one name from the list above>", "reason": "<one short sentence>"}\n'
|
| 108 |
-
"If the record does not give you enough to decide, answer "
|
| 109 |
-
'{"choice": "", "reason": "why not"} and a person will decide instead.'
|
| 110 |
-
)
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
def _parse(text, options):
|
| 114 |
-
"""The model's line β `(choice, reason)`. A choice that is not one of the offered stages is
|
| 115 |
-
DISCARDED, not fuzzy-matched: the offered list is a permission boundary, and a near-miss
|
| 116 |
-
resolved by string distance is how a card ends up somewhere nobody authorised."""
|
| 117 |
-
raw = str(text or "").strip()
|
| 118 |
-
obj = None
|
| 119 |
-
m = re.search(r"\{.*\}", raw, re.S)
|
| 120 |
-
if m:
|
| 121 |
-
try:
|
| 122 |
-
obj = json.loads(m.group(0))
|
| 123 |
-
except (ValueError, TypeError):
|
| 124 |
-
obj = None
|
| 125 |
-
if not isinstance(obj, dict):
|
| 126 |
-
return "", ""
|
| 127 |
-
choice = str(obj.get("choice") or "").strip()
|
| 128 |
-
reason = str(obj.get("reason") or "").strip()[:MAX_REASON]
|
| 129 |
-
for opt in options:
|
| 130 |
-
if choice.lower() == str(opt).lower():
|
| 131 |
-
return str(opt), reason # the OFFERED spelling wins, never the model's
|
| 132 |
-
return "", reason
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
# ββ WAVE 35 Β· W35-T41 / C7 β THE TWO TRANSPORTS RETURN THE RESPONSE BODY'S `usage`.
|
| 136 |
-
#
|
| 137 |
-
# β THIS FILE'S OWN HEADER USED TO BE THE PRODUCT'S CONFESSION THAT NOTHING COUNTED: `ai_enrich`'s
|
| 138 |
-
# docstring says *"`ai_review.decide` -- the product's only other LLM entry -- has no token
|
| 139 |
-
# accounting of ANY kind"*. R9 asks for ONE meter for every AI surface, so the counting has to reach
|
| 140 |
-
# this transport rather than be bolted onto its caller β the caller never sees the body.
|
| 141 |
-
#
|
| 142 |
-
# β A THIRD RETURN VALUE, NOT A MUTATED ARGUMENT, and both call sites are in `decide` below. The
|
| 143 |
-
# tuple grew from `(text, err)` to `(text, err, usage)`; `usage` is the raw `usage` OBJECT (or None),
|
| 144 |
-
# because reading it is `usage_ledger`'s job and a second reader here would be the two-normalizers
|
| 145 |
-
# shape this repo keeps paying for.
|
| 146 |
-
def _call_openai(p, model, system, user, timeout):
|
| 147 |
-
r = requests.post(p["url"], timeout=timeout,
|
| 148 |
-
headers={"Authorization": f"Bearer {os.environ[p['env']].strip()}",
|
| 149 |
-
"Content-Type": "application/json"},
|
| 150 |
-
json={"model": model, "max_tokens": 300, "temperature": 0,
|
| 151 |
-
"messages": [{"role": "system", "content": system},
|
| 152 |
-
{"role": "user", "content": user}]})
|
| 153 |
-
if r.status_code >= 400:
|
| 154 |
-
# β NO BODY ON A 4xx/5xx: an error envelope carries no usage, and a provider that refused
|
| 155 |
-
# before running the model has nothing to bill. The call is still COUNTED by `decide`.
|
| 156 |
-
return "", f"{p['name']} answered {r.status_code}", None
|
| 157 |
-
body = r.json()
|
| 158 |
-
choices = body.get("choices") or []
|
| 159 |
-
if not choices:
|
| 160 |
-
return "", f"{p['name']} returned no choices", body
|
| 161 |
-
return str(((choices[0] or {}).get("message") or {}).get("content") or ""), "", body
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
def _call_anthropic(p, model, system, user, timeout):
|
| 165 |
-
r = requests.post(p["url"], timeout=timeout,
|
| 166 |
-
headers={"x-api-key": os.environ[p["env"]].strip(),
|
| 167 |
-
"anthropic-version": ANTHROPIC_VERSION,
|
| 168 |
-
"content-type": "application/json"},
|
| 169 |
-
json={"model": model, "max_tokens": 300, "system": system,
|
| 170 |
-
"messages": [{"role": "user", "content": user}]})
|
| 171 |
-
if r.status_code >= 400:
|
| 172 |
-
return "", f"anthropic answered {r.status_code}", None
|
| 173 |
-
body = r.json()
|
| 174 |
-
# β stop_reason FIRST. A safety refusal is a successful 200 with an EMPTY content list, so
|
| 175 |
-
# reading content[0] before this check turns a refusal into an IndexError inside a run.
|
| 176 |
-
# β A refusal IS billed and its body carries `usage`, so the body rides back on this branch too.
|
| 177 |
-
if body.get("stop_reason") == "refusal":
|
| 178 |
-
return "", "anthropic declined to answer this record", body
|
| 179 |
-
parts = [b.get("text") or "" for b in (body.get("content") or [])
|
| 180 |
-
if isinstance(b, dict) and b.get("type") == "text"]
|
| 181 |
-
if not parts:
|
| 182 |
-
return "", "anthropic returned no text", body
|
| 183 |
-
return "".join(parts), "", body
|
| 184 |
-
|
| 185 |
-
|
| 186 |
-
def decide(*, prompt, options, row, fields=(), label="Review", timeout=None, st=None, user=""):
|
| 187 |
-
"""Pick this record's next stage. Returns `(choice, meta)`.
|
| 188 |
-
|
| 189 |
-
`choice` is "" whenever a person should decide β which is every failure mode there is.
|
| 190 |
-
`meta` carries `provider`, `model`, `reason` on success, and `problem` on refusal to answer.
|
| 191 |
-
|
| 192 |
-
β W35-T41 / C7 β `st` and `user` are the USAGE LEDGER's target. Both default to absent because
|
| 193 |
-
this function's caller is `automation_engine.ai_decide(rt, ...)`, in another lane's fence: it
|
| 194 |
-
HAS the runtime and does not pass it yet, so until it does, a review's tokens are counted as
|
| 195 |
-
unattributed and REPORTED by `GET /usage` rather than dropped. See `usage_ledger`'s header for
|
| 196 |
-
why they cannot be resolved implicitly (measured: a contextvar does not survive a FastAPI
|
| 197 |
-
dependency).
|
| 198 |
-
"""
|
| 199 |
-
import usage_ledger # noqa: PLC0415
|
| 200 |
-
opts = [str(o) for o in (options or []) if str(o).strip()]
|
| 201 |
-
if not opts:
|
| 202 |
-
return "", {"problem": "the review offers no next stages"}
|
| 203 |
-
if not str(prompt or "").strip():
|
| 204 |
-
return "", {"problem": "the review has no prompt for the model to follow"}
|
| 205 |
-
live = ladder()
|
| 206 |
-
if not live:
|
| 207 |
-
return "", {"problem": "no AI provider is configured on this deployment"}
|
| 208 |
-
system = _instruction(prompt, opts, label)
|
| 209 |
-
user = "Here is the record:\n\n" + _record_text(row, fields)
|
| 210 |
-
tmo = float(timeout or TIMEOUT_SECONDS)
|
| 211 |
-
override = (os.environ.get("AIOS_AI_MODEL") or "").strip()
|
| 212 |
-
problems = []
|
| 213 |
-
for p in live:
|
| 214 |
-
model = override or p["model"]
|
| 215 |
-
body = None
|
| 216 |
-
try:
|
| 217 |
-
text, err, body = (_call_anthropic if p["shape"] == "anthropic" else _call_openai)(
|
| 218 |
-
p, model, system, user, tmo)
|
| 219 |
-
except Exception as e: # noqa: BLE001
|
| 220 |
-
text, err = "", f"{p['name']} failed: {type(e).__name__}"
|
| 221 |
-
# ββ C7 β THE LEDGER LINE, BEFORE ANY BRANCH BELOW READS THE ANSWER.
|
| 222 |
-
# β IT IS WRITTEN ON EVERY OUTCOME THAT REACHED A PROVIDER, including a refusal and an
|
| 223 |
-
# unusable answer. A meter that books only successes reports a cheap week for a run that
|
| 224 |
-
# spent its budget being declined β and a declined call is billed. The only path that does
|
| 225 |
-
# NOT record is a transport that never reached the vendor (`body is None`), which spent
|
| 226 |
-
# nothing.
|
| 227 |
-
if body is not None:
|
| 228 |
-
ins, outs = usage_ledger.tokens_from(body)
|
| 229 |
-
usage_ledger.record("ai_review", p["name"], model, ins, outs,
|
| 230 |
-
total=usage_ledger.total_from(body), st=st, user=user)
|
| 231 |
-
if err:
|
| 232 |
-
problems.append(err)
|
| 233 |
-
continue # ladder: a dead provider degrades to the next one
|
| 234 |
-
choice, reason = _parse(text, opts)
|
| 235 |
-
if not choice:
|
| 236 |
-
# The provider ANSWERED and declined (or answered unusably). That is a decision about
|
| 237 |
-
# this record, not a fault in the provider, so it does NOT fall through to a more
|
| 238 |
-
# expensive one β the card goes to a human, which is what the model just asked for.
|
| 239 |
-
return "", {"provider": p["name"], "model": model,
|
| 240 |
-
"problem": reason or "the model did not choose one of the stages"}
|
| 241 |
-
return choice, {"provider": p["name"], "model": model, "reason": reason}
|
| 242 |
-
return "", {"problem": "; ".join(problems)[:300] or "no provider answered"}
|
| 243 |
-
|
| 244 |
-
|
| 245 |
-
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ the FLOW WRITER
|
| 246 |
-
# ββ WAVE 33 Β· W33-T54/T55 (owner item 7, ruling R3) β A PROMPT BECOMES A DRAFT AUTOMATION.
|
| 247 |
-
#
|
| 248 |
-
# Owner, verbatim: *"lay out the foundation with custom tools we provide"* β and the tools are the
|
| 249 |
-
# action catalog, which has been machine-readable since wave 23. So this writes no new vocabulary:
|
| 250 |
-
# it hands the model the SAME `ACTION_CATALOG` the menu paints, the SAME `ACTION_REQUIRED` the
|
| 251 |
-
# runner blocks on, and the SAME trigger keys `clean_trigger` accepts, and asks for one JSON object
|
| 252 |
-
# in that vocabulary.
|
| 253 |
-
#
|
| 254 |
-
# β WHY IT LIVES IN `ai_review.py` RATHER THAN A NEW FILE. Two reasons, both about ownership. This
|
| 255 |
-
# is the one module in the API package that already holds an LLM ladder and its credentials, so a
|
| 256 |
-
# second one would be a second place a rotated key has to be noticed; and a new `routes_*.py` needs
|
| 257 |
-
# a `main.py` line from another lane (contract C2) to be reachable at all, which is how wave 23
|
| 258 |
-
# shipped three finished routers 404-dead behind green gates. The DOOR is a route on
|
| 259 |
-
# `routes_automation.py`, which is already mounted.
|
| 260 |
-
#
|
| 261 |
-
# ββ CONTRACT C7 IS THE DESIGN, NOT A CHECK AT THE END: *"the flow writer may emit only kinds
|
| 262 |
-
# present in `ACTION_CATALOG`, and its output must pass `clean_actions` unchanged."* Both halves are
|
| 263 |
-
# enforced mechanically rather than asked for politely β the kind list is an `enum` in the schema
|
| 264 |
-
# the model fills, and the caller runs `clean_actions` and DIFFS the result. That diff matters
|
| 265 |
-
# because `clean_actions` has no disclosure channel (D-75): it drops a key it does not recognise
|
| 266 |
-
# and answers 200, so a draft accepted without the diff would show a person a flow that is not the
|
| 267 |
-
# flow the model described, with nothing anywhere saying so.
|
| 268 |
-
#
|
| 269 |
-
# β `chat` IS INJECTABLE, exactly as `routes_query._call_model`'s is, and it is the reason this door
|
| 270 |
-
# can be proven end to end with no API key and no spend. A gate that can only run where a
|
| 271 |
-
# credential exists is a gate that never runs.
|
| 272 |
-
|
| 273 |
-
#: Cerebras first, NOT the cheap-first order `ladder()` uses. Same reason `routes_query` inverts it:
|
| 274 |
-
#: this contract includes a REFUSAL, and refusing honestly ("I cannot build that from the steps you
|
| 275 |
-
#: have") is a model property. A cheaper rung answers an impossible request with a plausible flow,
|
| 276 |
-
#: which is worse than no flow. Override with `AIOS_FLOW_PROVIDER`.
|
| 277 |
-
FLOW_PROVIDER_ORDER = ("cerebras", "groq", "openrouter", "anthropic")
|
| 278 |
-
MAX_DRAFT_ACTIONS = 12 # a draft a person reads in one screen; the engine's own cap is 25
|
| 279 |
-
MAX_PROMPT_CHARS = 2000
|
| 280 |
-
|
| 281 |
-
|
| 282 |
-
def flow_providers(pin=None):
|
| 283 |
-
"""The rungs usable here, in THIS module's refusal-first order. Empty = the feature is off.
|
| 284 |
-
|
| 285 |
-
β `pin` IS ASK D-18 (2026-08-18): the Agent chat's model toggle must configure something, and
|
| 286 |
-
the draft door used to read `prompt` off the body and nothing else β so the key the client sent
|
| 287 |
-
was accepted and dropped, and the picker was a control over nothing.
|
| 288 |
-
β AN UNKNOWN OR UNCONFIGURED PIN FALLS BACK TO THE LADDER rather than refusing. A model the
|
| 289 |
-
ladder stopped offering must not turn every later draft into an error; the caller is told which
|
| 290 |
-
rung actually answered, which is the honest half.
|
| 291 |
-
"""
|
| 292 |
-
by_name = {p["name"]: p for p in PROVIDERS}
|
| 293 |
-
live = [n for n in FLOW_PROVIDER_ORDER
|
| 294 |
-
if n in by_name and (os.environ.get(by_name[n]["env"]) or "").strip()]
|
| 295 |
-
wanted = str(pin or os.environ.get("AIOS_FLOW_PROVIDER") or "").strip().lower()
|
| 296 |
-
if wanted and wanted in live:
|
| 297 |
-
return [by_name[wanted]]
|
| 298 |
-
return [by_name[n] for n in live]
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
def flow_schema(kinds, trigger_keys, table_keys):
|
| 302 |
-
"""The tool schema the model fills β the catalog's OWN key lists as enums.
|
| 303 |
-
|
| 304 |
-
β `kind` AND `trigger` ARE ENUMS, not free strings, and that is C7's first half enforced by the
|
| 305 |
-
transport rather than by a check afterwards. A model asked for "any action name" invents
|
| 306 |
-
`send_slack_message` on a deployment that has no such kind, and the failure then surfaces as a
|
| 307 |
-
400 from `clean_actions` carrying a sentence about a word the person never typed.
|
| 308 |
-
β `required` IS `["kind"]` ALONE, deliberately: a REFUSAL carries no name and no actions, and a
|
| 309 |
-
schema demanding them turns an honest refusal into a provider-side 400 that reads exactly like a
|
| 310 |
-
transport failure (`routes_query._spec_schema` carries the same note, for the same measured
|
| 311 |
-
reason).
|
| 312 |
-
"""
|
| 313 |
-
return {
|
| 314 |
-
"type": "object",
|
| 315 |
-
"properties": {
|
| 316 |
-
"kind": {"type": "string", "enum": ["flow", "refused"],
|
| 317 |
-
"description": "refused = this cannot be built from the steps available"},
|
| 318 |
-
"refusal": {"type": "string",
|
| 319 |
-
"description": "when kind=refused: ONE plain sentence naming what is "
|
| 320 |
-
"missing, in the words a non-technical person would use"},
|
| 321 |
-
"name": {"type": "string", "description": "a short title for the automation"},
|
| 322 |
-
"trigger": {"type": "string", "enum": sorted(trigger_keys),
|
| 323 |
-
"description": "what starts this automation"},
|
| 324 |
-
"table": {"type": "string", "enum": sorted(table_keys),
|
| 325 |
-
"description": "the database whose records this flow walks, if any"},
|
| 326 |
-
"actions": {
|
| 327 |
-
"type": "array",
|
| 328 |
-
"maxItems": MAX_DRAFT_ACTIONS,
|
| 329 |
-
"items": {
|
| 330 |
-
"type": "object",
|
| 331 |
-
"properties": {
|
| 332 |
-
"kind": {"type": "string", "enum": sorted(kinds)},
|
| 333 |
-
"why": {"type": "string",
|
| 334 |
-
"description": "one short sentence: why this step is here"},
|
| 335 |
-
"config": {"type": "object", "additionalProperties": True,
|
| 336 |
-
"description": "the step's settings, using ONLY the keys named "
|
| 337 |
-
"for that kind in the system message"},
|
| 338 |
-
},
|
| 339 |
-
"required": ["kind"],
|
| 340 |
-
},
|
| 341 |
-
},
|
| 342 |
-
},
|
| 343 |
-
"required": ["kind"],
|
| 344 |
-
}
|
| 345 |
-
|
| 346 |
-
|
| 347 |
-
def flow_system_prompt(catalog, required, triggers, tables):
|
| 348 |
-
"""What the model is told it may build with β DERIVED, never written down twice.
|
| 349 |
-
|
| 350 |
-
Every list here is the server's own: the catalog rows the menu paints, the required-key table
|
| 351 |
-
the runner blocks on, the trigger keys `clean_trigger` accepts, and this tenant's real databases
|
| 352 |
-
with their real columns. Nothing about the vocabulary is restated by hand, so a kind added to
|
| 353 |
-
the catalog is offered here on the same deploy and a kind removed stops being offered.
|
| 354 |
-
"""
|
| 355 |
-
lines = ["You build small automations for a business tool. You are given the EXACT set of "
|
| 356 |
-
"steps this tool can perform. You may use nothing else.",
|
| 357 |
-
"", "THE STEPS YOU MAY USE:"]
|
| 358 |
-
# β BOTH HALVES, `key (phrase)`. The KEY is what the model must write into `config` and the
|
| 359 |
-
# PHRASE is the only human wording of that requirement anywhere β the one the runner's own
|
| 360 |
-
# refusal sentence is built from. Keys alone leave the model guessing what `field` means on a
|
| 361 |
-
# `web_read` (it is a column to write into, not a form field); phrases alone leave it guessing
|
| 362 |
-
# what to call the setting. Handing it one and hoping is how a draft comes back configured
|
| 363 |
-
# against a key the validator drops.
|
| 364 |
-
req = {str(k): [f"{key} ({phrase})" for phrase, key in v] for k, v in (required or {}).items()}
|
| 365 |
-
for row in catalog or []:
|
| 366 |
-
if not row.get("ready"):
|
| 367 |
-
continue
|
| 368 |
-
kind = str(row.get("kind") or "")
|
| 369 |
-
need = req.get(kind) or []
|
| 370 |
-
lines.append(f"- {kind}: {row.get('label')} - {row.get('detail') or ''}"
|
| 371 |
-
+ (f" REQUIRED settings: {', '.join(need)}" if need else ""))
|
| 372 |
-
lines += ["", "WHAT CAN START AN AUTOMATION:"]
|
| 373 |
-
for t in triggers or []:
|
| 374 |
-
if t.get("planned") or not t.get("ready", True):
|
| 375 |
-
continue
|
| 376 |
-
lines.append(f"- {t.get('key')}: {t.get('label')}")
|
| 377 |
-
lines += ["", "THE DATABASES THIS PERSON HAS:"]
|
| 378 |
-
for t in (tables or [])[:40]:
|
| 379 |
-
cols = ", ".join(str(f.get("key")) for f in (t.get("fields") or [])[:25])
|
| 380 |
-
lines.append(f"- {t.get('key')} ({t.get('label')}): {cols or 'no columns yet'}")
|
| 381 |
-
if not tables:
|
| 382 |
-
lines.append("- (none - do not name a database)")
|
| 383 |
-
lines += [
|
| 384 |
-
"",
|
| 385 |
-
"RULES:",
|
| 386 |
-
"1. Use ONLY the step kinds listed above. If what is asked needs a step that is not there, "
|
| 387 |
-
"answer kind=refused and say plainly which capability is missing.",
|
| 388 |
-
"2. Fill in every REQUIRED setting you can from what the person told you. Leave one blank "
|
| 389 |
-
"rather than inventing a web address, a CSS selector or a column name.",
|
| 390 |
-
"3. A web address must start with http:// or https://.",
|
| 391 |
-
"4. To use a value from the record the flow is walking, write it as {{Column name}}.",
|
| 392 |
-
"5. Name a database only from the list above, by its key.",
|
| 393 |
-
"6. Keep it short. Fewer steps that work beat more steps that guess.",
|
| 394 |
-
]
|
| 395 |
-
return "\n".join(lines)
|
| 396 |
-
|
| 397 |
-
|
| 398 |
-
def _flow_from_tool_call(obj):
|
| 399 |
-
"""The model's tool arguments -> `(draft, refusal)`. A shape error is a refusal, never a crash."""
|
| 400 |
-
if not isinstance(obj, dict):
|
| 401 |
-
return None, "the assistant's answer could not be read"
|
| 402 |
-
if str(obj.get("kind") or "") == "refused":
|
| 403 |
-
return None, (str(obj.get("refusal") or "").strip()
|
| 404 |
-
or "the assistant could not build this from the steps available")
|
| 405 |
-
acts = obj.get("actions")
|
| 406 |
-
if not isinstance(acts, list) or not acts:
|
| 407 |
-
return None, ("the assistant did not produce any steps - try describing what should "
|
| 408 |
-
"happen, one action at a time")
|
| 409 |
-
out = []
|
| 410 |
-
for a in acts[:MAX_DRAFT_ACTIONS]:
|
| 411 |
-
if not isinstance(a, dict) or not str(a.get("kind") or "").strip():
|
| 412 |
-
continue
|
| 413 |
-
cfg = a.get("config")
|
| 414 |
-
out.append({"kind": str(a["kind"]).strip(),
|
| 415 |
-
"config": cfg if isinstance(cfg, dict) else {},
|
| 416 |
-
"why": str(a.get("why") or "").strip()[:200]})
|
| 417 |
-
if not out:
|
| 418 |
-
return None, "the assistant's steps could not be read"
|
| 419 |
-
return {"name": str(obj.get("name") or "").strip()[:80] or "New automation",
|
| 420 |
-
"trigger": str(obj.get("trigger") or "").strip(),
|
| 421 |
-
"table": str(obj.get("table") or "").strip(),
|
| 422 |
-
# ββ THE TRUNCATION IS REPORTED, and it is a STANDING RULE that it must be (owner,
|
| 423 |
-
# 2026-08-12, W30/R6 second sentence): *"if there is lag or it can't be done, you need
|
| 424 |
-
# to explicitly tell me why and recommend a fix"* β **a silent truncation IS the
|
| 425 |
-
# violation, not the limit.** `acts[:MAX_DRAFT_ACTIONS]` above dropped everything past
|
| 426 |
-
# the twelfth and said nothing, so a model that answered with a twenty-step journey had
|
| 427 |
-
# eight steps deleted between the answer and the screen, with no key anywhere in the
|
| 428 |
-
# response naming them. `MAX_ACTIONS` is 20, so those steps were STORABLE β this
|
| 429 |
-
# ceiling is the draft door's own, which makes reporting it the whole obligation.
|
| 430 |
-
# β Found by the verifier that checked this ticket, not by a gate.
|
| 431 |
-
"asked": len([a for a in acts if isinstance(a, dict)]),
|
| 432 |
-
"actions": out}, ""
|
| 433 |
-
|
| 434 |
-
|
| 435 |
-
def draft_flow(*, prompt, catalog, required, triggers, tables, chat=None, timeout=None,
|
| 436 |
-
st=None, user="", model=None):
|
| 437 |
-
"""A sentence -> `(draft, refusal_sentence, provider)`. β NOTHING IS SAVED HERE.
|
| 438 |
-
|
| 439 |
-
Exactly one of `draft` and `refusal_sentence` is truthy β the same contract
|
| 440 |
-
`web_agent.run_step` keeps, so a caller has no third case to get wrong.
|
| 441 |
-
|
| 442 |
-
β W35-T41 / C7 β `st`/`user` are the usage ledger's target, exactly as on `decide` above and for
|
| 443 |
-
the same reason: both of this function's callers (`automation_engine._ai_agent_plan` and
|
| 444 |
-
`routes_automation`'s draft door) are in lane D's fence. C7 says E adds the ledger line here and
|
| 445 |
-
D asserts it; the two keywords are what D has to pass for the line to be attributable.
|
| 446 |
-
"""
|
| 447 |
-
import usage_ledger # noqa: PLC0415
|
| 448 |
-
text = str(prompt or "").strip()[:MAX_PROMPT_CHARS]
|
| 449 |
-
if not text:
|
| 450 |
-
return None, "type what you want the automation to do", None
|
| 451 |
-
kinds = sorted({str(r.get("kind")) for r in (catalog or []) if r.get("ready")})
|
| 452 |
-
if not kinds:
|
| 453 |
-
return None, "this deployment offers no automation steps to build with", None
|
| 454 |
-
trigger_keys = sorted({str(t.get("key")) for t in (triggers or [])
|
| 455 |
-
if t.get("key") and not t.get("planned")}) or ["manual"]
|
| 456 |
-
table_keys = sorted({str(t.get("key")) for t in (tables or []) if t.get("key")}) or [""]
|
| 457 |
-
tools = [{"type": "function", "function": {
|
| 458 |
-
"name": "build_automation",
|
| 459 |
-
"description": "Emit the automation, or refuse.",
|
| 460 |
-
"parameters": flow_schema(kinds, trigger_keys, table_keys)}}]
|
| 461 |
-
messages = [{"role": "system",
|
| 462 |
-
"content": flow_system_prompt(catalog, required, triggers, tables)},
|
| 463 |
-
{"role": "user", "content": text}]
|
| 464 |
-
|
| 465 |
-
if chat is not None:
|
| 466 |
-
# β THE INJECTED PATH IS THE PROVEN PATH. It runs the SAME parse and the SAME refusal
|
| 467 |
-
# branches as a live call; only the transport is replaced.
|
| 468 |
-
draft, refusal = _flow_from_tool_call(chat(messages, tools))
|
| 469 |
-
return draft, refusal, "injected"
|
| 470 |
-
|
| 471 |
-
provs = flow_providers(model)
|
| 472 |
-
if not provs:
|
| 473 |
-
# β SAY SO. An AI feature that silently does nothing is indistinguishable from one that was
|
| 474 |
-
# never built [[flag-shipped-without-its-writer]].
|
| 475 |
-
return None, ("the assistant is not configured on this deployment, so an automation cannot "
|
| 476 |
-
"be drafted from a description yet"), None
|
| 477 |
-
tmo = float(timeout or TIMEOUT_SECONDS)
|
| 478 |
-
problems = []
|
| 479 |
-
import providers as _prov
|
| 480 |
-
for p in provs:
|
| 481 |
-
# β
|
| 482 |
-
# `problems.append(f"{p['name']}: tool calls are not wired for this shape")` and `continue`,
|
| 483 |
-
# so the sentence *"anthropic: tool calls are not wired for this shape"* appeared under the
|
| 484 |
-
# Agent module verbatim. R4: *"Anthropic becomes the tool-calling path that always works."*
|
| 485 |
-
# It is wired now, through the ONE wire in `providers`, and it matters more than it looks:
|
| 486 |
-
# every side rung on this account is refusing today (cerebras 402, groq 404, openrouter
|
| 487 |
-
# 402), so without this branch the drafter cannot answer at all.
|
| 488 |
-
# β NO `effort` ON THIS DOOR β `output_config.effort` errors on Haiku 4.5, the tier this
|
| 489 |
-
# ladder runs. The parameter is the caller's to send, which is why the wire takes it.
|
| 490 |
-
try:
|
| 491 |
-
if p["shape"] == "anthropic":
|
| 492 |
-
req = _prov.anthropic_request(
|
| 493 |
-
model=p["model"], key=os.environ[p["env"]].strip(), system=None,
|
| 494 |
-
messages=messages, tools=tools, max_tokens=1500, tool_choice="required")
|
| 495 |
-
r = requests.post(req["url"], timeout=tmo,
|
| 496 |
-
headers=req["headers"], json=req["json"])
|
| 497 |
-
else:
|
| 498 |
-
r = requests.post(p["url"], timeout=tmo,
|
| 499 |
-
headers={"Authorization": f"Bearer {os.environ[p['env']].strip()}",
|
| 500 |
-
"Content-Type": "application/json"},
|
| 501 |
-
json={"model": p["model"], "messages": messages, "tools": tools,
|
| 502 |
-
"tool_choice": "required", "temperature": 0.1,
|
| 503 |
-
"max_tokens": 1500})
|
| 504 |
-
except Exception as e: # noqa: BLE001
|
| 505 |
-
problems.append(f"{p['name']}: {type(e).__name__}")
|
| 506 |
-
continue
|
| 507 |
-
if r.status_code != 200:
|
| 508 |
-
# β A SENTENCE, NOT `HTTP 402` (R4's third clause). The owner read the status codes off
|
| 509 |
-
# this very door. `refusal_sentence` also decides whether this was a CREDIT failure, and
|
| 510 |
-
# the memo makes the next turn SKIP the empty account instead of paying for it again.
|
| 511 |
-
if _prov.is_credit_failure(r.status_code, r.text):
|
| 512 |
-
_prov.mark_no_credit(p["name"])
|
| 513 |
-
problems.append(_prov.refusal_sentence(p["name"], r.status_code, r.text))
|
| 514 |
-
continue
|
| 515 |
-
try:
|
| 516 |
-
body = r.json()
|
| 517 |
-
if p["shape"] == "anthropic":
|
| 518 |
-
_text, args, _refused = _prov.anthropic_read(body)
|
| 519 |
-
if _refused:
|
| 520 |
-
problems.append(f"{p['name']}: {_refused}")
|
| 521 |
-
continue
|
| 522 |
-
else:
|
| 523 |
-
calls = (((body.get("choices") or [{}])[0].get("message") or {})
|
| 524 |
-
.get("tool_calls") or [])
|
| 525 |
-
args = json.loads(calls[0]["function"]["arguments"]) if calls else None
|
| 526 |
-
except Exception as e: # noqa: BLE001
|
| 527 |
-
problems.append(f"{p['name']}: unreadable answer ({type(e).__name__})")
|
| 528 |
-
continue
|
| 529 |
-
# ββ C7 β THE LEDGER LINE. Placed after the parse rather than before it so an UNREADABLE
|
| 530 |
-
# answer is not double-counted by the `continue` above... β which means an unreadable answer
|
| 531 |
-
# is NOT counted at all, and that is a deliberate, disclosed loss: a body this code cannot
|
| 532 |
-
# parse is a body whose `usage` it also cannot trust, and `body` is out of scope in that
|
| 533 |
-
# branch by construction. The 200-with-junk case is rare and named here rather than
|
| 534 |
-
# silently rounded to zero.
|
| 535 |
-
ins, outs = usage_ledger.tokens_from(body)
|
| 536 |
-
usage_ledger.record("automation_draft", p["name"], p["model"], ins, outs,
|
| 537 |
-
total=usage_ledger.total_from(body), st=st, user=user)
|
| 538 |
-
draft, refusal = _flow_from_tool_call(args)
|
| 539 |
-
if draft is None and isinstance(args, dict) and str(args.get("kind")) == "refused":
|
| 540 |
-
# β A REFUSAL IS AN ANSWER, NOT A FAULT, so it does NOT fall through to a more
|
| 541 |
-
# expensive rung β the model has just said this cannot be built. `decide()` takes the
|
| 542 |
-
# same posture for the same reason.
|
| 543 |
-
return None, refusal, p["name"]
|
| 544 |
-
if draft is not None:
|
| 545 |
-
return draft, "", p["name"]
|
| 546 |
-
problems.append(f"{p['name']}: {refusal}")
|
| 547 |
-
return None, ("; ".join(problems)[:300] or "no provider answered"), None
|
|
|
|
| 1 |
+
"""AI REVIEW β let a model decide which stage a card moves to (wave 23, owner ruling R4/R14).
|
| 2 |
+
|
| 3 |
+
β WHAT THIS IS. A review stage holds a record until somebody decides where it goes next. R4 made
|
| 4 |
+
that somebody optionally a MODEL: the engine hands over the record's own values, the review's
|
| 5 |
+
prompt, and the list of stages the review is allowed to send a card to, and gets back ONE of
|
| 6 |
+
those stage labels plus a one-line reason. Every decision is written to the same `reviews` audit
|
| 7 |
+
log a human click writes to, tagged `by: "ai"` with the provider and model that made it.
|
| 8 |
+
|
| 9 |
+
β FAIL-CLOSED IN EVERY DIRECTION, and this is the whole safety story. No key configured, a
|
| 10 |
+
network failure, a slow answer, a malformed answer, or an answer naming a stage the review does
|
| 11 |
+
not offer β all return `("", {...})`, and the caller leaves the card exactly where a human would
|
| 12 |
+
have found it. The feature can be absent, broken or wrong and the worst outcome is a person doing
|
| 13 |
+
the work. Nothing here can move a card somewhere the review does not already permit.
|
| 14 |
+
|
| 15 |
+
β CHEAP FIRST (owner R14, verbatim: *"Claude is a bit too expensive"*). The ladder is
|
| 16 |
+
`groq β cerebras β openrouter β anthropic`; the first CONFIGURED provider wins, and Anthropic is
|
| 17 |
+
last rather than absent β it is the quality backstop, not the default. `AIOS_AI_REVIEW_PROVIDER`
|
| 18 |
+
pins one; `AIOS_AI_MODEL` overrides the model.
|
| 19 |
+
|
| 20 |
+
β WHY RAW HTTP RATHER THAN THE `anthropic` SDK, stated because it is a deliberate deviation from
|
| 21 |
+
the /claude-api skill's default and not an oversight. Three of the four legs are OpenAI-chat-shaped
|
| 22 |
+
endpoints with no shared SDK, so a ladder built on the SDK would be one SDK leg beside three
|
| 23 |
+
hand-rolled ones β two implementations of the same call, the seam this repo keeps closing. And
|
| 24 |
+
`aios-web/api/requirements.txt` is PINNED to what the verify battery proves ([[pin-deps-space-
|
| 25 |
+
rebuilds]]): adding a dependency there rebuilds the container, which is a real deploy risk to take
|
| 26 |
+
for one leg of an optional feature. `requests` is already a dependency and `harness/analyst.py`
|
| 27 |
+
established per-provider raw HTTP as the house pattern. Booked as a DEBT line so the integrator
|
| 28 |
+
can overturn it deliberately rather than by drift.
|
| 29 |
+
|
| 30 |
+
The Messages shape below is the current one: `x-api-key` + `anthropic-version: 2023-06-01`, and
|
| 31 |
+
`stop_reason: "refusal"` is checked BEFORE reading `content` β a refusal answers HTTP 200 with an
|
| 32 |
+
empty content list, so code that indexes `content[0]` unconditionally breaks on it.
|
| 33 |
+
"""
|
| 34 |
+
from __future__ import annotations
|
| 35 |
+
|
| 36 |
+
import json
|
| 37 |
+
import os
|
| 38 |
+
import re
|
| 39 |
+
|
| 40 |
+
import requests
|
| 41 |
+
|
| 42 |
+
#: The ladder. Order IS the policy (R14) β cheapest capable first, Anthropic last.
|
| 43 |
+
PROVIDERS = [
|
| 44 |
+
{"name": "groq", "env": "GROQ_API_KEY", "shape": "openai",
|
| 45 |
+
"url": "https://api.groq.com/openai/v1/chat/completions",
|
| 46 |
+
"model": "llama-3.3-70b-versatile"},
|
| 47 |
+
{"name": "cerebras", "env": "CEREBRAS_API_KEY", "shape": "openai",
|
| 48 |
+
"url": "https://api.cerebras.ai/v1/chat/completions",
|
| 49 |
+
"model": "gpt-oss-120b"},
|
| 50 |
+
{"name": "openrouter", "env": "OPENROUTER_API_KEY", "shape": "openai",
|
| 51 |
+
"url": "https://openrouter.ai/api/v1/chat/completions",
|
| 52 |
+
"model": "openai/gpt-4o-mini"},
|
| 53 |
+
# β haiku-class DELIBERATELY, not the default Opus tier: R14 put Anthropic on this ladder as
|
| 54 |
+
# the backstop for a one-line classification, and this is the cheapest current Claude that
|
| 55 |
+
# does it well. A bigger model here would be spending the owner's money to pick between two
|
| 56 |
+
# labels it already has in front of it.
|
| 57 |
+
{"name": "anthropic", "env": "ANTHROPIC_API_KEY", "shape": "anthropic",
|
| 58 |
+
"url": "https://api.anthropic.com/v1/messages",
|
| 59 |
+
# β AN ENV LEVER, NOT A NEW DEFAULT (W36-T35): haiku stays the choice for the reasons above,
|
| 60 |
+
# and a deployment whose side rungs are out of credit can raise the tier without a release.
|
| 61 |
+
"model": os.environ.get("AIOS_AI_REVIEW_ANTHROPIC_MODEL") or "claude-haiku-4-5"},
|
| 62 |
+
]
|
| 63 |
+
ANTHROPIC_VERSION = "2023-06-01"
|
| 64 |
+
TIMEOUT_SECONDS = float(os.environ.get("AIOS_AI_REVIEW_TIMEOUT") or 20)
|
| 65 |
+
MAX_FIELD_CHARS = 200 # per value handed to the model
|
| 66 |
+
MAX_FIELDS = 30 # columns handed to the model
|
| 67 |
+
MAX_REASON = 200
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def ladder():
|
| 71 |
+
"""The providers that are actually usable here, in order. Empty = the feature is off."""
|
| 72 |
+
pin = (os.environ.get("AIOS_AI_REVIEW_PROVIDER") or "").strip().lower()
|
| 73 |
+
live = [p for p in PROVIDERS if (os.environ.get(p["env"]) or "").strip()]
|
| 74 |
+
if pin:
|
| 75 |
+
live = [p for p in live if p["name"] == pin]
|
| 76 |
+
return live
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def configured():
|
| 80 |
+
return bool(ladder())
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def _record_text(row, fields):
|
| 84 |
+
"""The record, as the model sees it. Values are truncated and the column set is bounded β
|
| 85 |
+
an automation table can carry a 32 KB JSON blob per row (C7) and a review decision does not
|
| 86 |
+
need it. Machine bookkeeping columns are dropped: a stage cell naming the stage the card is
|
| 87 |
+
sitting at would be the model reading its own question back."""
|
| 88 |
+
keys = [k for k in (fields or list((row or {}).keys()))
|
| 89 |
+
if not str(k).startswith("stage_")][:MAX_FIELDS]
|
| 90 |
+
lines = []
|
| 91 |
+
for k in keys:
|
| 92 |
+
v = (row or {}).get(k)
|
| 93 |
+
if v is None or str(v).strip() == "":
|
| 94 |
+
continue
|
| 95 |
+
lines.append(f"{k}: {str(v)[:MAX_FIELD_CHARS]}")
|
| 96 |
+
return "\n".join(lines) or "(this record has no filled-in values)"
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
def _instruction(prompt, options, label):
|
| 100 |
+
return (
|
| 101 |
+
f"You are deciding what happens to one record waiting at a review step called "
|
| 102 |
+
f"{label!r} in a workflow.\n\n"
|
| 103 |
+
f"The person who built this workflow told you: {prompt}\n\n"
|
| 104 |
+
f"Choose EXACTLY ONE of these next steps, by its exact name:\n"
|
| 105 |
+
+ "\n".join(f"- {o}" for o in options)
|
| 106 |
+
+ "\n\nAnswer with one line of JSON and nothing else:\n"
|
| 107 |
+
'{"choice": "<one name from the list above>", "reason": "<one short sentence>"}\n'
|
| 108 |
+
"If the record does not give you enough to decide, answer "
|
| 109 |
+
'{"choice": "", "reason": "why not"} and a person will decide instead.'
|
| 110 |
+
)
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def _parse(text, options):
|
| 114 |
+
"""The model's line β `(choice, reason)`. A choice that is not one of the offered stages is
|
| 115 |
+
DISCARDED, not fuzzy-matched: the offered list is a permission boundary, and a near-miss
|
| 116 |
+
resolved by string distance is how a card ends up somewhere nobody authorised."""
|
| 117 |
+
raw = str(text or "").strip()
|
| 118 |
+
obj = None
|
| 119 |
+
m = re.search(r"\{.*\}", raw, re.S)
|
| 120 |
+
if m:
|
| 121 |
+
try:
|
| 122 |
+
obj = json.loads(m.group(0))
|
| 123 |
+
except (ValueError, TypeError):
|
| 124 |
+
obj = None
|
| 125 |
+
if not isinstance(obj, dict):
|
| 126 |
+
return "", ""
|
| 127 |
+
choice = str(obj.get("choice") or "").strip()
|
| 128 |
+
reason = str(obj.get("reason") or "").strip()[:MAX_REASON]
|
| 129 |
+
for opt in options:
|
| 130 |
+
if choice.lower() == str(opt).lower():
|
| 131 |
+
return str(opt), reason # the OFFERED spelling wins, never the model's
|
| 132 |
+
return "", reason
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
# ββ WAVE 35 Β· W35-T41 / C7 β THE TWO TRANSPORTS RETURN THE RESPONSE BODY'S `usage`.
|
| 136 |
+
#
|
| 137 |
+
# β THIS FILE'S OWN HEADER USED TO BE THE PRODUCT'S CONFESSION THAT NOTHING COUNTED: `ai_enrich`'s
|
| 138 |
+
# docstring says *"`ai_review.decide` -- the product's only other LLM entry -- has no token
|
| 139 |
+
# accounting of ANY kind"*. R9 asks for ONE meter for every AI surface, so the counting has to reach
|
| 140 |
+
# this transport rather than be bolted onto its caller β the caller never sees the body.
|
| 141 |
+
#
|
| 142 |
+
# β A THIRD RETURN VALUE, NOT A MUTATED ARGUMENT, and both call sites are in `decide` below. The
|
| 143 |
+
# tuple grew from `(text, err)` to `(text, err, usage)`; `usage` is the raw `usage` OBJECT (or None),
|
| 144 |
+
# because reading it is `usage_ledger`'s job and a second reader here would be the two-normalizers
|
| 145 |
+
# shape this repo keeps paying for.
|
| 146 |
+
def _call_openai(p, model, system, user, timeout):
|
| 147 |
+
r = requests.post(p["url"], timeout=timeout,
|
| 148 |
+
headers={"Authorization": f"Bearer {os.environ[p['env']].strip()}",
|
| 149 |
+
"Content-Type": "application/json"},
|
| 150 |
+
json={"model": model, "max_tokens": 300, "temperature": 0,
|
| 151 |
+
"messages": [{"role": "system", "content": system},
|
| 152 |
+
{"role": "user", "content": user}]})
|
| 153 |
+
if r.status_code >= 400:
|
| 154 |
+
# β NO BODY ON A 4xx/5xx: an error envelope carries no usage, and a provider that refused
|
| 155 |
+
# before running the model has nothing to bill. The call is still COUNTED by `decide`.
|
| 156 |
+
return "", f"{p['name']} answered {r.status_code}", None
|
| 157 |
+
body = r.json()
|
| 158 |
+
choices = body.get("choices") or []
|
| 159 |
+
if not choices:
|
| 160 |
+
return "", f"{p['name']} returned no choices", body
|
| 161 |
+
return str(((choices[0] or {}).get("message") or {}).get("content") or ""), "", body
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def _call_anthropic(p, model, system, user, timeout):
|
| 165 |
+
r = requests.post(p["url"], timeout=timeout,
|
| 166 |
+
headers={"x-api-key": os.environ[p["env"]].strip(),
|
| 167 |
+
"anthropic-version": ANTHROPIC_VERSION,
|
| 168 |
+
"content-type": "application/json"},
|
| 169 |
+
json={"model": model, "max_tokens": 300, "system": system,
|
| 170 |
+
"messages": [{"role": "user", "content": user}]})
|
| 171 |
+
if r.status_code >= 400:
|
| 172 |
+
return "", f"anthropic answered {r.status_code}", None
|
| 173 |
+
body = r.json()
|
| 174 |
+
# β stop_reason FIRST. A safety refusal is a successful 200 with an EMPTY content list, so
|
| 175 |
+
# reading content[0] before this check turns a refusal into an IndexError inside a run.
|
| 176 |
+
# β A refusal IS billed and its body carries `usage`, so the body rides back on this branch too.
|
| 177 |
+
if body.get("stop_reason") == "refusal":
|
| 178 |
+
return "", "anthropic declined to answer this record", body
|
| 179 |
+
parts = [b.get("text") or "" for b in (body.get("content") or [])
|
| 180 |
+
if isinstance(b, dict) and b.get("type") == "text"]
|
| 181 |
+
if not parts:
|
| 182 |
+
return "", "anthropic returned no text", body
|
| 183 |
+
return "".join(parts), "", body
|
| 184 |
+
|
| 185 |
+
|
| 186 |
+
def decide(*, prompt, options, row, fields=(), label="Review", timeout=None, st=None, user=""):
|
| 187 |
+
"""Pick this record's next stage. Returns `(choice, meta)`.
|
| 188 |
+
|
| 189 |
+
`choice` is "" whenever a person should decide β which is every failure mode there is.
|
| 190 |
+
`meta` carries `provider`, `model`, `reason` on success, and `problem` on refusal to answer.
|
| 191 |
+
|
| 192 |
+
β W35-T41 / C7 β `st` and `user` are the USAGE LEDGER's target. Both default to absent because
|
| 193 |
+
this function's caller is `automation_engine.ai_decide(rt, ...)`, in another lane's fence: it
|
| 194 |
+
HAS the runtime and does not pass it yet, so until it does, a review's tokens are counted as
|
| 195 |
+
unattributed and REPORTED by `GET /usage` rather than dropped. See `usage_ledger`'s header for
|
| 196 |
+
why they cannot be resolved implicitly (measured: a contextvar does not survive a FastAPI
|
| 197 |
+
dependency).
|
| 198 |
+
"""
|
| 199 |
+
import usage_ledger # noqa: PLC0415
|
| 200 |
+
opts = [str(o) for o in (options or []) if str(o).strip()]
|
| 201 |
+
if not opts:
|
| 202 |
+
return "", {"problem": "the review offers no next stages"}
|
| 203 |
+
if not str(prompt or "").strip():
|
| 204 |
+
return "", {"problem": "the review has no prompt for the model to follow"}
|
| 205 |
+
live = ladder()
|
| 206 |
+
if not live:
|
| 207 |
+
return "", {"problem": "no AI provider is configured on this deployment"}
|
| 208 |
+
system = _instruction(prompt, opts, label)
|
| 209 |
+
user = "Here is the record:\n\n" + _record_text(row, fields)
|
| 210 |
+
tmo = float(timeout or TIMEOUT_SECONDS)
|
| 211 |
+
override = (os.environ.get("AIOS_AI_MODEL") or "").strip()
|
| 212 |
+
problems = []
|
| 213 |
+
for p in live:
|
| 214 |
+
model = override or p["model"]
|
| 215 |
+
body = None
|
| 216 |
+
try:
|
| 217 |
+
text, err, body = (_call_anthropic if p["shape"] == "anthropic" else _call_openai)(
|
| 218 |
+
p, model, system, user, tmo)
|
| 219 |
+
except Exception as e: # noqa: BLE001
|
| 220 |
+
text, err = "", f"{p['name']} failed: {type(e).__name__}"
|
| 221 |
+
# ββ C7 β THE LEDGER LINE, BEFORE ANY BRANCH BELOW READS THE ANSWER.
|
| 222 |
+
# β IT IS WRITTEN ON EVERY OUTCOME THAT REACHED A PROVIDER, including a refusal and an
|
| 223 |
+
# unusable answer. A meter that books only successes reports a cheap week for a run that
|
| 224 |
+
# spent its budget being declined β and a declined call is billed. The only path that does
|
| 225 |
+
# NOT record is a transport that never reached the vendor (`body is None`), which spent
|
| 226 |
+
# nothing.
|
| 227 |
+
if body is not None:
|
| 228 |
+
ins, outs = usage_ledger.tokens_from(body)
|
| 229 |
+
usage_ledger.record("ai_review", p["name"], model, ins, outs,
|
| 230 |
+
total=usage_ledger.total_from(body), st=st, user=user)
|
| 231 |
+
if err:
|
| 232 |
+
problems.append(err)
|
| 233 |
+
continue # ladder: a dead provider degrades to the next one
|
| 234 |
+
choice, reason = _parse(text, opts)
|
| 235 |
+
if not choice:
|
| 236 |
+
# The provider ANSWERED and declined (or answered unusably). That is a decision about
|
| 237 |
+
# this record, not a fault in the provider, so it does NOT fall through to a more
|
| 238 |
+
# expensive one β the card goes to a human, which is what the model just asked for.
|
| 239 |
+
return "", {"provider": p["name"], "model": model,
|
| 240 |
+
"problem": reason or "the model did not choose one of the stages"}
|
| 241 |
+
return choice, {"provider": p["name"], "model": model, "reason": reason}
|
| 242 |
+
return "", {"problem": "; ".join(problems)[:300] or "no provider answered"}
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ the FLOW WRITER
|
| 246 |
+
# ββ WAVE 33 Β· W33-T54/T55 (owner item 7, ruling R3) β A PROMPT BECOMES A DRAFT AUTOMATION.
|
| 247 |
+
#
|
| 248 |
+
# Owner, verbatim: *"lay out the foundation with custom tools we provide"* β and the tools are the
|
| 249 |
+
# action catalog, which has been machine-readable since wave 23. So this writes no new vocabulary:
|
| 250 |
+
# it hands the model the SAME `ACTION_CATALOG` the menu paints, the SAME `ACTION_REQUIRED` the
|
| 251 |
+
# runner blocks on, and the SAME trigger keys `clean_trigger` accepts, and asks for one JSON object
|
| 252 |
+
# in that vocabulary.
|
| 253 |
+
#
|
| 254 |
+
# β WHY IT LIVES IN `ai_review.py` RATHER THAN A NEW FILE. Two reasons, both about ownership. This
|
| 255 |
+
# is the one module in the API package that already holds an LLM ladder and its credentials, so a
|
| 256 |
+
# second one would be a second place a rotated key has to be noticed; and a new `routes_*.py` needs
|
| 257 |
+
# a `main.py` line from another lane (contract C2) to be reachable at all, which is how wave 23
|
| 258 |
+
# shipped three finished routers 404-dead behind green gates. The DOOR is a route on
|
| 259 |
+
# `routes_automation.py`, which is already mounted.
|
| 260 |
+
#
|
| 261 |
+
# ββ CONTRACT C7 IS THE DESIGN, NOT A CHECK AT THE END: *"the flow writer may emit only kinds
|
| 262 |
+
# present in `ACTION_CATALOG`, and its output must pass `clean_actions` unchanged."* Both halves are
|
| 263 |
+
# enforced mechanically rather than asked for politely β the kind list is an `enum` in the schema
|
| 264 |
+
# the model fills, and the caller runs `clean_actions` and DIFFS the result. That diff matters
|
| 265 |
+
# because `clean_actions` has no disclosure channel (D-75): it drops a key it does not recognise
|
| 266 |
+
# and answers 200, so a draft accepted without the diff would show a person a flow that is not the
|
| 267 |
+
# flow the model described, with nothing anywhere saying so.
|
| 268 |
+
#
|
| 269 |
+
# β `chat` IS INJECTABLE, exactly as `routes_query._call_model`'s is, and it is the reason this door
|
| 270 |
+
# can be proven end to end with no API key and no spend. A gate that can only run where a
|
| 271 |
+
# credential exists is a gate that never runs.
|
| 272 |
+
|
| 273 |
+
#: Cerebras first, NOT the cheap-first order `ladder()` uses. Same reason `routes_query` inverts it:
|
| 274 |
+
#: this contract includes a REFUSAL, and refusing honestly ("I cannot build that from the steps you
|
| 275 |
+
#: have") is a model property. A cheaper rung answers an impossible request with a plausible flow,
|
| 276 |
+
#: which is worse than no flow. Override with `AIOS_FLOW_PROVIDER`.
|
| 277 |
+
FLOW_PROVIDER_ORDER = ("cerebras", "groq", "openrouter", "anthropic")
|
| 278 |
+
MAX_DRAFT_ACTIONS = 12 # a draft a person reads in one screen; the engine's own cap is 25
|
| 279 |
+
MAX_PROMPT_CHARS = 2000
|
| 280 |
+
|
| 281 |
+
|
| 282 |
+
def flow_providers(pin=None):
|
| 283 |
+
"""The rungs usable here, in THIS module's refusal-first order. Empty = the feature is off.
|
| 284 |
+
|
| 285 |
+
β `pin` IS ASK D-18 (2026-08-18): the Agent chat's model toggle must configure something, and
|
| 286 |
+
the draft door used to read `prompt` off the body and nothing else β so the key the client sent
|
| 287 |
+
was accepted and dropped, and the picker was a control over nothing.
|
| 288 |
+
β AN UNKNOWN OR UNCONFIGURED PIN FALLS BACK TO THE LADDER rather than refusing. A model the
|
| 289 |
+
ladder stopped offering must not turn every later draft into an error; the caller is told which
|
| 290 |
+
rung actually answered, which is the honest half.
|
| 291 |
+
"""
|
| 292 |
+
by_name = {p["name"]: p for p in PROVIDERS}
|
| 293 |
+
live = [n for n in FLOW_PROVIDER_ORDER
|
| 294 |
+
if n in by_name and (os.environ.get(by_name[n]["env"]) or "").strip()]
|
| 295 |
+
wanted = str(pin or os.environ.get("AIOS_FLOW_PROVIDER") or "").strip().lower()
|
| 296 |
+
if wanted and wanted in live:
|
| 297 |
+
return [by_name[wanted]]
|
| 298 |
+
return [by_name[n] for n in live]
|
| 299 |
+
|
| 300 |
+
|
| 301 |
+
def flow_schema(kinds, trigger_keys, table_keys):
|
| 302 |
+
"""The tool schema the model fills β the catalog's OWN key lists as enums.
|
| 303 |
+
|
| 304 |
+
β `kind` AND `trigger` ARE ENUMS, not free strings, and that is C7's first half enforced by the
|
| 305 |
+
transport rather than by a check afterwards. A model asked for "any action name" invents
|
| 306 |
+
`send_slack_message` on a deployment that has no such kind, and the failure then surfaces as a
|
| 307 |
+
400 from `clean_actions` carrying a sentence about a word the person never typed.
|
| 308 |
+
β `required` IS `["kind"]` ALONE, deliberately: a REFUSAL carries no name and no actions, and a
|
| 309 |
+
schema demanding them turns an honest refusal into a provider-side 400 that reads exactly like a
|
| 310 |
+
transport failure (`routes_query._spec_schema` carries the same note, for the same measured
|
| 311 |
+
reason).
|
| 312 |
+
"""
|
| 313 |
+
return {
|
| 314 |
+
"type": "object",
|
| 315 |
+
"properties": {
|
| 316 |
+
"kind": {"type": "string", "enum": ["flow", "refused"],
|
| 317 |
+
"description": "refused = this cannot be built from the steps available"},
|
| 318 |
+
"refusal": {"type": "string",
|
| 319 |
+
"description": "when kind=refused: ONE plain sentence naming what is "
|
| 320 |
+
"missing, in the words a non-technical person would use"},
|
| 321 |
+
"name": {"type": "string", "description": "a short title for the automation"},
|
| 322 |
+
"trigger": {"type": "string", "enum": sorted(trigger_keys),
|
| 323 |
+
"description": "what starts this automation"},
|
| 324 |
+
"table": {"type": "string", "enum": sorted(table_keys),
|
| 325 |
+
"description": "the database whose records this flow walks, if any"},
|
| 326 |
+
"actions": {
|
| 327 |
+
"type": "array",
|
| 328 |
+
"maxItems": MAX_DRAFT_ACTIONS,
|
| 329 |
+
"items": {
|
| 330 |
+
"type": "object",
|
| 331 |
+
"properties": {
|
| 332 |
+
"kind": {"type": "string", "enum": sorted(kinds)},
|
| 333 |
+
"why": {"type": "string",
|
| 334 |
+
"description": "one short sentence: why this step is here"},
|
| 335 |
+
"config": {"type": "object", "additionalProperties": True,
|
| 336 |
+
"description": "the step's settings, using ONLY the keys named "
|
| 337 |
+
"for that kind in the system message"},
|
| 338 |
+
},
|
| 339 |
+
"required": ["kind"],
|
| 340 |
+
},
|
| 341 |
+
},
|
| 342 |
+
},
|
| 343 |
+
"required": ["kind"],
|
| 344 |
+
}
|
| 345 |
+
|
| 346 |
+
|
| 347 |
+
def flow_system_prompt(catalog, required, triggers, tables):
|
| 348 |
+
"""What the model is told it may build with β DERIVED, never written down twice.
|
| 349 |
+
|
| 350 |
+
Every list here is the server's own: the catalog rows the menu paints, the required-key table
|
| 351 |
+
the runner blocks on, the trigger keys `clean_trigger` accepts, and this tenant's real databases
|
| 352 |
+
with their real columns. Nothing about the vocabulary is restated by hand, so a kind added to
|
| 353 |
+
the catalog is offered here on the same deploy and a kind removed stops being offered.
|
| 354 |
+
"""
|
| 355 |
+
lines = ["You build small automations for a business tool. You are given the EXACT set of "
|
| 356 |
+
"steps this tool can perform. You may use nothing else.",
|
| 357 |
+
"", "THE STEPS YOU MAY USE:"]
|
| 358 |
+
# β BOTH HALVES, `key (phrase)`. The KEY is what the model must write into `config` and the
|
| 359 |
+
# PHRASE is the only human wording of that requirement anywhere β the one the runner's own
|
| 360 |
+
# refusal sentence is built from. Keys alone leave the model guessing what `field` means on a
|
| 361 |
+
# `web_read` (it is a column to write into, not a form field); phrases alone leave it guessing
|
| 362 |
+
# what to call the setting. Handing it one and hoping is how a draft comes back configured
|
| 363 |
+
# against a key the validator drops.
|
| 364 |
+
req = {str(k): [f"{key} ({phrase})" for phrase, key in v] for k, v in (required or {}).items()}
|
| 365 |
+
for row in catalog or []:
|
| 366 |
+
if not row.get("ready"):
|
| 367 |
+
continue
|
| 368 |
+
kind = str(row.get("kind") or "")
|
| 369 |
+
need = req.get(kind) or []
|
| 370 |
+
lines.append(f"- {kind}: {row.get('label')} - {row.get('detail') or ''}"
|
| 371 |
+
+ (f" REQUIRED settings: {', '.join(need)}" if need else ""))
|
| 372 |
+
lines += ["", "WHAT CAN START AN AUTOMATION:"]
|
| 373 |
+
for t in triggers or []:
|
| 374 |
+
if t.get("planned") or not t.get("ready", True):
|
| 375 |
+
continue
|
| 376 |
+
lines.append(f"- {t.get('key')}: {t.get('label')}")
|
| 377 |
+
lines += ["", "THE DATABASES THIS PERSON HAS:"]
|
| 378 |
+
for t in (tables or [])[:40]:
|
| 379 |
+
cols = ", ".join(str(f.get("key")) for f in (t.get("fields") or [])[:25])
|
| 380 |
+
lines.append(f"- {t.get('key')} ({t.get('label')}): {cols or 'no columns yet'}")
|
| 381 |
+
if not tables:
|
| 382 |
+
lines.append("- (none - do not name a database)")
|
| 383 |
+
lines += [
|
| 384 |
+
"",
|
| 385 |
+
"RULES:",
|
| 386 |
+
"1. Use ONLY the step kinds listed above. If what is asked needs a step that is not there, "
|
| 387 |
+
"answer kind=refused and say plainly which capability is missing.",
|
| 388 |
+
"2. Fill in every REQUIRED setting you can from what the person told you. Leave one blank "
|
| 389 |
+
"rather than inventing a web address, a CSS selector or a column name.",
|
| 390 |
+
"3. A web address must start with http:// or https://.",
|
| 391 |
+
"4. To use a value from the record the flow is walking, write it as {{Column name}}.",
|
| 392 |
+
"5. Name a database only from the list above, by its key.",
|
| 393 |
+
"6. Keep it short. Fewer steps that work beat more steps that guess.",
|
| 394 |
+
]
|
| 395 |
+
return "\n".join(lines)
|
| 396 |
+
|
| 397 |
+
|
| 398 |
+
def _flow_from_tool_call(obj):
|
| 399 |
+
"""The model's tool arguments -> `(draft, refusal)`. A shape error is a refusal, never a crash."""
|
| 400 |
+
if not isinstance(obj, dict):
|
| 401 |
+
return None, "the assistant's answer could not be read"
|
| 402 |
+
if str(obj.get("kind") or "") == "refused":
|
| 403 |
+
return None, (str(obj.get("refusal") or "").strip()
|
| 404 |
+
or "the assistant could not build this from the steps available")
|
| 405 |
+
acts = obj.get("actions")
|
| 406 |
+
if not isinstance(acts, list) or not acts:
|
| 407 |
+
return None, ("the assistant did not produce any steps - try describing what should "
|
| 408 |
+
"happen, one action at a time")
|
| 409 |
+
out = []
|
| 410 |
+
for a in acts[:MAX_DRAFT_ACTIONS]:
|
| 411 |
+
if not isinstance(a, dict) or not str(a.get("kind") or "").strip():
|
| 412 |
+
continue
|
| 413 |
+
cfg = a.get("config")
|
| 414 |
+
out.append({"kind": str(a["kind"]).strip(),
|
| 415 |
+
"config": cfg if isinstance(cfg, dict) else {},
|
| 416 |
+
"why": str(a.get("why") or "").strip()[:200]})
|
| 417 |
+
if not out:
|
| 418 |
+
return None, "the assistant's steps could not be read"
|
| 419 |
+
return {"name": str(obj.get("name") or "").strip()[:80] or "New automation",
|
| 420 |
+
"trigger": str(obj.get("trigger") or "").strip(),
|
| 421 |
+
"table": str(obj.get("table") or "").strip(),
|
| 422 |
+
# ββ THE TRUNCATION IS REPORTED, and it is a STANDING RULE that it must be (owner,
|
| 423 |
+
# 2026-08-12, W30/R6 second sentence): *"if there is lag or it can't be done, you need
|
| 424 |
+
# to explicitly tell me why and recommend a fix"* β **a silent truncation IS the
|
| 425 |
+
# violation, not the limit.** `acts[:MAX_DRAFT_ACTIONS]` above dropped everything past
|
| 426 |
+
# the twelfth and said nothing, so a model that answered with a twenty-step journey had
|
| 427 |
+
# eight steps deleted between the answer and the screen, with no key anywhere in the
|
| 428 |
+
# response naming them. `MAX_ACTIONS` is 20, so those steps were STORABLE β this
|
| 429 |
+
# ceiling is the draft door's own, which makes reporting it the whole obligation.
|
| 430 |
+
# β Found by the verifier that checked this ticket, not by a gate.
|
| 431 |
+
"asked": len([a for a in acts if isinstance(a, dict)]),
|
| 432 |
+
"actions": out}, ""
|
| 433 |
+
|
| 434 |
+
|
| 435 |
+
def draft_flow(*, prompt, catalog, required, triggers, tables, chat=None, timeout=None,
|
| 436 |
+
st=None, user="", model=None):
|
| 437 |
+
"""A sentence -> `(draft, refusal_sentence, provider)`. β NOTHING IS SAVED HERE.
|
| 438 |
+
|
| 439 |
+
Exactly one of `draft` and `refusal_sentence` is truthy β the same contract
|
| 440 |
+
`web_agent.run_step` keeps, so a caller has no third case to get wrong.
|
| 441 |
+
|
| 442 |
+
β W35-T41 / C7 β `st`/`user` are the usage ledger's target, exactly as on `decide` above and for
|
| 443 |
+
the same reason: both of this function's callers (`automation_engine._ai_agent_plan` and
|
| 444 |
+
`routes_automation`'s draft door) are in lane D's fence. C7 says E adds the ledger line here and
|
| 445 |
+
D asserts it; the two keywords are what D has to pass for the line to be attributable.
|
| 446 |
+
"""
|
| 447 |
+
import usage_ledger # noqa: PLC0415
|
| 448 |
+
text = str(prompt or "").strip()[:MAX_PROMPT_CHARS]
|
| 449 |
+
if not text:
|
| 450 |
+
return None, "type what you want the automation to do", None
|
| 451 |
+
kinds = sorted({str(r.get("kind")) for r in (catalog or []) if r.get("ready")})
|
| 452 |
+
if not kinds:
|
| 453 |
+
return None, "this deployment offers no automation steps to build with", None
|
| 454 |
+
trigger_keys = sorted({str(t.get("key")) for t in (triggers or [])
|
| 455 |
+
if t.get("key") and not t.get("planned")}) or ["manual"]
|
| 456 |
+
table_keys = sorted({str(t.get("key")) for t in (tables or []) if t.get("key")}) or [""]
|
| 457 |
+
tools = [{"type": "function", "function": {
|
| 458 |
+
"name": "build_automation",
|
| 459 |
+
"description": "Emit the automation, or refuse.",
|
| 460 |
+
"parameters": flow_schema(kinds, trigger_keys, table_keys)}}]
|
| 461 |
+
messages = [{"role": "system",
|
| 462 |
+
"content": flow_system_prompt(catalog, required, triggers, tables)},
|
| 463 |
+
{"role": "user", "content": text}]
|
| 464 |
+
|
| 465 |
+
if chat is not None:
|
| 466 |
+
# β THE INJECTED PATH IS THE PROVEN PATH. It runs the SAME parse and the SAME refusal
|
| 467 |
+
# branches as a live call; only the transport is replaced.
|
| 468 |
+
draft, refusal = _flow_from_tool_call(chat(messages, tools))
|
| 469 |
+
return draft, refusal, "injected"
|
| 470 |
+
|
| 471 |
+
provs = flow_providers(model)
|
| 472 |
+
if not provs:
|
| 473 |
+
# β SAY SO. An AI feature that silently does nothing is indistinguishable from one that was
|
| 474 |
+
# never built [[flag-shipped-without-its-writer]].
|
| 475 |
+
return None, ("the assistant is not configured on this deployment, so an automation cannot "
|
| 476 |
+
"be drafted from a description yet"), None
|
| 477 |
+
tmo = float(timeout or TIMEOUT_SECONDS)
|
| 478 |
+
problems = []
|
| 479 |
+
import providers as _prov
|
| 480 |
+
for p in provs:
|
| 481 |
+
# βοΏ½οΏ½ W36-T35 / R4 β THE OWNER QUOTED THIS LINE BACK AT US. It used to read
|
| 482 |
+
# `problems.append(f"{p['name']}: tool calls are not wired for this shape")` and `continue`,
|
| 483 |
+
# so the sentence *"anthropic: tool calls are not wired for this shape"* appeared under the
|
| 484 |
+
# Agent module verbatim. R4: *"Anthropic becomes the tool-calling path that always works."*
|
| 485 |
+
# It is wired now, through the ONE wire in `providers`, and it matters more than it looks:
|
| 486 |
+
# every side rung on this account is refusing today (cerebras 402, groq 404, openrouter
|
| 487 |
+
# 402), so without this branch the drafter cannot answer at all.
|
| 488 |
+
# β NO `effort` ON THIS DOOR β `output_config.effort` errors on Haiku 4.5, the tier this
|
| 489 |
+
# ladder runs. The parameter is the caller's to send, which is why the wire takes it.
|
| 490 |
+
try:
|
| 491 |
+
if p["shape"] == "anthropic":
|
| 492 |
+
req = _prov.anthropic_request(
|
| 493 |
+
model=p["model"], key=os.environ[p["env"]].strip(), system=None,
|
| 494 |
+
messages=messages, tools=tools, max_tokens=1500, tool_choice="required")
|
| 495 |
+
r = requests.post(req["url"], timeout=tmo,
|
| 496 |
+
headers=req["headers"], json=req["json"])
|
| 497 |
+
else:
|
| 498 |
+
r = requests.post(p["url"], timeout=tmo,
|
| 499 |
+
headers={"Authorization": f"Bearer {os.environ[p['env']].strip()}",
|
| 500 |
+
"Content-Type": "application/json"},
|
| 501 |
+
json={"model": p["model"], "messages": messages, "tools": tools,
|
| 502 |
+
"tool_choice": "required", "temperature": 0.1,
|
| 503 |
+
"max_tokens": 1500})
|
| 504 |
+
except Exception as e: # noqa: BLE001
|
| 505 |
+
problems.append(f"{p['name']}: {type(e).__name__}")
|
| 506 |
+
continue
|
| 507 |
+
if r.status_code != 200:
|
| 508 |
+
# β A SENTENCE, NOT `HTTP 402` (R4's third clause). The owner read the status codes off
|
| 509 |
+
# this very door. `refusal_sentence` also decides whether this was a CREDIT failure, and
|
| 510 |
+
# the memo makes the next turn SKIP the empty account instead of paying for it again.
|
| 511 |
+
if _prov.is_credit_failure(r.status_code, r.text):
|
| 512 |
+
_prov.mark_no_credit(p["name"])
|
| 513 |
+
problems.append(_prov.refusal_sentence(p["name"], r.status_code, r.text))
|
| 514 |
+
continue
|
| 515 |
+
try:
|
| 516 |
+
body = r.json()
|
| 517 |
+
if p["shape"] == "anthropic":
|
| 518 |
+
_text, args, _refused = _prov.anthropic_read(body)
|
| 519 |
+
if _refused:
|
| 520 |
+
problems.append(f"{p['name']}: {_refused}")
|
| 521 |
+
continue
|
| 522 |
+
else:
|
| 523 |
+
calls = (((body.get("choices") or [{}])[0].get("message") or {})
|
| 524 |
+
.get("tool_calls") or [])
|
| 525 |
+
args = json.loads(calls[0]["function"]["arguments"]) if calls else None
|
| 526 |
+
except Exception as e: # noqa: BLE001
|
| 527 |
+
problems.append(f"{p['name']}: unreadable answer ({type(e).__name__})")
|
| 528 |
+
continue
|
| 529 |
+
# ββ C7 β THE LEDGER LINE. Placed after the parse rather than before it so an UNREADABLE
|
| 530 |
+
# answer is not double-counted by the `continue` above... β which means an unreadable answer
|
| 531 |
+
# is NOT counted at all, and that is a deliberate, disclosed loss: a body this code cannot
|
| 532 |
+
# parse is a body whose `usage` it also cannot trust, and `body` is out of scope in that
|
| 533 |
+
# branch by construction. The 200-with-junk case is rare and named here rather than
|
| 534 |
+
# silently rounded to zero.
|
| 535 |
+
ins, outs = usage_ledger.tokens_from(body)
|
| 536 |
+
usage_ledger.record("automation_draft", p["name"], p["model"], ins, outs,
|
| 537 |
+
total=usage_ledger.total_from(body), st=st, user=user)
|
| 538 |
+
draft, refusal = _flow_from_tool_call(args)
|
| 539 |
+
if draft is None and isinstance(args, dict) and str(args.get("kind")) == "refused":
|
| 540 |
+
# β A REFUSAL IS AN ANSWER, NOT A FAULT, so it does NOT fall through to a more
|
| 541 |
+
# expensive rung β the model has just said this cannot be built. `decide()` takes the
|
| 542 |
+
# same posture for the same reason.
|
| 543 |
+
return None, refusal, p["name"]
|
| 544 |
+
if draft is not None:
|
| 545 |
+
return draft, "", p["name"]
|
| 546 |
+
problems.append(f"{p['name']}: {refusal}")
|
| 547 |
+
return None, ("; ".join(problems)[:300] or "no provider answered"), None
|
api/automation_engine.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
api/connectors_tt.py
CHANGED
|
@@ -1,568 +1,568 @@
|
|
| 1 |
-
"""connectors_tt.py β the TIKTOK connector (wave 29 Β· item 7 Β· DEBT D-9 Β· rulings R1 + R2).
|
| 2 |
-
|
| 3 |
-
Everything in this file knows what a VENDOR's TikTok row looks like. Nothing in it knows what an
|
| 4 |
-
automation is. That split is `connectors_ig.py`'s (wave 27 item 23) and it is the reason this file
|
| 5 |
-
exists at all rather than another thousand lines inside the engine.
|
| 6 |
-
|
| 7 |
-
β **EVERY VENDOR FIELD NAME HERE WAS PROBED, NOT GUESSED.** The whole schema β 40 profile / 43
|
| 8 |
-
post / 17 comment fields, each with the vendor's own type, description and `pii` flag β was read
|
| 9 |
-
live from `GET /datasets/{id}/metadata` for **$0.00** and written down in
|
| 10 |
-
`.claude/wiki/waves/wave29/proto/tiktok-schema.md` (promoted to `tiktok-capture.md` at
|
| 11 |
-
close-out). That document is the AUTHORITY: do not re-probe it, and do not invent a key. Where a
|
| 12 |
-
name below reads through a candidate list it is because the vendor has two names for one fact
|
| 13 |
-
(`biography`/`signature`, `region`/`country`), never because the name is uncertain.
|
| 14 |
-
|
| 15 |
-
β **NOTHING HERE EVER AUTHENTICATES TO TIKTOK.** No login, no cookie, no account to get banned β
|
| 16 |
-
public data through a supplier, exactly the rail `connectors_ig.py` states for Instagram. The
|
| 17 |
-
vendor key is a key to a SUPPLIER.
|
| 18 |
-
|
| 19 |
-
β **THE TRANSPORT IS `connectors_bd.py` β SHARED, VENDOR-NAMED, AND NO LONGER BORROWED FROM THE
|
| 20 |
-
OTHER PLATFORM'S CONNECTOR** (WAVE 30 Β· T09, DEBT D-128). `bd_call`, `bd_scrape` and
|
| 21 |
-
`bd_filter_start` take the dataset id as a PARAMETER β they are Bright Data's wire, not
|
| 22 |
-
Instagram's β and re-implementing them here would be a second copy of the deferral handling, the
|
| 23 |
-
truncation guard, the SSRF rail and the snapshot-progress reader, i.e. five places for one bug.
|
| 24 |
-
Until wave 30 the code was right and the NAME was wrong: this file imported thirteen symbols from
|
| 25 |
-
`connectors_ig`, which read as a dependency on Instagram and was really a dependency on a supplier.
|
| 26 |
-
β **This file now imports ZERO names from `connectors_ig`, and a gate check asserts that**, because
|
| 27 |
-
the sentence above is the kind that quietly stops being true.
|
| 28 |
-
|
| 29 |
-
β **WHAT $0 COULD NOT BUY, so nobody reads this file as more measured than it is:**
|
| 30 |
-
1. the real ROW shape β `/metadata` describes a DATASET, and Instagram's rows carry undeclared
|
| 31 |
-
envelope keys (`timestamp`, `input`) that no metadata call mentions;
|
| 32 |
-
2. that a declared field POPULATES β Bright Data's Instagram Reels *declares* `views: number`
|
| 33 |
-
and delivers an account-grain wrong number (Β§4e). **Declared is not delivered**, and the one
|
| 34 |
-
TikTok claim that matters most (`play_count`) is exactly a declaration.
|
| 35 |
-
"""
|
| 36 |
-
from __future__ import annotations
|
| 37 |
-
|
| 38 |
-
import automation_engine as engine
|
| 39 |
-
# β T09 β FOUR NAMES CAME OFF THIS LIST AND NOTHING BROKE, which is the point of deriving an
|
| 40 |
-
# import block from the AST both ways. `bd_call`, `bd_filter_start`, `bd_key` and `bd_ready` were
|
| 41 |
-
# imported here under a comment claiming they were *"re-exported for the runners"*; no runner ever
|
| 42 |
-
# read them off this module (the engine imports them from the transport itself), so they were four
|
| 43 |
-
# lines of dependency nobody was paying for. `[[artifact-with-no-importer]]` in its smallest form.
|
| 44 |
-
from connectors_bd import (
|
| 45 |
-
_bd_first_url,
|
| 46 |
-
_bd_flag,
|
| 47 |
-
_bd_list,
|
| 48 |
-
_bd_source_payload,
|
| 49 |
-
_first,
|
| 50 |
-
_ig_int,
|
| 51 |
-
bd_scrape,
|
| 52 |
-
)
|
| 53 |
-
|
| 54 |
-
# ---------------------------------------------------------------------------------------------
|
| 55 |
-
# THE DATASETS
|
| 56 |
-
# ---------------------------------------------------------------------------------------------
|
| 57 |
-
# β CATALOGUE PRESENCE IS NOT ENTITLEMENT β the same finding Instagram produced. `GET
|
| 58 |
-
# /datasets/list` returned 1,735 rows of which 12 are TikTok; the three below answer 200 with a
|
| 59 |
-
# full field list, and the two DISCOVERY halves answer **404 for our key**:
|
| 60 |
-
# `gd_lj71gn6l68bz7y9hc` (posts by profile) and `gd_lilwhto81z415d9mdl` (posts by keyword).
|
| 61 |
-
# β TikTok discovery routes through the PROFILES dataset's corpus filter, exactly as Instagram's
|
| 62 |
-
# does. A ticket that reaches for a by-keyword endpoint is reaching for a 404.
|
| 63 |
-
TT_DS_PROFILES = "gd_l1villgoiiidt09ci" # TikTok - Profiles. 40 fields, 152,000,000 records
|
| 64 |
-
TT_DS_POSTS = "gd_lu702nij2f790tmv9h" # TikTok - Posts. 43 fields
|
| 65 |
-
TT_DS_COMMENTS = "gd_lkf2st302ap89utw5k" # TikTok - Comments. 17 fields
|
| 66 |
-
|
| 67 |
-
#: The vendor's two post-type tokens, verbatim from the dataset's own `ai_description`
|
| 68 |
-
#: (*"strictly these two"*, video = 99.5% of rows). Ours are `image`/`video`/`carousel`.
|
| 69 |
-
TT_POST_TYPE_VIDEO = "video"
|
| 70 |
-
TT_POST_TYPE_CONTENT = "content"
|
| 71 |
-
|
| 72 |
-
#: What a TikTok profile URL looks like, for the runner that has a handle and needs a URL. Kept
|
| 73 |
-
#: beside the dataset ids because it is the same class of vendor fact.
|
| 74 |
-
#: β ONE SPELLING. `platform/core/user_tables.profile_url(handle, 'tiktok')` builds the identical
|
| 75 |
-
#: string from `_PROFILE_RULES` (contract C2, E's half) β this exists for the connector's own
|
| 76 |
-
#: batch calls, and the gate asserts the two agree rather than trusting that they do.
|
| 77 |
-
TT_PROFILE_URL = "https://www.tiktok.com/@{handle}"
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
def tt_profile_url(handle):
|
| 81 |
-
"""`nurilab` β `https://www.tiktok.com/@nurilab`. `''` for a blank handle, never a bare `@`."""
|
| 82 |
-
h = str(handle or "").strip().lstrip("@")
|
| 83 |
-
return TT_PROFILE_URL.format(handle=h) if h else ""
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
# ---------------------------------------------------------------------------------------------
|
| 87 |
-
# THE FIELD MAPS
|
| 88 |
-
# ---------------------------------------------------------------------------------------------
|
| 89 |
-
# Each function turns ONE vendor row into the cell dict for one of the `ut_tt_*` schemas declared
|
| 90 |
-
# in `automation_engine`. The rules they all obey, stated once:
|
| 91 |
-
#
|
| 92 |
-
# * **BLANK MEANS NOT READ, NEVER "THEY HAVE NONE".** A key the vendor did not send is OMITTED,
|
| 93 |
-
# so a later, richer pull fills it instead of being overwritten by this one's silence. `_first`
|
| 94 |
-
# returns `None` (never 0) when nothing matches, which is what makes that possible.
|
| 95 |
-
# * **A zero from the vendor is a MEASUREMENT and survives.** (Instagram's `_ig_zero_is_blank`
|
| 96 |
-
# rule is scoped to a paid rung whose zeros were proven fictional; nothing here has earned it.)
|
| 97 |
-
# * **Every unpromoted vendor key stays whole in `source_payload`.** A schema addition on the
|
| 98 |
-
# vendor's side is preserved rather than silently discarded while our column model catches up.
|
| 99 |
-
# * **Nothing here writes a session token.** `tt_chain_token`, `secu_id` (~85% null), `short_id`
|
| 100 |
-
# (100% null in the sample), `ftc` (100% null) and `relation` are deliberately unmapped; they
|
| 101 |
-
# ride in `source_payload` where they make no claim.
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
def _tt_str(node, *names):
|
| 105 |
-
"""The first non-empty string among `names`, or None when the vendor sent nothing.
|
| 106 |
-
|
| 107 |
-
β `None`, NOT `""`. The callers below drop `None` keys, which is what keeps a blank honest β
|
| 108 |
-
an empty string written into a cell claims "we looked and it is empty".
|
| 109 |
-
"""
|
| 110 |
-
v = _first(node, *names)
|
| 111 |
-
if v is None:
|
| 112 |
-
return None
|
| 113 |
-
s = str(v).strip()
|
| 114 |
-
return s or None
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
def _tt_pct(node, *names):
|
| 118 |
-
"""A vendor 0β1 engagement fraction β our stored 0β100 percentage, or None.
|
| 119 |
-
|
| 120 |
-
β THE Γ100 IS NOT COSMETIC (wave 26, amendment C1-a). Our `pct` renderer appends the sign to
|
| 121 |
-
the STORED number, so writing the vendor's raw 0.0656 would print a 6.6% creator as `0.0%` β
|
| 122 |
-
measured on the Instagram side, and TikTok sends the same shape on all three of its rates.
|
| 123 |
-
"""
|
| 124 |
-
v = _first(node, *names)
|
| 125 |
-
if v is None:
|
| 126 |
-
return None
|
| 127 |
-
out = engine._pct100(v)
|
| 128 |
-
return out or None
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
def _tt_day(node, *names):
|
| 132 |
-
"""A vendor stamp β `YYYY-MM-DD`, or None. Our `date` columns store a day."""
|
| 133 |
-
v = _first(node, *names)
|
| 134 |
-
if v is None:
|
| 135 |
-
return None
|
| 136 |
-
return engine._day(v) or None
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
def _drop_blanks(row):
|
| 140 |
-
"""The one place a mapped row loses its `None`s β see the BLANK MEANS NOT READ rule above."""
|
| 141 |
-
return {k: v for k, v in row.items() if v is not None and v != ""}
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
def normalize_profile(node, handle=""):
|
| 145 |
-
"""A TikTok Profiles row β the `ut_tt_profile` / `ut_tt_snapshots` cell shape.
|
| 146 |
-
|
| 147 |
-
β TWO FIELDS READ THROUGH A CANDIDATE PAIR, and both pairs are the vendor's, not a guess:
|
| 148 |
-
* `biography` is PRIMARY and `signature` the FALLBACK β the probe measured `signature`
|
| 149 |
-
populated on 85% of rows and they carry the same text;
|
| 150 |
-
* `region` is PRIMARY and `country` the FALLBACK β `region` is the one with a documented
|
| 151 |
-
two-letter-ISO description, `country` has no description at all.
|
| 152 |
-
β `videos_count` is mapped to `posts_count` with a caveat recorded rather than hidden: its
|
| 153 |
-
`ai_description` ranges 1-89, so it may be a WINDOW rather than a lifetime total. It is the
|
| 154 |
-
only count of its kind the dataset offers.
|
| 155 |
-
"""
|
| 156 |
-
node = node if isinstance(node, dict) else {}
|
| 157 |
-
account = _tt_str(node, "account_id") or str(handle or "").strip().lstrip("@")
|
| 158 |
-
return _drop_blanks({
|
| 159 |
-
"platform": engine.PLATFORM_TIKTOK,
|
| 160 |
-
"handle": account,
|
| 161 |
-
"full_name": _tt_str(node, "nickname"),
|
| 162 |
-
"tt_id": _tt_str(node, "id"),
|
| 163 |
-
"profile_url": _tt_str(node, "url") or (tt_profile_url(account) or None),
|
| 164 |
-
"bio": _tt_str(node, "biography", "signature"),
|
| 165 |
-
"external_url": _bd_first_url(_first(node, "bio_link")),
|
| 166 |
-
"verified": _bd_flag(node, "is_verified"),
|
| 167 |
-
"is_private": _bd_flag(node, "is_private"),
|
| 168 |
-
# β APPROXIMATE, and the word is the vendor's: `is_commerce_user` has *"many null values"*
|
| 169 |
-
# on their own description. It is the closest thing TikTok has to Instagram's
|
| 170 |
-
# `is_business_account`, and `_bd_flag` writes nothing at all when the key is absent β so
|
| 171 |
-
# the approximation only ever fills a cell the vendor actually answered.
|
| 172 |
-
"is_business": _bd_flag(node, "is_commerce_user"),
|
| 173 |
-
"followers": _ig_int(_first(node, "followers")),
|
| 174 |
-
"following": _ig_int(_first(node, "following")),
|
| 175 |
-
"posts_count": _ig_int(_first(node, "videos_count")),
|
| 176 |
-
# β `likes` on a PROFILE row is likes RECEIVED across the account's videos (18-110,200, no
|
| 177 |
-
# nulls). It is not a post-level number and it is not our `likes` column, which is why it
|
| 178 |
-
# is stored under a different name.
|
| 179 |
-
"likes_received": _ig_int(_first(node, "likes")),
|
| 180 |
-
"avg_engagement": _tt_pct(node, "awg_engagement_rate"),
|
| 181 |
-
"like_engagement": _tt_pct(node, "like_engagement_rate"),
|
| 182 |
-
"comment_engagement": _tt_pct(node, "comment_engagement_rate"),
|
| 183 |
-
"country_code": _tt_str(node, "region", "country"),
|
| 184 |
-
"region": _tt_str(node, "region"),
|
| 185 |
-
"predicted_lang": _tt_str(node, "predicted_lang"),
|
| 186 |
-
# β ACCOUNT AGE, NOT A MEASUREMENT STAMP β `create_time` on a profile is when the ACCOUNT
|
| 187 |
-
# was made. TikTok stamps nothing with "when this number was true", exactly like Instagram,
|
| 188 |
-
# which is why the append law dates a snapshot by when WE read it.
|
| 189 |
-
"account_created_at": _tt_day(node, "create_time"),
|
| 190 |
-
"source_payload": _bd_source_payload(node),
|
| 191 |
-
})
|
| 192 |
-
|
| 193 |
-
|
| 194 |
-
def tt_post_type(node):
|
| 195 |
-
"""A TikTok post row β one of OUR three type options, or None.
|
| 196 |
-
|
| 197 |
-
β THE FIRST OF THE PROBE DOC'S TWO NAMED BLOCKERS. The vendor's vocabulary is `"video"` /
|
| 198 |
-
`"content"`; ours is `image` / `video` / `carousel` and has no `"content"`. Writing the
|
| 199 |
-
vendor's token would fail `_clean_field`'s option check on the way in and would put an
|
| 200 |
-
untranslated API word in front of a user on the way out.
|
| 201 |
-
|
| 202 |
-
So: `video` is `video`, and `content` β TikTok's photo-mode post β is `image`, EXCEPT when the
|
| 203 |
-
row carries more than one `carousel_images` entry, which is what a carousel IS on either
|
| 204 |
-
network. β The multi-image branch is decided from the IMAGES, never from the type token: the
|
| 205 |
-
token cannot express it, so inferring `carousel` from the word would be inventing a fact.
|
| 206 |
-
β An UNKNOWN token returns None rather than defaulting to `video` (99.5% of rows are video, and
|
| 207 |
-
that is exactly what would make the wrong default invisible).
|
| 208 |
-
"""
|
| 209 |
-
raw = str((node or {}).get("post_type") or "").strip().lower()
|
| 210 |
-
if raw == TT_POST_TYPE_VIDEO:
|
| 211 |
-
return "video"
|
| 212 |
-
if raw != TT_POST_TYPE_CONTENT:
|
| 213 |
-
return None
|
| 214 |
-
images = (node or {}).get("carousel_images")
|
| 215 |
-
return "carousel" if isinstance(images, list) and len(images) > 1 else "image"
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
def normalize_post(node):
|
| 219 |
-
"""A TikTok Posts row β the `ut_tt_posts` cell shape. None when it carries no identity.
|
| 220 |
-
|
| 221 |
-
β THE SECOND NAMED BLOCKER, RESOLVED HERE AND NOWHERE ELSE: `play_count` is ONE number and
|
| 222 |
-
Instagram's schema has TWO columns for it (`plays` and `views`). On TikTok they are the same
|
| 223 |
-
fact β `play_count` IS the count TikTok displays under a video β so it maps to `views` and
|
| 224 |
-
`ut_tt_posts` HAS NO `plays` COLUMN. Copying one vendor number into two of our columns would
|
| 225 |
-
manufacture a second measurement that a rollup could average or double-count, which is a worse
|
| 226 |
-
outcome than the missing column it would paper over.
|
| 227 |
-
|
| 228 |
-
β `shortcode` reads `shortcode` then `post_id`: both are 19-digit numerics on this dataset and
|
| 229 |
-
the probe measured them as the same shape. That equality is what lets the commentsβposts link
|
| 230 |
-
join with NO normaliser, which the Instagram side never had.
|
| 231 |
-
β `num_share_count` (a number) is preferred over `share_count` (typed TEXT by the vendor).
|
| 232 |
-
β `commerce_info` is a business/commerce LOCATION per its own description β cities and
|
| 233 |
-
countries. It is NOT a paid-partnership flag, and nothing in this dataset is: TikTok declares
|
| 234 |
-
no equivalent, so `paid_partnership`/`partner` have no column on this family at all.
|
| 235 |
-
"""
|
| 236 |
-
node = node if isinstance(node, dict) else {}
|
| 237 |
-
shortcode = _tt_str(node, "shortcode", "post_id")
|
| 238 |
-
if not shortcode:
|
| 239 |
-
return None
|
| 240 |
-
return _drop_blanks({
|
| 241 |
-
"platform": engine.PLATFORM_TIKTOK,
|
| 242 |
-
"shortcode": shortcode,
|
| 243 |
-
# ββ `account_id`, NOT `profile_username` β MEASURED on a real Posts row 2026-08-12.
|
| 244 |
-
# The vendor's `profile_username` is the DISPLAY NAME (`"Dina"`), while `account_id` is the
|
| 245 |
-
# @handle (`"d1na_th"`) β the same field `normalize_profile` already reads for `handle`, so
|
| 246 |
-
# one name means one thing across both corpora. Reading the display name silently broke the
|
| 247 |
-
# only join this table has: `ut_tt_posts.influencer_key` -> `ut_tt_profile.handle` matched
|
| 248 |
-
# NOTHING, so a person could not filter posts by creator and a rollup would count zero.
|
| 249 |
-
# β NO FALLBACK TO `profile_username`, deliberately. It is not a degraded handle, it is a
|
| 250 |
-
# different fact, and filling a join key with it is worse than leaving it blank β a blank
|
| 251 |
-
# is visibly missing, a display name looks like an answer [[one-question-two-normalizers]].
|
| 252 |
-
# The URL is the honest second source: it carries the handle by construction.
|
| 253 |
-
"influencer_key": (_tt_str(node, "account_id")
|
| 254 |
-
or tt_handle(_tt_str(node, "profile_url", "url") or "") or None),
|
| 255 |
-
"posted_at": _tt_day(node, "create_time"),
|
| 256 |
-
"type": tt_post_type(node),
|
| 257 |
-
"caption": _tt_str(node, "description"),
|
| 258 |
-
"url": _tt_str(node, "url"),
|
| 259 |
-
"hashtags": _bd_list(node, "hashtags"),
|
| 260 |
-
"tagged_location": _tt_str(node, "commerce_info"),
|
| 261 |
-
"views": _ig_int(_first(node, "play_count")),
|
| 262 |
-
"likes": _ig_int(_first(node, "digg_count")),
|
| 263 |
-
"comments": _ig_int(_first(node, "comment_count")),
|
| 264 |
-
"shares": _ig_int(_first(node, "num_share_count")),
|
| 265 |
-
"saves": _ig_int(_first(node, "collect_count")),
|
| 266 |
-
"video_duration": _ig_int(_first(node, "video_duration")),
|
| 267 |
-
"source_payload": _bd_source_payload(node),
|
| 268 |
-
})
|
| 269 |
-
|
| 270 |
-
|
| 271 |
-
def normalize_comment(node):
|
| 272 |
-
"""A TikTok Comments row β the `ut_tt_comments` cell shape. None without a comment id.
|
| 273 |
-
|
| 274 |
-
β THE NAME COLLISION, RESOLVED: TikTok's `replies` is an ARRAY of reply objects and OUR
|
| 275 |
-
`replies` column is an INT count. The count comes from `num_replies`; the array stays whole in
|
| 276 |
-
`source_payload`. Reading the array's length instead would be a second, disagreeing answer to
|
| 277 |
-
a question the vendor already answers β and it would disagree, because a page of replies is not
|
| 278 |
-
all of them.
|
| 279 |
-
β `date_created` is typed `date` by this vendor, unlike Instagram's `comment_date` which is
|
| 280 |
-
text and needs defensive parsing. It still goes through `_tt_day` β one date path, so a vendor
|
| 281 |
-
that changes its mind cannot change ours.
|
| 282 |
-
β The comment TEXT and every identifiable commenter field (`commenter_user_name` is flagged
|
| 283 |
-
PII) stay in `source_payload` and are promoted to no column, which is the same posture the
|
| 284 |
-
Instagram comment schema takes for the same D-24 reason.
|
| 285 |
-
"""
|
| 286 |
-
node = node if isinstance(node, dict) else {}
|
| 287 |
-
comment_key = _tt_str(node, "comment_id")
|
| 288 |
-
if not comment_key:
|
| 289 |
-
return None
|
| 290 |
-
return _drop_blanks({
|
| 291 |
-
"platform": engine.PLATFORM_TIKTOK,
|
| 292 |
-
"comment_key": comment_key,
|
| 293 |
-
"shortcode": _tt_str(node, "post_id"),
|
| 294 |
-
# ββ OWNER RULING 2026-08-12: the comment's CONTENT gets a column. `comment_text` is the
|
| 295 |
-
# vendor's own key and `comment_text_only` its stripped variant (the probe recorded both);
|
| 296 |
-
# primary first, so a row carrying the rich form is not silently served the plain one.
|
| 297 |
-
# β The commenter's identity is deliberately NOT promoted β see `TT_COMMENT_FIELDS`.
|
| 298 |
-
"text": _tt_str(node, "comment_text", "comment_text_only"),
|
| 299 |
-
"commented_at": _tt_day(node, "date_created"),
|
| 300 |
-
"likes": _ig_int(_first(node, "num_likes")),
|
| 301 |
-
"replies": _ig_int(_first(node, "num_replies")),
|
| 302 |
-
"source_payload": _bd_source_payload(node),
|
| 303 |
-
})
|
| 304 |
-
|
| 305 |
-
|
| 306 |
-
#: β The three maps, addressable by name β so a gate (and the runners in T04-T06) can walk them
|
| 307 |
-
#: rather than naming three functions, and so adding a fourth dataset is one entry.
|
| 308 |
-
TT_NORMALIZERS = {
|
| 309 |
-
"tt_profile": normalize_profile,
|
| 310 |
-
"tt_post_metrics": normalize_post,
|
| 311 |
-
"tt_comments": normalize_comment,
|
| 312 |
-
}
|
| 313 |
-
|
| 314 |
-
|
| 315 |
-
# ---------------------------------------------------------------------------------------------
|
| 316 |
-
# THE FETCH β wave 30 Β· W30-T08 (carrying wave-29's dropped T05)
|
| 317 |
-
# ---------------------------------------------------------------------------------------------
|
| 318 |
-
|
| 319 |
-
def tt_handle(url):
|
| 320 |
-
"""A TikTok profile URL **or** a bare handle β the handle. `''` when it is neither.
|
| 321 |
-
|
| 322 |
-
Deliberately permissive about the input and strict about the output, because the two callers
|
| 323 |
-
hand it different things: an automation stores whatever a person typed in the profile column
|
| 324 |
-
(`@nurilab`, `nurilab`, or the full URL), while the discovery runner already holds a clean
|
| 325 |
-
`account_id`. One normaliser, so a row found by discovery and a row typed by hand cannot
|
| 326 |
-
resolve to two different handles.
|
| 327 |
-
"""
|
| 328 |
-
s = str(url or "").strip()
|
| 329 |
-
if not s:
|
| 330 |
-
return ""
|
| 331 |
-
if "tiktok.com" in s.lower():
|
| 332 |
-
# Everything after the first `@`, up to the next path segment or query.
|
| 333 |
-
tail = s.split("@", 1)[1] if "@" in s else ""
|
| 334 |
-
s = tail.split("/")[0].split("?")[0].split("#")[0]
|
| 335 |
-
s = s.strip().lstrip("@").strip()
|
| 336 |
-
# A handle is the vendor's `account_id` shape: alphanumerics, dots and underscores.
|
| 337 |
-
return s if s and all(c.isalnum() or c in "._" for c in s) else ""
|
| 338 |
-
|
| 339 |
-
|
| 340 |
-
def tt_post_urls(node, limit=0):
|
| 341 |
-
"""β WAVE 30 Β· T10 β the profile row's own post permalinks, newest-first as the vendor sends.
|
| 342 |
-
|
| 343 |
-
β THIS IS WHY TIKTOK POST CAPTURE COSTS NO EXTRA DISCOVERY. `top_videos` rides the PROFILE row
|
| 344 |
-
we have already bought, so the posts read is a scrape of links we hold, never a search for them.
|
| 345 |
-
The two TikTok DISCOVERY datasets (posts-by-profile, posts-by-keyword) are **404 for our key**,
|
| 346 |
-
so a design that reached for either would not merely be dearer, it would not work.
|
| 347 |
-
|
| 348 |
-
ββ CORRECTED 2026-08-12 β THIS DOCSTRING USED TO SAY *"the probe MEASURED `top_videos` as an
|
| 349 |
-
array of video permalinks, NO empties"*, AND THAT SENTENCE IS WHAT SHIPPED THE BUG. The probe
|
| 350 |
-
read `/datasets/{id}/metadata` β a DATASET description β and the phrase quoted was the field's
|
| 351 |
-
`ai_description`, not an observation of a row. The real row sends **dicts keyed `video_url`**
|
| 352 |
-
(measured below), so the reader built against the quoted sentence found nothing, forever, in
|
| 353 |
-
silence. β The transferable half: *"the probe measured X"* and *"the probe read a declaration
|
| 354 |
-
of X"* are different claims, and prose cannot be told apart by a reader downstream β which is
|
| 355 |
-
why the correction names the method, not just the value.
|
| 356 |
-
|
| 357 |
-
β `top_posts_data` is deliberately NOT read: the probe calls it *"a thin dup of `top_videos`"*,
|
| 358 |
-
and preferring whichever happened to be longer is how one creator's window silently differs
|
| 359 |
-
from another's.
|
| 360 |
-
β `limit <= 0` means "everything the row carried". The CAP IS THE CALLER'S β `config.maxPosts`,
|
| 361 |
-
validated 1..12 β and it is applied here rather than after the scrape so an unwanted post is
|
| 362 |
-
never bought. [[a-constant-two-features-share]]: the 12 is the vendor's measured profile window,
|
| 363 |
-
not a number this function may invent.
|
| 364 |
-
"""
|
| 365 |
-
raw = (node or {}).get("top_videos")
|
| 366 |
-
out = []
|
| 367 |
-
for item in raw if isinstance(raw, list) else []:
|
| 368 |
-
# ββ MEASURED ON A REAL ROW 2026-08-12, AND IT IS NOT WHAT THE SCHEMA SAID.
|
| 369 |
-
# `top_videos` is NOT an array of permalink strings. One paid TikTok Profiles scrape of a
|
| 370 |
-
# live handle returns **19 DICTS**, keyed
|
| 371 |
-
# `video_url Β· video_id Β· playcount Β· diggcount Β· commentcount Β· share_count Β·
|
| 372 |
-
# favorites_count Β· create_date Β· cover_image`.
|
| 373 |
-
# The docstring above cites the $0 probe as having "measured" permalinks β it had not, and
|
| 374 |
-
# could not: `/datasets/{id}/metadata` describes a DATASET, and the probe's own verdict says
|
| 375 |
-
# so in terms (*"what $0 cannot buy β¦ the real ROW shape β¦ that a declared field
|
| 376 |
-
# POPULATES"*, `wave29/proto/tiktok-schema.md`). This is the SECOND time this vendor's
|
| 377 |
-
# declaration has diverged from its delivery on this exact axis; BD's IG Reels `views` was
|
| 378 |
-
# the first. [[reachable-is-not-the-same-as-built]]
|
| 379 |
-
# β The consequence, live: `TT_DS_POSTS` was never reached, because this returned an EMPTY
|
| 380 |
-
# list on every real profile β post capture could not have worked for anybody, and the
|
| 381 |
-
# T10 gate stayed green because its canned fixture encoded the DECLARED shape. A fixture
|
| 382 |
-
# written from a schema tests the schema.
|
| 383 |
-
# β `video_url` is FIRST because it is the key the vendor actually sends; `url` is kept
|
| 384 |
-
# because it costs nothing and is what a future corpus revision would most likely use. The
|
| 385 |
-
# bare-string branch stays for the same reason β this widens what is ACCEPTED and invents
|
| 386 |
-
# nothing: a shape that yields no `httpβ¦` value still degrades to "no posts", exactly as
|
| 387 |
-
# before, rather than to a URL built out of a guess.
|
| 388 |
-
# β `top_posts_data` is STILL not read (it carries `post_url` and would work): preferring
|
| 389 |
-
# whichever array happened to be longer is how one creator's window silently differs from
|
| 390 |
-
# another's, and that reasoning is unchanged by this correction.
|
| 391 |
-
if isinstance(item, dict):
|
| 392 |
-
u = str(item.get("video_url") or item.get("url") or "").strip()
|
| 393 |
-
else:
|
| 394 |
-
u = str(item or "").strip()
|
| 395 |
-
if u.lower().startswith("http") and u not in out:
|
| 396 |
-
out.append(u)
|
| 397 |
-
return out[:limit] if limit and limit > 0 else out
|
| 398 |
-
|
| 399 |
-
|
| 400 |
-
def pull_posts_tt(post_urls, log=print, deferred=None):
|
| 401 |
-
"""The TikTok Posts dataset for a list of permalinks β `(rows, note)`, already normalised.
|
| 402 |
-
|
| 403 |
-
β ONE CALL FOR THE WHOLE WINDOW. `bd_scrape` has always taken a list, and the Instagram side
|
| 404 |
-
measured what happens when a caller forgets: 25 records, one billed snapshot each, a walk still
|
| 405 |
-
running at 67 minutes. Nothing here loops per URL.
|
| 406 |
-
"""
|
| 407 |
-
urls = [str(u) for u in (post_urls or []) if str(u or "").strip()]
|
| 408 |
-
if not urls:
|
| 409 |
-
return [], ""
|
| 410 |
-
rows, note = bd_scrape(TT_DS_POSTS, urls, deferred=deferred)
|
| 411 |
-
if note:
|
| 412 |
-
log(f"[aios-tt] posts: {note}")
|
| 413 |
-
return [], note
|
| 414 |
-
out = [r for r in (normalize_post(n) for n in rows) if r]
|
| 415 |
-
return out, ""
|
| 416 |
-
|
| 417 |
-
|
| 418 |
-
def pull_comments_tt(post_urls, log=print, deferred=None):
|
| 419 |
-
"""The TikTok Comments dataset for a list of POST permalinks β `(rows, note)`, normalised.
|
| 420 |
-
|
| 421 |
-
β THE MOST EXPENSIVE THING THIS PRODUCT BUYS, and the reason `commentMetrics` defaults OFF on
|
| 422 |
-
both networks: a comments scrape ingests identifiable third parties who never entered anybody's
|
| 423 |
-
list (D-24). The mapper already keeps every commenter field in `source_payload` and promotes
|
| 424 |
-
none of them to a column; this function adds no new exposure, it just has to be asked for.
|
| 425 |
-
"""
|
| 426 |
-
urls = [str(u) for u in (post_urls or []) if str(u or "").strip()]
|
| 427 |
-
if not urls:
|
| 428 |
-
return [], ""
|
| 429 |
-
rows, note = bd_scrape(TT_DS_COMMENTS, urls, deferred=deferred)
|
| 430 |
-
if note:
|
| 431 |
-
log(f"[aios-tt] comments: {note}")
|
| 432 |
-
return [], note
|
| 433 |
-
out = [r for r in (normalize_comment(n) for n in rows) if r]
|
| 434 |
-
return out, ""
|
| 435 |
-
|
| 436 |
-
|
| 437 |
-
#: ββ WAVE 30 Β· D-156 β THE MEDIA DATASETS, AS A SET, SO THE HAND-OFF CAN FILTER ON IDENTITY.
|
| 438 |
-
#: `_media_deferrals` uses this to lift ONLY posts/comments snapshots out of the local deferral
|
| 439 |
-
#: list. That is what makes it structurally impossible to file a PROFILE snapshot in the engine's
|
| 440 |
-
#: metric queue β the defect a draft of T10 shipped and A-39 booked as "the wrong fix is worse
|
| 441 |
-
#: than the gap". A membership test cannot be got wrong by a later edit the way `if` order can.
|
| 442 |
-
TT_MEDIA_DATASETS = (TT_DS_POSTS, TT_DS_COMMENTS)
|
| 443 |
-
|
| 444 |
-
|
| 445 |
-
def _media_deferrals(deferred):
|
| 446 |
-
"""The POSTS/COMMENTS entries of a `bd_scrape` deferral list β never the profile's.
|
| 447 |
-
|
| 448 |
-
β The engine, not this module, decides what a deferral MEANS: it stamps `kind` and the
|
| 449 |
-
handle and files it. TikTok needs no `_tag_metric_deferrals` twin because one dataset is one
|
| 450 |
-
kind here, so the id already carries everything a mapper choice depends on β and importing
|
| 451 |
-
Instagram's tagger is not available anyway (W30-T09 gates ZERO `from connectors_ig` lines).
|
| 452 |
-
"""
|
| 453 |
-
out = []
|
| 454 |
-
for d in deferred or []:
|
| 455 |
-
if isinstance(d, dict) and str(d.get("datasetId") or "") in TT_MEDIA_DATASETS:
|
| 456 |
-
out.append(dict(d))
|
| 457 |
-
return out
|
| 458 |
-
|
| 459 |
-
|
| 460 |
-
def pull_profile_tt(url, log=print, pending_profile=None, prefetch=None,
|
| 461 |
-
max_posts=0, post_metrics=False, comment_metrics=False):
|
| 462 |
-
"""ONE TikTok profile from the vendor. Same return contract as `pull_profile`.
|
| 463 |
-
|
| 464 |
-
`{state, profile, posts, comments, via, note}` with `state β ok | partial | blocked | error`,
|
| 465 |
-
so the engine's enrich branch treats every network identically and no caller learns a new
|
| 466 |
-
shape.
|
| 467 |
-
|
| 468 |
-
β WAVE 30 Β· T10 β POSTS AND COMMENTS ARE REAL NOW, AND BOTH DEFAULT OFF, exactly as Instagram's
|
| 469 |
-
do. `post_metrics` scrapes the profile row's own `top_videos` permalinks (see `tt_post_urls` β
|
| 470 |
-
no discovery call, because both TikTok discovery datasets 404 for our key); `comment_metrics`
|
| 471 |
-
then scrapes the comments of the posts that came back. β COMMENTS REQUIRE POSTS by construction
|
| 472 |
-
rather than by a rule: their input IS a post permalink, so asking for comments with post capture
|
| 473 |
-
off is a request with no subject, and it returns none instead of quietly buying posts nobody
|
| 474 |
-
asked for.
|
| 475 |
-
|
| 476 |
-
β `partial` IS THE SUCCESS STATE WHENEVER NO MEDIA WAS READ, and that is deliberate rather than
|
| 477 |
-
pessimistic. The Instagram contract reads `ok` only when identity AND media both landed
|
| 478 |
-
(`pull_profile_bd`: *"identity without media is still partial ... a run that wrote a follower
|
| 479 |
-
count and no posts must not paint green over a posts table that did not grow"*). So: posts not
|
| 480 |
-
ASKED for β `partial`, saying so; posts asked for and landed β `ok`; asked for and none came β
|
| 481 |
-
`partial` with the vendor's reason. The state answers "did this pull deliver what it went for",
|
| 482 |
-
never "did the function finish".
|
| 483 |
-
|
| 484 |
-
β **NO FREE RUNG, AND NO FALLBACK CHAIN.** Instagram's `pull_profile` drops to Apify when the
|
| 485 |
-
paid rung refuses; `providers.DEFAULT_CHAINS["tt_profile"]` is deliberately single-provider,
|
| 486 |
-
with its own note explaining that a multi-provider chain is a promise something walks it and
|
| 487 |
-
that nothing walks Instagram's second name today either. So a refusal here is final, and it
|
| 488 |
-
says so instead of implying a retry somewhere.
|
| 489 |
-
"""
|
| 490 |
-
handle = tt_handle(url)
|
| 491 |
-
if not handle:
|
| 492 |
-
return {"state": "error", "profile": {}, "posts": [], "comments": [], "via": "",
|
| 493 |
-
"note": f"{url!r} is not a TikTok profile URL or handle"}
|
| 494 |
-
|
| 495 |
-
# β THE BATCH FAST PATH, same shape as the Instagram side: `prefetch` is `{handle: node}` from
|
| 496 |
-
# one multi-URL scrape covering a whole selection. A hit is a vendor round trip that does not
|
| 497 |
-
# happen; a miss falls through to the single-URL call below.
|
| 498 |
-
cached = prefetch.get(handle) if isinstance(prefetch, dict) else None
|
| 499 |
-
_deferred = []
|
| 500 |
-
if isinstance(cached, dict) and cached:
|
| 501 |
-
rows, note = [cached], ""
|
| 502 |
-
else:
|
| 503 |
-
rows, note = bd_scrape(TT_DS_PROFILES, [tt_profile_url(handle)], deferred=_deferred)
|
| 504 |
-
|
| 505 |
-
node = rows[0] if rows else {}
|
| 506 |
-
profile = normalize_profile(node, handle) if node else {}
|
| 507 |
-
# β THE READABILITY TEST IS `followers`/`following`, NOT "did we get a dict". `normalize_profile`
|
| 508 |
-
# drops blanks, so an unreadable row still returns `{"platform": β¦, "handle": β¦}` β truthy, and
|
| 509 |
-
# carrying nothing anybody asked for. The Instagram rung tests exactly this pair for exactly
|
| 510 |
-
# this reason, and answering "0 followers" instead is the failure it exists to prevent.
|
| 511 |
-
unreadable = profile.get("followers") is None and profile.get("following") is None
|
| 512 |
-
if note or unreadable:
|
| 513 |
-
# β THE DEFERRAL IS HANDED OVER RATHER THAN DISCARDED. A snapshot the vendor is still
|
| 514 |
-
# building HAS ALREADY BEEN PAID FOR; dropping its id bills again on the next run for the
|
| 515 |
-
# same record. That was live on the Instagram profile path until 2026-08-09 β measured on
|
| 516 |
-
# nurilab as two runs, two fresh snapshots, both abandoned β and it is not being
|
| 517 |
-
# reintroduced here by omission.
|
| 518 |
-
if isinstance(pending_profile, list):
|
| 519 |
-
for d in _deferred:
|
| 520 |
-
pending_profile.append({**d, "kind": "profile", "influencer": handle})
|
| 521 |
-
why = note or ("the scrape answered, but no follower/following counts were readable in it "
|
| 522 |
-
"(the field names may have moved - see tiktok-capture.md)")
|
| 523 |
-
return {"state": "blocked", "profile": {}, "posts": [], "comments": [], "via": "brightdata",
|
| 524 |
-
"deferredProfile": [d.get("snapshotId") for d in _deferred],
|
| 525 |
-
"note": why}
|
| 526 |
-
|
| 527 |
-
# --- W30-T10: THE MEDIA, ONLY WHEN IT WAS ASKED FOR. ------------------------------------
|
| 528 |
-
if not post_metrics:
|
| 529 |
-
return {"state": "partial", "profile": profile, "posts": [], "comments": [],
|
| 530 |
-
"via": "brightdata",
|
| 531 |
-
"note": note or "profile read; post capture is off for this step"}
|
| 532 |
-
urls = tt_post_urls(node, limit=max_posts)
|
| 533 |
-
if not urls:
|
| 534 |
-
# β NOT AN ERROR AND NOT A RETRY. A creator with no `top_videos` has nothing to buy, and
|
| 535 |
-
# saying so is what stops the next run paying to be told the same thing.
|
| 536 |
-
return {"state": "partial", "profile": profile, "posts": [], "comments": [],
|
| 537 |
-
"via": "brightdata",
|
| 538 |
-
"note": note or "profile read; this account's row carried no post links"}
|
| 539 |
-
posts, p_note = pull_posts_tt(urls, log=log, deferred=_deferred)
|
| 540 |
-
comments, c_note = ([], "")
|
| 541 |
-
if comment_metrics and posts:
|
| 542 |
-
# The comments dataset is keyed on a POST permalink, so it reads the posts we just bought β
|
| 543 |
-
# `url` from the mapper, never the profile's raw array, so a post the posts scrape refused
|
| 544 |
-
# is not silently asked about again one rung later.
|
| 545 |
-
comments, c_note = pull_comments_tt([p.get("url") for p in posts if p.get("url")],
|
| 546 |
-
log=log, deferred=_deferred)
|
| 547 |
-
# ββ WAVE 30 Β· D-156 β THE MEDIA DEFERRALS ARE HANDED BACK, and the shape of the hand-off is
|
| 548 |
-
# the whole lesson. An earlier draft of T10 appended every `_deferred` entry to
|
| 549 |
-
# `pending_profile` tagged `kind: "profile"`. By the time control reaches here a PROFILE
|
| 550 |
-
# deferral is impossible β the profile branch above returns `blocked` on any note β so **every
|
| 551 |
-
# id fanned out that way was a POSTS or COMMENTS snapshot in the PROFILE queue**, whose
|
| 552 |
-
# collector writes preset profile cells onto somebody's record from post rows. The engine keeps
|
| 553 |
-
# the two queues apart deliberately (`_pending_profile_tasks` vs `_pending_metric_tasks`).
|
| 554 |
-
# β So this returns them under their OWN key, filtered by dataset identity
|
| 555 |
-
# (`_media_deferrals`), and the engine files them in the metric queue with the handle it
|
| 556 |
-
# already holds. Returning rather than appending also keeps the queue's vocabulary out of a
|
| 557 |
-
# connector: this module knows which CORPUS deferred, never what the engine calls it.
|
| 558 |
-
# β `deferredMedia` rides BOTH returns on purpose. The empty-posts case is the one that
|
| 559 |
-
# matters most β that is exactly the run where the vendor took too long, so a caller reading
|
| 560 |
-
# the ids only from the success path would lose every batch it actually paid for.
|
| 561 |
-
deferred_media = _media_deferrals(_deferred)
|
| 562 |
-
if not posts:
|
| 563 |
-
return {"state": "partial", "profile": profile, "posts": [], "comments": [],
|
| 564 |
-
"via": "brightdata", "deferredMedia": deferred_media,
|
| 565 |
-
"note": p_note or note or "profile read; the post source returned nothing"}
|
| 566 |
-
return {"state": "ok", "profile": profile, "posts": posts, "comments": comments,
|
| 567 |
-
"via": "brightdata", "deferredMedia": deferred_media,
|
| 568 |
-
"note": c_note or note or ""}
|
|
|
|
| 1 |
+
"""connectors_tt.py β the TIKTOK connector (wave 29 Β· item 7 Β· DEBT D-9 Β· rulings R1 + R2).
|
| 2 |
+
|
| 3 |
+
Everything in this file knows what a VENDOR's TikTok row looks like. Nothing in it knows what an
|
| 4 |
+
automation is. That split is `connectors_ig.py`'s (wave 27 item 23) and it is the reason this file
|
| 5 |
+
exists at all rather than another thousand lines inside the engine.
|
| 6 |
+
|
| 7 |
+
β **EVERY VENDOR FIELD NAME HERE WAS PROBED, NOT GUESSED.** The whole schema β 40 profile / 43
|
| 8 |
+
post / 17 comment fields, each with the vendor's own type, description and `pii` flag β was read
|
| 9 |
+
live from `GET /datasets/{id}/metadata` for **$0.00** and written down in
|
| 10 |
+
`.claude/wiki/waves/wave29/proto/tiktok-schema.md` (promoted to `tiktok-capture.md` at
|
| 11 |
+
close-out). That document is the AUTHORITY: do not re-probe it, and do not invent a key. Where a
|
| 12 |
+
name below reads through a candidate list it is because the vendor has two names for one fact
|
| 13 |
+
(`biography`/`signature`, `region`/`country`), never because the name is uncertain.
|
| 14 |
+
|
| 15 |
+
β **NOTHING HERE EVER AUTHENTICATES TO TIKTOK.** No login, no cookie, no account to get banned β
|
| 16 |
+
public data through a supplier, exactly the rail `connectors_ig.py` states for Instagram. The
|
| 17 |
+
vendor key is a key to a SUPPLIER.
|
| 18 |
+
|
| 19 |
+
β **THE TRANSPORT IS `connectors_bd.py` β SHARED, VENDOR-NAMED, AND NO LONGER BORROWED FROM THE
|
| 20 |
+
OTHER PLATFORM'S CONNECTOR** (WAVE 30 Β· T09, DEBT D-128). `bd_call`, `bd_scrape` and
|
| 21 |
+
`bd_filter_start` take the dataset id as a PARAMETER β they are Bright Data's wire, not
|
| 22 |
+
Instagram's β and re-implementing them here would be a second copy of the deferral handling, the
|
| 23 |
+
truncation guard, the SSRF rail and the snapshot-progress reader, i.e. five places for one bug.
|
| 24 |
+
Until wave 30 the code was right and the NAME was wrong: this file imported thirteen symbols from
|
| 25 |
+
`connectors_ig`, which read as a dependency on Instagram and was really a dependency on a supplier.
|
| 26 |
+
β **This file now imports ZERO names from `connectors_ig`, and a gate check asserts that**, because
|
| 27 |
+
the sentence above is the kind that quietly stops being true.
|
| 28 |
+
|
| 29 |
+
β **WHAT $0 COULD NOT BUY, so nobody reads this file as more measured than it is:**
|
| 30 |
+
1. the real ROW shape β `/metadata` describes a DATASET, and Instagram's rows carry undeclared
|
| 31 |
+
envelope keys (`timestamp`, `input`) that no metadata call mentions;
|
| 32 |
+
2. that a declared field POPULATES β Bright Data's Instagram Reels *declares* `views: number`
|
| 33 |
+
and delivers an account-grain wrong number (Β§4e). **Declared is not delivered**, and the one
|
| 34 |
+
TikTok claim that matters most (`play_count`) is exactly a declaration.
|
| 35 |
+
"""
|
| 36 |
+
from __future__ import annotations
|
| 37 |
+
|
| 38 |
+
import automation_engine as engine
|
| 39 |
+
# β T09 β FOUR NAMES CAME OFF THIS LIST AND NOTHING BROKE, which is the point of deriving an
|
| 40 |
+
# import block from the AST both ways. `bd_call`, `bd_filter_start`, `bd_key` and `bd_ready` were
|
| 41 |
+
# imported here under a comment claiming they were *"re-exported for the runners"*; no runner ever
|
| 42 |
+
# read them off this module (the engine imports them from the transport itself), so they were four
|
| 43 |
+
# lines of dependency nobody was paying for. `[[artifact-with-no-importer]]` in its smallest form.
|
| 44 |
+
from connectors_bd import (
|
| 45 |
+
_bd_first_url,
|
| 46 |
+
_bd_flag,
|
| 47 |
+
_bd_list,
|
| 48 |
+
_bd_source_payload,
|
| 49 |
+
_first,
|
| 50 |
+
_ig_int,
|
| 51 |
+
bd_scrape,
|
| 52 |
+
)
|
| 53 |
+
|
| 54 |
+
# ---------------------------------------------------------------------------------------------
|
| 55 |
+
# THE DATASETS
|
| 56 |
+
# ---------------------------------------------------------------------------------------------
|
| 57 |
+
# β CATALOGUE PRESENCE IS NOT ENTITLEMENT β the same finding Instagram produced. `GET
|
| 58 |
+
# /datasets/list` returned 1,735 rows of which 12 are TikTok; the three below answer 200 with a
|
| 59 |
+
# full field list, and the two DISCOVERY halves answer **404 for our key**:
|
| 60 |
+
# `gd_lj71gn6l68bz7y9hc` (posts by profile) and `gd_lilwhto81z415d9mdl` (posts by keyword).
|
| 61 |
+
# β TikTok discovery routes through the PROFILES dataset's corpus filter, exactly as Instagram's
|
| 62 |
+
# does. A ticket that reaches for a by-keyword endpoint is reaching for a 404.
|
| 63 |
+
TT_DS_PROFILES = "gd_l1villgoiiidt09ci" # TikTok - Profiles. 40 fields, 152,000,000 records
|
| 64 |
+
TT_DS_POSTS = "gd_lu702nij2f790tmv9h" # TikTok - Posts. 43 fields
|
| 65 |
+
TT_DS_COMMENTS = "gd_lkf2st302ap89utw5k" # TikTok - Comments. 17 fields
|
| 66 |
+
|
| 67 |
+
#: The vendor's two post-type tokens, verbatim from the dataset's own `ai_description`
|
| 68 |
+
#: (*"strictly these two"*, video = 99.5% of rows). Ours are `image`/`video`/`carousel`.
|
| 69 |
+
TT_POST_TYPE_VIDEO = "video"
|
| 70 |
+
TT_POST_TYPE_CONTENT = "content"
|
| 71 |
+
|
| 72 |
+
#: What a TikTok profile URL looks like, for the runner that has a handle and needs a URL. Kept
|
| 73 |
+
#: beside the dataset ids because it is the same class of vendor fact.
|
| 74 |
+
#: β ONE SPELLING. `platform/core/user_tables.profile_url(handle, 'tiktok')` builds the identical
|
| 75 |
+
#: string from `_PROFILE_RULES` (contract C2, E's half) β this exists for the connector's own
|
| 76 |
+
#: batch calls, and the gate asserts the two agree rather than trusting that they do.
|
| 77 |
+
TT_PROFILE_URL = "https://www.tiktok.com/@{handle}"
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def tt_profile_url(handle):
|
| 81 |
+
"""`nurilab` β `https://www.tiktok.com/@nurilab`. `''` for a blank handle, never a bare `@`."""
|
| 82 |
+
h = str(handle or "").strip().lstrip("@")
|
| 83 |
+
return TT_PROFILE_URL.format(handle=h) if h else ""
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
# ---------------------------------------------------------------------------------------------
|
| 87 |
+
# THE FIELD MAPS
|
| 88 |
+
# ---------------------------------------------------------------------------------------------
|
| 89 |
+
# Each function turns ONE vendor row into the cell dict for one of the `ut_tt_*` schemas declared
|
| 90 |
+
# in `automation_engine`. The rules they all obey, stated once:
|
| 91 |
+
#
|
| 92 |
+
# * **BLANK MEANS NOT READ, NEVER "THEY HAVE NONE".** A key the vendor did not send is OMITTED,
|
| 93 |
+
# so a later, richer pull fills it instead of being overwritten by this one's silence. `_first`
|
| 94 |
+
# returns `None` (never 0) when nothing matches, which is what makes that possible.
|
| 95 |
+
# * **A zero from the vendor is a MEASUREMENT and survives.** (Instagram's `_ig_zero_is_blank`
|
| 96 |
+
# rule is scoped to a paid rung whose zeros were proven fictional; nothing here has earned it.)
|
| 97 |
+
# * **Every unpromoted vendor key stays whole in `source_payload`.** A schema addition on the
|
| 98 |
+
# vendor's side is preserved rather than silently discarded while our column model catches up.
|
| 99 |
+
# * **Nothing here writes a session token.** `tt_chain_token`, `secu_id` (~85% null), `short_id`
|
| 100 |
+
# (100% null in the sample), `ftc` (100% null) and `relation` are deliberately unmapped; they
|
| 101 |
+
# ride in `source_payload` where they make no claim.
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def _tt_str(node, *names):
|
| 105 |
+
"""The first non-empty string among `names`, or None when the vendor sent nothing.
|
| 106 |
+
|
| 107 |
+
β `None`, NOT `""`. The callers below drop `None` keys, which is what keeps a blank honest β
|
| 108 |
+
an empty string written into a cell claims "we looked and it is empty".
|
| 109 |
+
"""
|
| 110 |
+
v = _first(node, *names)
|
| 111 |
+
if v is None:
|
| 112 |
+
return None
|
| 113 |
+
s = str(v).strip()
|
| 114 |
+
return s or None
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
def _tt_pct(node, *names):
|
| 118 |
+
"""A vendor 0β1 engagement fraction β our stored 0β100 percentage, or None.
|
| 119 |
+
|
| 120 |
+
β THE Γ100 IS NOT COSMETIC (wave 26, amendment C1-a). Our `pct` renderer appends the sign to
|
| 121 |
+
the STORED number, so writing the vendor's raw 0.0656 would print a 6.6% creator as `0.0%` β
|
| 122 |
+
measured on the Instagram side, and TikTok sends the same shape on all three of its rates.
|
| 123 |
+
"""
|
| 124 |
+
v = _first(node, *names)
|
| 125 |
+
if v is None:
|
| 126 |
+
return None
|
| 127 |
+
out = engine._pct100(v)
|
| 128 |
+
return out or None
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
def _tt_day(node, *names):
|
| 132 |
+
"""A vendor stamp β `YYYY-MM-DD`, or None. Our `date` columns store a day."""
|
| 133 |
+
v = _first(node, *names)
|
| 134 |
+
if v is None:
|
| 135 |
+
return None
|
| 136 |
+
return engine._day(v) or None
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
def _drop_blanks(row):
|
| 140 |
+
"""The one place a mapped row loses its `None`s β see the BLANK MEANS NOT READ rule above."""
|
| 141 |
+
return {k: v for k, v in row.items() if v is not None and v != ""}
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
def normalize_profile(node, handle=""):
|
| 145 |
+
"""A TikTok Profiles row β the `ut_tt_profile` / `ut_tt_snapshots` cell shape.
|
| 146 |
+
|
| 147 |
+
β TWO FIELDS READ THROUGH A CANDIDATE PAIR, and both pairs are the vendor's, not a guess:
|
| 148 |
+
* `biography` is PRIMARY and `signature` the FALLBACK β the probe measured `signature`
|
| 149 |
+
populated on 85% of rows and they carry the same text;
|
| 150 |
+
* `region` is PRIMARY and `country` the FALLBACK β `region` is the one with a documented
|
| 151 |
+
two-letter-ISO description, `country` has no description at all.
|
| 152 |
+
β `videos_count` is mapped to `posts_count` with a caveat recorded rather than hidden: its
|
| 153 |
+
`ai_description` ranges 1-89, so it may be a WINDOW rather than a lifetime total. It is the
|
| 154 |
+
only count of its kind the dataset offers.
|
| 155 |
+
"""
|
| 156 |
+
node = node if isinstance(node, dict) else {}
|
| 157 |
+
account = _tt_str(node, "account_id") or str(handle or "").strip().lstrip("@")
|
| 158 |
+
return _drop_blanks({
|
| 159 |
+
"platform": engine.PLATFORM_TIKTOK,
|
| 160 |
+
"handle": account,
|
| 161 |
+
"full_name": _tt_str(node, "nickname"),
|
| 162 |
+
"tt_id": _tt_str(node, "id"),
|
| 163 |
+
"profile_url": _tt_str(node, "url") or (tt_profile_url(account) or None),
|
| 164 |
+
"bio": _tt_str(node, "biography", "signature"),
|
| 165 |
+
"external_url": _bd_first_url(_first(node, "bio_link")),
|
| 166 |
+
"verified": _bd_flag(node, "is_verified"),
|
| 167 |
+
"is_private": _bd_flag(node, "is_private"),
|
| 168 |
+
# β APPROXIMATE, and the word is the vendor's: `is_commerce_user` has *"many null values"*
|
| 169 |
+
# on their own description. It is the closest thing TikTok has to Instagram's
|
| 170 |
+
# `is_business_account`, and `_bd_flag` writes nothing at all when the key is absent β so
|
| 171 |
+
# the approximation only ever fills a cell the vendor actually answered.
|
| 172 |
+
"is_business": _bd_flag(node, "is_commerce_user"),
|
| 173 |
+
"followers": _ig_int(_first(node, "followers")),
|
| 174 |
+
"following": _ig_int(_first(node, "following")),
|
| 175 |
+
"posts_count": _ig_int(_first(node, "videos_count")),
|
| 176 |
+
# β `likes` on a PROFILE row is likes RECEIVED across the account's videos (18-110,200, no
|
| 177 |
+
# nulls). It is not a post-level number and it is not our `likes` column, which is why it
|
| 178 |
+
# is stored under a different name.
|
| 179 |
+
"likes_received": _ig_int(_first(node, "likes")),
|
| 180 |
+
"avg_engagement": _tt_pct(node, "awg_engagement_rate"),
|
| 181 |
+
"like_engagement": _tt_pct(node, "like_engagement_rate"),
|
| 182 |
+
"comment_engagement": _tt_pct(node, "comment_engagement_rate"),
|
| 183 |
+
"country_code": _tt_str(node, "region", "country"),
|
| 184 |
+
"region": _tt_str(node, "region"),
|
| 185 |
+
"predicted_lang": _tt_str(node, "predicted_lang"),
|
| 186 |
+
# β ACCOUNT AGE, NOT A MEASUREMENT STAMP β `create_time` on a profile is when the ACCOUNT
|
| 187 |
+
# was made. TikTok stamps nothing with "when this number was true", exactly like Instagram,
|
| 188 |
+
# which is why the append law dates a snapshot by when WE read it.
|
| 189 |
+
"account_created_at": _tt_day(node, "create_time"),
|
| 190 |
+
"source_payload": _bd_source_payload(node),
|
| 191 |
+
})
|
| 192 |
+
|
| 193 |
+
|
| 194 |
+
def tt_post_type(node):
|
| 195 |
+
"""A TikTok post row β one of OUR three type options, or None.
|
| 196 |
+
|
| 197 |
+
β THE FIRST OF THE PROBE DOC'S TWO NAMED BLOCKERS. The vendor's vocabulary is `"video"` /
|
| 198 |
+
`"content"`; ours is `image` / `video` / `carousel` and has no `"content"`. Writing the
|
| 199 |
+
vendor's token would fail `_clean_field`'s option check on the way in and would put an
|
| 200 |
+
untranslated API word in front of a user on the way out.
|
| 201 |
+
|
| 202 |
+
So: `video` is `video`, and `content` β TikTok's photo-mode post β is `image`, EXCEPT when the
|
| 203 |
+
row carries more than one `carousel_images` entry, which is what a carousel IS on either
|
| 204 |
+
network. β The multi-image branch is decided from the IMAGES, never from the type token: the
|
| 205 |
+
token cannot express it, so inferring `carousel` from the word would be inventing a fact.
|
| 206 |
+
β An UNKNOWN token returns None rather than defaulting to `video` (99.5% of rows are video, and
|
| 207 |
+
that is exactly what would make the wrong default invisible).
|
| 208 |
+
"""
|
| 209 |
+
raw = str((node or {}).get("post_type") or "").strip().lower()
|
| 210 |
+
if raw == TT_POST_TYPE_VIDEO:
|
| 211 |
+
return "video"
|
| 212 |
+
if raw != TT_POST_TYPE_CONTENT:
|
| 213 |
+
return None
|
| 214 |
+
images = (node or {}).get("carousel_images")
|
| 215 |
+
return "carousel" if isinstance(images, list) and len(images) > 1 else "image"
|
| 216 |
+
|
| 217 |
+
|
| 218 |
+
def normalize_post(node):
|
| 219 |
+
"""A TikTok Posts row β the `ut_tt_posts` cell shape. None when it carries no identity.
|
| 220 |
+
|
| 221 |
+
β THE SECOND NAMED BLOCKER, RESOLVED HERE AND NOWHERE ELSE: `play_count` is ONE number and
|
| 222 |
+
Instagram's schema has TWO columns for it (`plays` and `views`). On TikTok they are the same
|
| 223 |
+
fact β `play_count` IS the count TikTok displays under a video β so it maps to `views` and
|
| 224 |
+
`ut_tt_posts` HAS NO `plays` COLUMN. Copying one vendor number into two of our columns would
|
| 225 |
+
manufacture a second measurement that a rollup could average or double-count, which is a worse
|
| 226 |
+
outcome than the missing column it would paper over.
|
| 227 |
+
|
| 228 |
+
β `shortcode` reads `shortcode` then `post_id`: both are 19-digit numerics on this dataset and
|
| 229 |
+
the probe measured them as the same shape. That equality is what lets the commentsβposts link
|
| 230 |
+
join with NO normaliser, which the Instagram side never had.
|
| 231 |
+
β `num_share_count` (a number) is preferred over `share_count` (typed TEXT by the vendor).
|
| 232 |
+
β `commerce_info` is a business/commerce LOCATION per its own description β cities and
|
| 233 |
+
countries. It is NOT a paid-partnership flag, and nothing in this dataset is: TikTok declares
|
| 234 |
+
no equivalent, so `paid_partnership`/`partner` have no column on this family at all.
|
| 235 |
+
"""
|
| 236 |
+
node = node if isinstance(node, dict) else {}
|
| 237 |
+
shortcode = _tt_str(node, "shortcode", "post_id")
|
| 238 |
+
if not shortcode:
|
| 239 |
+
return None
|
| 240 |
+
return _drop_blanks({
|
| 241 |
+
"platform": engine.PLATFORM_TIKTOK,
|
| 242 |
+
"shortcode": shortcode,
|
| 243 |
+
# ββ `account_id`, NOT `profile_username` β MEASURED on a real Posts row 2026-08-12.
|
| 244 |
+
# The vendor's `profile_username` is the DISPLAY NAME (`"Dina"`), while `account_id` is the
|
| 245 |
+
# @handle (`"d1na_th"`) β the same field `normalize_profile` already reads for `handle`, so
|
| 246 |
+
# one name means one thing across both corpora. Reading the display name silently broke the
|
| 247 |
+
# only join this table has: `ut_tt_posts.influencer_key` -> `ut_tt_profile.handle` matched
|
| 248 |
+
# NOTHING, so a person could not filter posts by creator and a rollup would count zero.
|
| 249 |
+
# β NO FALLBACK TO `profile_username`, deliberately. It is not a degraded handle, it is a
|
| 250 |
+
# different fact, and filling a join key with it is worse than leaving it blank β a blank
|
| 251 |
+
# is visibly missing, a display name looks like an answer [[one-question-two-normalizers]].
|
| 252 |
+
# The URL is the honest second source: it carries the handle by construction.
|
| 253 |
+
"influencer_key": (_tt_str(node, "account_id")
|
| 254 |
+
or tt_handle(_tt_str(node, "profile_url", "url") or "") or None),
|
| 255 |
+
"posted_at": _tt_day(node, "create_time"),
|
| 256 |
+
"type": tt_post_type(node),
|
| 257 |
+
"caption": _tt_str(node, "description"),
|
| 258 |
+
"url": _tt_str(node, "url"),
|
| 259 |
+
"hashtags": _bd_list(node, "hashtags"),
|
| 260 |
+
"tagged_location": _tt_str(node, "commerce_info"),
|
| 261 |
+
"views": _ig_int(_first(node, "play_count")),
|
| 262 |
+
"likes": _ig_int(_first(node, "digg_count")),
|
| 263 |
+
"comments": _ig_int(_first(node, "comment_count")),
|
| 264 |
+
"shares": _ig_int(_first(node, "num_share_count")),
|
| 265 |
+
"saves": _ig_int(_first(node, "collect_count")),
|
| 266 |
+
"video_duration": _ig_int(_first(node, "video_duration")),
|
| 267 |
+
"source_payload": _bd_source_payload(node),
|
| 268 |
+
})
|
| 269 |
+
|
| 270 |
+
|
| 271 |
+
def normalize_comment(node):
|
| 272 |
+
"""A TikTok Comments row β the `ut_tt_comments` cell shape. None without a comment id.
|
| 273 |
+
|
| 274 |
+
β THE NAME COLLISION, RESOLVED: TikTok's `replies` is an ARRAY of reply objects and OUR
|
| 275 |
+
`replies` column is an INT count. The count comes from `num_replies`; the array stays whole in
|
| 276 |
+
`source_payload`. Reading the array's length instead would be a second, disagreeing answer to
|
| 277 |
+
a question the vendor already answers β and it would disagree, because a page of replies is not
|
| 278 |
+
all of them.
|
| 279 |
+
β `date_created` is typed `date` by this vendor, unlike Instagram's `comment_date` which is
|
| 280 |
+
text and needs defensive parsing. It still goes through `_tt_day` β one date path, so a vendor
|
| 281 |
+
that changes its mind cannot change ours.
|
| 282 |
+
β The comment TEXT and every identifiable commenter field (`commenter_user_name` is flagged
|
| 283 |
+
PII) stay in `source_payload` and are promoted to no column, which is the same posture the
|
| 284 |
+
Instagram comment schema takes for the same D-24 reason.
|
| 285 |
+
"""
|
| 286 |
+
node = node if isinstance(node, dict) else {}
|
| 287 |
+
comment_key = _tt_str(node, "comment_id")
|
| 288 |
+
if not comment_key:
|
| 289 |
+
return None
|
| 290 |
+
return _drop_blanks({
|
| 291 |
+
"platform": engine.PLATFORM_TIKTOK,
|
| 292 |
+
"comment_key": comment_key,
|
| 293 |
+
"shortcode": _tt_str(node, "post_id"),
|
| 294 |
+
# ββ OWNER RULING 2026-08-12: the comment's CONTENT gets a column. `comment_text` is the
|
| 295 |
+
# vendor's own key and `comment_text_only` its stripped variant (the probe recorded both);
|
| 296 |
+
# primary first, so a row carrying the rich form is not silently served the plain one.
|
| 297 |
+
# β The commenter's identity is deliberately NOT promoted β see `TT_COMMENT_FIELDS`.
|
| 298 |
+
"text": _tt_str(node, "comment_text", "comment_text_only"),
|
| 299 |
+
"commented_at": _tt_day(node, "date_created"),
|
| 300 |
+
"likes": _ig_int(_first(node, "num_likes")),
|
| 301 |
+
"replies": _ig_int(_first(node, "num_replies")),
|
| 302 |
+
"source_payload": _bd_source_payload(node),
|
| 303 |
+
})
|
| 304 |
+
|
| 305 |
+
|
| 306 |
+
#: β The three maps, addressable by name β so a gate (and the runners in T04-T06) can walk them
|
| 307 |
+
#: rather than naming three functions, and so adding a fourth dataset is one entry.
|
| 308 |
+
TT_NORMALIZERS = {
|
| 309 |
+
"tt_profile": normalize_profile,
|
| 310 |
+
"tt_post_metrics": normalize_post,
|
| 311 |
+
"tt_comments": normalize_comment,
|
| 312 |
+
}
|
| 313 |
+
|
| 314 |
+
|
| 315 |
+
# ---------------------------------------------------------------------------------------------
|
| 316 |
+
# THE FETCH β wave 30 Β· W30-T08 (carrying wave-29's dropped T05)
|
| 317 |
+
# ---------------------------------------------------------------------------------------------
|
| 318 |
+
|
| 319 |
+
def tt_handle(url):
|
| 320 |
+
"""A TikTok profile URL **or** a bare handle β the handle. `''` when it is neither.
|
| 321 |
+
|
| 322 |
+
Deliberately permissive about the input and strict about the output, because the two callers
|
| 323 |
+
hand it different things: an automation stores whatever a person typed in the profile column
|
| 324 |
+
(`@nurilab`, `nurilab`, or the full URL), while the discovery runner already holds a clean
|
| 325 |
+
`account_id`. One normaliser, so a row found by discovery and a row typed by hand cannot
|
| 326 |
+
resolve to two different handles.
|
| 327 |
+
"""
|
| 328 |
+
s = str(url or "").strip()
|
| 329 |
+
if not s:
|
| 330 |
+
return ""
|
| 331 |
+
if "tiktok.com" in s.lower():
|
| 332 |
+
# Everything after the first `@`, up to the next path segment or query.
|
| 333 |
+
tail = s.split("@", 1)[1] if "@" in s else ""
|
| 334 |
+
s = tail.split("/")[0].split("?")[0].split("#")[0]
|
| 335 |
+
s = s.strip().lstrip("@").strip()
|
| 336 |
+
# A handle is the vendor's `account_id` shape: alphanumerics, dots and underscores.
|
| 337 |
+
return s if s and all(c.isalnum() or c in "._" for c in s) else ""
|
| 338 |
+
|
| 339 |
+
|
| 340 |
+
def tt_post_urls(node, limit=0):
|
| 341 |
+
"""β WAVE 30 Β· T10 β the profile row's own post permalinks, newest-first as the vendor sends.
|
| 342 |
+
|
| 343 |
+
β THIS IS WHY TIKTOK POST CAPTURE COSTS NO EXTRA DISCOVERY. `top_videos` rides the PROFILE row
|
| 344 |
+
we have already bought, so the posts read is a scrape of links we hold, never a search for them.
|
| 345 |
+
The two TikTok DISCOVERY datasets (posts-by-profile, posts-by-keyword) are **404 for our key**,
|
| 346 |
+
so a design that reached for either would not merely be dearer, it would not work.
|
| 347 |
+
|
| 348 |
+
ββ CORRECTED 2026-08-12 β THIS DOCSTRING USED TO SAY *"the probe MEASURED `top_videos` as an
|
| 349 |
+
array of video permalinks, NO empties"*, AND THAT SENTENCE IS WHAT SHIPPED THE BUG. The probe
|
| 350 |
+
read `/datasets/{id}/metadata` β a DATASET description β and the phrase quoted was the field's
|
| 351 |
+
`ai_description`, not an observation of a row. The real row sends **dicts keyed `video_url`**
|
| 352 |
+
(measured below), so the reader built against the quoted sentence found nothing, forever, in
|
| 353 |
+
silence. β The transferable half: *"the probe measured X"* and *"the probe read a declaration
|
| 354 |
+
of X"* are different claims, and prose cannot be told apart by a reader downstream β which is
|
| 355 |
+
why the correction names the method, not just the value.
|
| 356 |
+
|
| 357 |
+
β `top_posts_data` is deliberately NOT read: the probe calls it *"a thin dup of `top_videos`"*,
|
| 358 |
+
and preferring whichever happened to be longer is how one creator's window silently differs
|
| 359 |
+
from another's.
|
| 360 |
+
β `limit <= 0` means "everything the row carried". The CAP IS THE CALLER'S β `config.maxPosts`,
|
| 361 |
+
validated 1..12 β and it is applied here rather than after the scrape so an unwanted post is
|
| 362 |
+
never bought. [[a-constant-two-features-share]]: the 12 is the vendor's measured profile window,
|
| 363 |
+
not a number this function may invent.
|
| 364 |
+
"""
|
| 365 |
+
raw = (node or {}).get("top_videos")
|
| 366 |
+
out = []
|
| 367 |
+
for item in raw if isinstance(raw, list) else []:
|
| 368 |
+
# ββ MEASURED ON A REAL ROW 2026-08-12, AND IT IS NOT WHAT THE SCHEMA SAID.
|
| 369 |
+
# `top_videos` is NOT an array of permalink strings. One paid TikTok Profiles scrape of a
|
| 370 |
+
# live handle returns **19 DICTS**, keyed
|
| 371 |
+
# `video_url Β· video_id Β· playcount Β· diggcount Β· commentcount Β· share_count Β·
|
| 372 |
+
# favorites_count Β· create_date Β· cover_image`.
|
| 373 |
+
# The docstring above cites the $0 probe as having "measured" permalinks β it had not, and
|
| 374 |
+
# could not: `/datasets/{id}/metadata` describes a DATASET, and the probe's own verdict says
|
| 375 |
+
# so in terms (*"what $0 cannot buy β¦ the real ROW shape β¦ that a declared field
|
| 376 |
+
# POPULATES"*, `wave29/proto/tiktok-schema.md`). This is the SECOND time this vendor's
|
| 377 |
+
# declaration has diverged from its delivery on this exact axis; BD's IG Reels `views` was
|
| 378 |
+
# the first. [[reachable-is-not-the-same-as-built]]
|
| 379 |
+
# β The consequence, live: `TT_DS_POSTS` was never reached, because this returned an EMPTY
|
| 380 |
+
# list on every real profile β post capture could not have worked for anybody, and the
|
| 381 |
+
# T10 gate stayed green because its canned fixture encoded the DECLARED shape. A fixture
|
| 382 |
+
# written from a schema tests the schema.
|
| 383 |
+
# β `video_url` is FIRST because it is the key the vendor actually sends; `url` is kept
|
| 384 |
+
# because it costs nothing and is what a future corpus revision would most likely use. The
|
| 385 |
+
# bare-string branch stays for the same reason β this widens what is ACCEPTED and invents
|
| 386 |
+
# nothing: a shape that yields no `httpβ¦` value still degrades to "no posts", exactly as
|
| 387 |
+
# before, rather than to a URL built out of a guess.
|
| 388 |
+
# β `top_posts_data` is STILL not read (it carries `post_url` and would work): preferring
|
| 389 |
+
# whichever array happened to be longer is how one creator's window silently differs from
|
| 390 |
+
# another's, and that reasoning is unchanged by this correction.
|
| 391 |
+
if isinstance(item, dict):
|
| 392 |
+
u = str(item.get("video_url") or item.get("url") or "").strip()
|
| 393 |
+
else:
|
| 394 |
+
u = str(item or "").strip()
|
| 395 |
+
if u.lower().startswith("http") and u not in out:
|
| 396 |
+
out.append(u)
|
| 397 |
+
return out[:limit] if limit and limit > 0 else out
|
| 398 |
+
|
| 399 |
+
|
| 400 |
+
def pull_posts_tt(post_urls, log=print, deferred=None):
|
| 401 |
+
"""The TikTok Posts dataset for a list of permalinks β `(rows, note)`, already normalised.
|
| 402 |
+
|
| 403 |
+
β ONE CALL FOR THE WHOLE WINDOW. `bd_scrape` has always taken a list, and the Instagram side
|
| 404 |
+
measured what happens when a caller forgets: 25 records, one billed snapshot each, a walk still
|
| 405 |
+
running at 67 minutes. Nothing here loops per URL.
|
| 406 |
+
"""
|
| 407 |
+
urls = [str(u) for u in (post_urls or []) if str(u or "").strip()]
|
| 408 |
+
if not urls:
|
| 409 |
+
return [], ""
|
| 410 |
+
rows, note = bd_scrape(TT_DS_POSTS, urls, deferred=deferred)
|
| 411 |
+
if note:
|
| 412 |
+
log(f"[aios-tt] posts: {note}")
|
| 413 |
+
return [], note
|
| 414 |
+
out = [r for r in (normalize_post(n) for n in rows) if r]
|
| 415 |
+
return out, ""
|
| 416 |
+
|
| 417 |
+
|
| 418 |
+
def pull_comments_tt(post_urls, log=print, deferred=None):
|
| 419 |
+
"""The TikTok Comments dataset for a list of POST permalinks β `(rows, note)`, normalised.
|
| 420 |
+
|
| 421 |
+
β THE MOST EXPENSIVE THING THIS PRODUCT BUYS, and the reason `commentMetrics` defaults OFF on
|
| 422 |
+
both networks: a comments scrape ingests identifiable third parties who never entered anybody's
|
| 423 |
+
list (D-24). The mapper already keeps every commenter field in `source_payload` and promotes
|
| 424 |
+
none of them to a column; this function adds no new exposure, it just has to be asked for.
|
| 425 |
+
"""
|
| 426 |
+
urls = [str(u) for u in (post_urls or []) if str(u or "").strip()]
|
| 427 |
+
if not urls:
|
| 428 |
+
return [], ""
|
| 429 |
+
rows, note = bd_scrape(TT_DS_COMMENTS, urls, deferred=deferred)
|
| 430 |
+
if note:
|
| 431 |
+
log(f"[aios-tt] comments: {note}")
|
| 432 |
+
return [], note
|
| 433 |
+
out = [r for r in (normalize_comment(n) for n in rows) if r]
|
| 434 |
+
return out, ""
|
| 435 |
+
|
| 436 |
+
|
| 437 |
+
#: ββ WAVE 30 Β· D-156 β THE MEDIA DATASETS, AS A SET, SO THE HAND-OFF CAN FILTER ON IDENTITY.
|
| 438 |
+
#: `_media_deferrals` uses this to lift ONLY posts/comments snapshots out of the local deferral
|
| 439 |
+
#: list. That is what makes it structurally impossible to file a PROFILE snapshot in the engine's
|
| 440 |
+
#: metric queue β the defect a draft of T10 shipped and A-39 booked as "the wrong fix is worse
|
| 441 |
+
#: than the gap". A membership test cannot be got wrong by a later edit the way `if` order can.
|
| 442 |
+
TT_MEDIA_DATASETS = (TT_DS_POSTS, TT_DS_COMMENTS)
|
| 443 |
+
|
| 444 |
+
|
| 445 |
+
def _media_deferrals(deferred):
|
| 446 |
+
"""The POSTS/COMMENTS entries of a `bd_scrape` deferral list β never the profile's.
|
| 447 |
+
|
| 448 |
+
β The engine, not this module, decides what a deferral MEANS: it stamps `kind` and the
|
| 449 |
+
handle and files it. TikTok needs no `_tag_metric_deferrals` twin because one dataset is one
|
| 450 |
+
kind here, so the id already carries everything a mapper choice depends on β and importing
|
| 451 |
+
Instagram's tagger is not available anyway (W30-T09 gates ZERO `from connectors_ig` lines).
|
| 452 |
+
"""
|
| 453 |
+
out = []
|
| 454 |
+
for d in deferred or []:
|
| 455 |
+
if isinstance(d, dict) and str(d.get("datasetId") or "") in TT_MEDIA_DATASETS:
|
| 456 |
+
out.append(dict(d))
|
| 457 |
+
return out
|
| 458 |
+
|
| 459 |
+
|
| 460 |
+
def pull_profile_tt(url, log=print, pending_profile=None, prefetch=None,
|
| 461 |
+
max_posts=0, post_metrics=False, comment_metrics=False):
|
| 462 |
+
"""ONE TikTok profile from the vendor. Same return contract as `pull_profile`.
|
| 463 |
+
|
| 464 |
+
`{state, profile, posts, comments, via, note}` with `state β ok | partial | blocked | error`,
|
| 465 |
+
so the engine's enrich branch treats every network identically and no caller learns a new
|
| 466 |
+
shape.
|
| 467 |
+
|
| 468 |
+
β WAVE 30 Β· T10 β POSTS AND COMMENTS ARE REAL NOW, AND BOTH DEFAULT OFF, exactly as Instagram's
|
| 469 |
+
do. `post_metrics` scrapes the profile row's own `top_videos` permalinks (see `tt_post_urls` β
|
| 470 |
+
no discovery call, because both TikTok discovery datasets 404 for our key); `comment_metrics`
|
| 471 |
+
then scrapes the comments of the posts that came back. β COMMENTS REQUIRE POSTS by construction
|
| 472 |
+
rather than by a rule: their input IS a post permalink, so asking for comments with post capture
|
| 473 |
+
off is a request with no subject, and it returns none instead of quietly buying posts nobody
|
| 474 |
+
asked for.
|
| 475 |
+
|
| 476 |
+
β `partial` IS THE SUCCESS STATE WHENEVER NO MEDIA WAS READ, and that is deliberate rather than
|
| 477 |
+
pessimistic. The Instagram contract reads `ok` only when identity AND media both landed
|
| 478 |
+
(`pull_profile_bd`: *"identity without media is still partial ... a run that wrote a follower
|
| 479 |
+
count and no posts must not paint green over a posts table that did not grow"*). So: posts not
|
| 480 |
+
ASKED for β `partial`, saying so; posts asked for and landed β `ok`; asked for and none came β
|
| 481 |
+
`partial` with the vendor's reason. The state answers "did this pull deliver what it went for",
|
| 482 |
+
never "did the function finish".
|
| 483 |
+
|
| 484 |
+
β **NO FREE RUNG, AND NO FALLBACK CHAIN.** Instagram's `pull_profile` drops to Apify when the
|
| 485 |
+
paid rung refuses; `providers.DEFAULT_CHAINS["tt_profile"]` is deliberately single-provider,
|
| 486 |
+
with its own note explaining that a multi-provider chain is a promise something walks it and
|
| 487 |
+
that nothing walks Instagram's second name today either. So a refusal here is final, and it
|
| 488 |
+
says so instead of implying a retry somewhere.
|
| 489 |
+
"""
|
| 490 |
+
handle = tt_handle(url)
|
| 491 |
+
if not handle:
|
| 492 |
+
return {"state": "error", "profile": {}, "posts": [], "comments": [], "via": "",
|
| 493 |
+
"note": f"{url!r} is not a TikTok profile URL or handle"}
|
| 494 |
+
|
| 495 |
+
# β THE BATCH FAST PATH, same shape as the Instagram side: `prefetch` is `{handle: node}` from
|
| 496 |
+
# one multi-URL scrape covering a whole selection. A hit is a vendor round trip that does not
|
| 497 |
+
# happen; a miss falls through to the single-URL call below.
|
| 498 |
+
cached = prefetch.get(handle) if isinstance(prefetch, dict) else None
|
| 499 |
+
_deferred = []
|
| 500 |
+
if isinstance(cached, dict) and cached:
|
| 501 |
+
rows, note = [cached], ""
|
| 502 |
+
else:
|
| 503 |
+
rows, note = bd_scrape(TT_DS_PROFILES, [tt_profile_url(handle)], deferred=_deferred)
|
| 504 |
+
|
| 505 |
+
node = rows[0] if rows else {}
|
| 506 |
+
profile = normalize_profile(node, handle) if node else {}
|
| 507 |
+
# β THE READABILITY TEST IS `followers`/`following`, NOT "did we get a dict". `normalize_profile`
|
| 508 |
+
# drops blanks, so an unreadable row still returns `{"platform": β¦, "handle": β¦}` β truthy, and
|
| 509 |
+
# carrying nothing anybody asked for. The Instagram rung tests exactly this pair for exactly
|
| 510 |
+
# this reason, and answering "0 followers" instead is the failure it exists to prevent.
|
| 511 |
+
unreadable = profile.get("followers") is None and profile.get("following") is None
|
| 512 |
+
if note or unreadable:
|
| 513 |
+
# β THE DEFERRAL IS HANDED OVER RATHER THAN DISCARDED. A snapshot the vendor is still
|
| 514 |
+
# building HAS ALREADY BEEN PAID FOR; dropping its id bills again on the next run for the
|
| 515 |
+
# same record. That was live on the Instagram profile path until 2026-08-09 β measured on
|
| 516 |
+
# nurilab as two runs, two fresh snapshots, both abandoned β and it is not being
|
| 517 |
+
# reintroduced here by omission.
|
| 518 |
+
if isinstance(pending_profile, list):
|
| 519 |
+
for d in _deferred:
|
| 520 |
+
pending_profile.append({**d, "kind": "profile", "influencer": handle})
|
| 521 |
+
why = note or ("the scrape answered, but no follower/following counts were readable in it "
|
| 522 |
+
"(the field names may have moved - see tiktok-capture.md)")
|
| 523 |
+
return {"state": "blocked", "profile": {}, "posts": [], "comments": [], "via": "brightdata",
|
| 524 |
+
"deferredProfile": [d.get("snapshotId") for d in _deferred],
|
| 525 |
+
"note": why}
|
| 526 |
+
|
| 527 |
+
# --- W30-T10: THE MEDIA, ONLY WHEN IT WAS ASKED FOR. ------------------------------------
|
| 528 |
+
if not post_metrics:
|
| 529 |
+
return {"state": "partial", "profile": profile, "posts": [], "comments": [],
|
| 530 |
+
"via": "brightdata",
|
| 531 |
+
"note": note or "profile read; post capture is off for this step"}
|
| 532 |
+
urls = tt_post_urls(node, limit=max_posts)
|
| 533 |
+
if not urls:
|
| 534 |
+
# β NOT AN ERROR AND NOT A RETRY. A creator with no `top_videos` has nothing to buy, and
|
| 535 |
+
# saying so is what stops the next run paying to be told the same thing.
|
| 536 |
+
return {"state": "partial", "profile": profile, "posts": [], "comments": [],
|
| 537 |
+
"via": "brightdata",
|
| 538 |
+
"note": note or "profile read; this account's row carried no post links"}
|
| 539 |
+
posts, p_note = pull_posts_tt(urls, log=log, deferred=_deferred)
|
| 540 |
+
comments, c_note = ([], "")
|
| 541 |
+
if comment_metrics and posts:
|
| 542 |
+
# The comments dataset is keyed on a POST permalink, so it reads the posts we just bought β
|
| 543 |
+
# `url` from the mapper, never the profile's raw array, so a post the posts scrape refused
|
| 544 |
+
# is not silently asked about again one rung later.
|
| 545 |
+
comments, c_note = pull_comments_tt([p.get("url") for p in posts if p.get("url")],
|
| 546 |
+
log=log, deferred=_deferred)
|
| 547 |
+
# ββ WAVE 30 Β· D-156 β THE MEDIA DEFERRALS ARE HANDED BACK, and the shape of the hand-off is
|
| 548 |
+
# the whole lesson. An earlier draft of T10 appended every `_deferred` entry to
|
| 549 |
+
# `pending_profile` tagged `kind: "profile"`. By the time control reaches here a PROFILE
|
| 550 |
+
# deferral is impossible β the profile branch above returns `blocked` on any note β so **every
|
| 551 |
+
# id fanned out that way was a POSTS or COMMENTS snapshot in the PROFILE queue**, whose
|
| 552 |
+
# collector writes preset profile cells onto somebody's record from post rows. The engine keeps
|
| 553 |
+
# the two queues apart deliberately (`_pending_profile_tasks` vs `_pending_metric_tasks`).
|
| 554 |
+
# β So this returns them under their OWN key, filtered by dataset identity
|
| 555 |
+
# (`_media_deferrals`), and the engine files them in the metric queue with the handle it
|
| 556 |
+
# already holds. Returning rather than appending also keeps the queue's vocabulary out of a
|
| 557 |
+
# connector: this module knows which CORPUS deferred, never what the engine calls it.
|
| 558 |
+
# β `deferredMedia` rides BOTH returns on purpose. The empty-posts case is the one that
|
| 559 |
+
# matters most β that is exactly the run where the vendor took too long, so a caller reading
|
| 560 |
+
# the ids only from the success path would lose every batch it actually paid for.
|
| 561 |
+
deferred_media = _media_deferrals(_deferred)
|
| 562 |
+
if not posts:
|
| 563 |
+
return {"state": "partial", "profile": profile, "posts": [], "comments": [],
|
| 564 |
+
"via": "brightdata", "deferredMedia": deferred_media,
|
| 565 |
+
"note": p_note or note or "profile read; the post source returned nothing"}
|
| 566 |
+
return {"state": "ok", "profile": profile, "posts": posts, "comments": comments,
|
| 567 |
+
"via": "brightdata", "deferredMedia": deferred_media,
|
| 568 |
+
"note": c_note or note or ""}
|
api/deps.py
CHANGED
|
@@ -60,7 +60,12 @@ class Session:
|
|
| 60 |
that tenant's admin, or a freshly provisioned Nurilab admin (role=admin β may_open
|
| 61 |
everything) would open Royal's Odoo-backed surfaces through any module route. One
|
| 62 |
chokepoint: `module_gate` and every inline `session.require` pass through here."""
|
| 63 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
raise err(403, "forbidden", f"your account does not have access to {module_key}")
|
| 65 |
tcfg = getattr(self.runtime.tenant, "config", None) or {}
|
| 66 |
tmods = tcfg.get("modules", "all")
|
|
|
|
| 60 |
that tenant's admin, or a freshly provisioned Nurilab admin (role=admin β may_open
|
| 61 |
everything) would open Royal's Odoo-backed surfaces through any module route. One
|
| 62 |
chokepoint: `module_gate` and every inline `session.require` pass through here."""
|
| 63 |
+
# β THE SAME PREDICATE THE NAV USES, or the two disagree and the disagreement is
|
| 64 |
+
# invisible: a row the rail offers that the route 403s, or the reverse. The
|
| 65 |
+
# editor's explicit grant lands here too, so a database an administrator ticked
|
| 66 |
+
# is reachable by URL as well as visible in the rail.
|
| 67 |
+
import core.perm_scope as _perm_scope
|
| 68 |
+
if not _perm_scope.nav_may_open(self.user, module_key):
|
| 69 |
raise err(403, "forbidden", f"your account does not have access to {module_key}")
|
| 70 |
tcfg = getattr(self.runtime.tenant, "config", None) or {}
|
| 71 |
tmods = tcfg.get("modules", "all")
|
api/main.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
api/odoo_relational.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
api/providers.py
CHANGED
|
@@ -1,770 +1,770 @@
|
|
| 1 |
-
"""THE PROVIDER LAYER β many vendors behind one capability, chosen per FIELD and per cost.
|
| 2 |
-
|
| 3 |
-
ββ OWNER RULING 2026-08-08: *"Let's do a combination scraper. Do Bright Data first, and if
|
| 4 |
-
anything fails, we use APIfy or vice versa. Make this dynamic somehow. remember to cost optimize
|
| 5 |
-
when we scale to thousands+ customer β¦ make sure our code is customizable and modular."*
|
| 6 |
-
|
| 7 |
-
β THE DESIGN DECISION THAT MATTERS, AND IT IS NOT "TRY A, THEN TRY B".
|
| 8 |
-
A provider-level failover ("if Bright Data errors, re-run the whole thing on Apify") is the obvious
|
| 9 |
-
shape and it is the expensive one. MEASURED: Bright Data answers profile, likes and comments
|
| 10 |
-
correctly and CANNOT answer view counts on any route; Apify answers view counts correctly. A
|
| 11 |
-
provider-level fallback would notice the missing views and re-buy the profile, the likes and the
|
| 12 |
-
comments from Apify as well β **paying twice for the 90% that already worked**. At one profile that
|
| 13 |
-
is noise; at thousands of customers Γ twelve posts each it is the whole bill.
|
| 14 |
-
|
| 15 |
-
So routing is per CAPABILITY, and a capability is the smallest independently-billable thing:
|
| 16 |
-
|
| 17 |
-
ig_profile -> brightdata (36 fields, works)
|
| 18 |
-
ig_post_metrics -> brightdata (likes + comments, works)
|
| 19 |
-
ig_post_views -> apify (Bright Data is INCAPABLE β see `capable=False` below)
|
| 20 |
-
ig_comments -> brightdata
|
| 21 |
-
|
| 22 |
-
Each capability has an ORDERED list of providers. The first one that is configured AND capable AND
|
| 23 |
-
succeeds wins; the rest are never called, so the common path costs exactly one vendor record.
|
| 24 |
-
|
| 25 |
-
β "FAILURE" INCLUDES ANSWERING WITH THE FIELD BLANK. A vendor that returns HTTP 200 and a null in
|
| 26 |
-
the one column you asked for has failed for our purposes, and a chain that only catches exceptions
|
| 27 |
-
would stop at it forever. `run()` takes a `satisfied` predicate and falls through on an unsatisfying
|
| 28 |
-
answer exactly as it would on a 500.
|
| 29 |
-
|
| 30 |
-
β ORDER IS DATA, NOT CODE. `AIOS_PROVIDER_ORDER` overrides any chain at deploy time
|
| 31 |
-
(`ig_post_views=apify,brightdata;ig_profile=apify`), which is what "or vice versa" means and what
|
| 32 |
-
lets a tenant be moved off a vendor without a release.
|
| 33 |
-
"""
|
| 34 |
-
from __future__ import annotations
|
| 35 |
-
|
| 36 |
-
import json
|
| 37 |
-
import os
|
| 38 |
-
import time
|
| 39 |
-
from dataclasses import dataclass, field
|
| 40 |
-
from typing import Callable
|
| 41 |
-
|
| 42 |
-
#: β APPROXIMATE, AND DELIBERATELY SO. These are list prices per RECORD in USD, used only to RANK
|
| 43 |
-
#: providers and to estimate a run's spend for the operator. They are not billing truth β the
|
| 44 |
-
#: vendor's own dashboard is. Wrong by 2x still ranks correctly; wrong by 100x does not, which is
|
| 45 |
-
#: why they are named and dated rather than guessed silently.
|
| 46 |
-
#: Bright Data Instagram datasets ~ $0.0015/record (2026-08 list). Apify instagram-scraper
|
| 47 |
-
#: ~ $0.0027/result (2026-08 list). Both re-checked when a provider is added.
|
| 48 |
-
#: β WAVE 30 Β· T16 β APIFY CORRECTED 0.0023 -> 0.0027, and the direction matters: the old number
|
| 49 |
-
#: made the fallback look CHEAPER than it is, and `estimate()` is what a person is shown before
|
| 50 |
-
#: they authorise a run. An under-stated price is the one rounding error a cost guard cannot catch.
|
| 51 |
-
#: β Both are LIST rates used to rank and to estimate. They are not invoices, and nothing here
|
| 52 |
-
#: reads a live price β a modelled figure that says so is honest; one that pretends is not.
|
| 53 |
-
_COST_BRIGHTDATA = float(os.environ.get("AIOS_COST_BRIGHTDATA") or 0.0015)
|
| 54 |
-
_COST_APIFY = float(os.environ.get("AIOS_COST_APIFY") or 0.0027)
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
@dataclass(frozen=True)
|
| 58 |
-
class Capability:
|
| 59 |
-
"""What ONE provider can do for ONE capability, and what it costs to ask."""
|
| 60 |
-
#: β `False` means MEASURED INCAPABLE, not "untried". A provider declared incapable is never
|
| 61 |
-
#: called for this capability at all β it cannot be reached by a fallback, cannot be put first
|
| 62 |
-
#: by an env override, and cannot silently start being billed because somebody reordered a
|
| 63 |
-
#: list. Bright Data's `ig_post_views` is False on the strength of Β§4e-Β§4h: every route
|
| 64 |
-
#: (Posts x /p/, Posts x /reel/, Reels x /p/, Reels x /reel/, discover-by-profile) on four
|
| 65 |
-
#: accounts from 13K to 268M followers returned an account-grain constant or a null.
|
| 66 |
-
capable: bool = True
|
| 67 |
-
cost_per_record: float = 0.0
|
| 68 |
-
#: Free-text, shown to an operator deciding where their money went.
|
| 69 |
-
note: str = ""
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
@dataclass
|
| 73 |
-
class Provider:
|
| 74 |
-
key: str
|
| 75 |
-
label: str
|
| 76 |
-
#: The env var holding this provider's credential. NEVER the credential itself β this module
|
| 77 |
-
#: is imported by surfaces that serialise their config.
|
| 78 |
-
key_env: str = ""
|
| 79 |
-
caps: dict = field(default_factory=dict)
|
| 80 |
-
|
| 81 |
-
def configured(self) -> bool:
|
| 82 |
-
return bool((os.environ.get(self.key_env) or "").strip()) if self.key_env else True
|
| 83 |
-
|
| 84 |
-
def cap(self, capability: str) -> Capability | None:
|
| 85 |
-
return self.caps.get(capability)
|
| 86 |
-
|
| 87 |
-
def can(self, capability: str) -> bool:
|
| 88 |
-
c = self.cap(capability)
|
| 89 |
-
return bool(c and c.capable and self.configured())
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
#: β ADDING A PROVIDER IS A REGISTRY ENTRY PLUS A RUNNER β no change to any caller. That is the
|
| 93 |
-
#: "modular/customizable" half of the ruling, and the reason the chains below name STRINGS.
|
| 94 |
-
PROVIDERS: dict[str, Provider] = {
|
| 95 |
-
"brightdata": Provider(
|
| 96 |
-
key="brightdata", label="Bright Data", key_env="AIOS_BRIGHTDATA_KEY",
|
| 97 |
-
caps={
|
| 98 |
-
"ig_profile": Capability(True, _COST_BRIGHTDATA, "36 fields incl. bio/email/category"),
|
| 99 |
-
"ig_post_metrics": Capability(True, _COST_BRIGHTDATA, "likes + comments + captions"),
|
| 100 |
-
"ig_comments": Capability(True, _COST_BRIGHTDATA, "separate paid dataset, opt-in"),
|
| 101 |
-
# β THE ONE THAT COSTS US NOTHING TO GET RIGHT AND EVERYTHING TO GET WRONG.
|
| 102 |
-
"ig_post_views": Capability(
|
| 103 |
-
False, _COST_BRIGHTDATA,
|
| 104 |
-
"MEASURED INCAPABLE: returns one account-grain number for every reel of a "
|
| 105 |
-
"creator (identical for two shortcodes in one call, ticking upward between "
|
| 106 |
-
"calls) and never populates video_play_count. Their own documented example "
|
| 107 |
-
"account no longer reproduces it."),
|
| 108 |
-
# ββ WAVE 29 (item 7 / D-9 / R1) β TIKTOK. Three datasets, all reachable with our
|
| 109 |
-
# existing key, all schema-probed for $0.00 (40/43/17 fields with the vendor's own
|
| 110 |
-
# types): `gd_l1villgoiiidt09ci` Β· `gd_lu702nij2f790tmv9h` Β· `gd_lkf2st302ap89utw5k`.
|
| 111 |
-
"tt_profile": Capability(True, _COST_BRIGHTDATA,
|
| 112 |
-
"40 fields incl. bio/engagement rates/region"),
|
| 113 |
-
# β THERE IS NO `tt_post_views` CAPABILITY, AND ITS ABSENCE IS THE CLAIM.
|
| 114 |
-
# `TikTok - Posts` declares `play_count: number` and the vendor's own sample says
|
| 115 |
-
# "no empty values or zeros" β so on TikTok the view count arrives INSIDE the post
|
| 116 |
-
# record and a separate view rung would be a second bill for a number we already have.
|
| 117 |
-
# β DECLARED, NOT MEASURED, and that distinction is the whole scar tissue of Β§4e:
|
| 118 |
-
# Bright Data's Instagram Reels also DECLARES `views: number` and delivers an
|
| 119 |
-
# account-grain wrong one. One live pull settles it (W29-T06, spend-gated). If it
|
| 120 |
-
# fails, the honest repair is a `tt_post_views` capability routed elsewhere β never a
|
| 121 |
-
# quiet fallback bolted onto this one.
|
| 122 |
-
"tt_post_metrics": Capability(
|
| 123 |
-
True, _COST_BRIGHTDATA,
|
| 124 |
-
"likes + comments + shares + saves, and play_count inline (DECLARED "
|
| 125 |
-
"no-empties-or-zeros, UNPROVEN until one live pull)"),
|
| 126 |
-
"tt_comments": Capability(True, _COST_BRIGHTDATA,
|
| 127 |
-
"separate paid dataset, opt-in: 17 fields"),
|
| 128 |
-
}),
|
| 129 |
-
"apify": Provider(
|
| 130 |
-
key="apify", label="Apify", key_env="AIOS_APIFY_KEY",
|
| 131 |
-
caps={
|
| 132 |
-
# β MEASURED 2026-08-08 against the public Reels grid: `videoPlayCount` 137,684 and
|
| 133 |
-
# 299,493 vs a browser-read ground truth of 134K-137K and 299K. Exact.
|
| 134 |
-
"ig_post_views": Capability(True, _COST_APIFY,
|
| 135 |
-
"videoPlayCount, matches Instagram's displayed views"),
|
| 136 |
-
"ig_post_metrics": Capability(True, _COST_APIFY, "likes + comments (fallback)"),
|
| 137 |
-
"ig_profile": Capability(True, _COST_APIFY, "profile fields (fallback)"),
|
| 138 |
-
}),
|
| 139 |
-
}
|
| 140 |
-
|
| 141 |
-
#: The DEFAULT chain per capability, cheapest-capable-first. Overridable β see `chain()`.
|
| 142 |
-
DEFAULT_CHAINS: dict[str, tuple] = {
|
| 143 |
-
"ig_profile": ("brightdata", "apify"),
|
| 144 |
-
"ig_post_metrics": ("brightdata", "apify"),
|
| 145 |
-
# Only one entry, and that is the point: Bright Data is declared incapable, so listing it here
|
| 146 |
-
# would be a lie that costs a wasted call on every single post.
|
| 147 |
-
"ig_post_views": ("apify",),
|
| 148 |
-
"ig_comments": ("brightdata",),
|
| 149 |
-
# β WAVE 29 β TikTok, and every chain is deliberately SINGLE-PROVIDER.
|
| 150 |
-
# β A MULTI-PROVIDER CHAIN IS A PROMISE SOMETHING WALKS IT. `ig_post_metrics` has declared a
|
| 151 |
-
# two-provider fallback since wave 28 and NOTHING reaches the second name: if Bright Data
|
| 152 |
-
# answers with the likes blank, Apify is never asked. `verify_automation`'s E2b pins that gap
|
| 153 |
-
# BY NAME so a third unwalked chain turns it red β which is exactly what a second name here
|
| 154 |
-
# would be today. Apify does sell TikTok (`clockworks/tiktok-profile-scraper`, $0.003/result,
|
| 155 |
-
# 0.7% 30-day failure rate β measured from its own store record), so the fallback is buildable;
|
| 156 |
-
# it is not declared until a runner walks it.
|
| 157 |
-
"tt_profile": ("brightdata",),
|
| 158 |
-
"tt_post_metrics": ("brightdata",),
|
| 159 |
-
"tt_comments": ("brightdata",),
|
| 160 |
-
}
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
def chain(capability: str) -> list:
|
| 164 |
-
"""The provider order for `capability` β env override first, then the default.
|
| 165 |
-
|
| 166 |
-
`AIOS_PROVIDER_ORDER` is a `;`-separated list of `capability=p1,p2` clauses. This is how the
|
| 167 |
-
owner's "or vice versa" is expressed WITHOUT a release, and how one tenant can be moved off a
|
| 168 |
-
vendor that is having a bad day.
|
| 169 |
-
β An override may REORDER and may DROP, but it can never make an incapable provider capable β
|
| 170 |
-
`can()` still gates every name. A chain that reads `ig_post_views=brightdata` therefore
|
| 171 |
-
resolves to EMPTY rather than to a provider that would return a wrong number, because a
|
| 172 |
-
plausible wrong number is worse than an honest refusal.
|
| 173 |
-
"""
|
| 174 |
-
raw = (os.environ.get("AIOS_PROVIDER_ORDER") or "").strip()
|
| 175 |
-
names = None
|
| 176 |
-
for clause in raw.split(";"):
|
| 177 |
-
if "=" in clause:
|
| 178 |
-
cap_name, _, order = clause.partition("=")
|
| 179 |
-
if cap_name.strip() == capability:
|
| 180 |
-
names = [x.strip() for x in order.split(",") if x.strip()]
|
| 181 |
-
if names is None:
|
| 182 |
-
names = list(DEFAULT_CHAINS.get(capability) or ())
|
| 183 |
-
return [PROVIDERS[n] for n in names if n in PROVIDERS and PROVIDERS[n].can(capability)]
|
| 184 |
-
|
| 185 |
-
|
| 186 |
-
def estimate(capability: str, records: int) -> dict:
|
| 187 |
-
"""What the FIRST capable provider would cost for `records` β the number an operator plans on.
|
| 188 |
-
|
| 189 |
-
Reported per capability rather than per run because that is the unit that scales: at a thousand
|
| 190 |
-
tenants the question is never "what did this run cost", it is "what does adding view counts to
|
| 191 |
-
every post cost per month".
|
| 192 |
-
"""
|
| 193 |
-
ch = chain(capability)
|
| 194 |
-
if not ch:
|
| 195 |
-
return {"capability": capability, "records": records, "provider": None, "usd": 0.0,
|
| 196 |
-
"note": "no configured, capable provider"}
|
| 197 |
-
p = ch[0]
|
| 198 |
-
c = p.cap(capability)
|
| 199 |
-
return {"capability": capability, "records": records, "provider": p.key,
|
| 200 |
-
"usd": round((c.cost_per_record or 0.0) * max(0, int(records)), 4),
|
| 201 |
-
"note": c.note}
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
@dataclass
|
| 205 |
-
class Attempt:
|
| 206 |
-
provider: str
|
| 207 |
-
ok: bool
|
| 208 |
-
note: str = ""
|
| 209 |
-
records: int = 0
|
| 210 |
-
seconds: float = 0.0
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
def run(capability, work, satisfied=None, log=None):
|
| 214 |
-
"""Walk the chain for `capability` until one provider gives a SATISFYING answer.
|
| 215 |
-
|
| 216 |
-
`work(provider) -> (result, note)`; a non-empty note means it did not answer.
|
| 217 |
-
`satisfied(result) -> bool` decides whether an answer is good enough to stop. Default: any
|
| 218 |
-
truthy result.
|
| 219 |
-
|
| 220 |
-
Returns `(result, attempts)`. `attempts` is the audit trail β every provider tried, whether it
|
| 221 |
-
satisfied, and how long it took β because "where did this number come from and what did it
|
| 222 |
-
cost" is a question somebody asks about a bill, not about a stack trace.
|
| 223 |
-
|
| 224 |
-
β THE `satisfied` HOOK IS THE WHOLE POINT. Without it this is an error-handler, and the
|
| 225 |
-
failure it must catch is not an error: a vendor answering 200 with the one field you needed
|
| 226 |
-
left blank. That is exactly how Bright Data behaves on view counts, and a chain that only
|
| 227 |
-
caught exceptions would have stopped there forever and never reached Apify.
|
| 228 |
-
"""
|
| 229 |
-
log = log or (lambda *_a: None)
|
| 230 |
-
ok = satisfied or (lambda r: bool(r))
|
| 231 |
-
attempts, last = [], None
|
| 232 |
-
for provider in chain(capability):
|
| 233 |
-
started = time.time()
|
| 234 |
-
try:
|
| 235 |
-
result, note = work(provider)
|
| 236 |
-
except Exception as e: # noqa: BLE001
|
| 237 |
-
# β TYPE NAME ONLY. A vendor client's str() can carry the URL, and the credential is
|
| 238 |
-
# one refactor away from being a query param; this line must not be what leaks it.
|
| 239 |
-
result, note = None, f"{type(e).__name__}"
|
| 240 |
-
secs = round(time.time() - started, 2)
|
| 241 |
-
if note:
|
| 242 |
-
attempts.append(Attempt(provider.key, False, note, 0, secs))
|
| 243 |
-
log(f" {provider.label}: {note}, falling through")
|
| 244 |
-
continue
|
| 245 |
-
if not ok(result):
|
| 246 |
-
attempts.append(Attempt(provider.key, False, "answered without the field asked for",
|
| 247 |
-
0, secs))
|
| 248 |
-
log(f" {provider.label}: answered, but not with what was asked for, falling through")
|
| 249 |
-
last = result if last is None else last
|
| 250 |
-
continue
|
| 251 |
-
attempts.append(Attempt(provider.key, True, "", _count(result), secs))
|
| 252 |
-
log(f" {provider.label}: ok ({_count(result)} records, {secs}s)")
|
| 253 |
-
return result, attempts
|
| 254 |
-
return last, attempts
|
| 255 |
-
|
| 256 |
-
|
| 257 |
-
def _count(result):
|
| 258 |
-
if isinstance(result, (list, tuple, set)):
|
| 259 |
-
return len(result)
|
| 260 |
-
if isinstance(result, dict):
|
| 261 |
-
return len(result)
|
| 262 |
-
return 1 if result else 0
|
| 263 |
-
|
| 264 |
-
|
| 265 |
-
def wire():
|
| 266 |
-
"""What the Settings surface renders. Booleans and labels β NEVER a credential.
|
| 267 |
-
|
| 268 |
-
Mirrors the `hikerReady` rule the rest of the product follows: a surface is told WHETHER a
|
| 269 |
-
provider is usable, never what the key is.
|
| 270 |
-
"""
|
| 271 |
-
return {
|
| 272 |
-
"providers": [
|
| 273 |
-
{"key": p.key, "label": p.label, "configured": p.configured(),
|
| 274 |
-
"capabilities": sorted(k for k, c in p.caps.items() if c.capable)}
|
| 275 |
-
for p in PROVIDERS.values()
|
| 276 |
-
],
|
| 277 |
-
"chains": {cap: [p.key for p in chain(cap)] for cap in DEFAULT_CHAINS},
|
| 278 |
-
"incapable": {
|
| 279 |
-
f"{p.key}:{cap}": c.note
|
| 280 |
-
for p in PROVIDERS.values() for cap, c in p.caps.items() if not c.capable
|
| 281 |
-
},
|
| 282 |
-
}
|
| 283 |
-
|
| 284 |
-
|
| 285 |
-
|
| 286 |
-
|
| 287 |
-
# βββββββββ
|
| 288 |
-
#
|
| 289 |
-
# Owner item 4, verbatim (2026-08-18): *"I'm also getting errors everywhere when I want to use the
|
| 290 |
-
# assisntant: 'the assistant could not be reached just now (openrouter: HTTP 402)' and 'cerebras:
|
| 291 |
-
# HTTP 402; groq: HTTP 404; openrouter: HTTP 402; anthropic: tool calls are not wired for this
|
| 292 |
-
# shape'."*
|
| 293 |
-
#
|
| 294 |
-
# ββ R4 IS THREE CLAUSES AND THEY LAND IN THREE DIFFERENT PLACES. (1) Anthropic becomes the
|
| 295 |
-
# tool-calling path that always works β a WIRE, in `routes_query`. (2) A provider with no credit is
|
| 296 |
-
# SKIPPED rather than tried β a memo, `mark_no_credit` below. (3) No raw `HTTP 402` ever reaches a
|
| 297 |
-
# screen β a SENTENCE, `refusal_sentence` below. The declaration here is what the first two read.
|
| 298 |
-
#
|
| 299 |
-
# β WHY A SECOND REGISTRY RATHER THAN ROWS IN `PROVIDERS` ABOVE. A scraping provider is
|
| 300 |
-
# `(key_env, caps)` and is billed per RECORD; an LLM provider is `(env, url, model, wire)` and is
|
| 301 |
-
# billed per TOKEN. Folding them into one dict would mean four fields that are meaningless for half
|
| 302 |
-
# the rows and an `estimate()` that answers $0.00 for anything LLM-shaped. What they SHARE is the
|
| 303 |
-
# thing worth sharing: `Capability`, so "MEASURED INCAPABLE" means exactly the same thing on both
|
| 304 |
-
# sides, and `llm_chain()` refuses an incapable row exactly as `chain()` does.
|
| 305 |
-
#
|
| 306 |
-
# β THE DECLARATION IS THE POINT (staged item 3). `_FAILED_GEN` in `routes_query` is a REGEX that
|
| 307 |
-
# recovers a tool call out of a provider's 400 β the evidence that guessing at capability failed.
|
| 308 |
-
# A row that says `llm_tool_calling: Capability(False, β¦)` is never offered for a tool-calling
|
| 309 |
-
# turn at all, so the guess never has to be made.
|
| 310 |
-
|
| 311 |
-
#: How long a provider stays skipped after it tells us it is out of credit. β A MEMO, NOT A FACT:
|
| 312 |
-
#: the balance can be topped up at any moment, so this expires rather than latching. Fifteen
|
| 313 |
-
#: minutes is long enough that a chat session does not re-pay the timeout on every turn, and short
|
| 314 |
-
#: enough that a top-up is picked up without a restart.
|
| 315 |
-
CREDIT_COOLDOWN_S = float(os.environ.get("AIOS_CREDIT_COOLDOWN_S") or 900)
|
| 316 |
-
|
| 317 |
-
#: `{provider name: unix ts when the memo expires}`. β PROCESS-LOCAL AND DELIBERATELY SO β it is a
|
| 318 |
-
#: latency optimisation, not a billing record. A second container learns the same thing from its
|
| 319 |
-
#: own first 402, and neither one can be wrong for longer than the cooldown.
|
| 320 |
-
_NO_CREDIT: dict[str, float] = {}
|
| 321 |
-
|
| 322 |
-
|
| 323 |
-
@dataclass(frozen=True)
|
| 324 |
-
class LlmProvider:
|
| 325 |
-
"""One chat-completions endpoint, and what it is DECLARED able to do."""
|
| 326 |
-
name: str
|
| 327 |
-
label: str
|
| 328 |
-
env: str
|
| 329 |
-
url: str
|
| 330 |
-
model: str
|
| 331 |
-
#: `openai` = the OpenAI-compatible `/chat/completions` shape. `anthropic` = the Messages API,
|
| 332 |
-
#: which is a different body, a different auth header and a different result shape.
|
| 333 |
-
wire: str
|
| 334 |
-
caps: dict = field(default_factory=dict)
|
| 335 |
-
|
| 336 |
-
def configured(self) -> bool:
|
| 337 |
-
return bool((os.environ.get(self.env) or "").strip())
|
| 338 |
-
|
| 339 |
-
def can(self, capability: str) -> bool:
|
| 340 |
-
cap = self.caps.get(capability)
|
| 341 |
-
return bool(cap and cap.capable and self.configured())
|
| 342 |
-
|
| 343 |
-
|
| 344 |
-
#: β ORDER IS THE LADDER, AND ANTHROPIC IS FIRST BECAUSE OF R4. `routes_query`'s old comment put
|
| 345 |
-
#: cerebras first *"because this path needs tool calling and cerebras carries this account's
|
| 346 |
-
#: tool-capable model"* β R4 replaces that premise: Anthropic is the tool-calling path that always
|
| 347 |
-
#: works, and the others are the cheap seats it falls through to.
|
| 348 |
-
LLM_PROVIDERS: dict[str, LlmProvider] = {
|
| 349 |
-
"anthropic": LlmProvider(
|
| 350 |
-
name="anthropic", label="Anthropic", env="ANTHROPIC_API_KEY",
|
| 351 |
-
url="https://api.anthropic.com/v1/messages",
|
| 352 |
-
# β Claude Opus 5, $5 / $25 per million tokens (2026-06 list). Override per deployment with
|
| 353 |
-
# `AIOS_ANTHROPIC_MODEL` β the id is read at call time, so a cheaper tier
|
| 354 |
-
# (`claude-sonnet-5`, $3 / $15) is an environment change, not a release.
|
| 355 |
-
model=os.environ.get("AIOS_ANTHROPIC_MODEL") or "claude-opus-5",
|
| 356 |
-
wire="anthropic",
|
| 357 |
-
caps={
|
| 358 |
-
# β MEASURED, and it is the whole of R4's first clause: the Messages API answers with a
|
| 359 |
-
# typed `tool_use` content block carrying parsed `input`. There is nothing to recover
|
| 360 |
-
# out of a 400 and no regex in the path β which is exactly what `_FAILED_GEN` exists to
|
| 361 |
-
# apologise for on the other wire.
|
| 362 |
-
"llm_tool_calling": Capability(True, 0.0,
|
| 363 |
-
"typed tool_use content block; no text recovery path"),
|
| 364 |
-
"llm_chat": Capability(True, 0.0, "Messages API"),
|
| 365 |
-
"llm_json_mode": Capability(True, 0.0, "output_config.format, schema-constrained"),
|
| 366 |
-
}),
|
| 367 |
-
"cerebras": LlmProvider(
|
| 368 |
-
name="cerebras", label="Cerebras", env="CEREBRAS_API_KEY",
|
| 369 |
-
url="https://api.cerebras.ai/v1/chat/completions",
|
| 370 |
-
model="gpt-oss-120b", wire="openai",
|
| 371 |
-
caps={
|
| 372 |
-
"llm_tool_calling": Capability(True, 0.0,
|
| 373 |
-
"DECLARED, not measured: this account's tool-capable "
|
| 374 |
-
"model per the wave-32 ladder note"),
|
| 375 |
-
"llm_chat": Capability(True, 0.0, "OpenAI-compatible"),
|
| 376 |
-
"llm_json_mode": Capability(True, 0.0, "response_format json_object"),
|
| 377 |
-
}),
|
| 378 |
-
"groq": LlmProvider(
|
| 379 |
-
name="groq", label="Groq", env="GROQ_API_KEY",
|
| 380 |
-
url="https://api.groq.com/openai/v1/chat/completions",
|
| 381 |
-
model="llama-3.3-70b-versatile", wire="openai",
|
| 382 |
-
caps={
|
| 383 |
-
"llm_tool_calling": Capability(True, 0.0,
|
| 384 |
-
"DECLARED, not measured. β `routes_query._FAILED_GEN` "
|
| 385 |
-
"exists because SOME provider on this wire answers 400 "
|
| 386 |
-
"with the call in `failed_generation`; the repo never "
|
| 387 |
-
"recorded which. Flip this to False the day it is"),
|
| 388 |
-
"llm_chat": Capability(True, 0.0, "OpenAI-compatible"),
|
| 389 |
-
"llm_json_mode": Capability(True, 0.0, "response_format json_object"),
|
| 390 |
-
}),
|
| 391 |
-
"openrouter": LlmProvider(
|
| 392 |
-
name="openrouter", label="OpenRouter", env="OPENROUTER_API_KEY",
|
| 393 |
-
url="https://openrouter.ai/api/v1/chat/completions",
|
| 394 |
-
model="openai/gpt-4o-mini", wire="openai",
|
| 395 |
-
caps={
|
| 396 |
-
"llm_tool_calling": Capability(True, 0.0, "DECLARED, not measured"),
|
| 397 |
-
"llm_chat": Capability(True, 0.0, "OpenAI-compatible"),
|
| 398 |
-
"llm_json_mode": Capability(True, 0.0, "response_format json_object"),
|
| 399 |
-
}),
|
| 400 |
-
}
|
| 401 |
-
|
| 402 |
-
LLM_DEFAULT_ORDER = ("anthropic", "cerebras", "groq", "openrouter")
|
| 403 |
-
|
| 404 |
-
|
| 405 |
-
def mark_no_credit(name, seconds=None):
|
| 406 |
-
"""Remember that `name` said it is out of credit, so the next turn SKIPS it (R4).
|
| 407 |
-
|
| 408 |
-
β THE SECOND CLAUSE OF R4 IS "SKIPPED, NOT TRIED", and without a memo there is nowhere for
|
| 409 |
-
that to live: a stateless ladder re-tries the empty account on every single turn, pays its
|
| 410 |
-
round trip, and shows the reader a longer error each time. This is that memo.
|
| 411 |
-
"""
|
| 412 |
-
_NO_CREDIT[str(name)] = time.time() + float(
|
| 413 |
-
CREDIT_COOLDOWN_S if seconds is None else seconds)
|
| 414 |
-
return _NO_CREDIT[str(name)]
|
| 415 |
-
|
| 416 |
-
|
| 417 |
-
def no_credit(name):
|
| 418 |
-
"""Is this provider inside its out-of-credit cooldown? Expiry is checked, never assumed."""
|
| 419 |
-
until = _NO_CREDIT.get(str(name))
|
| 420 |
-
if not until:
|
| 421 |
-
return False
|
| 422 |
-
if time.time() >= until:
|
| 423 |
-
_NO_CREDIT.pop(str(name), None)
|
| 424 |
-
return False
|
| 425 |
-
return True
|
| 426 |
-
|
| 427 |
-
|
| 428 |
-
def clear_credit_memo(name=None):
|
| 429 |
-
"""Forget one memo, or all of them. For a gate, and for an operator after a top-up."""
|
| 430 |
-
if name is None:
|
| 431 |
-
_NO_CREDIT.clear()
|
| 432 |
-
else:
|
| 433 |
-
_NO_CREDIT.pop(str(name), None)
|
| 434 |
-
|
| 435 |
-
|
| 436 |
-
def llm_chain(capability="llm_tool_calling"):
|
| 437 |
-
"""The provider order for `capability` β declaration first, credit memo second.
|
| 438 |
-
|
| 439 |
-
Three filters, in this order, and each removes a DIFFERENT kind of row:
|
| 440 |
-
1. `can()` β declared capable AND configured. An incapable row is never offered, so a
|
| 441 |
-
turn cannot be spent discovering it (the `chain()` rule, one layer up).
|
| 442 |
-
2. `no_credit()` β R4's skip. A provider that told us its balance is empty is passed over
|
| 443 |
-
until the memo expires.
|
| 444 |
-
3. the ORDER itself, overridable with `AIOS_LLM_ORDER` (`anthropic,groq`) so a deployment
|
| 445 |
-
can be moved off a vendor without a release β the same clause `AIOS_PROVIDER_ORDER`
|
| 446 |
-
carries for the scraping side.
|
| 447 |
-
"""
|
| 448 |
-
raw = (os.environ.get("AIOS_LLM_ORDER") or "").strip()
|
| 449 |
-
names = [x.strip() for x in raw.split(",") if x.strip()] or list(LLM_DEFAULT_ORDER)
|
| 450 |
-
return [LLM_PROVIDERS[n] for n in names
|
| 451 |
-
if n in LLM_PROVIDERS and LLM_PROVIDERS[n].can(capability) and not no_credit(n)]
|
| 452 |
-
|
| 453 |
-
|
| 454 |
-
#: What an HTTP status MEANS, in words a person can act on. ββ R4's THIRD CLAUSE LIVES HERE AND
|
| 455 |
-
#: IT IS NOT COSMETIC: `HTTP 402` on a screen tells a reader nothing they can do, and the owner
|
| 456 |
-
#: quoted it back at us twice. Every sentence names the VENDOR and the ACTION.
|
| 457 |
-
_STATUS_WORDS = {
|
| 458 |
-
401: "{label} would not accept our key",
|
| 459 |
-
403: "{label} would not accept our key",
|
| 460 |
-
402: "{label} is out of credit",
|
| 461 |
-
404: "{label} does not offer the model we asked it for",
|
| 462 |
-
408: "{label} took too long",
|
| 463 |
-
413: "the question was too long for {label}",
|
| 464 |
-
429: "{label} is rate limiting us right now",
|
| 465 |
-
}
|
| 466 |
-
|
| 467 |
-
#: Substrings that mean "no money" on a wire that does not use 402. β Anthropic answers a spent
|
| 468 |
-
#: balance with a 400 or 403 carrying a message, not with a status code of its own, so this is the
|
| 469 |
-
#: one place a body has to be read. It is a LOWERCASE substring test on the vendor's own words and
|
| 470 |
-
#: it only ever decides whether to SKIP a provider, never whether to trust one.
|
| 471 |
-
_CREDIT_WORDS = ("credit balance", "insufficient credit", "insufficient_quota", "out of credit",
|
| 472 |
-
"quota exceeded", "billing", "payment required", "add credits")
|
| 473 |
-
|
| 474 |
-
|
| 475 |
-
def is_credit_failure(status, body=""):
|
| 476 |
-
"""Did this response mean "the account is empty"? Status first, then the vendor's own words."""
|
| 477 |
-
if int(status or 0) == 402:
|
| 478 |
-
return True
|
| 479 |
-
if int(status or 0) not in (400, 403, 429):
|
| 480 |
-
return False
|
| 481 |
-
return any(word in str(body or "").lower() for word in _CREDIT_WORDS)
|
| 482 |
-
|
| 483 |
-
|
| 484 |
-
def refusal_sentence(name, status, body=""):
|
| 485 |
-
"""One provider's failure, as a SENTENCE. Never a bare status code, never a vendor stack trace.
|
| 486 |
-
|
| 487 |
-
β THE BODY IS READ AND NEVER QUOTED. A provider's error body can carry an account id, a key
|
| 488 |
-
prefix or an internal trace; the only thing taken out of it is the yes/no answer to "is this a
|
| 489 |
-
credit problem", and what reaches the caller is this module's own wording.
|
| 490 |
-
"""
|
| 491 |
-
label = (LLM_PROVIDERS.get(str(name)) or LlmProvider(name, str(name), "", "", "", "")).label
|
| 492 |
-
if is_credit_failure(status, body):
|
| 493 |
-
return f"{label} is out of credit"
|
| 494 |
-
code = int(status or 0)
|
| 495 |
-
if code in _STATUS_WORDS:
|
| 496 |
-
return _STATUS_WORDS[code].format(label=label)
|
| 497 |
-
if 500 <= code <= 599:
|
| 498 |
-
return f"{label} is having trouble at their end"
|
| 499 |
-
return f"{label} did not answer"
|
| 500 |
-
|
| 501 |
-
|
| 502 |
-
# βββββββββββ THE ANTHROPIC WIRE, ONCE (W36-T35 / ASK D-18, ruling R4) βββββββββββββββββββββββββββ
|
| 503 |
-
#
|
| 504 |
-
# ββ TWO DOORS IN THIS PRODUCT CALL ANTHROPIC AND THEY MUST NOT EACH LEARN THE MESSAGES API.
|
| 505 |
-
# `routes_query._call_model` (the Assistant and Query) and `ai_review.draft_flow` (the automation
|
| 506 |
-
# drafter) both need it, and the owner quoted an error from EACH of them in one breath:
|
| 507 |
-
# *"the assistant could not be reached just now (openrouter: HTTP 402)"* and *"anthropic: tool
|
| 508 |
-
# calls are not wired for this shape"*. Two implementations of one wire is
|
| 509 |
-
# [[one-question-two-normalizers]] before a line is written, so the wire lives here, beside the
|
| 510 |
-
# ladder that declares the rung.
|
| 511 |
-
#
|
| 512 |
-
# FOUR THINGS THE OPENAI-COMPATIBLE SHAPE GETS WRONG, each a 400 on its own:
|
| 513 |
-
# 1. the system prompt is a TOP-LEVEL field, not a `{"role": "system"}` message
|
| 514 |
-
# 2. a tool is `{name, description, input_schema}` FLAT, not nested under `function`
|
| 515 |
-
# 3. `temperature` and friends are REMOVED on the current model family
|
| 516 |
-
# 4. `tool_choice` is an OBJECT (`{"type": "auto"}` / `{"type": "any"}`), not a string
|
| 517 |
-
#
|
| 518 |
-
# β AND ONE THING THAT IS NOT A SHAPE: `effort` is model-gated. `output_config.effort` errors on
|
| 519 |
-
# Haiku 4.5, so it is a PARAMETER here and the caller decides β the drafter runs on haiku and omits
|
| 520 |
-
# it, the assistant runs on the Opus tier and sends it.
|
| 521 |
-
|
| 522 |
-
#: The Messages API version header. A DATE that pins the WIRE FORMAT, never a model.
|
| 523 |
-
ANTHROPIC_VERSION = "2023-06-01"
|
| 524 |
-
|
| 525 |
-
|
| 526 |
-
def anthropic_request(*, model, key, system, messages, tools, max_tokens,
|
| 527 |
-
tool_choice="auto", effort=None):
|
| 528 |
-
"""`{url, headers, json}` for one Messages API call. Pure: reads no environment, sends nothing.
|
| 529 |
-
|
| 530 |
-
`messages` is the OpenAI-shaped list this product already builds; the `system` turns are lifted
|
| 531 |
-
out of it, because that is where this API wants them. `tools` is the OpenAI-shaped tool list,
|
| 532 |
-
re-addressed rather than re-derived, so a schema change happens in one place.
|
| 533 |
-
"""
|
| 534 |
-
system_text = "\n\n".join(str(m.get("content") or "") for m in messages
|
| 535 |
-
if m.get("role") == "system")
|
| 536 |
-
if system:
|
| 537 |
-
system_text = (system_text + "\n\n" + str(system)).strip() if system_text else str(system)
|
| 538 |
-
turns = [{"role": ("assistant" if m.get("role") == "assistant" else "user"),
|
| 539 |
-
"content": str(m.get("content") or "")}
|
| 540 |
-
for m in messages if m.get("role") != "system" and str(m.get("content") or "").strip()]
|
| 541 |
-
body = {
|
| 542 |
-
"model": str(model),
|
| 543 |
-
"max_tokens": int(max_tokens),
|
| 544 |
-
"system": system_text,
|
| 545 |
-
"messages": turns,
|
| 546 |
-
"tools": [{"name": t["function"]["name"],
|
| 547 |
-
"description": t["function"]["description"],
|
| 548 |
-
"input_schema": t["function"]["parameters"]} for t in (tools or [])],
|
| 549 |
-
"tool_choice": {"type": "any" if tool_choice in ("required", "any") else "auto"},
|
| 550 |
-
}
|
| 551 |
-
if effort:
|
| 552 |
-
body["output_config"] = {"effort": str(effort)}
|
| 553 |
-
return {"url": LLM_PROVIDERS["anthropic"].url,
|
| 554 |
-
"headers": {"x-api-key": str(key),
|
| 555 |
-
"anthropic-version": ANTHROPIC_VERSION,
|
| 556 |
-
"content-type": "application/json"},
|
| 557 |
-
"json": body}
|
| 558 |
-
|
| 559 |
-
|
| 560 |
-
def anthropic_read(body):
|
| 561 |
-
"""`(text, tool_input, refusal)` out of a Messages API answer.
|
| 562 |
-
|
| 563 |
-
β `stop_reason` IS CHECKED BEFORE `content` IS READ. A safety decline answers **HTTP 200** with
|
| 564 |
-
`stop_reason: "refusal"` and an empty or partial `content`, so code that indexes `content[0]`
|
| 565 |
-
unconditionally breaks on exactly the turn a person most needs explained.
|
| 566 |
-
β AND THE TOOL CALL ARRIVES PARSED. `tool_use.input` is already a dict β no `json.loads`, and
|
| 567 |
-
no regex recovering a call out of a 400, which is what the other wire needs.
|
| 568 |
-
"""
|
| 569 |
-
body = body if isinstance(body, dict) else {}
|
| 570 |
-
if str(body.get("stop_reason") or "") == "refusal":
|
| 571 |
-
return "", None, "the assistant declined to answer that one"
|
| 572 |
-
blocks = [b for b in (body.get("content") or []) if isinstance(b, dict)]
|
| 573 |
-
text = " ".join(str(b.get("text") or "") for b in blocks if b.get("type") == "text").strip()
|
| 574 |
-
calls = [b for b in blocks if b.get("type") == "tool_use"]
|
| 575 |
-
got = calls[0].get("input") if calls else None
|
| 576 |
-
return text, (got if isinstance(got, dict) else None), None
|
| 577 |
-
|
| 578 |
-
|
| 579 |
-
def llm_status(capability="llm_tool_calling"):
|
| 580 |
-
"""Per provider: configured, declared-capable, in cooldown, and WHY β contract C5's payload.
|
| 581 |
-
|
| 582 |
-
β ONE LIST, ONE DOOR. The Assistant's model picker and the Agent chat's toggle (W36-T34) read
|
| 583 |
-
THIS, so a model offered in one place cannot be missing from the other, and neither can offer a
|
| 584 |
-
provider the ladder would refuse to call [[permitted-is-not-answerable]].
|
| 585 |
-
"""
|
| 586 |
-
rows = []
|
| 587 |
-
for name in LLM_DEFAULT_ORDER:
|
| 588 |
-
p = LLM_PROVIDERS[name]
|
| 589 |
-
cap = p.caps.get(capability)
|
| 590 |
-
rows.append({
|
| 591 |
-
"provider": p.name, "label": p.label, "model": p.model, "wire": p.wire,
|
| 592 |
-
"configured": p.configured(),
|
| 593 |
-
"toolCalling": bool((p.caps.get("llm_tool_calling") or Capability(False)).capable),
|
| 594 |
-
"jsonMode": bool((p.caps.get("llm_json_mode") or Capability(False)).capable),
|
| 595 |
-
"capable": bool(cap and cap.capable),
|
| 596 |
-
"outOfCredit": no_credit(name),
|
| 597 |
-
"note": (cap.note if cap else ""),
|
| 598 |
-
})
|
| 599 |
-
return rows
|
| 600 |
-
|
| 601 |
-
|
| 602 |
-
# =============================================================================================
|
| 603 |
-
# THE CANONICAL SCHEMA β owner ruling 2026-08-08:
|
| 604 |
-
# *"standardize the schema between Bright Data and APIfy so we keep using the same pre-set
|
| 605 |
-
# database even if the underlying engine changes"*
|
| 606 |
-
#
|
| 607 |
-
# β THE PRESET TABLES ARE THE CONTRACT; A VENDOR IS AN IMPLEMENTATION DETAIL. `ut_ig_posts` and
|
| 608 |
-
# `ut_ig_post_snapshots` must not gain, lose or rename a column because a chain was reordered β a
|
| 609 |
-
# tenant's saved views, filters, rollups and forms all bind to these keys, and a schema that moves
|
| 610 |
-
# with the vendor turns a routing change into a data migration.
|
| 611 |
-
#
|
| 612 |
-
# So every provider normalises INTO the keys below and nothing reads a vendor row downstream.
|
| 613 |
-
# `verify_automation` asserts both normalisers emit exactly `CANONICAL_POST_KEYS` on a fixture, so
|
| 614 |
-
# a third provider cannot ship with a near-miss key like `viewCount` and silently write a column
|
| 615 |
-
# nobody declared.
|
| 616 |
-
#
|
| 617 |
-
# β ONE NAME PER MEASUREMENT, AND `views` IS THE MEASUREMENT INSTAGRAM DISPLAYS. Meta folded
|
| 618 |
-
# Impressions/Plays/Video Views into a single **Views** metric on 2025-04-10, so carrying both a
|
| 619 |
-
# `views` and a `plays` column would be modelling a distinction the platform deleted β and it is
|
| 620 |
-
# exactly the distinction that let an account-grain number wear the "Views" label for a month.
|
| 621 |
-
# β THE VENDOR KEY THAT LOOKS RIGHT IS THE WRONG ONE, ON BOTH VENDORS. Apify ships BOTH
|
| 622 |
-
# `videoViewCount` (10,678) and `videoPlayCount` (137,684) for one reel whose true displayed count
|
| 623 |
-
# is ~137K β and Bright Data's useless `views` is 10,638 for that same reel. The two vendors' junk
|
| 624 |
-
# fields AGREE with each other, which is precisely what makes picking by name so dangerous.
|
| 625 |
-
# canonical `views` <- apify `videoPlayCount` β
matches the grid
|
| 626 |
-
# canonical `views` <- apify `videoViewCount` β off by 13x
|
| 627 |
-
# canonical `views` <- brightdata `views` β off by 13x AND account-grain
|
| 628 |
-
# =============================================================================================
|
| 629 |
-
|
| 630 |
-
#: Every key a normalised POST row may carry. Absent > blank: a key is omitted when the provider
|
| 631 |
-
#: did not answer, because `upsert_rows` merges and an empty string would ERASE what an earlier
|
| 632 |
-
#: paid run learned.
|
| 633 |
-
CANONICAL_POST_KEYS = (
|
| 634 |
-
"shortcode", "url", "influencer_key", "posted_at", "type", "caption",
|
| 635 |
-
"likes", "comments", "views", "paid_partnership", "partner", "hashtags",
|
| 636 |
-
"alt_text", "tagged_location", "source_payload",
|
| 637 |
-
# β 2026-08-09 β the three the SECOND provider answers and the first does not. Each has a
|
| 638 |
-
# matching `field_def` in the engine's POST_FIELDS; a key here without a column there is a
|
| 639 |
-
# value that normalises cleanly and is then dropped by the write door, silently.
|
| 640 |
-
"plays", "video_duration", "comments_disabled",
|
| 641 |
-
)
|
| 642 |
-
|
| 643 |
-
|
| 644 |
-
def _int_or_none(v):
|
| 645 |
-
"""`-1` IS NOT A COUNT. Apify returns `likesCount: -1` when the creator HIDES their like
|
| 646 |
-
count β a real state that is not a measurement. Writing -1 would render as a negative like
|
| 647 |
-
count; writing 0 would claim nobody liked it. Both are lies, so the key is omitted."""
|
| 648 |
-
try:
|
| 649 |
-
n = int(v)
|
| 650 |
-
except (TypeError, ValueError):
|
| 651 |
-
return None
|
| 652 |
-
return None if n < 0 else n
|
| 653 |
-
|
| 654 |
-
|
| 655 |
-
def normalize_post_apify(row):
|
| 656 |
-
"""One Apify `instagram-scraper` item -> the canonical post row."""
|
| 657 |
-
if not isinstance(row, dict):
|
| 658 |
-
return None
|
| 659 |
-
code = str(row.get("shortCode") or "").strip()
|
| 660 |
-
if not code:
|
| 661 |
-
return None
|
| 662 |
-
out = {
|
| 663 |
-
"shortcode": code,
|
| 664 |
-
"url": str(row.get("url") or f"https://www.instagram.com/reel/{code}/"),
|
| 665 |
-
"influencer_key": str(row.get("ownerUsername") or ""),
|
| 666 |
-
"posted_at": str(row.get("timestamp") or "").replace("T", " ")[:16],
|
| 667 |
-
"type": "video" if str(row.get("type") or "").lower() == "video" else
|
| 668 |
-
("carousel" if row.get("childPosts") else "image"),
|
| 669 |
-
"caption": str(row.get("caption") or ""),
|
| 670 |
-
# β THE FIELD THIS WHOLE PROVIDER EXISTS FOR. `videoPlayCount`, never `videoViewCount`:
|
| 671 |
-
# the wrong one agreed with Bright Data's junk (10,678 vs 10,638) on a reel whose true
|
| 672 |
-
# count was ~137K, and `videoPlayCount` matched a browser read to the digit.
|
| 673 |
-
"views": _int_or_none(row.get("videoPlayCount")),
|
| 674 |
-
# ββ 2026-08-09 (owner: *"Video plays (# Plays) field isn't in APIfy? i believe it is"*).
|
| 675 |
-
# They were right, and the `plays` COLUMN had been empty on all 816 rows because nothing
|
| 676 |
-
# ever wrote it β the field existed with no writer. MEASURED on two of their own reels:
|
| 677 |
-
# `videoPlayCount` 216,904 / 95,331 and `videoViewCount` 115,929 / 30,871. Two different
|
| 678 |
-
# real numbers, and we were storing only one of them.
|
| 679 |
-
# β `plays` TAKES THE PLAY COUNT, which is also what `views` carries today β so the two
|
| 680 |
-
# columns will agree until somebody decides otherwise, and that decision is the OWNER'S:
|
| 681 |
-
# re-sourcing `views` to `videoViewCount` would change what the ~480 rows already captured
|
| 682 |
-
# mean, and a column whose meaning changes halfway down is the one thing worse than a
|
| 683 |
-
# column with no data. Flagged rather than done.
|
| 684 |
-
"plays": _int_or_none(row.get("videoPlayCount")),
|
| 685 |
-
"likes": _int_or_none(row.get("likesCount")),
|
| 686 |
-
"comments": _int_or_none(row.get("commentsCount")),
|
| 687 |
-
# β FIELDS BRIGHT DATA DOES NOT RETURN AT ALL, kept because they are already paid for in
|
| 688 |
-
# this same response (owner: *"whatever APIfy has more than BD pls use it"*).
|
| 689 |
-
"video_duration": _int_or_none(row.get("videoDuration")),
|
| 690 |
-
"comments_disabled": "1" if row.get("isCommentsDisabled") else "",
|
| 691 |
-
"hashtags": ", ".join(str(h) for h in (row.get("hashtags") or []) if h),
|
| 692 |
-
"alt_text": str(row.get("alt") or ""),
|
| 693 |
-
"paid_partnership": "1" if row.get("paidPartnership") else "",
|
| 694 |
-
"partner": ", ".join(str((s or {}).get("username") or "")
|
| 695 |
-
for s in (row.get("sponsors") or []) if isinstance(s, dict)),
|
| 696 |
-
"tagged_location": str((row.get("locationName") or "")),
|
| 697 |
-
"source_payload": json.dumps(row, default=str)[:32_000_000],
|
| 698 |
-
}
|
| 699 |
-
return {k: v for k, v in out.items() if v not in (None, "")}
|
| 700 |
-
|
| 701 |
-
|
| 702 |
-
def normalize_profile_apify(row):
|
| 703 |
-
"""One Apify `instagram-scraper` PROFILE item (`resultsType: "details"`) -> our snapshot shape.
|
| 704 |
-
|
| 705 |
-
β WHY THIS EXISTS (owner report 2026-08-09: *"I need Bright Data and Apify to work correctly
|
| 706 |
-
in tandem"*). `PROVIDERS["apify"]` has declared `ig_profile` capable since 2026-08-08 and
|
| 707 |
-
`DEFAULT_CHAINS["ig_profile"]` has read `("brightdata", "apify")` β but **nothing ever ran
|
| 708 |
-
that chain.** `connectors_ig.pull_profile` went Bright Data -> anonymous HTML rungs and Apify
|
| 709 |
-
was never asked, so a profile Bright Data cannot scrape came back `blocked` while a
|
| 710 |
-
configured, declared-capable provider sat unused. A registry entry with no runner is a
|
| 711 |
-
promise the product does not keep ([[flag-shipped-without-its-writer]]).
|
| 712 |
-
|
| 713 |
-
β THE KEYS ARE APIFY'S camelCase, and mapping them HERE is the boundary rule this module
|
| 714 |
-
already enforces for posts: nothing downstream may ever see a vendor-shaped key, so the
|
| 715 |
-
snapshot writer cannot tell which vendor answered β which is what makes the fallback
|
| 716 |
-
invisible to every consumer instead of a second schema.
|
| 717 |
-
|
| 718 |
-
β OMIT, NEVER ZERO. A field Apify did not send is dropped, exactly as `normalize_post_apify`
|
| 719 |
-
drops a missing count: writing 0 followers would claim we measured an empty account, and the
|
| 720 |
-
caller's `satisfied` hook reads absence as "did not answer" and falls through. A zero would
|
| 721 |
-
stop the chain on a lie.
|
| 722 |
-
"""
|
| 723 |
-
if not isinstance(row, dict):
|
| 724 |
-
return None
|
| 725 |
-
# β AN ERROR ENVELOPE IS NOT A PROFILE (measured 2026-08-09). Apify answers a dead handle
|
| 726 |
-
# with a 200 and a well-formed item β `{"username": β¦, "error": "not_found",
|
| 727 |
-
# "errorDescription": "Post does not exist"}` β and this function used to build a "profile"
|
| 728 |
-
# out of it: a username, a url and a source_payload, with every measured field absent. It
|
| 729 |
-
# then read to the caller as a vendor that answered, so the reason was replaced by a shrug.
|
| 730 |
-
# Refused HERE as well as in `connectors_ig.apify_profile` on purpose: the boundary rule this
|
| 731 |
-
# module exists for is that nothing downstream ever sees a vendor-shaped key, and a vendor's
|
| 732 |
-
# ERROR shape is the one that must never become a row.
|
| 733 |
-
# β TRUTHINESS, NOT KEY PRESENCE β a successful item carries `error: null` (measured on
|
| 734 |
-
# `sriyynntt`), so `"error" in row` would refuse every good profile.
|
| 735 |
-
if row.get("error"):
|
| 736 |
-
return None
|
| 737 |
-
handle = str(row.get("username") or "").strip()
|
| 738 |
-
if not handle:
|
| 739 |
-
return None
|
| 740 |
-
out = {
|
| 741 |
-
"username": handle,
|
| 742 |
-
"full_name": str(row.get("fullName") or ""),
|
| 743 |
-
"bio": str(row.get("biography") or ""),
|
| 744 |
-
"followers": _int_or_none(row.get("followersCount")),
|
| 745 |
-
"following": _int_or_none(row.get("followsCount")),
|
| 746 |
-
"posts_count": _int_or_none(row.get("postsCount")),
|
| 747 |
-
"verified": "1" if row.get("verified") else "",
|
| 748 |
-
"external_url": str(row.get("externalUrl") or ""),
|
| 749 |
-
"ig_id": str(row.get("id") or ""),
|
| 750 |
-
"profile_url": str(row.get("url") or f"https://www.instagram.com/{handle}/"),
|
| 751 |
-
"business_category": str(row.get("businessCategoryName") or ""),
|
| 752 |
-
"is_business": "1" if row.get("isBusinessAccount") else "",
|
| 753 |
-
"is_private": "1" if row.get("private") else "",
|
| 754 |
-
"highlights_count": _int_or_none(row.get("highlightReelCount")),
|
| 755 |
-
"source_payload": json.dumps(row, default=str)[:32_000_000],
|
| 756 |
-
}
|
| 757 |
-
# β `followers`/`following` are ints and 0 is a LEGITIMATE value for them, so the filter
|
| 758 |
-
# below must not treat 0 as absent the way the post normaliser can β an account really can
|
| 759 |
-
# have zero followers. Only None and "" are dropped.
|
| 760 |
-
return {k: v for k, v in out.items() if v is not None and v != ""}
|
| 761 |
-
|
| 762 |
-
|
| 763 |
-
def canonical_gaps(row, want=("views",)):
|
| 764 |
-
"""Which requested canonical fields this row does NOT carry β the `satisfied` input.
|
| 765 |
-
|
| 766 |
-
Named rather than inlined because "did the vendor actually answer the question" is the whole
|
| 767 |
-
fallback trigger, and a chain that asks it differently in two places will drift.
|
| 768 |
-
"""
|
| 769 |
-
row = row if isinstance(row, dict) else {}
|
| 770 |
-
return [k for k in want if row.get(k) in (None, "")]
|
|
|
|
| 1 |
+
"""THE PROVIDER LAYER β many vendors behind one capability, chosen per FIELD and per cost.
|
| 2 |
+
|
| 3 |
+
ββ OWNER RULING 2026-08-08: *"Let's do a combination scraper. Do Bright Data first, and if
|
| 4 |
+
anything fails, we use APIfy or vice versa. Make this dynamic somehow. remember to cost optimize
|
| 5 |
+
when we scale to thousands+ customer β¦ make sure our code is customizable and modular."*
|
| 6 |
+
|
| 7 |
+
β THE DESIGN DECISION THAT MATTERS, AND IT IS NOT "TRY A, THEN TRY B".
|
| 8 |
+
A provider-level failover ("if Bright Data errors, re-run the whole thing on Apify") is the obvious
|
| 9 |
+
shape and it is the expensive one. MEASURED: Bright Data answers profile, likes and comments
|
| 10 |
+
correctly and CANNOT answer view counts on any route; Apify answers view counts correctly. A
|
| 11 |
+
provider-level fallback would notice the missing views and re-buy the profile, the likes and the
|
| 12 |
+
comments from Apify as well β **paying twice for the 90% that already worked**. At one profile that
|
| 13 |
+
is noise; at thousands of customers Γ twelve posts each it is the whole bill.
|
| 14 |
+
|
| 15 |
+
So routing is per CAPABILITY, and a capability is the smallest independently-billable thing:
|
| 16 |
+
|
| 17 |
+
ig_profile -> brightdata (36 fields, works)
|
| 18 |
+
ig_post_metrics -> brightdata (likes + comments, works)
|
| 19 |
+
ig_post_views -> apify (Bright Data is INCAPABLE β see `capable=False` below)
|
| 20 |
+
ig_comments -> brightdata
|
| 21 |
+
|
| 22 |
+
Each capability has an ORDERED list of providers. The first one that is configured AND capable AND
|
| 23 |
+
succeeds wins; the rest are never called, so the common path costs exactly one vendor record.
|
| 24 |
+
|
| 25 |
+
β "FAILURE" INCLUDES ANSWERING WITH THE FIELD BLANK. A vendor that returns HTTP 200 and a null in
|
| 26 |
+
the one column you asked for has failed for our purposes, and a chain that only catches exceptions
|
| 27 |
+
would stop at it forever. `run()` takes a `satisfied` predicate and falls through on an unsatisfying
|
| 28 |
+
answer exactly as it would on a 500.
|
| 29 |
+
|
| 30 |
+
β ORDER IS DATA, NOT CODE. `AIOS_PROVIDER_ORDER` overrides any chain at deploy time
|
| 31 |
+
(`ig_post_views=apify,brightdata;ig_profile=apify`), which is what "or vice versa" means and what
|
| 32 |
+
lets a tenant be moved off a vendor without a release.
|
| 33 |
+
"""
|
| 34 |
+
from __future__ import annotations
|
| 35 |
+
|
| 36 |
+
import json
|
| 37 |
+
import os
|
| 38 |
+
import time
|
| 39 |
+
from dataclasses import dataclass, field
|
| 40 |
+
from typing import Callable
|
| 41 |
+
|
| 42 |
+
#: β APPROXIMATE, AND DELIBERATELY SO. These are list prices per RECORD in USD, used only to RANK
|
| 43 |
+
#: providers and to estimate a run's spend for the operator. They are not billing truth β the
|
| 44 |
+
#: vendor's own dashboard is. Wrong by 2x still ranks correctly; wrong by 100x does not, which is
|
| 45 |
+
#: why they are named and dated rather than guessed silently.
|
| 46 |
+
#: Bright Data Instagram datasets ~ $0.0015/record (2026-08 list). Apify instagram-scraper
|
| 47 |
+
#: ~ $0.0027/result (2026-08 list). Both re-checked when a provider is added.
|
| 48 |
+
#: β WAVE 30 Β· T16 β APIFY CORRECTED 0.0023 -> 0.0027, and the direction matters: the old number
|
| 49 |
+
#: made the fallback look CHEAPER than it is, and `estimate()` is what a person is shown before
|
| 50 |
+
#: they authorise a run. An under-stated price is the one rounding error a cost guard cannot catch.
|
| 51 |
+
#: β Both are LIST rates used to rank and to estimate. They are not invoices, and nothing here
|
| 52 |
+
#: reads a live price β a modelled figure that says so is honest; one that pretends is not.
|
| 53 |
+
_COST_BRIGHTDATA = float(os.environ.get("AIOS_COST_BRIGHTDATA") or 0.0015)
|
| 54 |
+
_COST_APIFY = float(os.environ.get("AIOS_COST_APIFY") or 0.0027)
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
@dataclass(frozen=True)
|
| 58 |
+
class Capability:
|
| 59 |
+
"""What ONE provider can do for ONE capability, and what it costs to ask."""
|
| 60 |
+
#: β `False` means MEASURED INCAPABLE, not "untried". A provider declared incapable is never
|
| 61 |
+
#: called for this capability at all β it cannot be reached by a fallback, cannot be put first
|
| 62 |
+
#: by an env override, and cannot silently start being billed because somebody reordered a
|
| 63 |
+
#: list. Bright Data's `ig_post_views` is False on the strength of Β§4e-Β§4h: every route
|
| 64 |
+
#: (Posts x /p/, Posts x /reel/, Reels x /p/, Reels x /reel/, discover-by-profile) on four
|
| 65 |
+
#: accounts from 13K to 268M followers returned an account-grain constant or a null.
|
| 66 |
+
capable: bool = True
|
| 67 |
+
cost_per_record: float = 0.0
|
| 68 |
+
#: Free-text, shown to an operator deciding where their money went.
|
| 69 |
+
note: str = ""
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
@dataclass
|
| 73 |
+
class Provider:
|
| 74 |
+
key: str
|
| 75 |
+
label: str
|
| 76 |
+
#: The env var holding this provider's credential. NEVER the credential itself β this module
|
| 77 |
+
#: is imported by surfaces that serialise their config.
|
| 78 |
+
key_env: str = ""
|
| 79 |
+
caps: dict = field(default_factory=dict)
|
| 80 |
+
|
| 81 |
+
def configured(self) -> bool:
|
| 82 |
+
return bool((os.environ.get(self.key_env) or "").strip()) if self.key_env else True
|
| 83 |
+
|
| 84 |
+
def cap(self, capability: str) -> Capability | None:
|
| 85 |
+
return self.caps.get(capability)
|
| 86 |
+
|
| 87 |
+
def can(self, capability: str) -> bool:
|
| 88 |
+
c = self.cap(capability)
|
| 89 |
+
return bool(c and c.capable and self.configured())
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
#: β ADDING A PROVIDER IS A REGISTRY ENTRY PLUS A RUNNER β no change to any caller. That is the
|
| 93 |
+
#: "modular/customizable" half of the ruling, and the reason the chains below name STRINGS.
|
| 94 |
+
PROVIDERS: dict[str, Provider] = {
|
| 95 |
+
"brightdata": Provider(
|
| 96 |
+
key="brightdata", label="Bright Data", key_env="AIOS_BRIGHTDATA_KEY",
|
| 97 |
+
caps={
|
| 98 |
+
"ig_profile": Capability(True, _COST_BRIGHTDATA, "36 fields incl. bio/email/category"),
|
| 99 |
+
"ig_post_metrics": Capability(True, _COST_BRIGHTDATA, "likes + comments + captions"),
|
| 100 |
+
"ig_comments": Capability(True, _COST_BRIGHTDATA, "separate paid dataset, opt-in"),
|
| 101 |
+
# β THE ONE THAT COSTS US NOTHING TO GET RIGHT AND EVERYTHING TO GET WRONG.
|
| 102 |
+
"ig_post_views": Capability(
|
| 103 |
+
False, _COST_BRIGHTDATA,
|
| 104 |
+
"MEASURED INCAPABLE: returns one account-grain number for every reel of a "
|
| 105 |
+
"creator (identical for two shortcodes in one call, ticking upward between "
|
| 106 |
+
"calls) and never populates video_play_count. Their own documented example "
|
| 107 |
+
"account no longer reproduces it."),
|
| 108 |
+
# ββ WAVE 29 (item 7 / D-9 / R1) β TIKTOK. Three datasets, all reachable with our
|
| 109 |
+
# existing key, all schema-probed for $0.00 (40/43/17 fields with the vendor's own
|
| 110 |
+
# types): `gd_l1villgoiiidt09ci` Β· `gd_lu702nij2f790tmv9h` Β· `gd_lkf2st302ap89utw5k`.
|
| 111 |
+
"tt_profile": Capability(True, _COST_BRIGHTDATA,
|
| 112 |
+
"40 fields incl. bio/engagement rates/region"),
|
| 113 |
+
# β THERE IS NO `tt_post_views` CAPABILITY, AND ITS ABSENCE IS THE CLAIM.
|
| 114 |
+
# `TikTok - Posts` declares `play_count: number` and the vendor's own sample says
|
| 115 |
+
# "no empty values or zeros" β so on TikTok the view count arrives INSIDE the post
|
| 116 |
+
# record and a separate view rung would be a second bill for a number we already have.
|
| 117 |
+
# β DECLARED, NOT MEASURED, and that distinction is the whole scar tissue of Β§4e:
|
| 118 |
+
# Bright Data's Instagram Reels also DECLARES `views: number` and delivers an
|
| 119 |
+
# account-grain wrong one. One live pull settles it (W29-T06, spend-gated). If it
|
| 120 |
+
# fails, the honest repair is a `tt_post_views` capability routed elsewhere β never a
|
| 121 |
+
# quiet fallback bolted onto this one.
|
| 122 |
+
"tt_post_metrics": Capability(
|
| 123 |
+
True, _COST_BRIGHTDATA,
|
| 124 |
+
"likes + comments + shares + saves, and play_count inline (DECLARED "
|
| 125 |
+
"no-empties-or-zeros, UNPROVEN until one live pull)"),
|
| 126 |
+
"tt_comments": Capability(True, _COST_BRIGHTDATA,
|
| 127 |
+
"separate paid dataset, opt-in: 17 fields"),
|
| 128 |
+
}),
|
| 129 |
+
"apify": Provider(
|
| 130 |
+
key="apify", label="Apify", key_env="AIOS_APIFY_KEY",
|
| 131 |
+
caps={
|
| 132 |
+
# β MEASURED 2026-08-08 against the public Reels grid: `videoPlayCount` 137,684 and
|
| 133 |
+
# 299,493 vs a browser-read ground truth of 134K-137K and 299K. Exact.
|
| 134 |
+
"ig_post_views": Capability(True, _COST_APIFY,
|
| 135 |
+
"videoPlayCount, matches Instagram's displayed views"),
|
| 136 |
+
"ig_post_metrics": Capability(True, _COST_APIFY, "likes + comments (fallback)"),
|
| 137 |
+
"ig_profile": Capability(True, _COST_APIFY, "profile fields (fallback)"),
|
| 138 |
+
}),
|
| 139 |
+
}
|
| 140 |
+
|
| 141 |
+
#: The DEFAULT chain per capability, cheapest-capable-first. Overridable β see `chain()`.
|
| 142 |
+
DEFAULT_CHAINS: dict[str, tuple] = {
|
| 143 |
+
"ig_profile": ("brightdata", "apify"),
|
| 144 |
+
"ig_post_metrics": ("brightdata", "apify"),
|
| 145 |
+
# Only one entry, and that is the point: Bright Data is declared incapable, so listing it here
|
| 146 |
+
# would be a lie that costs a wasted call on every single post.
|
| 147 |
+
"ig_post_views": ("apify",),
|
| 148 |
+
"ig_comments": ("brightdata",),
|
| 149 |
+
# β WAVE 29 β TikTok, and every chain is deliberately SINGLE-PROVIDER.
|
| 150 |
+
# β A MULTI-PROVIDER CHAIN IS A PROMISE SOMETHING WALKS IT. `ig_post_metrics` has declared a
|
| 151 |
+
# two-provider fallback since wave 28 and NOTHING reaches the second name: if Bright Data
|
| 152 |
+
# answers with the likes blank, Apify is never asked. `verify_automation`'s E2b pins that gap
|
| 153 |
+
# BY NAME so a third unwalked chain turns it red β which is exactly what a second name here
|
| 154 |
+
# would be today. Apify does sell TikTok (`clockworks/tiktok-profile-scraper`, $0.003/result,
|
| 155 |
+
# 0.7% 30-day failure rate β measured from its own store record), so the fallback is buildable;
|
| 156 |
+
# it is not declared until a runner walks it.
|
| 157 |
+
"tt_profile": ("brightdata",),
|
| 158 |
+
"tt_post_metrics": ("brightdata",),
|
| 159 |
+
"tt_comments": ("brightdata",),
|
| 160 |
+
}
|
| 161 |
+
|
| 162 |
+
|
| 163 |
+
def chain(capability: str) -> list:
|
| 164 |
+
"""The provider order for `capability` β env override first, then the default.
|
| 165 |
+
|
| 166 |
+
`AIOS_PROVIDER_ORDER` is a `;`-separated list of `capability=p1,p2` clauses. This is how the
|
| 167 |
+
owner's "or vice versa" is expressed WITHOUT a release, and how one tenant can be moved off a
|
| 168 |
+
vendor that is having a bad day.
|
| 169 |
+
β An override may REORDER and may DROP, but it can never make an incapable provider capable β
|
| 170 |
+
`can()` still gates every name. A chain that reads `ig_post_views=brightdata` therefore
|
| 171 |
+
resolves to EMPTY rather than to a provider that would return a wrong number, because a
|
| 172 |
+
plausible wrong number is worse than an honest refusal.
|
| 173 |
+
"""
|
| 174 |
+
raw = (os.environ.get("AIOS_PROVIDER_ORDER") or "").strip()
|
| 175 |
+
names = None
|
| 176 |
+
for clause in raw.split(";"):
|
| 177 |
+
if "=" in clause:
|
| 178 |
+
cap_name, _, order = clause.partition("=")
|
| 179 |
+
if cap_name.strip() == capability:
|
| 180 |
+
names = [x.strip() for x in order.split(",") if x.strip()]
|
| 181 |
+
if names is None:
|
| 182 |
+
names = list(DEFAULT_CHAINS.get(capability) or ())
|
| 183 |
+
return [PROVIDERS[n] for n in names if n in PROVIDERS and PROVIDERS[n].can(capability)]
|
| 184 |
+
|
| 185 |
+
|
| 186 |
+
def estimate(capability: str, records: int) -> dict:
|
| 187 |
+
"""What the FIRST capable provider would cost for `records` β the number an operator plans on.
|
| 188 |
+
|
| 189 |
+
Reported per capability rather than per run because that is the unit that scales: at a thousand
|
| 190 |
+
tenants the question is never "what did this run cost", it is "what does adding view counts to
|
| 191 |
+
every post cost per month".
|
| 192 |
+
"""
|
| 193 |
+
ch = chain(capability)
|
| 194 |
+
if not ch:
|
| 195 |
+
return {"capability": capability, "records": records, "provider": None, "usd": 0.0,
|
| 196 |
+
"note": "no configured, capable provider"}
|
| 197 |
+
p = ch[0]
|
| 198 |
+
c = p.cap(capability)
|
| 199 |
+
return {"capability": capability, "records": records, "provider": p.key,
|
| 200 |
+
"usd": round((c.cost_per_record or 0.0) * max(0, int(records)), 4),
|
| 201 |
+
"note": c.note}
|
| 202 |
+
|
| 203 |
+
|
| 204 |
+
@dataclass
|
| 205 |
+
class Attempt:
|
| 206 |
+
provider: str
|
| 207 |
+
ok: bool
|
| 208 |
+
note: str = ""
|
| 209 |
+
records: int = 0
|
| 210 |
+
seconds: float = 0.0
|
| 211 |
+
|
| 212 |
+
|
| 213 |
+
def run(capability, work, satisfied=None, log=None):
|
| 214 |
+
"""Walk the chain for `capability` until one provider gives a SATISFYING answer.
|
| 215 |
+
|
| 216 |
+
`work(provider) -> (result, note)`; a non-empty note means it did not answer.
|
| 217 |
+
`satisfied(result) -> bool` decides whether an answer is good enough to stop. Default: any
|
| 218 |
+
truthy result.
|
| 219 |
+
|
| 220 |
+
Returns `(result, attempts)`. `attempts` is the audit trail β every provider tried, whether it
|
| 221 |
+
satisfied, and how long it took β because "where did this number come from and what did it
|
| 222 |
+
cost" is a question somebody asks about a bill, not about a stack trace.
|
| 223 |
+
|
| 224 |
+
β THE `satisfied` HOOK IS THE WHOLE POINT. Without it this is an error-handler, and the
|
| 225 |
+
failure it must catch is not an error: a vendor answering 200 with the one field you needed
|
| 226 |
+
left blank. That is exactly how Bright Data behaves on view counts, and a chain that only
|
| 227 |
+
caught exceptions would have stopped there forever and never reached Apify.
|
| 228 |
+
"""
|
| 229 |
+
log = log or (lambda *_a: None)
|
| 230 |
+
ok = satisfied or (lambda r: bool(r))
|
| 231 |
+
attempts, last = [], None
|
| 232 |
+
for provider in chain(capability):
|
| 233 |
+
started = time.time()
|
| 234 |
+
try:
|
| 235 |
+
result, note = work(provider)
|
| 236 |
+
except Exception as e: # noqa: BLE001
|
| 237 |
+
# β TYPE NAME ONLY. A vendor client's str() can carry the URL, and the credential is
|
| 238 |
+
# one refactor away from being a query param; this line must not be what leaks it.
|
| 239 |
+
result, note = None, f"{type(e).__name__}"
|
| 240 |
+
secs = round(time.time() - started, 2)
|
| 241 |
+
if note:
|
| 242 |
+
attempts.append(Attempt(provider.key, False, note, 0, secs))
|
| 243 |
+
log(f" {provider.label}: {note}, falling through")
|
| 244 |
+
continue
|
| 245 |
+
if not ok(result):
|
| 246 |
+
attempts.append(Attempt(provider.key, False, "answered without the field asked for",
|
| 247 |
+
0, secs))
|
| 248 |
+
log(f" {provider.label}: answered, but not with what was asked for, falling through")
|
| 249 |
+
last = result if last is None else last
|
| 250 |
+
continue
|
| 251 |
+
attempts.append(Attempt(provider.key, True, "", _count(result), secs))
|
| 252 |
+
log(f" {provider.label}: ok ({_count(result)} records, {secs}s)")
|
| 253 |
+
return result, attempts
|
| 254 |
+
return last, attempts
|
| 255 |
+
|
| 256 |
+
|
| 257 |
+
def _count(result):
|
| 258 |
+
if isinstance(result, (list, tuple, set)):
|
| 259 |
+
return len(result)
|
| 260 |
+
if isinstance(result, dict):
|
| 261 |
+
return len(result)
|
| 262 |
+
return 1 if result else 0
|
| 263 |
+
|
| 264 |
+
|
| 265 |
+
def wire():
|
| 266 |
+
"""What the Settings surface renders. Booleans and labels β NEVER a credential.
|
| 267 |
+
|
| 268 |
+
Mirrors the `hikerReady` rule the rest of the product follows: a surface is told WHETHER a
|
| 269 |
+
provider is usable, never what the key is.
|
| 270 |
+
"""
|
| 271 |
+
return {
|
| 272 |
+
"providers": [
|
| 273 |
+
{"key": p.key, "label": p.label, "configured": p.configured(),
|
| 274 |
+
"capabilities": sorted(k for k, c in p.caps.items() if c.capable)}
|
| 275 |
+
for p in PROVIDERS.values()
|
| 276 |
+
],
|
| 277 |
+
"chains": {cap: [p.key for p in chain(cap)] for cap in DEFAULT_CHAINS},
|
| 278 |
+
"incapable": {
|
| 279 |
+
f"{p.key}:{cap}": c.note
|
| 280 |
+
for p in PROVIDERS.values() for cap, c in p.caps.items() if not c.capable
|
| 281 |
+
},
|
| 282 |
+
}
|
| 283 |
+
|
| 284 |
+
|
| 285 |
+
|
| 286 |
+
|
| 287 |
+
# βββββββββββββββββββ WAVE 36 Β· W36-T35 (ruling R4, contract C5) β THE LLM LADDER ββββββββββββββββ
|
| 288 |
+
#
|
| 289 |
+
# Owner item 4, verbatim (2026-08-18): *"I'm also getting errors everywhere when I want to use the
|
| 290 |
+
# assisntant: 'the assistant could not be reached just now (openrouter: HTTP 402)' and 'cerebras:
|
| 291 |
+
# HTTP 402; groq: HTTP 404; openrouter: HTTP 402; anthropic: tool calls are not wired for this
|
| 292 |
+
# shape'."*
|
| 293 |
+
#
|
| 294 |
+
# ββ R4 IS THREE CLAUSES AND THEY LAND IN THREE DIFFERENT PLACES. (1) Anthropic becomes the
|
| 295 |
+
# tool-calling path that always works β a WIRE, in `routes_query`. (2) A provider with no credit is
|
| 296 |
+
# SKIPPED rather than tried β a memo, `mark_no_credit` below. (3) No raw `HTTP 402` ever reaches a
|
| 297 |
+
# screen β a SENTENCE, `refusal_sentence` below. The declaration here is what the first two read.
|
| 298 |
+
#
|
| 299 |
+
# β WHY A SECOND REGISTRY RATHER THAN ROWS IN `PROVIDERS` ABOVE. A scraping provider is
|
| 300 |
+
# `(key_env, caps)` and is billed per RECORD; an LLM provider is `(env, url, model, wire)` and is
|
| 301 |
+
# billed per TOKEN. Folding them into one dict would mean four fields that are meaningless for half
|
| 302 |
+
# the rows and an `estimate()` that answers $0.00 for anything LLM-shaped. What they SHARE is the
|
| 303 |
+
# thing worth sharing: `Capability`, so "MEASURED INCAPABLE" means exactly the same thing on both
|
| 304 |
+
# sides, and `llm_chain()` refuses an incapable row exactly as `chain()` does.
|
| 305 |
+
#
|
| 306 |
+
# β THE DECLARATION IS THE POINT (staged item 3). `_FAILED_GEN` in `routes_query` is a REGEX that
|
| 307 |
+
# recovers a tool call out of a provider's 400 β the evidence that guessing at capability failed.
|
| 308 |
+
# A row that says `llm_tool_calling: Capability(False, β¦)` is never offered for a tool-calling
|
| 309 |
+
# turn at all, so the guess never has to be made.
|
| 310 |
+
|
| 311 |
+
#: How long a provider stays skipped after it tells us it is out of credit. β A MEMO, NOT A FACT:
|
| 312 |
+
#: the balance can be topped up at any moment, so this expires rather than latching. Fifteen
|
| 313 |
+
#: minutes is long enough that a chat session does not re-pay the timeout on every turn, and short
|
| 314 |
+
#: enough that a top-up is picked up without a restart.
|
| 315 |
+
CREDIT_COOLDOWN_S = float(os.environ.get("AIOS_CREDIT_COOLDOWN_S") or 900)
|
| 316 |
+
|
| 317 |
+
#: `{provider name: unix ts when the memo expires}`. β PROCESS-LOCAL AND DELIBERATELY SO β it is a
|
| 318 |
+
#: latency optimisation, not a billing record. A second container learns the same thing from its
|
| 319 |
+
#: own first 402, and neither one can be wrong for longer than the cooldown.
|
| 320 |
+
_NO_CREDIT: dict[str, float] = {}
|
| 321 |
+
|
| 322 |
+
|
| 323 |
+
@dataclass(frozen=True)
|
| 324 |
+
class LlmProvider:
|
| 325 |
+
"""One chat-completions endpoint, and what it is DECLARED able to do."""
|
| 326 |
+
name: str
|
| 327 |
+
label: str
|
| 328 |
+
env: str
|
| 329 |
+
url: str
|
| 330 |
+
model: str
|
| 331 |
+
#: `openai` = the OpenAI-compatible `/chat/completions` shape. `anthropic` = the Messages API,
|
| 332 |
+
#: which is a different body, a different auth header and a different result shape.
|
| 333 |
+
wire: str
|
| 334 |
+
caps: dict = field(default_factory=dict)
|
| 335 |
+
|
| 336 |
+
def configured(self) -> bool:
|
| 337 |
+
return bool((os.environ.get(self.env) or "").strip())
|
| 338 |
+
|
| 339 |
+
def can(self, capability: str) -> bool:
|
| 340 |
+
cap = self.caps.get(capability)
|
| 341 |
+
return bool(cap and cap.capable and self.configured())
|
| 342 |
+
|
| 343 |
+
|
| 344 |
+
#: β ORDER IS THE LADDER, AND ANTHROPIC IS FIRST BECAUSE OF R4. `routes_query`'s old comment put
|
| 345 |
+
#: cerebras first *"because this path needs tool calling and cerebras carries this account's
|
| 346 |
+
#: tool-capable model"* β R4 replaces that premise: Anthropic is the tool-calling path that always
|
| 347 |
+
#: works, and the others are the cheap seats it falls through to.
|
| 348 |
+
LLM_PROVIDERS: dict[str, LlmProvider] = {
|
| 349 |
+
"anthropic": LlmProvider(
|
| 350 |
+
name="anthropic", label="Anthropic", env="ANTHROPIC_API_KEY",
|
| 351 |
+
url="https://api.anthropic.com/v1/messages",
|
| 352 |
+
# β Claude Opus 5, $5 / $25 per million tokens (2026-06 list). Override per deployment with
|
| 353 |
+
# `AIOS_ANTHROPIC_MODEL` β the id is read at call time, so a cheaper tier
|
| 354 |
+
# (`claude-sonnet-5`, $3 / $15) is an environment change, not a release.
|
| 355 |
+
model=os.environ.get("AIOS_ANTHROPIC_MODEL") or "claude-opus-5",
|
| 356 |
+
wire="anthropic",
|
| 357 |
+
caps={
|
| 358 |
+
# β MEASURED, and it is the whole of R4's first clause: the Messages API answers with a
|
| 359 |
+
# typed `tool_use` content block carrying parsed `input`. There is nothing to recover
|
| 360 |
+
# out of a 400 and no regex in the path β which is exactly what `_FAILED_GEN` exists to
|
| 361 |
+
# apologise for on the other wire.
|
| 362 |
+
"llm_tool_calling": Capability(True, 0.0,
|
| 363 |
+
"typed tool_use content block; no text recovery path"),
|
| 364 |
+
"llm_chat": Capability(True, 0.0, "Messages API"),
|
| 365 |
+
"llm_json_mode": Capability(True, 0.0, "output_config.format, schema-constrained"),
|
| 366 |
+
}),
|
| 367 |
+
"cerebras": LlmProvider(
|
| 368 |
+
name="cerebras", label="Cerebras", env="CEREBRAS_API_KEY",
|
| 369 |
+
url="https://api.cerebras.ai/v1/chat/completions",
|
| 370 |
+
model="gpt-oss-120b", wire="openai",
|
| 371 |
+
caps={
|
| 372 |
+
"llm_tool_calling": Capability(True, 0.0,
|
| 373 |
+
"DECLARED, not measured: this account's tool-capable "
|
| 374 |
+
"model per the wave-32 ladder note"),
|
| 375 |
+
"llm_chat": Capability(True, 0.0, "OpenAI-compatible"),
|
| 376 |
+
"llm_json_mode": Capability(True, 0.0, "response_format json_object"),
|
| 377 |
+
}),
|
| 378 |
+
"groq": LlmProvider(
|
| 379 |
+
name="groq", label="Groq", env="GROQ_API_KEY",
|
| 380 |
+
url="https://api.groq.com/openai/v1/chat/completions",
|
| 381 |
+
model="llama-3.3-70b-versatile", wire="openai",
|
| 382 |
+
caps={
|
| 383 |
+
"llm_tool_calling": Capability(True, 0.0,
|
| 384 |
+
"DECLARED, not measured. β `routes_query._FAILED_GEN` "
|
| 385 |
+
"exists because SOME provider on this wire answers 400 "
|
| 386 |
+
"with the call in `failed_generation`; the repo never "
|
| 387 |
+
"recorded which. Flip this to False the day it is"),
|
| 388 |
+
"llm_chat": Capability(True, 0.0, "OpenAI-compatible"),
|
| 389 |
+
"llm_json_mode": Capability(True, 0.0, "response_format json_object"),
|
| 390 |
+
}),
|
| 391 |
+
"openrouter": LlmProvider(
|
| 392 |
+
name="openrouter", label="OpenRouter", env="OPENROUTER_API_KEY",
|
| 393 |
+
url="https://openrouter.ai/api/v1/chat/completions",
|
| 394 |
+
model="openai/gpt-4o-mini", wire="openai",
|
| 395 |
+
caps={
|
| 396 |
+
"llm_tool_calling": Capability(True, 0.0, "DECLARED, not measured"),
|
| 397 |
+
"llm_chat": Capability(True, 0.0, "OpenAI-compatible"),
|
| 398 |
+
"llm_json_mode": Capability(True, 0.0, "response_format json_object"),
|
| 399 |
+
}),
|
| 400 |
+
}
|
| 401 |
+
|
| 402 |
+
LLM_DEFAULT_ORDER = ("anthropic", "cerebras", "groq", "openrouter")
|
| 403 |
+
|
| 404 |
+
|
| 405 |
+
def mark_no_credit(name, seconds=None):
|
| 406 |
+
"""Remember that `name` said it is out of credit, so the next turn SKIPS it (R4).
|
| 407 |
+
|
| 408 |
+
β THE SECOND CLAUSE OF R4 IS "SKIPPED, NOT TRIED", and without a memo there is nowhere for
|
| 409 |
+
that to live: a stateless ladder re-tries the empty account on every single turn, pays its
|
| 410 |
+
round trip, and shows the reader a longer error each time. This is that memo.
|
| 411 |
+
"""
|
| 412 |
+
_NO_CREDIT[str(name)] = time.time() + float(
|
| 413 |
+
CREDIT_COOLDOWN_S if seconds is None else seconds)
|
| 414 |
+
return _NO_CREDIT[str(name)]
|
| 415 |
+
|
| 416 |
+
|
| 417 |
+
def no_credit(name):
|
| 418 |
+
"""Is this provider inside its out-of-credit cooldown? Expiry is checked, never assumed."""
|
| 419 |
+
until = _NO_CREDIT.get(str(name))
|
| 420 |
+
if not until:
|
| 421 |
+
return False
|
| 422 |
+
if time.time() >= until:
|
| 423 |
+
_NO_CREDIT.pop(str(name), None)
|
| 424 |
+
return False
|
| 425 |
+
return True
|
| 426 |
+
|
| 427 |
+
|
| 428 |
+
def clear_credit_memo(name=None):
|
| 429 |
+
"""Forget one memo, or all of them. For a gate, and for an operator after a top-up."""
|
| 430 |
+
if name is None:
|
| 431 |
+
_NO_CREDIT.clear()
|
| 432 |
+
else:
|
| 433 |
+
_NO_CREDIT.pop(str(name), None)
|
| 434 |
+
|
| 435 |
+
|
| 436 |
+
def llm_chain(capability="llm_tool_calling"):
|
| 437 |
+
"""The provider order for `capability` β declaration first, credit memo second.
|
| 438 |
+
|
| 439 |
+
Three filters, in this order, and each removes a DIFFERENT kind of row:
|
| 440 |
+
1. `can()` β declared capable AND configured. An incapable row is never offered, so a
|
| 441 |
+
turn cannot be spent discovering it (the `chain()` rule, one layer up).
|
| 442 |
+
2. `no_credit()` β R4's skip. A provider that told us its balance is empty is passed over
|
| 443 |
+
until the memo expires.
|
| 444 |
+
3. the ORDER itself, overridable with `AIOS_LLM_ORDER` (`anthropic,groq`) so a deployment
|
| 445 |
+
can be moved off a vendor without a release β the same clause `AIOS_PROVIDER_ORDER`
|
| 446 |
+
carries for the scraping side.
|
| 447 |
+
"""
|
| 448 |
+
raw = (os.environ.get("AIOS_LLM_ORDER") or "").strip()
|
| 449 |
+
names = [x.strip() for x in raw.split(",") if x.strip()] or list(LLM_DEFAULT_ORDER)
|
| 450 |
+
return [LLM_PROVIDERS[n] for n in names
|
| 451 |
+
if n in LLM_PROVIDERS and LLM_PROVIDERS[n].can(capability) and not no_credit(n)]
|
| 452 |
+
|
| 453 |
+
|
| 454 |
+
#: What an HTTP status MEANS, in words a person can act on. ββ R4's THIRD CLAUSE LIVES HERE AND
|
| 455 |
+
#: IT IS NOT COSMETIC: `HTTP 402` on a screen tells a reader nothing they can do, and the owner
|
| 456 |
+
#: quoted it back at us twice. Every sentence names the VENDOR and the ACTION.
|
| 457 |
+
_STATUS_WORDS = {
|
| 458 |
+
401: "{label} would not accept our key",
|
| 459 |
+
403: "{label} would not accept our key",
|
| 460 |
+
402: "{label} is out of credit",
|
| 461 |
+
404: "{label} does not offer the model we asked it for",
|
| 462 |
+
408: "{label} took too long",
|
| 463 |
+
413: "the question was too long for {label}",
|
| 464 |
+
429: "{label} is rate limiting us right now",
|
| 465 |
+
}
|
| 466 |
+
|
| 467 |
+
#: Substrings that mean "no money" on a wire that does not use 402. β Anthropic answers a spent
|
| 468 |
+
#: balance with a 400 or 403 carrying a message, not with a status code of its own, so this is the
|
| 469 |
+
#: one place a body has to be read. It is a LOWERCASE substring test on the vendor's own words and
|
| 470 |
+
#: it only ever decides whether to SKIP a provider, never whether to trust one.
|
| 471 |
+
_CREDIT_WORDS = ("credit balance", "insufficient credit", "insufficient_quota", "out of credit",
|
| 472 |
+
"quota exceeded", "billing", "payment required", "add credits")
|
| 473 |
+
|
| 474 |
+
|
| 475 |
+
def is_credit_failure(status, body=""):
|
| 476 |
+
"""Did this response mean "the account is empty"? Status first, then the vendor's own words."""
|
| 477 |
+
if int(status or 0) == 402:
|
| 478 |
+
return True
|
| 479 |
+
if int(status or 0) not in (400, 403, 429):
|
| 480 |
+
return False
|
| 481 |
+
return any(word in str(body or "").lower() for word in _CREDIT_WORDS)
|
| 482 |
+
|
| 483 |
+
|
| 484 |
+
def refusal_sentence(name, status, body=""):
|
| 485 |
+
"""One provider's failure, as a SENTENCE. Never a bare status code, never a vendor stack trace.
|
| 486 |
+
|
| 487 |
+
β THE BODY IS READ AND NEVER QUOTED. A provider's error body can carry an account id, a key
|
| 488 |
+
prefix or an internal trace; the only thing taken out of it is the yes/no answer to "is this a
|
| 489 |
+
credit problem", and what reaches the caller is this module's own wording.
|
| 490 |
+
"""
|
| 491 |
+
label = (LLM_PROVIDERS.get(str(name)) or LlmProvider(name, str(name), "", "", "", "")).label
|
| 492 |
+
if is_credit_failure(status, body):
|
| 493 |
+
return f"{label} is out of credit"
|
| 494 |
+
code = int(status or 0)
|
| 495 |
+
if code in _STATUS_WORDS:
|
| 496 |
+
return _STATUS_WORDS[code].format(label=label)
|
| 497 |
+
if 500 <= code <= 599:
|
| 498 |
+
return f"{label} is having trouble at their end"
|
| 499 |
+
return f"{label} did not answer"
|
| 500 |
+
|
| 501 |
+
|
| 502 |
+
# βββββββββββ THE ANTHROPIC WIRE, ONCE (W36-T35 / ASK D-18, ruling R4) βββββββββββββββββββββββββββ
|
| 503 |
+
#
|
| 504 |
+
# ββ TWO DOORS IN THIS PRODUCT CALL ANTHROPIC AND THEY MUST NOT EACH LEARN THE MESSAGES API.
|
| 505 |
+
# `routes_query._call_model` (the Assistant and Query) and `ai_review.draft_flow` (the automation
|
| 506 |
+
# drafter) both need it, and the owner quoted an error from EACH of them in one breath:
|
| 507 |
+
# *"the assistant could not be reached just now (openrouter: HTTP 402)"* and *"anthropic: tool
|
| 508 |
+
# calls are not wired for this shape"*. Two implementations of one wire is
|
| 509 |
+
# [[one-question-two-normalizers]] before a line is written, so the wire lives here, beside the
|
| 510 |
+
# ladder that declares the rung.
|
| 511 |
+
#
|
| 512 |
+
# FOUR THINGS THE OPENAI-COMPATIBLE SHAPE GETS WRONG, each a 400 on its own:
|
| 513 |
+
# 1. the system prompt is a TOP-LEVEL field, not a `{"role": "system"}` message
|
| 514 |
+
# 2. a tool is `{name, description, input_schema}` FLAT, not nested under `function`
|
| 515 |
+
# 3. `temperature` and friends are REMOVED on the current model family
|
| 516 |
+
# 4. `tool_choice` is an OBJECT (`{"type": "auto"}` / `{"type": "any"}`), not a string
|
| 517 |
+
#
|
| 518 |
+
# β AND ONE THING THAT IS NOT A SHAPE: `effort` is model-gated. `output_config.effort` errors on
|
| 519 |
+
# Haiku 4.5, so it is a PARAMETER here and the caller decides β the drafter runs on haiku and omits
|
| 520 |
+
# it, the assistant runs on the Opus tier and sends it.
|
| 521 |
+
|
| 522 |
+
#: The Messages API version header. A DATE that pins the WIRE FORMAT, never a model.
|
| 523 |
+
ANTHROPIC_VERSION = "2023-06-01"
|
| 524 |
+
|
| 525 |
+
|
| 526 |
+
def anthropic_request(*, model, key, system, messages, tools, max_tokens,
|
| 527 |
+
tool_choice="auto", effort=None):
|
| 528 |
+
"""`{url, headers, json}` for one Messages API call. Pure: reads no environment, sends nothing.
|
| 529 |
+
|
| 530 |
+
`messages` is the OpenAI-shaped list this product already builds; the `system` turns are lifted
|
| 531 |
+
out of it, because that is where this API wants them. `tools` is the OpenAI-shaped tool list,
|
| 532 |
+
re-addressed rather than re-derived, so a schema change happens in one place.
|
| 533 |
+
"""
|
| 534 |
+
system_text = "\n\n".join(str(m.get("content") or "") for m in messages
|
| 535 |
+
if m.get("role") == "system")
|
| 536 |
+
if system:
|
| 537 |
+
system_text = (system_text + "\n\n" + str(system)).strip() if system_text else str(system)
|
| 538 |
+
turns = [{"role": ("assistant" if m.get("role") == "assistant" else "user"),
|
| 539 |
+
"content": str(m.get("content") or "")}
|
| 540 |
+
for m in messages if m.get("role") != "system" and str(m.get("content") or "").strip()]
|
| 541 |
+
body = {
|
| 542 |
+
"model": str(model),
|
| 543 |
+
"max_tokens": int(max_tokens),
|
| 544 |
+
"system": system_text,
|
| 545 |
+
"messages": turns,
|
| 546 |
+
"tools": [{"name": t["function"]["name"],
|
| 547 |
+
"description": t["function"]["description"],
|
| 548 |
+
"input_schema": t["function"]["parameters"]} for t in (tools or [])],
|
| 549 |
+
"tool_choice": {"type": "any" if tool_choice in ("required", "any") else "auto"},
|
| 550 |
+
}
|
| 551 |
+
if effort:
|
| 552 |
+
body["output_config"] = {"effort": str(effort)}
|
| 553 |
+
return {"url": LLM_PROVIDERS["anthropic"].url,
|
| 554 |
+
"headers": {"x-api-key": str(key),
|
| 555 |
+
"anthropic-version": ANTHROPIC_VERSION,
|
| 556 |
+
"content-type": "application/json"},
|
| 557 |
+
"json": body}
|
| 558 |
+
|
| 559 |
+
|
| 560 |
+
def anthropic_read(body):
|
| 561 |
+
"""`(text, tool_input, refusal)` out of a Messages API answer.
|
| 562 |
+
|
| 563 |
+
β `stop_reason` IS CHECKED BEFORE `content` IS READ. A safety decline answers **HTTP 200** with
|
| 564 |
+
`stop_reason: "refusal"` and an empty or partial `content`, so code that indexes `content[0]`
|
| 565 |
+
unconditionally breaks on exactly the turn a person most needs explained.
|
| 566 |
+
β AND THE TOOL CALL ARRIVES PARSED. `tool_use.input` is already a dict β no `json.loads`, and
|
| 567 |
+
no regex recovering a call out of a 400, which is what the other wire needs.
|
| 568 |
+
"""
|
| 569 |
+
body = body if isinstance(body, dict) else {}
|
| 570 |
+
if str(body.get("stop_reason") or "") == "refusal":
|
| 571 |
+
return "", None, "the assistant declined to answer that one"
|
| 572 |
+
blocks = [b for b in (body.get("content") or []) if isinstance(b, dict)]
|
| 573 |
+
text = " ".join(str(b.get("text") or "") for b in blocks if b.get("type") == "text").strip()
|
| 574 |
+
calls = [b for b in blocks if b.get("type") == "tool_use"]
|
| 575 |
+
got = calls[0].get("input") if calls else None
|
| 576 |
+
return text, (got if isinstance(got, dict) else None), None
|
| 577 |
+
|
| 578 |
+
|
| 579 |
+
def llm_status(capability="llm_tool_calling"):
|
| 580 |
+
"""Per provider: configured, declared-capable, in cooldown, and WHY β contract C5's payload.
|
| 581 |
+
|
| 582 |
+
β ONE LIST, ONE DOOR. The Assistant's model picker and the Agent chat's toggle (W36-T34) read
|
| 583 |
+
THIS, so a model offered in one place cannot be missing from the other, and neither can offer a
|
| 584 |
+
provider the ladder would refuse to call [[permitted-is-not-answerable]].
|
| 585 |
+
"""
|
| 586 |
+
rows = []
|
| 587 |
+
for name in LLM_DEFAULT_ORDER:
|
| 588 |
+
p = LLM_PROVIDERS[name]
|
| 589 |
+
cap = p.caps.get(capability)
|
| 590 |
+
rows.append({
|
| 591 |
+
"provider": p.name, "label": p.label, "model": p.model, "wire": p.wire,
|
| 592 |
+
"configured": p.configured(),
|
| 593 |
+
"toolCalling": bool((p.caps.get("llm_tool_calling") or Capability(False)).capable),
|
| 594 |
+
"jsonMode": bool((p.caps.get("llm_json_mode") or Capability(False)).capable),
|
| 595 |
+
"capable": bool(cap and cap.capable),
|
| 596 |
+
"outOfCredit": no_credit(name),
|
| 597 |
+
"note": (cap.note if cap else ""),
|
| 598 |
+
})
|
| 599 |
+
return rows
|
| 600 |
+
|
| 601 |
+
|
| 602 |
+
# =============================================================================================
|
| 603 |
+
# THE CANONICAL SCHEMA β owner ruling 2026-08-08:
|
| 604 |
+
# *"standardize the schema between Bright Data and APIfy so we keep using the same pre-set
|
| 605 |
+
# database even if the underlying engine changes"*
|
| 606 |
+
#
|
| 607 |
+
# β THE PRESET TABLES ARE THE CONTRACT; A VENDOR IS AN IMPLEMENTATION DETAIL. `ut_ig_posts` and
|
| 608 |
+
# `ut_ig_post_snapshots` must not gain, lose or rename a column because a chain was reordered β a
|
| 609 |
+
# tenant's saved views, filters, rollups and forms all bind to these keys, and a schema that moves
|
| 610 |
+
# with the vendor turns a routing change into a data migration.
|
| 611 |
+
#
|
| 612 |
+
# So every provider normalises INTO the keys below and nothing reads a vendor row downstream.
|
| 613 |
+
# `verify_automation` asserts both normalisers emit exactly `CANONICAL_POST_KEYS` on a fixture, so
|
| 614 |
+
# a third provider cannot ship with a near-miss key like `viewCount` and silently write a column
|
| 615 |
+
# nobody declared.
|
| 616 |
+
#
|
| 617 |
+
# β ONE NAME PER MEASUREMENT, AND `views` IS THE MEASUREMENT INSTAGRAM DISPLAYS. Meta folded
|
| 618 |
+
# Impressions/Plays/Video Views into a single **Views** metric on 2025-04-10, so carrying both a
|
| 619 |
+
# `views` and a `plays` column would be modelling a distinction the platform deleted β and it is
|
| 620 |
+
# exactly the distinction that let an account-grain number wear the "Views" label for a month.
|
| 621 |
+
# β THE VENDOR KEY THAT LOOKS RIGHT IS THE WRONG ONE, ON BOTH VENDORS. Apify ships BOTH
|
| 622 |
+
# `videoViewCount` (10,678) and `videoPlayCount` (137,684) for one reel whose true displayed count
|
| 623 |
+
# is ~137K β and Bright Data's useless `views` is 10,638 for that same reel. The two vendors' junk
|
| 624 |
+
# fields AGREE with each other, which is precisely what makes picking by name so dangerous.
|
| 625 |
+
# canonical `views` <- apify `videoPlayCount` β
matches the grid
|
| 626 |
+
# canonical `views` <- apify `videoViewCount` β off by 13x
|
| 627 |
+
# canonical `views` <- brightdata `views` β off by 13x AND account-grain
|
| 628 |
+
# =============================================================================================
|
| 629 |
+
|
| 630 |
+
#: Every key a normalised POST row may carry. Absent > blank: a key is omitted when the provider
|
| 631 |
+
#: did not answer, because `upsert_rows` merges and an empty string would ERASE what an earlier
|
| 632 |
+
#: paid run learned.
|
| 633 |
+
CANONICAL_POST_KEYS = (
|
| 634 |
+
"shortcode", "url", "influencer_key", "posted_at", "type", "caption",
|
| 635 |
+
"likes", "comments", "views", "paid_partnership", "partner", "hashtags",
|
| 636 |
+
"alt_text", "tagged_location", "source_payload",
|
| 637 |
+
# β 2026-08-09 β the three the SECOND provider answers and the first does not. Each has a
|
| 638 |
+
# matching `field_def` in the engine's POST_FIELDS; a key here without a column there is a
|
| 639 |
+
# value that normalises cleanly and is then dropped by the write door, silently.
|
| 640 |
+
"plays", "video_duration", "comments_disabled",
|
| 641 |
+
)
|
| 642 |
+
|
| 643 |
+
|
| 644 |
+
def _int_or_none(v):
|
| 645 |
+
"""`-1` IS NOT A COUNT. Apify returns `likesCount: -1` when the creator HIDES their like
|
| 646 |
+
count β a real state that is not a measurement. Writing -1 would render as a negative like
|
| 647 |
+
count; writing 0 would claim nobody liked it. Both are lies, so the key is omitted."""
|
| 648 |
+
try:
|
| 649 |
+
n = int(v)
|
| 650 |
+
except (TypeError, ValueError):
|
| 651 |
+
return None
|
| 652 |
+
return None if n < 0 else n
|
| 653 |
+
|
| 654 |
+
|
| 655 |
+
def normalize_post_apify(row):
|
| 656 |
+
"""One Apify `instagram-scraper` item -> the canonical post row."""
|
| 657 |
+
if not isinstance(row, dict):
|
| 658 |
+
return None
|
| 659 |
+
code = str(row.get("shortCode") or "").strip()
|
| 660 |
+
if not code:
|
| 661 |
+
return None
|
| 662 |
+
out = {
|
| 663 |
+
"shortcode": code,
|
| 664 |
+
"url": str(row.get("url") or f"https://www.instagram.com/reel/{code}/"),
|
| 665 |
+
"influencer_key": str(row.get("ownerUsername") or ""),
|
| 666 |
+
"posted_at": str(row.get("timestamp") or "").replace("T", " ")[:16],
|
| 667 |
+
"type": "video" if str(row.get("type") or "").lower() == "video" else
|
| 668 |
+
("carousel" if row.get("childPosts") else "image"),
|
| 669 |
+
"caption": str(row.get("caption") or ""),
|
| 670 |
+
# β THE FIELD THIS WHOLE PROVIDER EXISTS FOR. `videoPlayCount`, never `videoViewCount`:
|
| 671 |
+
# the wrong one agreed with Bright Data's junk (10,678 vs 10,638) on a reel whose true
|
| 672 |
+
# count was ~137K, and `videoPlayCount` matched a browser read to the digit.
|
| 673 |
+
"views": _int_or_none(row.get("videoPlayCount")),
|
| 674 |
+
# ββ 2026-08-09 (owner: *"Video plays (# Plays) field isn't in APIfy? i believe it is"*).
|
| 675 |
+
# They were right, and the `plays` COLUMN had been empty on all 816 rows because nothing
|
| 676 |
+
# ever wrote it β the field existed with no writer. MEASURED on two of their own reels:
|
| 677 |
+
# `videoPlayCount` 216,904 / 95,331 and `videoViewCount` 115,929 / 30,871. Two different
|
| 678 |
+
# real numbers, and we were storing only one of them.
|
| 679 |
+
# β `plays` TAKES THE PLAY COUNT, which is also what `views` carries today β so the two
|
| 680 |
+
# columns will agree until somebody decides otherwise, and that decision is the OWNER'S:
|
| 681 |
+
# re-sourcing `views` to `videoViewCount` would change what the ~480 rows already captured
|
| 682 |
+
# mean, and a column whose meaning changes halfway down is the one thing worse than a
|
| 683 |
+
# column with no data. Flagged rather than done.
|
| 684 |
+
"plays": _int_or_none(row.get("videoPlayCount")),
|
| 685 |
+
"likes": _int_or_none(row.get("likesCount")),
|
| 686 |
+
"comments": _int_or_none(row.get("commentsCount")),
|
| 687 |
+
# β FIELDS BRIGHT DATA DOES NOT RETURN AT ALL, kept because they are already paid for in
|
| 688 |
+
# this same response (owner: *"whatever APIfy has more than BD pls use it"*).
|
| 689 |
+
"video_duration": _int_or_none(row.get("videoDuration")),
|
| 690 |
+
"comments_disabled": "1" if row.get("isCommentsDisabled") else "",
|
| 691 |
+
"hashtags": ", ".join(str(h) for h in (row.get("hashtags") or []) if h),
|
| 692 |
+
"alt_text": str(row.get("alt") or ""),
|
| 693 |
+
"paid_partnership": "1" if row.get("paidPartnership") else "",
|
| 694 |
+
"partner": ", ".join(str((s or {}).get("username") or "")
|
| 695 |
+
for s in (row.get("sponsors") or []) if isinstance(s, dict)),
|
| 696 |
+
"tagged_location": str((row.get("locationName") or "")),
|
| 697 |
+
"source_payload": json.dumps(row, default=str)[:32_000_000],
|
| 698 |
+
}
|
| 699 |
+
return {k: v for k, v in out.items() if v not in (None, "")}
|
| 700 |
+
|
| 701 |
+
|
| 702 |
+
def normalize_profile_apify(row):
|
| 703 |
+
"""One Apify `instagram-scraper` PROFILE item (`resultsType: "details"`) -> our snapshot shape.
|
| 704 |
+
|
| 705 |
+
β WHY THIS EXISTS (owner report 2026-08-09: *"I need Bright Data and Apify to work correctly
|
| 706 |
+
in tandem"*). `PROVIDERS["apify"]` has declared `ig_profile` capable since 2026-08-08 and
|
| 707 |
+
`DEFAULT_CHAINS["ig_profile"]` has read `("brightdata", "apify")` β but **nothing ever ran
|
| 708 |
+
that chain.** `connectors_ig.pull_profile` went Bright Data -> anonymous HTML rungs and Apify
|
| 709 |
+
was never asked, so a profile Bright Data cannot scrape came back `blocked` while a
|
| 710 |
+
configured, declared-capable provider sat unused. A registry entry with no runner is a
|
| 711 |
+
promise the product does not keep ([[flag-shipped-without-its-writer]]).
|
| 712 |
+
|
| 713 |
+
β THE KEYS ARE APIFY'S camelCase, and mapping them HERE is the boundary rule this module
|
| 714 |
+
already enforces for posts: nothing downstream may ever see a vendor-shaped key, so the
|
| 715 |
+
snapshot writer cannot tell which vendor answered β which is what makes the fallback
|
| 716 |
+
invisible to every consumer instead of a second schema.
|
| 717 |
+
|
| 718 |
+
β OMIT, NEVER ZERO. A field Apify did not send is dropped, exactly as `normalize_post_apify`
|
| 719 |
+
drops a missing count: writing 0 followers would claim we measured an empty account, and the
|
| 720 |
+
caller's `satisfied` hook reads absence as "did not answer" and falls through. A zero would
|
| 721 |
+
stop the chain on a lie.
|
| 722 |
+
"""
|
| 723 |
+
if not isinstance(row, dict):
|
| 724 |
+
return None
|
| 725 |
+
# β AN ERROR ENVELOPE IS NOT A PROFILE (measured 2026-08-09). Apify answers a dead handle
|
| 726 |
+
# with a 200 and a well-formed item β `{"username": β¦, "error": "not_found",
|
| 727 |
+
# "errorDescription": "Post does not exist"}` β and this function used to build a "profile"
|
| 728 |
+
# out of it: a username, a url and a source_payload, with every measured field absent. It
|
| 729 |
+
# then read to the caller as a vendor that answered, so the reason was replaced by a shrug.
|
| 730 |
+
# Refused HERE as well as in `connectors_ig.apify_profile` on purpose: the boundary rule this
|
| 731 |
+
# module exists for is that nothing downstream ever sees a vendor-shaped key, and a vendor's
|
| 732 |
+
# ERROR shape is the one that must never become a row.
|
| 733 |
+
# β TRUTHINESS, NOT KEY PRESENCE β a successful item carries `error: null` (measured on
|
| 734 |
+
# `sriyynntt`), so `"error" in row` would refuse every good profile.
|
| 735 |
+
if row.get("error"):
|
| 736 |
+
return None
|
| 737 |
+
handle = str(row.get("username") or "").strip()
|
| 738 |
+
if not handle:
|
| 739 |
+
return None
|
| 740 |
+
out = {
|
| 741 |
+
"username": handle,
|
| 742 |
+
"full_name": str(row.get("fullName") or ""),
|
| 743 |
+
"bio": str(row.get("biography") or ""),
|
| 744 |
+
"followers": _int_or_none(row.get("followersCount")),
|
| 745 |
+
"following": _int_or_none(row.get("followsCount")),
|
| 746 |
+
"posts_count": _int_or_none(row.get("postsCount")),
|
| 747 |
+
"verified": "1" if row.get("verified") else "",
|
| 748 |
+
"external_url": str(row.get("externalUrl") or ""),
|
| 749 |
+
"ig_id": str(row.get("id") or ""),
|
| 750 |
+
"profile_url": str(row.get("url") or f"https://www.instagram.com/{handle}/"),
|
| 751 |
+
"business_category": str(row.get("businessCategoryName") or ""),
|
| 752 |
+
"is_business": "1" if row.get("isBusinessAccount") else "",
|
| 753 |
+
"is_private": "1" if row.get("private") else "",
|
| 754 |
+
"highlights_count": _int_or_none(row.get("highlightReelCount")),
|
| 755 |
+
"source_payload": json.dumps(row, default=str)[:32_000_000],
|
| 756 |
+
}
|
| 757 |
+
# β `followers`/`following` are ints and 0 is a LEGITIMATE value for them, so the filter
|
| 758 |
+
# below must not treat 0 as absent the way the post normaliser can β an account really can
|
| 759 |
+
# have zero followers. Only None and "" are dropped.
|
| 760 |
+
return {k: v for k, v in out.items() if v is not None and v != ""}
|
| 761 |
+
|
| 762 |
+
|
| 763 |
+
def canonical_gaps(row, want=("views",)):
|
| 764 |
+
"""Which requested canonical fields this row does NOT carry β the `satisfied` input.
|
| 765 |
+
|
| 766 |
+
Named rather than inlined because "did the vendor actually answer the question" is the whole
|
| 767 |
+
fallback trigger, and a chain that asks it differently in two places will drift.
|
| 768 |
+
"""
|
| 769 |
+
row = row if isinstance(row, dict) else {}
|
| 770 |
+
return [k for k in want if row.get(k) in (None, "")]
|
api/routes_admin.py
CHANGED
|
@@ -81,7 +81,7 @@ def _registry():
|
|
| 81 |
return reg
|
| 82 |
|
| 83 |
|
| 84 |
-
def _view(uname, rec, governed=None, st=None):
|
| 85 |
"""What an administrator may see about an account. NEVER `salt` or `hash` β this function is
|
| 86 |
the only projection these routes use, so there is one place that can leak and it does not.
|
| 87 |
|
|
@@ -105,11 +105,12 @@ def _view(uname, rec, governed=None, st=None):
|
|
| 105 |
# S3 ask 4 β one sentence for the roster's Access column, so the list does not need
|
| 106 |
# a /perms round trip per row to fill one cell. Computed from the same record the
|
| 107 |
# editor will open, so the two cannot disagree.
|
| 108 |
-
"accessSummary": _access_summary(rec, uname=uname, governed=governed, st=st
|
|
|
|
| 109 |
"perms_v": int(rec.get("perms_v") or 0)}
|
| 110 |
|
| 111 |
|
| 112 |
-
def _access_summary(rec, uname="", governed=None, st=None):
|
| 113 |
"""One sentence describing what this account may reach. Deliberately says LESS than the
|
| 114 |
editor: it is a signpost, not a rule listing, and a summary that tried to spell out filters
|
| 115 |
would be wrong the moment a filter got interesting.
|
|
@@ -143,7 +144,17 @@ def _access_summary(rec, uname="", governed=None, st=None):
|
|
| 143 |
entries = (rec.get("perms") or {})
|
| 144 |
if governed:
|
| 145 |
principal = users._public(uname, rec) if uname else rec
|
| 146 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 147 |
else:
|
| 148 |
opened = [k for k, e in entries.items() if isinstance(e, dict) and e.get("access", True)]
|
| 149 |
restricted = sum(1 for k in opened
|
|
@@ -250,13 +261,16 @@ def list_users(session: Session = Depends(admin_gate)):
|
|
| 250 |
# `_access_summary` can agree with the editor without paying a document copy per question.
|
| 251 |
import core.user_tables as _ut
|
| 252 |
try:
|
| 253 |
-
|
|
|
|
|
|
|
| 254 |
lent = _ut.lend_defs(session.runtime)
|
| 255 |
except Exception: # noqa: BLE001
|
| 256 |
# A store that cannot list the tenant's databases must not take the roster down. The
|
| 257 |
# summary degrades to the stored-entry reading, which is what it always was.
|
| 258 |
-
governed, lent = None, None
|
| 259 |
-
return {"users": [_view(u, reg[u], governed=governed, st=lent
|
|
|
|
| 260 |
if _tenant_of(reg[u]) == mine]}
|
| 261 |
|
| 262 |
|
|
@@ -434,7 +448,7 @@ def _governable_catalogue():
|
|
| 434 |
_PERM_MODULES = tuple(m["key"] for m in _governable_catalogue())
|
| 435 |
|
| 436 |
|
| 437 |
-
def _perm_modules(session):
|
| 438 |
"""The databases THIS tenant's permission editor governs: `[{key, label, enforced}]`.
|
| 439 |
|
| 440 |
Three inputs, none of them a literal: the topic catalogue above, the tenant's provisioned
|
|
@@ -457,8 +471,23 @@ def _perm_modules(session):
|
|
| 457 |
# A store that cannot list the tenant's databases must not take the whole editor down β
|
| 458 |
# the topic half is still administrable, and an empty `ut_*` half is visibly empty.
|
| 459 |
ut = ()
|
| 460 |
-
|
| 461 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 462 |
|
| 463 |
|
| 464 |
#: β `_enforced_keys` IS DELETED (W36-T22 / CONTRACT C2). It asked `perms.enforced_module_keys`
|
|
@@ -512,6 +541,13 @@ def _module_fields(key, session=None):
|
|
| 512 |
"""
|
| 513 |
import aios_grid
|
| 514 |
key = str(key or "")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 515 |
if key.startswith("ut_"):
|
| 516 |
if session is None:
|
| 517 |
raise err(400, "ungoverned_module",
|
|
@@ -959,7 +995,7 @@ def get_perms(username: str, session: Session = Depends(admin_gate)):
|
|
| 959 |
# nurilab or gtmlab admin opened Manage user and saw Royal Imports' Customer and Product
|
| 960 |
# databases. `_perm_modules` applies the tenant's own provisioning (the rule `routes_nav`
|
| 961 |
# has applied since wave 18) and merges the tenant's own `ut_*` databases.
|
| 962 |
-
modules = _perm_modules(session)
|
| 963 |
# ββ W36-T22 / C2 β EVERY listed database, not an `enforced` subset of them. See
|
| 964 |
# `perms.tenant_governable_modules` for why the flag is deleted rather than defaulted.
|
| 965 |
governed = [m["key"] for m in modules]
|
|
@@ -983,10 +1019,20 @@ def get_perms(username: str, session: Session = Depends(admin_gate)):
|
|
| 983 |
# `users.json` and does not carry one INSIDE it, so `may_read` would hand `may_open` a
|
| 984 |
# `None` viewer and get a fail-closed False β the very outage this line exists to prevent,
|
| 985 |
# arriving through the fix for it. The same trap cost a gate double an hour earlier today.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 986 |
perms_out[k] = e if isinstance(e, dict) else {
|
| 987 |
-
"access": bool(
|
| 988 |
-
st=session.runtime)),
|
| 989 |
-
"filter": None, "hiddenFields": []}
|
| 990 |
return {"username": uname,
|
| 991 |
"perms": perms_out,
|
| 992 |
# S3 ask 2 β the marker rides the GET. An absent module entry means DENY on a
|
|
@@ -1042,7 +1088,7 @@ def put_perms(username: str, body: dict = Body(default=None),
|
|
| 1042 |
raise err(400, "self_perms",
|
| 1043 |
"you cannot set permissions on your own account β ask another admin")
|
| 1044 |
|
| 1045 |
-
_mods = _perm_modules(session)
|
| 1046 |
# ββ W36-T22 / C2 β ONE key set. It was two ("listed" and "enforced") with a refusal in the
|
| 1047 |
# gap; W36-T21 closed the gap, so every database the editor shows is one whose wall the table
|
| 1048 |
# routes apply. `session` rides so a `ut_*` schema can be read for THIS tenant.
|
|
|
|
| 81 |
return reg
|
| 82 |
|
| 83 |
|
| 84 |
+
def _view(uname, rec, governed=None, st=None, surface_keys=None):
|
| 85 |
"""What an administrator may see about an account. NEVER `salt` or `hash` β this function is
|
| 86 |
the only projection these routes use, so there is one place that can leak and it does not.
|
| 87 |
|
|
|
|
| 105 |
# S3 ask 4 β one sentence for the roster's Access column, so the list does not need
|
| 106 |
# a /perms round trip per row to fill one cell. Computed from the same record the
|
| 107 |
# editor will open, so the two cannot disagree.
|
| 108 |
+
"accessSummary": _access_summary(rec, uname=uname, governed=governed, st=st,
|
| 109 |
+
surface_keys=surface_keys),
|
| 110 |
"perms_v": int(rec.get("perms_v") or 0)}
|
| 111 |
|
| 112 |
|
| 113 |
+
def _access_summary(rec, uname="", governed=None, st=None, surface_keys=None):
|
| 114 |
"""One sentence describing what this account may reach. Deliberately says LESS than the
|
| 115 |
editor: it is a signpost, not a rule listing, and a summary that tried to spell out filters
|
| 116 |
would be wrong the moment a filter got interesting.
|
|
|
|
| 144 |
entries = (rec.get("perms") or {})
|
| 145 |
if governed:
|
| 146 |
principal = users._public(uname, rec) if uname else rec
|
| 147 |
+
# β PER KIND, MIRRORING `get_perms`' OWN DEFAULTS β the first draft asked
|
| 148 |
+
# `nav_may_open` for EVERY key and that was a second idea of the count: for a migrated
|
| 149 |
+
# record with no topic entry the editor's default is `may_read` (DENY), so a blanket
|
| 150 |
+
# admission read would make the roster say "3 modules" about an account whose editor
|
| 151 |
+
# shows one box ticked. The roster's whole contract (leg 6 of the T22 section) is that
|
| 152 |
+
# it counts what the editor would show; a SURFACE key defaults from admission there,
|
| 153 |
+
# everything else from `may_read`, so this does exactly the same, key by key.
|
| 154 |
+
opened = [k for k in governed
|
| 155 |
+
if (perm_scope.nav_may_open(principal, k, st=st)
|
| 156 |
+
if k in (surface_keys or ()) else
|
| 157 |
+
perm_scope.may_read(principal, k, st=st))]
|
| 158 |
else:
|
| 159 |
opened = [k for k, e in entries.items() if isinstance(e, dict) and e.get("access", True)]
|
| 160 |
restricted = sum(1 for k in opened
|
|
|
|
| 261 |
# `_access_summary` can agree with the editor without paying a document copy per question.
|
| 262 |
import core.user_tables as _ut
|
| 263 |
try:
|
| 264 |
+
_rows = _perm_modules(session, surfaces=True)
|
| 265 |
+
governed = [m["key"] for m in _rows]
|
| 266 |
+
surface_keys = {m["key"] for m in _rows if m.get("surface")}
|
| 267 |
lent = _ut.lend_defs(session.runtime)
|
| 268 |
except Exception: # noqa: BLE001
|
| 269 |
# A store that cannot list the tenant's databases must not take the roster down. The
|
| 270 |
# summary degrades to the stored-entry reading, which is what it always was.
|
| 271 |
+
governed, lent, surface_keys = None, None, None
|
| 272 |
+
return {"users": [_view(u, reg[u], governed=governed, st=lent, surface_keys=surface_keys)
|
| 273 |
+
for u in sorted(reg)
|
| 274 |
if _tenant_of(reg[u]) == mine]}
|
| 275 |
|
| 276 |
|
|
|
|
| 448 |
_PERM_MODULES = tuple(m["key"] for m in _governable_catalogue())
|
| 449 |
|
| 450 |
|
| 451 |
+
def _perm_modules(session, surfaces=False):
|
| 452 |
"""The databases THIS tenant's permission editor governs: `[{key, label, enforced}]`.
|
| 453 |
|
| 454 |
Three inputs, none of them a literal: the topic catalogue above, the tenant's provisioned
|
|
|
|
| 471 |
# A store that cannot list the tenant's databases must not take the whole editor down β
|
| 472 |
# the topic half is still administrable, and an empty `ut_*` half is visibly empty.
|
| 473 |
ut = ()
|
| 474 |
+
cat = _governable_catalogue()
|
| 475 |
+
skeys = set()
|
| 476 |
+
if surfaces:
|
| 477 |
+
# ββ OWNER RULING 2026-08-18 β the Assistant and Agents surfaces are
|
| 478 |
+
# governable ACCESS toggles in the ACCOUNT editor. `surfaces` is caller-explicit and
|
| 479 |
+
# defaults CLOSED because the other consumer of this list is the CHANNEL-AGENT editor
|
| 480 |
+
# (`routes_slack.get_channel_agent`), whose principal is enforced per DATABASE
|
| 481 |
+
# (`agent_rows`/`agent_may_read`) and never opens an app surface: an Assistant toggle
|
| 482 |
+
# there would be a stored rule nothing applies, the exact class C2 deleted.
|
| 483 |
+
surf = _perms.governable_surface_modules()
|
| 484 |
+
skeys = {s["key"] for s in surf}
|
| 485 |
+
cat = cat + surf
|
| 486 |
+
rows = _perms.tenant_governable_modules(session.runtime, cat, ut_entries=ut)
|
| 487 |
+
# The marker survives OUTSIDE `tenant_governable_modules`, which builds bare {key,label}
|
| 488 |
+
# rows: the client needs it only to choose the schemaless help copy, and the server needs
|
| 489 |
+
# it only to pick the admission default in `get_perms`.
|
| 490 |
+
return [dict(r, surface=True) if r["key"] in skeys else r for r in rows]
|
| 491 |
|
| 492 |
|
| 493 |
#: β `_enforced_keys` IS DELETED (W36-T22 / CONTRACT C2). It asked `perms.enforced_module_keys`
|
|
|
|
| 541 |
"""
|
| 542 |
import aios_grid
|
| 543 |
key = str(key or "")
|
| 544 |
+
if key in {s["key"] for s in perms.governable_surface_modules()}:
|
| 545 |
+
# β A SURFACE (Assistant, Agents) has NO field schema, and `[]` is the honest
|
| 546 |
+
# answer rather than the wave-26 empty-option trap: the editor's schemaless rendering
|
| 547 |
+
# is a deliberate state (access toggle only, R9), and `_clean_perms` refuses any
|
| 548 |
+
# filter or hiddenFields validated against this empty vocabulary. An unknown key
|
| 549 |
+
# still RAISES below β this arm admits exactly the derived surface list.
|
| 550 |
+
return []
|
| 551 |
if key.startswith("ut_"):
|
| 552 |
if session is None:
|
| 553 |
raise err(400, "ungoverned_module",
|
|
|
|
| 995 |
# nurilab or gtmlab admin opened Manage user and saw Royal Imports' Customer and Product
|
| 996 |
# databases. `_perm_modules` applies the tenant's own provisioning (the rule `routes_nav`
|
| 997 |
# has applied since wave 18) and merges the tenant's own `ut_*` databases.
|
| 998 |
+
modules = _perm_modules(session, surfaces=True)
|
| 999 |
# ββ W36-T22 / C2 β EVERY listed database, not an `enforced` subset of them. See
|
| 1000 |
# `perms.tenant_governable_modules` for why the flag is deleted rather than defaulted.
|
| 1001 |
governed = [m["key"] for m in modules]
|
|
|
|
| 1019 |
# `users.json` and does not carry one INSIDE it, so `may_read` would hand `may_open` a
|
| 1020 |
# `None` viewer and get a fail-closed False β the very outage this line exists to prevent,
|
| 1021 |
# arriving through the fix for it. The same trap cost a gate double an hour earlier today.
|
| 1022 |
+
# ββ A SURFACE KEY DEFAULTS FROM `nav_may_open`, NOT `may_read`, AND THE
|
| 1023 |
+
# DIFFERENCE IS A MASS REVOCATION. `may_read` on a non-`ut_` key falls through to
|
| 1024 |
+
# `may_access`, which reads migrated-and-undeclared as DENY β correct for a topic
|
| 1025 |
+
# (every save writes topic entries) and false for a surface, because NO surface entry
|
| 1026 |
+
# was storable before 2026-08-18, so every migrated account carries none. The editor
|
| 1027 |
+
# would paint Assistant unchecked for an account that opens it every day, and the next
|
| 1028 |
+
# save of ANY unrelated change would write the explicit deny for real. `nav_may_open`
|
| 1029 |
+
# is the evaluator the nav and the route gate actually ask, legacy fallback included,
|
| 1030 |
+
# so the box shows what the account can reach and an untouched save stays a no-op.
|
| 1031 |
+
_default = (perm_scope.nav_may_open(users._public(uname, rec), k)
|
| 1032 |
+
if any(m.get("surface") and m["key"] == k for m in modules) else
|
| 1033 |
+
perm_scope.may_read(users._public(uname, rec), k, st=session.runtime))
|
| 1034 |
perms_out[k] = e if isinstance(e, dict) else {
|
| 1035 |
+
"access": bool(_default), "filter": None, "hiddenFields": []}
|
|
|
|
|
|
|
| 1036 |
return {"username": uname,
|
| 1037 |
"perms": perms_out,
|
| 1038 |
# S3 ask 2 β the marker rides the GET. An absent module entry means DENY on a
|
|
|
|
| 1088 |
raise err(400, "self_perms",
|
| 1089 |
"you cannot set permissions on your own account β ask another admin")
|
| 1090 |
|
| 1091 |
+
_mods = _perm_modules(session, surfaces=True)
|
| 1092 |
# ββ W36-T22 / C2 β ONE key set. It was two ("listed" and "enforced") with a refusal in the
|
| 1093 |
# gap; W36-T21 closed the gap, so every database the editor shows is one whose wall the table
|
| 1094 |
# routes apply. `session` rides so a `ut_*` schema can be read for THIS tenant.
|
api/routes_agent_harness.py
CHANGED
|
@@ -1,455 +1,455 @@
|
|
| 1 |
-
"""routes_agent_harness.py β CONTRACT C4: the agent's HARNESS, kept as versioned files.
|
| 2 |
-
|
| 3 |
-
Owner item 3, verbatim (2026-08-18): *"This new module under agent is supposed to host any file
|
| 4 |
-
pertaining to the agent skills, router etc. So we build a user can build a custom harness for each
|
| 5 |
-
agent through the chat interface."*
|
| 6 |
-
|
| 7 |
-
GET /api/v1/agents/{id}/harness the file LIST (no bodies)
|
| 8 |
-
GET /api/v1/agents/{id}/harness?path=β¦ one file: current body + its versions
|
| 9 |
-
GET /api/v1/agents/{id}/harness?path=β¦&version=N one older body, verbatim
|
| 10 |
-
PUT /api/v1/agents/{id}/harness write a NEW version of one file
|
| 11 |
-
DELETE /api/v1/agents/{id}/harness?path=β¦ drop a file and its history
|
| 12 |
-
|
| 13 |
-
ββ **R8 IS THE WHOLE SHAPE: BOTH PRINCIPALS WRITE, AND NOTHING IS EVER OVERWRITTEN.** A write
|
| 14 |
-
appends a version; a roll-back is a write of an older body (`restoredFrom`), never a delete. So the
|
| 15 |
-
history is a record of what happened rather than of what somebody last wanted it to look like.
|
| 16 |
-
|
| 17 |
-
β **R8 IS DELIBERATELY NOT R9.** An agent-authored ACTION's configuration is agent-only (item 8,
|
| 18 |
-
`routes_automation`); a harness file is not. Do not copy this file's posture over there or that
|
| 19 |
-
one's over here β the two rulings differ on purpose and they sit one screen apart in the product.
|
| 20 |
-
|
| 21 |
-
β **`author` IS THE SESSION, `authorKind` IS THE PROVENANCE, AND THEY ARE DIFFERENT FACTS.** Every
|
| 22 |
-
write through this router is made BY a signed-in administrator, so `author` is stamped from the
|
| 23 |
-
session and can never be supplied by the caller. `authorKind` says whether the BODY was drafted by
|
| 24 |
-
a person or by the agent in the chat β a claim the client is entitled to make, because both are
|
| 25 |
-
permitted (R8) and so nothing is bought by forging it. `record_version()` below is the server-side
|
| 26 |
-
door the automation engine uses when the agent writes with no session at all; that one stamps the
|
| 27 |
-
agent's own id as the author, which is the only case where `author` is not a username.
|
| 28 |
-
|
| 29 |
-
β **THE VERSION LIST IS CAPPED AND THE CAP IS REPORTED, NEVER SILENT.** `MAX_VERSIONS` versions of
|
| 30 |
-
one file are kept; past that the OLDEST are dropped and the count of what was dropped rides in
|
| 31 |
-
`trimmed` on every payload that mentions the file, so a reader can see that the history is partial
|
| 32 |
-
rather than infer that the file was only ever saved twice. This is the tenant document, which is
|
| 33 |
-
already 28.6 MB on tenant #0 and is deep-copied on every read: an unbounded per-agent history is
|
| 34 |
-
a store-sized leak with a UI in front of it.
|
| 35 |
-
"""
|
| 36 |
-
from datetime import datetime, timezone
|
| 37 |
-
|
| 38 |
-
from fastapi import APIRouter, Body, Depends
|
| 39 |
-
|
| 40 |
-
from deps import Session, err, require_session
|
| 41 |
-
|
| 42 |
-
router = APIRouter(prefix="/api/v1")
|
| 43 |
-
|
| 44 |
-
#: The tenant's harness files: `{agent_id: {path: file_record}}`. A per-tenant bucket, so it rides
|
| 45 |
-
#: `runtime.store_key`'s prefix and never lands in tenant #0's namespace β the same rule
|
| 46 |
-
#: `routes_slack.AGENTS_KEY` follows for the agent records these hang off.
|
| 47 |
-
HARNESS_KEY = "agent_harness"
|
| 48 |
-
|
| 49 |
-
#: One body. Generous for a skill or a router file and far under the point where a single write
|
| 50 |
-
#: would move the tenant document measurably. A larger body is a REFUSAL at the door, not a
|
| 51 |
-
#: truncation: a silently truncated skill file is a harness that does not do what its text says.
|
| 52 |
-
MAX_BODY_BYTES = 128 * 1024
|
| 53 |
-
|
| 54 |
-
#: Files per agent. A refusal, not a trim β creating the 65th file is a different act from saving
|
| 55 |
-
#: the 101st version of one, and only the second can be a routine consequence of ordinary editing.
|
| 56 |
-
MAX_FILES = 64
|
| 57 |
-
|
| 58 |
-
#: Versions kept per file. Past this the oldest go and `trimmed` counts them (see the header).
|
| 59 |
-
MAX_VERSIONS = 100
|
| 60 |
-
|
| 61 |
-
MAX_PATH = 200
|
| 62 |
-
|
| 63 |
-
#: What a path may contain. β THIS IS NOT A FILESYSTEM PATH AND NOTHING HERE EVER TOUCHES A DISK β
|
| 64 |
-
#: the "files" are keys in a store bucket. The character rule exists so the key is displayable, is
|
| 65 |
-
#: safe to put in a URL, and cannot carry a traversal sequence that would look meaningful to a
|
| 66 |
-
#: future reader who assumes it IS a filesystem path. Fail-closed on the character set, not on a
|
| 67 |
-
#: list of forbidden sequences: an allow-list cannot be walked around by a spelling.
|
| 68 |
-
_PATH_OK = set("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789._-/")
|
| 69 |
-
|
| 70 |
-
AUTHOR_KINDS = ("user", "agent")
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
def _now():
|
| 74 |
-
return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
def _all(rt):
|
| 78 |
-
"""`{agent_id: {path: record}}` for one tenant. `{}` on any failure β an unreadable bucket must
|
| 79 |
-
degrade to "this agent has no harness files", never to a 500 on the pane that lists them."""
|
| 80 |
-
try:
|
| 81 |
-
found = rt.get(HARNESS_KEY) or {}
|
| 82 |
-
except Exception: # noqa: BLE001
|
| 83 |
-
return {}
|
| 84 |
-
return found if isinstance(found, dict) else {}
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
def _files(rt, agent_id):
|
| 88 |
-
found = _all(rt).get(str(agent_id))
|
| 89 |
-
return found if isinstance(found, dict) else {}
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
#: What kind of principal an id names. `None` = this tenant has no agent with that id.
|
| 93 |
-
#:
|
| 94 |
-
#: ββ THERE ARE TWO AGENT REGISTRIES IN THIS PRODUCT AND THE FIRST DRAFT OF THIS FILE KNEW ONLY
|
| 95 |
-
#: ONE (ASK D-5, 2026-08-18). `routes_slack._agents` is the per-Slack-channel permission wall under
|
| 96 |
-
#: Manage users. The **Agents module** the owner's item 3 is about is the AUTOMATION surface β
|
| 97 |
-
#: `Shell.tsx` mounts `<Lazily surface="Agents"><AutomationSurface /></Lazily>`, and
|
| 98 |
-
#: `AutomationDetail`'s `panelTabs` is literally the `Properties | Run history` strip R7 adds
|
| 99 |
-
#: "Harness" to. Keyed on the Slack bucket alone, every `GET/PUT /agents/{id}/harness` from that
|
| 100 |
-
#: tab would have answered **404 no_agent**: whole, gate-green and dead on arrival
|
| 101 |
-
#: [[reachable-is-not-the-same-as-built]]. Verified independently before acting, not taken on
|
| 102 |
-
#: report: `surface="Agents"` mounts `AutomationSurface`, and `ManageAgentPane` contains ZERO
|
| 103 |
-
#: occurrences of Canvas, Properties, Run history or panelTabs.
|
| 104 |
-
#:
|
| 105 |
-
#: β ONE STORE, BOTH PRINCIPALS β never a second bucket keyed by surface. The owner's "agent
|
| 106 |
-
#: skills, router" files in two places is the parallel code path item 13 exists to refuse.
|
| 107 |
-
AGENT_SLACK = "slack"
|
| 108 |
-
AGENT_AUTOMATION = "automation"
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
def agent_kind(rt, agent_id):
|
| 112 |
-
"""Which registry holds this id β `AGENT_SLACK`, `AGENT_AUTOMATION`, or `None`.
|
| 113 |
-
|
| 114 |
-
β A SYNTHETIC ROW IS NOT AN AGENT FOR THIS PURPOSE. `field:` and `system:` ids are DERIVED at
|
| 115 |
-
read time from a column definition or a connector schedule; they have no stored home, so a
|
| 116 |
-
harness file hung off one is an orphan the moment the column changes. `patch_automation`
|
| 117 |
-
already refuses those ids for the neighbouring reason, and this refuses them by simply not
|
| 118 |
-
finding them β `all_definitions` holds stored automations only.
|
| 119 |
-
"""
|
| 120 |
-
import routes_slack
|
| 121 |
-
aid = str(agent_id or "")
|
| 122 |
-
if isinstance(routes_slack._agents(rt).get(aid), dict):
|
| 123 |
-
return AGENT_SLACK
|
| 124 |
-
import automation_engine as engine
|
| 125 |
-
try:
|
| 126 |
-
known = engine.all_definitions(rt) or {}
|
| 127 |
-
except Exception: # noqa: BLE001
|
| 128 |
-
return None
|
| 129 |
-
return AGENT_AUTOMATION if aid in known else None
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
def _known_agent(rt, agent_id):
|
| 133 |
-
"""Does this tenant have an agent with this id, in EITHER registry?"""
|
| 134 |
-
return agent_kind(rt, agent_id) is not None
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
def agent_wall(session, agent_id):
|
| 138 |
-
"""404 for an unknown id, else apply the wall THAT PRINCIPAL'S OWN SURFACE applies.
|
| 139 |
-
|
| 140 |
-
ββ ONE DOOR, TWO WALLS, AND THAT IS NOT A SECOND CODE PATH β it is the refusal to invent a
|
| 141 |
-
THIRD wall. A Slack channel agent is administered under Manage users and every `/agents/*` door
|
| 142 |
-
in `routes_slack` is `admin_gate`; an automation lives in the Agents module and every door in
|
| 143 |
-
`routes_automation` is `module_gate("automation")`. A harness file is configuration OF the
|
| 144 |
-
agent it hangs off, so it is reached by whoever may already configure that agent. Picking one
|
| 145 |
-
of the two walls for both would either lock the Agents module's own users out of a tab the
|
| 146 |
-
owner asked for, or hand the Slack permission wall to anyone with an automation grant.
|
| 147 |
-
"""
|
| 148 |
-
kind = agent_kind(session.runtime, agent_id)
|
| 149 |
-
if kind is None:
|
| 150 |
-
raise err(404, "no_agent", "there is no agent with that id in this workspace")
|
| 151 |
-
if kind == AGENT_SLACK:
|
| 152 |
-
import core.perms as perms
|
| 153 |
-
if not perms.is_admin(session.user):
|
| 154 |
-
raise err(403, "forbidden", "administrators only")
|
| 155 |
-
else:
|
| 156 |
-
session.require("automation")
|
| 157 |
-
return kind
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
def normalize_path(raw):
|
| 161 |
-
"""THE path rule, in ONE place. Returns the cleaned path, or `None` if it is not acceptable.
|
| 162 |
-
|
| 163 |
-
ββ ONE RULE, TWO DOORS, AND THAT IS WHY THIS IS A FUNCTION RATHER THAN TWO IF-BLOCKS. There
|
| 164 |
-
are two ways into this store β the HTTP route (which must answer 400) and `record_version()`
|
| 165 |
-
(which must raise `ValueError`, having no response to put a status into). Written twice, the
|
| 166 |
-
two copies are [[one-question-two-normalizers]] waiting to happen: the first weakening of one
|
| 167 |
-
copy is invisible because the other still refuses, so nothing goes red and the wall is now
|
| 168 |
-
half there. Written once, a change to the rule is felt at both doors and by the gate.
|
| 169 |
-
"""
|
| 170 |
-
path = str(raw or "").strip().strip("/")
|
| 171 |
-
if not path or len(path) > MAX_PATH:
|
| 172 |
-
return None
|
| 173 |
-
if set(path) - _PATH_OK or ".." in path or "//" in path:
|
| 174 |
-
return None
|
| 175 |
-
return path
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
def _clean_path(raw):
|
| 179 |
-
"""`normalize_path` at the HTTP door, where a refusal is a 400 with a reason."""
|
| 180 |
-
path = normalize_path(raw)
|
| 181 |
-
if path is None:
|
| 182 |
-
if not str(raw or "").strip().strip("/"):
|
| 183 |
-
raise err(400, "no_path", "a harness file needs a path, for example skills/router.md")
|
| 184 |
-
if len(str(raw)) > MAX_PATH:
|
| 185 |
-
raise err(400, "path_too_long", f"a harness path is at most {MAX_PATH} characters")
|
| 186 |
-
raise err(400, "bad_path",
|
| 187 |
-
"a harness path may use letters, digits, dot, dash, underscore and / only")
|
| 188 |
-
return path
|
| 189 |
-
|
| 190 |
-
|
| 191 |
-
def _blank(path, author, author_kind):
|
| 192 |
-
return {"path": path, "versions": [], "trimmed": 0,
|
| 193 |
-
"created": _now(), "createdBy": author, "createdKind": author_kind}
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
def _append(record, body, author, author_kind, restored_from=None):
|
| 197 |
-
"""Append ONE version to a file record, in place, and report what was trimmed.
|
| 198 |
-
|
| 199 |
-
The version NUMBER is monotonic and survives trimming β it counts writes, not stored entries.
|
| 200 |
-
A version list whose numbers restart at 1 after a trim would make two different bodies share a
|
| 201 |
-
name, and `restoredFrom` would then point at whichever one happened to be in the window.
|
| 202 |
-
"""
|
| 203 |
-
# β A NEW LIST, NEVER `versions.append(...)` ON THE STORED ONE. `update()` hands the callback
|
| 204 |
-
# the live document and may run it more than once; appending in place would then stack two
|
| 205 |
-
# copies of the same version into the history on a retry.
|
| 206 |
-
prior = record.get("versions") if isinstance(record.get("versions"), list) else []
|
| 207 |
-
last = max((int(v.get("version") or 0) for v in prior if isinstance(v, dict)), default=0)
|
| 208 |
-
entry = {"version": last + 1, "body": body, "author": author, "authorKind": author_kind,
|
| 209 |
-
"created": _now(), "bytes": len(body.encode("utf-8"))}
|
| 210 |
-
if restored_from:
|
| 211 |
-
entry["restoredFrom"] = int(restored_from)
|
| 212 |
-
versions = [*prior, entry]
|
| 213 |
-
dropped = max(0, len(versions) - MAX_VERSIONS)
|
| 214 |
-
if dropped:
|
| 215 |
-
versions = versions[dropped:]
|
| 216 |
-
record["versions"] = versions
|
| 217 |
-
record["trimmed"] = int(record.get("trimmed") or 0) + dropped
|
| 218 |
-
return entry
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
def _head(record):
|
| 222 |
-
"""The newest version of a file record, or `None` for a record with no versions at all."""
|
| 223 |
-
versions = record.get("versions") if isinstance(record.get("versions"), list) else []
|
| 224 |
-
return versions[-1] if versions else None
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
def _row(record):
|
| 228 |
-
"""One file, as the LIST door reports it: everything except the bodies.
|
| 229 |
-
|
| 230 |
-
β NO BODY, AND THAT IS THE POINT. A list door that carried every version of every file would
|
| 231 |
-
ship the whole harness on every pane render; the Harness tab lists first and opens one file
|
| 232 |
-
second, which is exactly the shape this answers.
|
| 233 |
-
"""
|
| 234 |
-
head = _head(record) or {}
|
| 235 |
-
versions = record.get("versions") if isinstance(record.get("versions"), list) else []
|
| 236 |
-
return {"path": record.get("path") or "",
|
| 237 |
-
"version": int(head.get("version") or 0),
|
| 238 |
-
"bytes": int(head.get("bytes") or 0),
|
| 239 |
-
"author": head.get("author") or record.get("createdBy") or "",
|
| 240 |
-
"authorKind": head.get("authorKind") or record.get("createdKind") or "user",
|
| 241 |
-
"updated": head.get("created") or record.get("created") or "",
|
| 242 |
-
"created": record.get("created") or "",
|
| 243 |
-
"versions": len(versions),
|
| 244 |
-
"trimmed": int(record.get("trimmed") or 0)}
|
| 245 |
-
|
| 246 |
-
|
| 247 |
-
def _version_rows(record):
|
| 248 |
-
"""The history of one file, newest first, WITHOUT the bodies.
|
| 249 |
-
|
| 250 |
-
A body per version is what makes a diff possible, and it is also what makes this payload big:
|
| 251 |
-
100 versions of a 128 KB file is 12 MB. The client asks for the two bodies it is diffing
|
| 252 |
-
(`?path=β¦&version=N`), which is two round trips for a diff and none for a history list.
|
| 253 |
-
"""
|
| 254 |
-
versions = record.get("versions") if isinstance(record.get("versions"), list) else []
|
| 255 |
-
out = []
|
| 256 |
-
for entry in reversed(versions):
|
| 257 |
-
if not isinstance(entry, dict):
|
| 258 |
-
continue
|
| 259 |
-
row = {"version": int(entry.get("version") or 0),
|
| 260 |
-
"author": entry.get("author") or "",
|
| 261 |
-
"authorKind": entry.get("authorKind") or "user",
|
| 262 |
-
"created": entry.get("created") or "",
|
| 263 |
-
"bytes": int(entry.get("bytes") or 0)}
|
| 264 |
-
if entry.get("restoredFrom"):
|
| 265 |
-
row["restoredFrom"] = int(entry["restoredFrom"])
|
| 266 |
-
out.append(row)
|
| 267 |
-
return out
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
def _limits():
|
| 271 |
-
"""The caps, IN the payload, so a client can say "this file is full" before a write fails.
|
| 272 |
-
|
| 273 |
-
β A limit the client cannot see is a limit the user meets as an error. `trimmed` reports the
|
| 274 |
-
one cap that acts without refusing; these report the three that refuse.
|
| 275 |
-
"""
|
| 276 |
-
return {"maxBodyBytes": MAX_BODY_BYTES, "maxFiles": MAX_FILES,
|
| 277 |
-
"maxVersions": MAX_VERSIONS, "maxPath": MAX_PATH}
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
# ββ the write door the SERVER uses (no session) ββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 281 |
-
def record_version(runtime, agent_id, path, body, author="", author_kind="agent",
|
| 282 |
-
restored_from=None):
|
| 283 |
-
"""Write one version from INSIDE the server β the agent's own half of R8.
|
| 284 |
-
|
| 285 |
-
β THIS IS THE FUNCTION, NOT THE ROUTE, THAT MAKES "editable by BOTH the user and the agent"
|
| 286 |
-
true. An agent acting inside an automation run holds no session and no cookie; if its only way
|
| 287 |
-
to write were the HTTP door it would have to borrow a person's identity, and the authorship
|
| 288 |
-
column would then be a record of who was logged in rather than of what wrote the file.
|
| 289 |
-
|
| 290 |
-
Returns the appended entry. Raises nothing the caller cannot handle: an unknown agent is a
|
| 291 |
-
`ValueError`, because a server-side caller has no HTTP response to put a 404 into.
|
| 292 |
-
"""
|
| 293 |
-
agent_id = str(agent_id or "")
|
| 294 |
-
if not _known_agent(runtime, agent_id):
|
| 295 |
-
raise ValueError(f"no agent {agent_id!r} in this workspace")
|
| 296 |
-
path = normalize_path(path)
|
| 297 |
-
if path is None:
|
| 298 |
-
raise ValueError("that is not an acceptable harness path")
|
| 299 |
-
body = str(body or "")
|
| 300 |
-
if len(body.encode("utf-8")) > MAX_BODY_BYTES:
|
| 301 |
-
raise ValueError("harness body is over the size limit")
|
| 302 |
-
kind = author_kind if author_kind in AUTHOR_KINDS else "agent"
|
| 303 |
-
author = str(author or agent_id)
|
| 304 |
-
# β THE FILE-COUNT CEILING IS CHECKED HERE, NOT INSIDE `_set`. An exception raised inside a
|
| 305 |
-
# store `update` callback propagates out of a half-run read-modify-write, and the one thing a
|
| 306 |
-
# refusal must never do is leave the caller unsure whether the write happened.
|
| 307 |
-
existing = _files(runtime, agent_id)
|
| 308 |
-
if path not in existing and len(existing) >= MAX_FILES:
|
| 309 |
-
raise ValueError(f"this agent already has {MAX_FILES} harness files")
|
| 310 |
-
appended = {}
|
| 311 |
-
|
| 312 |
-
def _set(cur):
|
| 313 |
-
cur = dict(cur or {})
|
| 314 |
-
files = dict(cur.get(agent_id) or {}) if isinstance(cur.get(agent_id), dict) else {}
|
| 315 |
-
record = dict(files[path]) if isinstance(files.get(path), dict) else _blank(path, author, kind)
|
| 316 |
-
appended.clear()
|
| 317 |
-
appended.update(_append(record, body, author, kind, restored_from))
|
| 318 |
-
files[path] = record
|
| 319 |
-
cur[agent_id] = files
|
| 320 |
-
return cur
|
| 321 |
-
|
| 322 |
-
runtime.update(HARNESS_KEY, _set, flush="sync")
|
| 323 |
-
return appended
|
| 324 |
-
|
| 325 |
-
|
| 326 |
-
# ββ the routes ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 327 |
-
@router.get("/agents/{agent_id}/harness")
|
| 328 |
-
def get_harness(agent_id: str, path: str = "", version: int = 0,
|
| 329 |
-
session: Session = Depends(require_session)):
|
| 330 |
-
"""The list, one file, or one older body β decided by the query string (C4).
|
| 331 |
-
|
| 332 |
-
β ADMIN-GATED LIKE EVERY OTHER AGENT DOOR (`routes_slack`), and for a stronger reason than
|
| 333 |
-
consistency: a harness file is what the agent is INSTRUCTED to do, so writing one is closer to
|
| 334 |
-
editing a permission than to editing a document.
|
| 335 |
-
"""
|
| 336 |
-
agent_id = str(agent_id or "")
|
| 337 |
-
agent_wall(session, agent_id)
|
| 338 |
-
files = _files(session.runtime, agent_id)
|
| 339 |
-
if not path:
|
| 340 |
-
rows = [_row(rec) for _p, rec in sorted(files.items()) if isinstance(rec, dict)]
|
| 341 |
-
return {"agent": agent_id, "files": rows, "limits": _limits()}
|
| 342 |
-
|
| 343 |
-
wanted = _clean_path(path)
|
| 344 |
-
record = files.get(wanted)
|
| 345 |
-
if not isinstance(record, dict):
|
| 346 |
-
raise err(404, "no_file", f"this agent has no harness file at {wanted}")
|
| 347 |
-
versions = record.get("versions") if isinstance(record.get("versions"), list) else []
|
| 348 |
-
if version:
|
| 349 |
-
for entry in versions:
|
| 350 |
-
if isinstance(entry, dict) and int(entry.get("version") or 0) == int(version):
|
| 351 |
-
return {"agent": agent_id, **_row(record), "body": entry.get("body") or "",
|
| 352 |
-
"atVersion": int(version), "history": _version_rows(record),
|
| 353 |
-
"limits": _limits()}
|
| 354 |
-
# β A TRIMMED VERSION IS A NAMED REFUSAL, NOT A 404 SHAPED LIKE A TYPO. The client asked
|
| 355 |
-
# for something that existed and no longer does, and telling it apart from a bad number is
|
| 356 |
-
# the difference between "roll back to v3" failing loudly and failing as if v3 never was.
|
| 357 |
-
trimmed = int(record.get("trimmed") or 0)
|
| 358 |
-
if trimmed and int(version) <= trimmed:
|
| 359 |
-
raise err(410, "version_trimmed",
|
| 360 |
-
f"version {int(version)} is older than the {MAX_VERSIONS} versions kept "
|
| 361 |
-
f"for this file, and its body is gone")
|
| 362 |
-
raise err(404, "no_version", f"this file has no version {int(version)}")
|
| 363 |
-
head = _head(record) or {}
|
| 364 |
-
return {"agent": agent_id, **_row(record), "body": head.get("body") or "",
|
| 365 |
-
"atVersion": int(head.get("version") or 0), "history": _version_rows(record),
|
| 366 |
-
"limits": _limits()}
|
| 367 |
-
|
| 368 |
-
|
| 369 |
-
@router.put("/agents/{agent_id}/harness")
|
| 370 |
-
def put_harness(agent_id: str, body: dict = Body(default=None),
|
| 371 |
-
session: Session = Depends(require_session)):
|
| 372 |
-
"""Write a NEW version of one harness file. `{path, body, authorKind?, restoredFrom?}` (C4).
|
| 373 |
-
|
| 374 |
-
β THERE IS NO OVERWRITE HERE AND THERE IS NO EDIT-IN-PLACE. R8's "every version kept" is not a
|
| 375 |
-
UI affordance; it is this function refusing to have a code path that replaces a body. A
|
| 376 |
-
roll-back arrives as an ordinary write carrying `restoredFrom`, so the history records that
|
| 377 |
-
somebody went back rather than pretending the intervening versions never happened.
|
| 378 |
-
"""
|
| 379 |
-
agent_id = str(agent_id or "")
|
| 380 |
-
agent_wall(session, agent_id)
|
| 381 |
-
body = body if isinstance(body, dict) else {}
|
| 382 |
-
path = _clean_path(body.get("path"))
|
| 383 |
-
text = body.get("body")
|
| 384 |
-
if not isinstance(text, str):
|
| 385 |
-
raise err(400, "no_body", "a harness file needs a body, even an empty one")
|
| 386 |
-
if len(text.encode("utf-8")) > MAX_BODY_BYTES:
|
| 387 |
-
raise err(413, "body_too_long",
|
| 388 |
-
f"a harness file is at most {MAX_BODY_BYTES // 1024} KB; this one is larger")
|
| 389 |
-
|
| 390 |
-
# β THE CALLER DECLARES THE PROVENANCE AND THE SERVER STAMPS THE IDENTITY. `authorKind` is a
|
| 391 |
-
# claim about who WROTE the text (the person, or the agent in the chat panel); `author` is the
|
| 392 |
-
# session and is never read off the request. Nothing is bought by forging the first β both
|
| 393 |
-
# principals may write (R8) β and everything would be bought by forging the second.
|
| 394 |
-
kind = str(body.get("authorKind") or "user").strip().lower()
|
| 395 |
-
if kind not in AUTHOR_KINDS:
|
| 396 |
-
raise err(400, "bad_author_kind", "authorKind is either user or agent")
|
| 397 |
-
restored = body.get("restoredFrom")
|
| 398 |
-
try:
|
| 399 |
-
restored = int(restored) if restored else None
|
| 400 |
-
except (TypeError, ValueError):
|
| 401 |
-
raise err(400, "bad_version", "restoredFrom must be a version number")
|
| 402 |
-
|
| 403 |
-
files = _files(session.runtime, agent_id)
|
| 404 |
-
if path not in files and len(files) >= MAX_FILES:
|
| 405 |
-
raise err(409, "too_many_files",
|
| 406 |
-
f"this agent already has {MAX_FILES} harness files; delete one to add another")
|
| 407 |
-
try:
|
| 408 |
-
record_version(session.runtime, agent_id, path, text,
|
| 409 |
-
author=session.uname, author_kind=kind, restored_from=restored)
|
| 410 |
-
except ValueError as exc:
|
| 411 |
-
raise err(400, "refused", str(exc))
|
| 412 |
-
|
| 413 |
-
fresh = _files(session.runtime, agent_id).get(path)
|
| 414 |
-
if not isinstance(fresh, dict) or not _head(fresh):
|
| 415 |
-
# The store took the write and did not record it. A 200 here would tell an administrator
|
| 416 |
-
# their skill file was saved when it was not β the shape `routes_slack` refuses too.
|
| 417 |
-
raise err(503, "store_unavailable", "the harness file was NOT saved")
|
| 418 |
-
head = _head(fresh)
|
| 419 |
-
return {"agent": agent_id, **_row(fresh), "body": head.get("body") or "",
|
| 420 |
-
"atVersion": int(head.get("version") or 0), "history": _version_rows(fresh),
|
| 421 |
-
"limits": _limits()}
|
| 422 |
-
|
| 423 |
-
|
| 424 |
-
@router.delete("/agents/{agent_id}/harness")
|
| 425 |
-
def delete_harness(agent_id: str, path: str = "", session: Session = Depends(require_session)):
|
| 426 |
-
"""Drop one harness file AND its history.
|
| 427 |
-
|
| 428 |
-
β THIS IS NOT THE THING R8 FORBIDS. R8 forbids a ROLL-BACK implemented as a delete β losing
|
| 429 |
-
versions as a side effect of an edit. Deleting a file is a person deciding the file should not
|
| 430 |
-
exist, which is a different act with a different button, and a store with no way to remove a
|
| 431 |
-
file is one where a typo'd path is permanent.
|
| 432 |
-
"""
|
| 433 |
-
agent_id = str(agent_id or "")
|
| 434 |
-
agent_wall(session, agent_id)
|
| 435 |
-
wanted = _clean_path(path)
|
| 436 |
-
if wanted not in _files(session.runtime, agent_id):
|
| 437 |
-
raise err(404, "no_file", f"this agent has no harness file at {wanted}")
|
| 438 |
-
|
| 439 |
-
def _set(cur):
|
| 440 |
-
cur = dict(cur or {})
|
| 441 |
-
files = dict(cur.get(agent_id) or {}) if isinstance(cur.get(agent_id), dict) else {}
|
| 442 |
-
files.pop(wanted, None)
|
| 443 |
-
# An agent with no harness files leaves NO key behind. An empty dict per agent id is how a
|
| 444 |
-
# bucket accumulates a row for every agent anybody ever opened the tab on.
|
| 445 |
-
if files:
|
| 446 |
-
cur[agent_id] = files
|
| 447 |
-
else:
|
| 448 |
-
cur.pop(agent_id, None)
|
| 449 |
-
return cur
|
| 450 |
-
|
| 451 |
-
session.runtime.update(HARNESS_KEY, _set, flush="sync")
|
| 452 |
-
return {"agent": agent_id, "deleted": wanted,
|
| 453 |
-
"files": [_row(rec) for _p, rec in sorted(_files(session.runtime, agent_id).items())
|
| 454 |
-
if isinstance(rec, dict)],
|
| 455 |
-
"limits": _limits()}
|
|
|
|
| 1 |
+
"""routes_agent_harness.py β CONTRACT C4: the agent's HARNESS, kept as versioned files.
|
| 2 |
+
|
| 3 |
+
Owner item 3, verbatim (2026-08-18): *"This new module under agent is supposed to host any file
|
| 4 |
+
pertaining to the agent skills, router etc. So we build a user can build a custom harness for each
|
| 5 |
+
agent through the chat interface."*
|
| 6 |
+
|
| 7 |
+
GET /api/v1/agents/{id}/harness the file LIST (no bodies)
|
| 8 |
+
GET /api/v1/agents/{id}/harness?path=β¦ one file: current body + its versions
|
| 9 |
+
GET /api/v1/agents/{id}/harness?path=β¦&version=N one older body, verbatim
|
| 10 |
+
PUT /api/v1/agents/{id}/harness write a NEW version of one file
|
| 11 |
+
DELETE /api/v1/agents/{id}/harness?path=β¦ drop a file and its history
|
| 12 |
+
|
| 13 |
+
ββ **R8 IS THE WHOLE SHAPE: BOTH PRINCIPALS WRITE, AND NOTHING IS EVER OVERWRITTEN.** A write
|
| 14 |
+
appends a version; a roll-back is a write of an older body (`restoredFrom`), never a delete. So the
|
| 15 |
+
history is a record of what happened rather than of what somebody last wanted it to look like.
|
| 16 |
+
|
| 17 |
+
β **R8 IS DELIBERATELY NOT R9.** An agent-authored ACTION's configuration is agent-only (item 8,
|
| 18 |
+
`routes_automation`); a harness file is not. Do not copy this file's posture over there or that
|
| 19 |
+
one's over here β the two rulings differ on purpose and they sit one screen apart in the product.
|
| 20 |
+
|
| 21 |
+
β **`author` IS THE SESSION, `authorKind` IS THE PROVENANCE, AND THEY ARE DIFFERENT FACTS.** Every
|
| 22 |
+
write through this router is made BY a signed-in administrator, so `author` is stamped from the
|
| 23 |
+
session and can never be supplied by the caller. `authorKind` says whether the BODY was drafted by
|
| 24 |
+
a person or by the agent in the chat β a claim the client is entitled to make, because both are
|
| 25 |
+
permitted (R8) and so nothing is bought by forging it. `record_version()` below is the server-side
|
| 26 |
+
door the automation engine uses when the agent writes with no session at all; that one stamps the
|
| 27 |
+
agent's own id as the author, which is the only case where `author` is not a username.
|
| 28 |
+
|
| 29 |
+
β **THE VERSION LIST IS CAPPED AND THE CAP IS REPORTED, NEVER SILENT.** `MAX_VERSIONS` versions of
|
| 30 |
+
one file are kept; past that the OLDEST are dropped and the count of what was dropped rides in
|
| 31 |
+
`trimmed` on every payload that mentions the file, so a reader can see that the history is partial
|
| 32 |
+
rather than infer that the file was only ever saved twice. This is the tenant document, which is
|
| 33 |
+
already 28.6 MB on tenant #0 and is deep-copied on every read: an unbounded per-agent history is
|
| 34 |
+
a store-sized leak with a UI in front of it.
|
| 35 |
+
"""
|
| 36 |
+
from datetime import datetime, timezone
|
| 37 |
+
|
| 38 |
+
from fastapi import APIRouter, Body, Depends
|
| 39 |
+
|
| 40 |
+
from deps import Session, err, require_session
|
| 41 |
+
|
| 42 |
+
router = APIRouter(prefix="/api/v1")
|
| 43 |
+
|
| 44 |
+
#: The tenant's harness files: `{agent_id: {path: file_record}}`. A per-tenant bucket, so it rides
|
| 45 |
+
#: `runtime.store_key`'s prefix and never lands in tenant #0's namespace β the same rule
|
| 46 |
+
#: `routes_slack.AGENTS_KEY` follows for the agent records these hang off.
|
| 47 |
+
HARNESS_KEY = "agent_harness"
|
| 48 |
+
|
| 49 |
+
#: One body. Generous for a skill or a router file and far under the point where a single write
|
| 50 |
+
#: would move the tenant document measurably. A larger body is a REFUSAL at the door, not a
|
| 51 |
+
#: truncation: a silently truncated skill file is a harness that does not do what its text says.
|
| 52 |
+
MAX_BODY_BYTES = 128 * 1024
|
| 53 |
+
|
| 54 |
+
#: Files per agent. A refusal, not a trim β creating the 65th file is a different act from saving
|
| 55 |
+
#: the 101st version of one, and only the second can be a routine consequence of ordinary editing.
|
| 56 |
+
MAX_FILES = 64
|
| 57 |
+
|
| 58 |
+
#: Versions kept per file. Past this the oldest go and `trimmed` counts them (see the header).
|
| 59 |
+
MAX_VERSIONS = 100
|
| 60 |
+
|
| 61 |
+
MAX_PATH = 200
|
| 62 |
+
|
| 63 |
+
#: What a path may contain. β THIS IS NOT A FILESYSTEM PATH AND NOTHING HERE EVER TOUCHES A DISK β
|
| 64 |
+
#: the "files" are keys in a store bucket. The character rule exists so the key is displayable, is
|
| 65 |
+
#: safe to put in a URL, and cannot carry a traversal sequence that would look meaningful to a
|
| 66 |
+
#: future reader who assumes it IS a filesystem path. Fail-closed on the character set, not on a
|
| 67 |
+
#: list of forbidden sequences: an allow-list cannot be walked around by a spelling.
|
| 68 |
+
_PATH_OK = set("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789._-/")
|
| 69 |
+
|
| 70 |
+
AUTHOR_KINDS = ("user", "agent")
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def _now():
|
| 74 |
+
return datetime.now(timezone.utc).isoformat(timespec="seconds")
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def _all(rt):
|
| 78 |
+
"""`{agent_id: {path: record}}` for one tenant. `{}` on any failure β an unreadable bucket must
|
| 79 |
+
degrade to "this agent has no harness files", never to a 500 on the pane that lists them."""
|
| 80 |
+
try:
|
| 81 |
+
found = rt.get(HARNESS_KEY) or {}
|
| 82 |
+
except Exception: # noqa: BLE001
|
| 83 |
+
return {}
|
| 84 |
+
return found if isinstance(found, dict) else {}
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def _files(rt, agent_id):
|
| 88 |
+
found = _all(rt).get(str(agent_id))
|
| 89 |
+
return found if isinstance(found, dict) else {}
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
#: What kind of principal an id names. `None` = this tenant has no agent with that id.
|
| 93 |
+
#:
|
| 94 |
+
#: ββ THERE ARE TWO AGENT REGISTRIES IN THIS PRODUCT AND THE FIRST DRAFT OF THIS FILE KNEW ONLY
|
| 95 |
+
#: ONE (ASK D-5, 2026-08-18). `routes_slack._agents` is the per-Slack-channel permission wall under
|
| 96 |
+
#: Manage users. The **Agents module** the owner's item 3 is about is the AUTOMATION surface β
|
| 97 |
+
#: `Shell.tsx` mounts `<Lazily surface="Agents"><AutomationSurface /></Lazily>`, and
|
| 98 |
+
#: `AutomationDetail`'s `panelTabs` is literally the `Properties | Run history` strip R7 adds
|
| 99 |
+
#: "Harness" to. Keyed on the Slack bucket alone, every `GET/PUT /agents/{id}/harness` from that
|
| 100 |
+
#: tab would have answered **404 no_agent**: whole, gate-green and dead on arrival
|
| 101 |
+
#: [[reachable-is-not-the-same-as-built]]. Verified independently before acting, not taken on
|
| 102 |
+
#: report: `surface="Agents"` mounts `AutomationSurface`, and `ManageAgentPane` contains ZERO
|
| 103 |
+
#: occurrences of Canvas, Properties, Run history or panelTabs.
|
| 104 |
+
#:
|
| 105 |
+
#: β ONE STORE, BOTH PRINCIPALS β never a second bucket keyed by surface. The owner's "agent
|
| 106 |
+
#: skills, router" files in two places is the parallel code path item 13 exists to refuse.
|
| 107 |
+
AGENT_SLACK = "slack"
|
| 108 |
+
AGENT_AUTOMATION = "automation"
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def agent_kind(rt, agent_id):
|
| 112 |
+
"""Which registry holds this id β `AGENT_SLACK`, `AGENT_AUTOMATION`, or `None`.
|
| 113 |
+
|
| 114 |
+
β A SYNTHETIC ROW IS NOT AN AGENT FOR THIS PURPOSE. `field:` and `system:` ids are DERIVED at
|
| 115 |
+
read time from a column definition or a connector schedule; they have no stored home, so a
|
| 116 |
+
harness file hung off one is an orphan the moment the column changes. `patch_automation`
|
| 117 |
+
already refuses those ids for the neighbouring reason, and this refuses them by simply not
|
| 118 |
+
finding them β `all_definitions` holds stored automations only.
|
| 119 |
+
"""
|
| 120 |
+
import routes_slack
|
| 121 |
+
aid = str(agent_id or "")
|
| 122 |
+
if isinstance(routes_slack._agents(rt).get(aid), dict):
|
| 123 |
+
return AGENT_SLACK
|
| 124 |
+
import automation_engine as engine
|
| 125 |
+
try:
|
| 126 |
+
known = engine.all_definitions(rt) or {}
|
| 127 |
+
except Exception: # noqa: BLE001
|
| 128 |
+
return None
|
| 129 |
+
return AGENT_AUTOMATION if aid in known else None
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def _known_agent(rt, agent_id):
|
| 133 |
+
"""Does this tenant have an agent with this id, in EITHER registry?"""
|
| 134 |
+
return agent_kind(rt, agent_id) is not None
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
def agent_wall(session, agent_id):
|
| 138 |
+
"""404 for an unknown id, else apply the wall THAT PRINCIPAL'S OWN SURFACE applies.
|
| 139 |
+
|
| 140 |
+
ββ ONE DOOR, TWO WALLS, AND THAT IS NOT A SECOND CODE PATH β it is the refusal to invent a
|
| 141 |
+
THIRD wall. A Slack channel agent is administered under Manage users and every `/agents/*` door
|
| 142 |
+
in `routes_slack` is `admin_gate`; an automation lives in the Agents module and every door in
|
| 143 |
+
`routes_automation` is `module_gate("automation")`. A harness file is configuration OF the
|
| 144 |
+
agent it hangs off, so it is reached by whoever may already configure that agent. Picking one
|
| 145 |
+
of the two walls for both would either lock the Agents module's own users out of a tab the
|
| 146 |
+
owner asked for, or hand the Slack permission wall to anyone with an automation grant.
|
| 147 |
+
"""
|
| 148 |
+
kind = agent_kind(session.runtime, agent_id)
|
| 149 |
+
if kind is None:
|
| 150 |
+
raise err(404, "no_agent", "there is no agent with that id in this workspace")
|
| 151 |
+
if kind == AGENT_SLACK:
|
| 152 |
+
import core.perms as perms
|
| 153 |
+
if not perms.is_admin(session.user):
|
| 154 |
+
raise err(403, "forbidden", "administrators only")
|
| 155 |
+
else:
|
| 156 |
+
session.require("automation")
|
| 157 |
+
return kind
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def normalize_path(raw):
|
| 161 |
+
"""THE path rule, in ONE place. Returns the cleaned path, or `None` if it is not acceptable.
|
| 162 |
+
|
| 163 |
+
ββ ONE RULE, TWO DOORS, AND THAT IS WHY THIS IS A FUNCTION RATHER THAN TWO IF-BLOCKS. There
|
| 164 |
+
are two ways into this store β the HTTP route (which must answer 400) and `record_version()`
|
| 165 |
+
(which must raise `ValueError`, having no response to put a status into). Written twice, the
|
| 166 |
+
two copies are [[one-question-two-normalizers]] waiting to happen: the first weakening of one
|
| 167 |
+
copy is invisible because the other still refuses, so nothing goes red and the wall is now
|
| 168 |
+
half there. Written once, a change to the rule is felt at both doors and by the gate.
|
| 169 |
+
"""
|
| 170 |
+
path = str(raw or "").strip().strip("/")
|
| 171 |
+
if not path or len(path) > MAX_PATH:
|
| 172 |
+
return None
|
| 173 |
+
if set(path) - _PATH_OK or ".." in path or "//" in path:
|
| 174 |
+
return None
|
| 175 |
+
return path
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
def _clean_path(raw):
|
| 179 |
+
"""`normalize_path` at the HTTP door, where a refusal is a 400 with a reason."""
|
| 180 |
+
path = normalize_path(raw)
|
| 181 |
+
if path is None:
|
| 182 |
+
if not str(raw or "").strip().strip("/"):
|
| 183 |
+
raise err(400, "no_path", "a harness file needs a path, for example skills/router.md")
|
| 184 |
+
if len(str(raw)) > MAX_PATH:
|
| 185 |
+
raise err(400, "path_too_long", f"a harness path is at most {MAX_PATH} characters")
|
| 186 |
+
raise err(400, "bad_path",
|
| 187 |
+
"a harness path may use letters, digits, dot, dash, underscore and / only")
|
| 188 |
+
return path
|
| 189 |
+
|
| 190 |
+
|
| 191 |
+
def _blank(path, author, author_kind):
|
| 192 |
+
return {"path": path, "versions": [], "trimmed": 0,
|
| 193 |
+
"created": _now(), "createdBy": author, "createdKind": author_kind}
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
def _append(record, body, author, author_kind, restored_from=None):
|
| 197 |
+
"""Append ONE version to a file record, in place, and report what was trimmed.
|
| 198 |
+
|
| 199 |
+
The version NUMBER is monotonic and survives trimming β it counts writes, not stored entries.
|
| 200 |
+
A version list whose numbers restart at 1 after a trim would make two different bodies share a
|
| 201 |
+
name, and `restoredFrom` would then point at whichever one happened to be in the window.
|
| 202 |
+
"""
|
| 203 |
+
# β A NEW LIST, NEVER `versions.append(...)` ON THE STORED ONE. `update()` hands the callback
|
| 204 |
+
# the live document and may run it more than once; appending in place would then stack two
|
| 205 |
+
# copies of the same version into the history on a retry.
|
| 206 |
+
prior = record.get("versions") if isinstance(record.get("versions"), list) else []
|
| 207 |
+
last = max((int(v.get("version") or 0) for v in prior if isinstance(v, dict)), default=0)
|
| 208 |
+
entry = {"version": last + 1, "body": body, "author": author, "authorKind": author_kind,
|
| 209 |
+
"created": _now(), "bytes": len(body.encode("utf-8"))}
|
| 210 |
+
if restored_from:
|
| 211 |
+
entry["restoredFrom"] = int(restored_from)
|
| 212 |
+
versions = [*prior, entry]
|
| 213 |
+
dropped = max(0, len(versions) - MAX_VERSIONS)
|
| 214 |
+
if dropped:
|
| 215 |
+
versions = versions[dropped:]
|
| 216 |
+
record["versions"] = versions
|
| 217 |
+
record["trimmed"] = int(record.get("trimmed") or 0) + dropped
|
| 218 |
+
return entry
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
def _head(record):
|
| 222 |
+
"""The newest version of a file record, or `None` for a record with no versions at all."""
|
| 223 |
+
versions = record.get("versions") if isinstance(record.get("versions"), list) else []
|
| 224 |
+
return versions[-1] if versions else None
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
def _row(record):
|
| 228 |
+
"""One file, as the LIST door reports it: everything except the bodies.
|
| 229 |
+
|
| 230 |
+
β NO BODY, AND THAT IS THE POINT. A list door that carried every version of every file would
|
| 231 |
+
ship the whole harness on every pane render; the Harness tab lists first and opens one file
|
| 232 |
+
second, which is exactly the shape this answers.
|
| 233 |
+
"""
|
| 234 |
+
head = _head(record) or {}
|
| 235 |
+
versions = record.get("versions") if isinstance(record.get("versions"), list) else []
|
| 236 |
+
return {"path": record.get("path") or "",
|
| 237 |
+
"version": int(head.get("version") or 0),
|
| 238 |
+
"bytes": int(head.get("bytes") or 0),
|
| 239 |
+
"author": head.get("author") or record.get("createdBy") or "",
|
| 240 |
+
"authorKind": head.get("authorKind") or record.get("createdKind") or "user",
|
| 241 |
+
"updated": head.get("created") or record.get("created") or "",
|
| 242 |
+
"created": record.get("created") or "",
|
| 243 |
+
"versions": len(versions),
|
| 244 |
+
"trimmed": int(record.get("trimmed") or 0)}
|
| 245 |
+
|
| 246 |
+
|
| 247 |
+
def _version_rows(record):
|
| 248 |
+
"""The history of one file, newest first, WITHOUT the bodies.
|
| 249 |
+
|
| 250 |
+
A body per version is what makes a diff possible, and it is also what makes this payload big:
|
| 251 |
+
100 versions of a 128 KB file is 12 MB. The client asks for the two bodies it is diffing
|
| 252 |
+
(`?path=β¦&version=N`), which is two round trips for a diff and none for a history list.
|
| 253 |
+
"""
|
| 254 |
+
versions = record.get("versions") if isinstance(record.get("versions"), list) else []
|
| 255 |
+
out = []
|
| 256 |
+
for entry in reversed(versions):
|
| 257 |
+
if not isinstance(entry, dict):
|
| 258 |
+
continue
|
| 259 |
+
row = {"version": int(entry.get("version") or 0),
|
| 260 |
+
"author": entry.get("author") or "",
|
| 261 |
+
"authorKind": entry.get("authorKind") or "user",
|
| 262 |
+
"created": entry.get("created") or "",
|
| 263 |
+
"bytes": int(entry.get("bytes") or 0)}
|
| 264 |
+
if entry.get("restoredFrom"):
|
| 265 |
+
row["restoredFrom"] = int(entry["restoredFrom"])
|
| 266 |
+
out.append(row)
|
| 267 |
+
return out
|
| 268 |
+
|
| 269 |
+
|
| 270 |
+
def _limits():
|
| 271 |
+
"""The caps, IN the payload, so a client can say "this file is full" before a write fails.
|
| 272 |
+
|
| 273 |
+
β A limit the client cannot see is a limit the user meets as an error. `trimmed` reports the
|
| 274 |
+
one cap that acts without refusing; these report the three that refuse.
|
| 275 |
+
"""
|
| 276 |
+
return {"maxBodyBytes": MAX_BODY_BYTES, "maxFiles": MAX_FILES,
|
| 277 |
+
"maxVersions": MAX_VERSIONS, "maxPath": MAX_PATH}
|
| 278 |
+
|
| 279 |
+
|
| 280 |
+
# ββ the write door the SERVER uses (no session) ββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 281 |
+
def record_version(runtime, agent_id, path, body, author="", author_kind="agent",
|
| 282 |
+
restored_from=None):
|
| 283 |
+
"""Write one version from INSIDE the server β the agent's own half of R8.
|
| 284 |
+
|
| 285 |
+
β THIS IS THE FUNCTION, NOT THE ROUTE, THAT MAKES "editable by BOTH the user and the agent"
|
| 286 |
+
true. An agent acting inside an automation run holds no session and no cookie; if its only way
|
| 287 |
+
to write were the HTTP door it would have to borrow a person's identity, and the authorship
|
| 288 |
+
column would then be a record of who was logged in rather than of what wrote the file.
|
| 289 |
+
|
| 290 |
+
Returns the appended entry. Raises nothing the caller cannot handle: an unknown agent is a
|
| 291 |
+
`ValueError`, because a server-side caller has no HTTP response to put a 404 into.
|
| 292 |
+
"""
|
| 293 |
+
agent_id = str(agent_id or "")
|
| 294 |
+
if not _known_agent(runtime, agent_id):
|
| 295 |
+
raise ValueError(f"no agent {agent_id!r} in this workspace")
|
| 296 |
+
path = normalize_path(path)
|
| 297 |
+
if path is None:
|
| 298 |
+
raise ValueError("that is not an acceptable harness path")
|
| 299 |
+
body = str(body or "")
|
| 300 |
+
if len(body.encode("utf-8")) > MAX_BODY_BYTES:
|
| 301 |
+
raise ValueError("harness body is over the size limit")
|
| 302 |
+
kind = author_kind if author_kind in AUTHOR_KINDS else "agent"
|
| 303 |
+
author = str(author or agent_id)
|
| 304 |
+
# β THE FILE-COUNT CEILING IS CHECKED HERE, NOT INSIDE `_set`. An exception raised inside a
|
| 305 |
+
# store `update` callback propagates out of a half-run read-modify-write, and the one thing a
|
| 306 |
+
# refusal must never do is leave the caller unsure whether the write happened.
|
| 307 |
+
existing = _files(runtime, agent_id)
|
| 308 |
+
if path not in existing and len(existing) >= MAX_FILES:
|
| 309 |
+
raise ValueError(f"this agent already has {MAX_FILES} harness files")
|
| 310 |
+
appended = {}
|
| 311 |
+
|
| 312 |
+
def _set(cur):
|
| 313 |
+
cur = dict(cur or {})
|
| 314 |
+
files = dict(cur.get(agent_id) or {}) if isinstance(cur.get(agent_id), dict) else {}
|
| 315 |
+
record = dict(files[path]) if isinstance(files.get(path), dict) else _blank(path, author, kind)
|
| 316 |
+
appended.clear()
|
| 317 |
+
appended.update(_append(record, body, author, kind, restored_from))
|
| 318 |
+
files[path] = record
|
| 319 |
+
cur[agent_id] = files
|
| 320 |
+
return cur
|
| 321 |
+
|
| 322 |
+
runtime.update(HARNESS_KEY, _set, flush="sync")
|
| 323 |
+
return appended
|
| 324 |
+
|
| 325 |
+
|
| 326 |
+
# ββ the routes ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 327 |
+
@router.get("/agents/{agent_id}/harness")
|
| 328 |
+
def get_harness(agent_id: str, path: str = "", version: int = 0,
|
| 329 |
+
session: Session = Depends(require_session)):
|
| 330 |
+
"""The list, one file, or one older body β decided by the query string (C4).
|
| 331 |
+
|
| 332 |
+
β ADMIN-GATED LIKE EVERY OTHER AGENT DOOR (`routes_slack`), and for a stronger reason than
|
| 333 |
+
consistency: a harness file is what the agent is INSTRUCTED to do, so writing one is closer to
|
| 334 |
+
editing a permission than to editing a document.
|
| 335 |
+
"""
|
| 336 |
+
agent_id = str(agent_id or "")
|
| 337 |
+
agent_wall(session, agent_id)
|
| 338 |
+
files = _files(session.runtime, agent_id)
|
| 339 |
+
if not path:
|
| 340 |
+
rows = [_row(rec) for _p, rec in sorted(files.items()) if isinstance(rec, dict)]
|
| 341 |
+
return {"agent": agent_id, "files": rows, "limits": _limits()}
|
| 342 |
+
|
| 343 |
+
wanted = _clean_path(path)
|
| 344 |
+
record = files.get(wanted)
|
| 345 |
+
if not isinstance(record, dict):
|
| 346 |
+
raise err(404, "no_file", f"this agent has no harness file at {wanted}")
|
| 347 |
+
versions = record.get("versions") if isinstance(record.get("versions"), list) else []
|
| 348 |
+
if version:
|
| 349 |
+
for entry in versions:
|
| 350 |
+
if isinstance(entry, dict) and int(entry.get("version") or 0) == int(version):
|
| 351 |
+
return {"agent": agent_id, **_row(record), "body": entry.get("body") or "",
|
| 352 |
+
"atVersion": int(version), "history": _version_rows(record),
|
| 353 |
+
"limits": _limits()}
|
| 354 |
+
# β A TRIMMED VERSION IS A NAMED REFUSAL, NOT A 404 SHAPED LIKE A TYPO. The client asked
|
| 355 |
+
# for something that existed and no longer does, and telling it apart from a bad number is
|
| 356 |
+
# the difference between "roll back to v3" failing loudly and failing as if v3 never was.
|
| 357 |
+
trimmed = int(record.get("trimmed") or 0)
|
| 358 |
+
if trimmed and int(version) <= trimmed:
|
| 359 |
+
raise err(410, "version_trimmed",
|
| 360 |
+
f"version {int(version)} is older than the {MAX_VERSIONS} versions kept "
|
| 361 |
+
f"for this file, and its body is gone")
|
| 362 |
+
raise err(404, "no_version", f"this file has no version {int(version)}")
|
| 363 |
+
head = _head(record) or {}
|
| 364 |
+
return {"agent": agent_id, **_row(record), "body": head.get("body") or "",
|
| 365 |
+
"atVersion": int(head.get("version") or 0), "history": _version_rows(record),
|
| 366 |
+
"limits": _limits()}
|
| 367 |
+
|
| 368 |
+
|
| 369 |
+
@router.put("/agents/{agent_id}/harness")
|
| 370 |
+
def put_harness(agent_id: str, body: dict = Body(default=None),
|
| 371 |
+
session: Session = Depends(require_session)):
|
| 372 |
+
"""Write a NEW version of one harness file. `{path, body, authorKind?, restoredFrom?}` (C4).
|
| 373 |
+
|
| 374 |
+
β THERE IS NO OVERWRITE HERE AND THERE IS NO EDIT-IN-PLACE. R8's "every version kept" is not a
|
| 375 |
+
UI affordance; it is this function refusing to have a code path that replaces a body. A
|
| 376 |
+
roll-back arrives as an ordinary write carrying `restoredFrom`, so the history records that
|
| 377 |
+
somebody went back rather than pretending the intervening versions never happened.
|
| 378 |
+
"""
|
| 379 |
+
agent_id = str(agent_id or "")
|
| 380 |
+
agent_wall(session, agent_id)
|
| 381 |
+
body = body if isinstance(body, dict) else {}
|
| 382 |
+
path = _clean_path(body.get("path"))
|
| 383 |
+
text = body.get("body")
|
| 384 |
+
if not isinstance(text, str):
|
| 385 |
+
raise err(400, "no_body", "a harness file needs a body, even an empty one")
|
| 386 |
+
if len(text.encode("utf-8")) > MAX_BODY_BYTES:
|
| 387 |
+
raise err(413, "body_too_long",
|
| 388 |
+
f"a harness file is at most {MAX_BODY_BYTES // 1024} KB; this one is larger")
|
| 389 |
+
|
| 390 |
+
# β THE CALLER DECLARES THE PROVENANCE AND THE SERVER STAMPS THE IDENTITY. `authorKind` is a
|
| 391 |
+
# claim about who WROTE the text (the person, or the agent in the chat panel); `author` is the
|
| 392 |
+
# session and is never read off the request. Nothing is bought by forging the first β both
|
| 393 |
+
# principals may write (R8) β and everything would be bought by forging the second.
|
| 394 |
+
kind = str(body.get("authorKind") or "user").strip().lower()
|
| 395 |
+
if kind not in AUTHOR_KINDS:
|
| 396 |
+
raise err(400, "bad_author_kind", "authorKind is either user or agent")
|
| 397 |
+
restored = body.get("restoredFrom")
|
| 398 |
+
try:
|
| 399 |
+
restored = int(restored) if restored else None
|
| 400 |
+
except (TypeError, ValueError):
|
| 401 |
+
raise err(400, "bad_version", "restoredFrom must be a version number")
|
| 402 |
+
|
| 403 |
+
files = _files(session.runtime, agent_id)
|
| 404 |
+
if path not in files and len(files) >= MAX_FILES:
|
| 405 |
+
raise err(409, "too_many_files",
|
| 406 |
+
f"this agent already has {MAX_FILES} harness files; delete one to add another")
|
| 407 |
+
try:
|
| 408 |
+
record_version(session.runtime, agent_id, path, text,
|
| 409 |
+
author=session.uname, author_kind=kind, restored_from=restored)
|
| 410 |
+
except ValueError as exc:
|
| 411 |
+
raise err(400, "refused", str(exc))
|
| 412 |
+
|
| 413 |
+
fresh = _files(session.runtime, agent_id).get(path)
|
| 414 |
+
if not isinstance(fresh, dict) or not _head(fresh):
|
| 415 |
+
# The store took the write and did not record it. A 200 here would tell an administrator
|
| 416 |
+
# their skill file was saved when it was not β the shape `routes_slack` refuses too.
|
| 417 |
+
raise err(503, "store_unavailable", "the harness file was NOT saved")
|
| 418 |
+
head = _head(fresh)
|
| 419 |
+
return {"agent": agent_id, **_row(fresh), "body": head.get("body") or "",
|
| 420 |
+
"atVersion": int(head.get("version") or 0), "history": _version_rows(fresh),
|
| 421 |
+
"limits": _limits()}
|
| 422 |
+
|
| 423 |
+
|
| 424 |
+
@router.delete("/agents/{agent_id}/harness")
|
| 425 |
+
def delete_harness(agent_id: str, path: str = "", session: Session = Depends(require_session)):
|
| 426 |
+
"""Drop one harness file AND its history.
|
| 427 |
+
|
| 428 |
+
β THIS IS NOT THE THING R8 FORBIDS. R8 forbids a ROLL-BACK implemented as a delete β losing
|
| 429 |
+
versions as a side effect of an edit. Deleting a file is a person deciding the file should not
|
| 430 |
+
exist, which is a different act with a different button, and a store with no way to remove a
|
| 431 |
+
file is one where a typo'd path is permanent.
|
| 432 |
+
"""
|
| 433 |
+
agent_id = str(agent_id or "")
|
| 434 |
+
agent_wall(session, agent_id)
|
| 435 |
+
wanted = _clean_path(path)
|
| 436 |
+
if wanted not in _files(session.runtime, agent_id):
|
| 437 |
+
raise err(404, "no_file", f"this agent has no harness file at {wanted}")
|
| 438 |
+
|
| 439 |
+
def _set(cur):
|
| 440 |
+
cur = dict(cur or {})
|
| 441 |
+
files = dict(cur.get(agent_id) or {}) if isinstance(cur.get(agent_id), dict) else {}
|
| 442 |
+
files.pop(wanted, None)
|
| 443 |
+
# An agent with no harness files leaves NO key behind. An empty dict per agent id is how a
|
| 444 |
+
# bucket accumulates a row for every agent anybody ever opened the tab on.
|
| 445 |
+
if files:
|
| 446 |
+
cur[agent_id] = files
|
| 447 |
+
else:
|
| 448 |
+
cur.pop(agent_id, None)
|
| 449 |
+
return cur
|
| 450 |
+
|
| 451 |
+
session.runtime.update(HARNESS_KEY, _set, flush="sync")
|
| 452 |
+
return {"agent": agent_id, "deleted": wanted,
|
| 453 |
+
"files": [_row(rec) for _p, rec in sorted(_files(session.runtime, agent_id).items())
|
| 454 |
+
if isinstance(rec, dict)],
|
| 455 |
+
"limits": _limits()}
|
api/routes_alerts.py
CHANGED
|
@@ -1,668 +1,668 @@
|
|
| 1 |
-
"""routes_alerts.py β the Alerts module (wave 20, owner item 25, contract C-ALERT).
|
| 2 |
-
|
| 3 |
-
GET /api/v1/alerts -> {alerts:[...]}
|
| 4 |
-
POST /api/v1/alerts <- {viewId, topic, label?}
|
| 5 |
-
DELETE /api/v1/alerts/{alert_id}
|
| 6 |
-
POST /api/v1/alerts/{alert_id}/run -> evaluate now (the pane's manual refresh)
|
| 7 |
-
GET /api/v1/notifications -> {unread, items:[...]}
|
| 8 |
-
POST /api/v1/notifications/read <- {ids:[...]|null, read?:bool}
|
| 9 |
-
|
| 10 |
-
The semantics β an alert is a view plus a remembered matched set, a notification is a NEW
|
| 11 |
-
ENTRANT, and the first evaluation seeds silently β live in `core.alerts` with the reasoning.
|
| 12 |
-
This file owns the two things a route must: WHO may do it, and HOW the view gets evaluated.
|
| 13 |
-
|
| 14 |
-
β **THE EVALUATION RUNS AS THE ALERT'S OWNER, NOT AS THE CALLER.** `_run_alert` builds the pool
|
| 15 |
-
for `rec['owner']`, never for whoever tripped the write hook. Any other choice leaks: a
|
| 16 |
-
full-access admin editing a cell would otherwise evaluate a BU-scoped user's alert over the whole
|
| 17 |
-
book, and the notification would name customers that user may not see β a permission leak wearing
|
| 18 |
-
a notification's clothes. The owner's own scope is the only correct basis for their alert.
|
| 19 |
-
|
| 20 |
-
β **AN ALERT IS NOT A SECOND READ PATH.** It resolves rows through the same
|
| 21 |
-
`routes_customers.grid_assembly` / `routes_tables.ut_assembly` the grid uses, so a row that an
|
| 22 |
-
alert can see is by construction a row its owner could open. Re-implementing the filter here
|
| 23 |
-
would be a second definition of "matches", and those two would drift.
|
| 24 |
-
"""
|
| 25 |
-
import re
|
| 26 |
-
|
| 27 |
-
from fastapi import APIRouter, Body, Depends
|
| 28 |
-
|
| 29 |
-
import core.alerts as alerts
|
| 30 |
-
from deps import Session, err, require_session
|
| 31 |
-
|
| 32 |
-
router = APIRouter(prefix="/api/v1")
|
| 33 |
-
|
| 34 |
-
#: The alert-bearing surfaces. `ut_` tables are admitted by prefix, like everywhere else.
|
| 35 |
-
_TOPICS = ("customer", "product")
|
| 36 |
-
|
| 37 |
-
# ββ ββ WAVE 32 Β· T20 Β· CONTRACT C3 β THE INBOX SHAPE, DERIVED ON READ ββββββββββββββββββββββββ
|
| 38 |
-
#
|
| 39 |
-
# `GET /notifications` gains `subject`, `kind` and `target` per item (`read` was always there).
|
| 40 |
-
#
|
| 41 |
-
# β DERIVED, NEVER STORED, AND THAT IS THE WHOLE OF WHY THIS WAVE EXISTS. Stamping the three
|
| 42 |
-
# keys onto the record at write time would give them to notifications minted AFTER the deploy and
|
| 43 |
-
# to nothing else β every notification already sitting in every tenant's inbox would open nothing,
|
| 44 |
-
# and the feature would be correct in the source and absent from the product
|
| 45 |
-
# ([[a-migration-that-runs-on-the-next-write]], D-201). A read-side derivation reaches a
|
| 46 |
-
# notification queued last month. It also keeps the store shape out of `core/alerts.py`, which is
|
| 47 |
-
# another lane's file this wave β but that is the convenience, not the reason.
|
| 48 |
-
#
|
| 49 |
-
# β TWO PRODUCERS WRITE TWO SHAPES into one inbox, and the vocabulary below is what tells them
|
| 50 |
-
# apart. `_queue` (a record ENTERED a watched view) sets `topic`+`viewId`. `notify()` sets
|
| 51 |
-
# `topic='automation'` and puts the producer's key in `alertId`, leaving `viewId` empty. Deciding
|
| 52 |
-
# here means the client branches on ONE field instead of re-deriving the same split.
|
| 53 |
-
#
|
| 54 |
-
# β **D-101 IS CLOSED HERE, BY SUBTRACTION.** There was a THIRD shape β `kind='automation_review'`
|
| 55 |
-
# + `autoId`, a card arriving at a review stage β and its producer `notify_review` was deleted by
|
| 56 |
-
# W27/R3 with the review lanes. `automation_engine.py`'s own tombstone (search `notify_review`)
|
| 57 |
-
# records the 2026-08-12 sweep: **no `.py` file anywhere produces one**, while the client branch,
|
| 58 |
-
# its route and three gate legs stayed fully alive. D-101's exit condition is *"the client review
|
| 59 |
-
# branch is deleted in the same change as any remaining residue, OR `notify_review` gains its real
|
| 60 |
-
# caller"* β the residue is zero, so the branch goes. It is not carried into the Inbox: a stored
|
| 61 |
-
# review notification (if any survives in a tenant from the wave-23 era) derives as an ordinary
|
| 62 |
-
# `alert` with no target, i.e. an honest unclickable row, which is correct β the board it pointed
|
| 63 |
-
# at was deleted two waves ago.
|
| 64 |
-
|
| 65 |
-
#: C3's `kind` vocabulary. Plain strings on the wire β the client must never union over them
|
| 66 |
-
#: (alertsModel's wave-9 law: a client union turns "the server grew a kind" into a dropped row).
|
| 67 |
-
NOTIF_KIND_ALERT = "alert"
|
| 68 |
-
NOTIF_KIND_AUTOMATION = "automation"
|
| 69 |
-
NOTIF_KIND_SHARE = "share"
|
| 70 |
-
|
| 71 |
-
#: C3's `target.module` vocabulary, and the automation sub-selection.
|
| 72 |
-
TARGET_MODULE_DATABASE = "database"
|
| 73 |
-
TARGET_MODULE_AUTOMATION = "automation"
|
| 74 |
-
TARGET_TAB_RUNS = "runs"
|
| 75 |
-
|
| 76 |
-
#: The topic `notify()` carries for a SHARE (W32-T28 writes it; nothing does yet, and a kind with
|
| 77 |
-
#: no producer is a string that reads as a feature β the reason this constant is named here and
|
| 78 |
-
#: cited from `routes_shares` rather than typed twice).
|
| 79 |
-
SHARE_TOPIC = "share"
|
| 80 |
-
|
| 81 |
-
#: `core.alerts.notify`'s default topic for a run outcome. Mirrors `inboxModel.AUTOMATION_TOPIC`.
|
| 82 |
-
AUTOMATION_TOPIC = "automation"
|
| 83 |
-
|
| 84 |
-
_UT_TOPIC = re.compile(r"ut_[A-Za-z0-9_]+\Z")
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
def route_for_topic(topic):
|
| 88 |
-
"""A grid SCOPE key -> the registry route that renders it, or None.
|
| 89 |
-
|
| 90 |
-
β THE SAME TABLE AS `alertsModel.routeForTopic`, and the parity is GATED
|
| 91 |
-
(`verify_alerts.py`'s vocabulary scan) rather than trusted. The two built-ins are the only
|
| 92 |
-
pair that differ β the registry names the surface (`customer_data`) while the grid names the
|
| 93 |
-
scope (`customer`) β so a topic passed through as a route sends every click to a page that
|
| 94 |
-
does not exist. `None` for anything else: a target this product cannot resolve must be ABSENT
|
| 95 |
-
rather than plausible, because an absent target renders as a row that does not pretend to be
|
| 96 |
-
clickable, and a wrong one renders as a click that silently goes nowhere.
|
| 97 |
-
"""
|
| 98 |
-
t = str(topic or "").strip()
|
| 99 |
-
if t == "customer":
|
| 100 |
-
return "customer_data"
|
| 101 |
-
if t == "product":
|
| 102 |
-
return "product_data"
|
| 103 |
-
if _UT_TOPIC.match(t):
|
| 104 |
-
return t
|
| 105 |
-
return None
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
def _refusal_code(exc):
|
| 109 |
-
"""The `error.code` an `HTTPException` raised by `deps.err()` carries, or `""`.
|
| 110 |
-
|
| 111 |
-
β W32-T22. Four refusals travel up the assembly chain β `unknown_table` (404), `forbidden`
|
| 112 |
-
(403), `window_required` (409) and `store_not_ready` (503) β and each already names its own
|
| 113 |
-
cause. Anything that reduces all four to one word is throwing away the only information the
|
| 114 |
-
reader could have acted on. Returns `""` for a plain exception, so a caller can tell
|
| 115 |
-
"refused, and here is why" apart from "broke, and we do not know why".
|
| 116 |
-
"""
|
| 117 |
-
detail = getattr(exc, "detail", None)
|
| 118 |
-
if isinstance(detail, dict):
|
| 119 |
-
inner = detail.get("error")
|
| 120 |
-
if isinstance(inner, dict):
|
| 121 |
-
return str(inner.get("code") or "")
|
| 122 |
-
return ""
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
def notification_view(item):
|
| 126 |
-
"""One STORED notification -> the shape the Inbox renders. PURE, and total.
|
| 127 |
-
|
| 128 |
-
Never raises and never drops a row: an item it cannot classify comes back as an `alert` with
|
| 129 |
-
no `target`, which the client renders as an unclickable row rather than hiding. An inbox that
|
| 130 |
-
silently omits what it does not understand is the one failure a reader cannot detect.
|
| 131 |
-
"""
|
| 132 |
-
if not isinstance(item, dict):
|
| 133 |
-
return item
|
| 134 |
-
topic = str(item.get("topic") or "").strip()
|
| 135 |
-
alert_id = str(item.get("alertId") or "").strip()
|
| 136 |
-
|
| 137 |
-
# β THE ID TEST IS HALF OF EVERY BRANCH, and it is the load-bearing half. A row whose topic
|
| 138 |
-
# says `automation` but whose producer key never arrived (a truncated payload, a server
|
| 139 |
-
# mid-deploy) would otherwise be handed a target naming NOTHING β a click that appears to work
|
| 140 |
-
# and silently does not, which is this repo's most-repeated failure shape. Failing the test
|
| 141 |
-
# drops it to the `alert` branch, where `route_for_topic` refuses out loud by answering None.
|
| 142 |
-
if topic == AUTOMATION_TOPIC and alert_id:
|
| 143 |
-
kind = NOTIF_KIND_AUTOMATION
|
| 144 |
-
target = {"module": TARGET_MODULE_AUTOMATION, "id": alert_id, "tab": TARGET_TAB_RUNS}
|
| 145 |
-
elif topic == SHARE_TOPIC and alert_id:
|
| 146 |
-
# β W32-T28: the sharer writes `key=<the ROUTE to open>` and, for a shared VIEW,
|
| 147 |
-
# `row_id=<the view to select>`.
|
| 148 |
-
#
|
| 149 |
-
# β `key` IS ALREADY A ROUTE, NOT A RAW OBJECT ID, and the first version of this got it
|
| 150 |
-
# wrong in a way worth recording: a shared VIEW put the VIEW's id in `alertId`, so the
|
| 151 |
-
# target read `{module: "database", id: "view_42"}` β an instruction to open a database
|
| 152 |
-
# called `view_42`. It looked right in the payload and would have opened nothing. The
|
| 153 |
-
# producer resolves the object to its topic and hands over the route; this branch only
|
| 154 |
-
# shapes what it is given.
|
| 155 |
-
kind = NOTIF_KIND_SHARE
|
| 156 |
-
row_id = str(item.get("rowId") or "").strip()
|
| 157 |
-
target = {"module": TARGET_MODULE_DATABASE, "id": alert_id,
|
| 158 |
-
**({"tab": row_id} if row_id else {})}
|
| 159 |
-
else:
|
| 160 |
-
kind = NOTIF_KIND_ALERT
|
| 161 |
-
route = route_for_topic(topic)
|
| 162 |
-
view_id = str(item.get("viewId") or "").strip()
|
| 163 |
-
target = None if route is None else (
|
| 164 |
-
{"module": TARGET_MODULE_DATABASE, "id": route,
|
| 165 |
-
**({"tab": view_id} if view_id else {})})
|
| 166 |
-
|
| 167 |
-
# The email split: `subject` is the HEADER (what this is about β the alert, the automation,
|
| 168 |
-
# the database), `label` stays the BODY (what happened β the record that entered, the run
|
| 169 |
-
# summary). They were one field, which is why a notification read as a sentence with no
|
| 170 |
-
# sender and the pane could not be laid out like mail.
|
| 171 |
-
subject = str(item.get("alertLabel") or "").strip() or str(item.get("label") or "").strip()
|
| 172 |
-
# β `kind` is OVERWRITTEN, not merged. There was one stored value (`automation_review`) and it
|
| 173 |
-
# is D-101's dead one; leaving it through would give the client two vocabularies for one
|
| 174 |
-
# question, which is the defect this wave's item 6 is about in a different file.
|
| 175 |
-
# ββ W33-T28 (owner: "the Inbox reads like email") β THE SENDER, WHICH DID NOT EXIST.
|
| 176 |
-
#
|
| 177 |
-
# β A `verifier` reading the finished wave-32 surface found that the row's sender POSITION was
|
| 178 |
-
# occupied by `kindLabel(n.kind)` β the literals "Alert" / "Automation" / "Shared with you" β
|
| 179 |
-
# i.e. a CATEGORY standing where a who belongs, and no sender field anywhere on the wire, in
|
| 180 |
-
# the model or in the markup. Mail has a from. This is it.
|
| 181 |
-
#
|
| 182 |
-
# β IT IS DERIVED HERE, NOT STORED, FOR EVERY KIND BUT ONE β and the exception is the point.
|
| 183 |
-
# An alert firing and an automation landing rows have no person behind them; their honest
|
| 184 |
-
# sender is the machine that did it, named as the thing the reader recognises. A SHARE has a
|
| 185 |
-
# real person, and only the producer knows who: `routes_shares.py` writes it as `actor` and
|
| 186 |
-
# this reads it back. β It is NOT parsed out of the body prose ("<name> shared this with
|
| 187 |
-
# you") β a sender recovered by regexing a sentence breaks the first time the sentence is
|
| 188 |
-
# reworded, and it would break silently, in the header.
|
| 189 |
-
#
|
| 190 |
-
# β FALLS BACK, NEVER BLANK. A share queued BEFORE `actor` existed has none, and a row with an
|
| 191 |
-
# empty from column reads as a broken inbox rather than as an old notification.
|
| 192 |
-
actor = str(item.get("actor") or "").strip()
|
| 193 |
-
if kind == NOTIF_KIND_SHARE:
|
| 194 |
-
sender = actor or "A teammate"
|
| 195 |
-
elif kind == NOTIF_KIND_AUTOMATION:
|
| 196 |
-
# β "Agents", not "Automation" (W34-T40, corrected at QA 2026-08-17). The client's
|
| 197 |
-
# `senderOf` already falls back to `AGENTS_MODULE_LABEL` β but `if (sent) return sent`
|
| 198 |
-
# runs FIRST, so this server literal won and every actor-less automation notification
|
| 199 |
-
# showed the retired module name in the inbox's From column.
|
| 200 |
-
sender = actor or "Agents"
|
| 201 |
-
else:
|
| 202 |
-
sender = actor or "Alerts"
|
| 203 |
-
out = {**item, "read": bool(item.get("read")), "kind": kind,
|
| 204 |
-
"subject": subject or "Notification", "sender": sender}
|
| 205 |
-
if target is not None:
|
| 206 |
-
out["target"] = target
|
| 207 |
-
return out
|
| 208 |
-
|
| 209 |
-
|
| 210 |
-
def inbox_view(box):
|
| 211 |
-
"""`core.alerts.inbox()`'s answer, with every item put through {@link notification_view}.
|
| 212 |
-
|
| 213 |
-
β `unread` IS NOT RECOUNTED. It is the ACCOUNT's number and `items` is one page of it; a
|
| 214 |
-
recount here would make the badge a function of whatever this page happened to include, which
|
| 215 |
-
is the exact defect `alertsModel.parseInbox`'s own header records from the other side.
|
| 216 |
-
"""
|
| 217 |
-
if not isinstance(box, dict):
|
| 218 |
-
return box
|
| 219 |
-
items = box.get("items")
|
| 220 |
-
if not isinstance(items, list):
|
| 221 |
-
return box
|
| 222 |
-
# ββ W33-T28 / D-208 β THE SERVER'S CLOCK RIDES WITH THE PAGE, and it is what lets the client
|
| 223 |
-
# render a mail-shaped stamp ("09:41" today, "Aug 12" beyond) instead of `2026-08-13 09:41`.
|
| 224 |
-
#
|
| 225 |
-
# β THE CLIENT MUST NOT READ ITS OWN CLOCK, which is D-208's exit condition word for word and
|
| 226 |
-
# is why this key exists rather than a `new Date()` in the browser. `at` is sent as UTC WITH
|
| 227 |
-
# its offset (D-18) precisely so every reader sees the same instant; deciding "is this today?"
|
| 228 |
-
# against a browser clock would re-introduce the drift the offset exists to remove β a reader a
|
| 229 |
-
# day ahead being told an event happened tomorrow [[date-window-vocabulary]]. Both operands
|
| 230 |
-
# now come from the same machine.
|
| 231 |
-
# β Same funnel as the enrichment, so the read door and the mark-read door cannot disagree β
|
| 232 |
-
# the note two lines up records what happened last time only one of them was enriched.
|
| 233 |
-
return {**box, "now": alerts._now_iso(),
|
| 234 |
-
"items": [notification_view(n) for n in items]}
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
def _view_by_id(g, view_id):
|
| 238 |
-
"""One saved view out of an assembly, by id. `None` when there is no such view.
|
| 239 |
-
|
| 240 |
-
ββ W33-T29 (owner: *"Alert me about new records"* answering "Something went wrong") β THIS
|
| 241 |
-
FUNCTION EXISTS BECAUSE TWO CALL SITES BOTH WROTE `(g.get("views") or {}).get(view_id)`, AND
|
| 242 |
-
`g["views"] IS A LIST`. `aios_grid.views_from_defs` returns `[{...}]`, `workspace_wire` passes
|
| 243 |
-
it straight out and both `ut_assembly` and `grid_assembly` return it unchanged β so `.get` on
|
| 244 |
-
it raises `AttributeError`, and `views_from_defs` always returns at least one element, so the
|
| 245 |
-
`or {}` never fires. **It raised on EVERY call, on every topic, since wave 20.**
|
| 246 |
-
|
| 247 |
-
β AND THE TWO SITES FAILED DIFFERENTLY, WHICH IS WHY ONLY ONE WAS EVER REPORTED. In
|
| 248 |
-
`_require_filtered_view` the raise lands ABOVE the handler's own `try`, so it leaves as a bare
|
| 249 |
-
FastAPI 500 and the client's `errorMessage` turns any 5xx into *"Something went wrong on our
|
| 250 |
-
side"* β the exact sentence the owner reported (D-107's shape, again: an attribute error above
|
| 251 |
-
the guard arrives as plain text rather than as our envelope). In `_evaluate` the identical
|
| 252 |
-
line is swallowed by `/notifications`' `except Exception: continue`, so **every stored
|
| 253 |
-
view-alert was silently dropped from the Inbox** and nobody had anything to report at all.
|
| 254 |
-
One expression, one loud symptom and one silent one.
|
| 255 |
-
|
| 256 |
-
β SO IT IS A FUNCTION, NOT TWO FIXED LINES. Two copies of "find the view" is what let one site
|
| 257 |
-
be discussed for three waves while its twin went unnoticed [[one-question-two-normalizers]].
|
| 258 |
-
|
| 259 |
-
β It accepts a dict too, and that is not defensive noise: `verify_alerts`' door fixture was
|
| 260 |
-
keyed `{id: view}` β which is precisely why the gate was green while production raised on
|
| 261 |
-
every call. The fixture is moving to the production shape in this same change, and tolerating
|
| 262 |
-
both here means a caller that legitimately holds one cannot resurrect the bug.
|
| 263 |
-
"""
|
| 264 |
-
want = str(view_id or "")
|
| 265 |
-
if not want:
|
| 266 |
-
return None
|
| 267 |
-
views = (g or {}).get("views")
|
| 268 |
-
if isinstance(views, dict):
|
| 269 |
-
found = views.get(want)
|
| 270 |
-
return found if isinstance(found, dict) else None
|
| 271 |
-
if not isinstance(views, list):
|
| 272 |
-
return None
|
| 273 |
-
for v in views:
|
| 274 |
-
if isinstance(v, dict) and str(v.get("id") or "") == want:
|
| 275 |
-
return v
|
| 276 |
-
return None
|
| 277 |
-
|
| 278 |
-
|
| 279 |
-
def _topic_or_400(raw):
|
| 280 |
-
topic = str(raw or "").strip().lower()
|
| 281 |
-
if topic.startswith("ut_") or topic in _TOPICS:
|
| 282 |
-
return topic
|
| 283 |
-
raise err(400, "bad_topic", f"topic must be one of {', '.join(_TOPICS)} or a ut_ table")
|
| 284 |
-
|
| 285 |
-
|
| 286 |
-
def _owner_session(session: Session, owner: str):
|
| 287 |
-
"""A `Session` for the alert's OWNER (see the module note on why the owner, not the caller).
|
| 288 |
-
|
| 289 |
-
β `Session` exposes `uname`/`admin` as PROPERTIES derived from `user`, not as fields β so an
|
| 290 |
-
owner session is built by swapping the `user` RECORD and letting both derive themselves. An
|
| 291 |
-
earlier version passed `uname=`/`admin=` to the constructor, which would have raised on the
|
| 292 |
-
first write hook of the wave; the properties are the single definition of who a session is,
|
| 293 |
-
and going around them is how a session with an admin flag and a non-admin record exists.
|
| 294 |
-
|
| 295 |
-
Returns None when the owner is gone or deactivated β their alerts then stop evaluating rather
|
| 296 |
-
than evaluating as somebody else, which is the fail-closed direction.
|
| 297 |
-
"""
|
| 298 |
-
import core.users as users
|
| 299 |
-
|
| 300 |
-
if str(owner) == str(session.uname):
|
| 301 |
-
return session
|
| 302 |
-
rec = (users.registry() or {}).get(str(owner))
|
| 303 |
-
if not isinstance(rec, dict) or not rec.get("active", True):
|
| 304 |
-
return None
|
| 305 |
-
# `_public` is THE definition of what a session may know about its own account (never a hash
|
| 306 |
-
# or a salt) β the same one `routes_auth` uses. Building the dict by hand here would be a
|
| 307 |
-
# second definition, and the one that leaks is always the copy.
|
| 308 |
-
return Session(tenant=session.tenant, user=users._public(str(owner), rec),
|
| 309 |
-
claims=session.claims, runtime=session.runtime)
|
| 310 |
-
|
| 311 |
-
|
| 312 |
-
def _evaluate(session: Session, rec: dict, assemblies=None):
|
| 313 |
-
"""Resolve `rec`'s view over its topic AS THE ALERT'S OWNER, then fold the result in.
|
| 314 |
-
|
| 315 |
-
ββ W31-T24 β `assemblies` IS A PER-REQUEST MEMO, KEYED `(topic, owner)`, and it is the whole
|
| 316 |
-
of this ticket's server half. `/notifications` re-evaluates EVERY alert inline on read and each
|
| 317 |
-
one built a FULL assembly β the pool, the workspace, `rows_from_pool` over every row. Two
|
| 318 |
-
alerts on one view built that table twice; ten built it ten times. Nothing dedupes them,
|
| 319 |
-
because each `_evaluate` was a closed call.
|
| 320 |
-
β `(topic, owner)` and not `topic`: the assembly is built as the alert's OWNER (see the module
|
| 321 |
-
note β evaluating a BU-scoped user's alert on a full-access admin's pool is a permission leak
|
| 322 |
-
wearing a notification's clothes), so two owners on one topic are two DIFFERENT tables and
|
| 323 |
-
must never share an entry. Getting that key wrong is the one way this optimisation could leak.
|
| 324 |
-
β Passing nothing keeps the old behaviour exactly, which is what the create/run doors want:
|
| 325 |
-
they evaluate ONE alert and a memo for a single call is pure overhead.
|
| 326 |
-
"""
|
| 327 |
-
import aios_grid
|
| 328 |
-
from harness import filter_eval
|
| 329 |
-
|
| 330 |
-
owner_sess = _owner_session(session, rec.get("owner"))
|
| 331 |
-
if owner_sess is None:
|
| 332 |
-
return {"skipped": "owner_unavailable"}
|
| 333 |
-
topic = str(rec.get("topic") or "")
|
| 334 |
-
memo_key = (topic, str(owner_sess.uname))
|
| 335 |
-
g = assemblies.get(memo_key) if isinstance(assemblies, dict) else None
|
| 336 |
-
if g is None:
|
| 337 |
-
try:
|
| 338 |
-
if topic.startswith("ut_"):
|
| 339 |
-
from routes_tables import ut_assembly
|
| 340 |
-
# β `consume_corrections=False`, and the default was a REAL BUG, not a tidy-up.
|
| 341 |
-
# `ut_assembly` defaults it True, so every `/notifications` read CONSUMED the
|
| 342 |
-
# one-shot field-name correction acks for every `ut_` topic that has an alert β
|
| 343 |
-
# taking them from the `/workspace` refresh that exists to show them to the person
|
| 344 |
-
# who made the edit. The customer branch below has always passed False; this one
|
| 345 |
-
# inherited a default nobody re-read. An inbox poll must never consume a one-shot.
|
| 346 |
-
g = ut_assembly(owner_sess, topic,
|
| 347 |
-
storage_key=f"{owner_sess.tenant}:{topic}:{owner_sess.uname}",
|
| 348 |
-
consume_corrections=False)
|
| 349 |
-
else:
|
| 350 |
-
from routes_customers import grid_assembly
|
| 351 |
-
g = grid_assembly(owner_sess, scope=topic, consume_corrections=False)
|
| 352 |
-
except Exception as e: # noqa: BLE001
|
| 353 |
-
# β W32-T22 β SKIPPING IS FINE HERE; SKIPPING ANONYMOUSLY IS NOT. This one must not
|
| 354 |
-
# raise (one bad alert cannot empty an inbox), so unlike `_require_filtered_view` it
|
| 355 |
-
# keeps a blanket catch β but it now reports the refusal's OWN code where there is
|
| 356 |
-
# one. `type(e).__name__` said `HTTPException` for four different causes, and
|
| 357 |
-
# `lastError` is the only place a user ever learns why an alert stopped firing.
|
| 358 |
-
#
|
| 359 |
-
# β `with_rows=True` STAYS on this path, deliberately: unlike the create door, an
|
| 360 |
-
# evaluation genuinely needs the rows to run the filter over. So an alert on a
|
| 361 |
-
# read-through grid is created (T22) and then skips at evaluation with
|
| 362 |
-
# `window_required` naming why β which is D-184's remaining half, and it is a
|
| 363 |
-
# SENTENCE now rather than silence.
|
| 364 |
-
return {"skipped": _refusal_code(e) or "unavailable", "detail": type(e).__name__}
|
| 365 |
-
if isinstance(assemblies, dict):
|
| 366 |
-
assemblies[memo_key] = g
|
| 367 |
-
|
| 368 |
-
view = _view_by_id(g, rec.get("viewId"))
|
| 369 |
-
if not isinstance(view, dict):
|
| 370 |
-
# Deleted, or un-shared out from under the alert. Say so on the RECORD rather than
|
| 371 |
-
# deleting the alert: an alert that silently vanishes is indistinguishable from one that
|
| 372 |
-
# never fires, and the user cannot debug what is not there.
|
| 373 |
-
return {"skipped": "view_missing"}
|
| 374 |
-
|
| 375 |
-
# The SAME row build the grid and `/customers` use β `rows_from_pool` is what puts derived
|
| 376 |
-
# and overlay values on a row. Evaluating a filter against raw pool dicts would silently
|
| 377 |
-
# never match any condition on a user-created or measure column.
|
| 378 |
-
#
|
| 379 |
-
# ββ AND "THE SAME ROW BUILD" WAS NOT TRUE, WHICH MADE EVERY ALERT ON A `ut_*` DATABASE BLIND
|
| 380 |
-
# TO IMPORTED DATA. Found by a verifier driving one real assembly through both paths.
|
| 381 |
-
#
|
| 382 |
-
# `routes_tables.table_rows` β the grid the person is looking at β merges the DEFINITION rows
|
| 383 |
-
# underneath the overlay ("base first, overlay wins"; that merge is itself the fix for owner
|
| 384 |
-
# item 3, *"it all got reseted"*). `_evaluate` is a second copy of that read and never got it:
|
| 385 |
-
# it handed `ws['overlays']` to `rows_from_pool` raw, so for a `ut_*` table every base cell
|
| 386 |
-
# evaluated as BLANK. Measured on one assembly, same view, same rows:
|
| 387 |
-
# rows_src state='unpaid' / 'paid'
|
| 388 |
-
# _evaluate saw state='' / '' β every base cell blank
|
| 389 |
-
# the GRID saw state='unpaid' / 'paid'
|
| 390 |
-
# so `state eq unpaid` matched NOTHING while the view showed one row, and `state isEmpty`
|
| 391 |
-
# matched EVERYTHING while the view showed none. **The alert did not merely miss rows β it
|
| 392 |
-
# inverted.** End to end: a row whose value arrived by import, automation, paste or the create
|
| 393 |
-
# door never fired; only a value typed as a hand EDIT did.
|
| 394 |
-
#
|
| 395 |
-
# β Scope, so nobody widens the fix past its cause: materialised `ut_*` tables are hit;
|
| 396 |
-
# `customer`/`product` are not (their fields are `source: "odoo"` and read off `rows_src`);
|
| 397 |
-
# `ut_odoo_*` never reaches here (`with_rows=True` refuses first and returns
|
| 398 |
-
# `skipped: window_required`).
|
| 399 |
-
# β ORDER IS LOAD-BEARING AND IS THE GRID'S: base underneath, overlay ON TOP. Inverting it
|
| 400 |
-
# would let a stale definition value shadow an edit the user has just made β the same defect
|
| 401 |
-
# `table_rows`' own note records, arriving from the other side.
|
| 402 |
-
_ov = (g.get("ws") or {}).get("overlays") or {}
|
| 403 |
-
_merged = {}
|
| 404 |
-
for _r in g["rows_src"]:
|
| 405 |
-
_pid = str(_r.get("pid"))
|
| 406 |
-
_cells = {k: v for k, v in _r.items() if k != "pid"}
|
| 407 |
-
_o = _ov.get(_pid)
|
| 408 |
-
if isinstance(_o, dict):
|
| 409 |
-
_cells.update(_o)
|
| 410 |
-
_merged[_pid] = _cells
|
| 411 |
-
rows = aios_grid.rows_from_pool(g["rows_src"], g["fields"], _merged,
|
| 412 |
-
derived=g.get("derived"))
|
| 413 |
-
config = view.get("config") or view
|
| 414 |
-
ctx = filter_eval.EvalCtx(
|
| 415 |
-
cohort_sets={str(k): {str(p) for p in (v.get("memberPids") or ())}
|
| 416 |
-
for k, v in (g.get("lists") or {}).items() if isinstance(v, dict)},
|
| 417 |
-
measure_sets=g.get("measure_sets") or {},
|
| 418 |
-
today=g.get("today"))
|
| 419 |
-
pids = filter_eval.visible_pids(config.get("filters") or [], rows, g["fields"], ctx,
|
| 420 |
-
member_pids=config.get("memberPids"))
|
| 421 |
-
labels = {str(r.get("pid")): str(r.get("name") or r.get("pid")) for r in rows}
|
| 422 |
-
return alerts.evaluate(rec.get("id"), [str(p) for p in pids],
|
| 423 |
-
labels=labels, partial=False, st=session.runtime)
|
| 424 |
-
|
| 425 |
-
|
| 426 |
-
@router.get("/alerts")
|
| 427 |
-
def list_alerts(session: Session = Depends(require_session)):
|
| 428 |
-
return {"alerts": alerts.list_alerts(user=session.uname, is_admin=session.admin,
|
| 429 |
-
st=session.runtime)}
|
| 430 |
-
|
| 431 |
-
|
| 432 |
-
@router.post("/alerts")
|
| 433 |
-
def create_alert(body: dict = Body(default=None), session: Session = Depends(require_session)):
|
| 434 |
-
body = body or {}
|
| 435 |
-
view_id = str(body.get("viewId") or "").strip()
|
| 436 |
-
if not view_id:
|
| 437 |
-
raise err(400, "bad_view", "an alert needs the id of the view it watches")
|
| 438 |
-
topic = _topic_or_400(body.get("topic"))
|
| 439 |
-
_require_filtered_view(session, topic, view_id)
|
| 440 |
-
import uuid
|
| 441 |
-
aid = f"al_{uuid.uuid4().hex[:12]}"
|
| 442 |
-
rec = alerts.create(aid, view_id=view_id, topic=topic, owner=session.uname,
|
| 443 |
-
label=body.get("label") or "", st=session.runtime)
|
| 444 |
-
# SEED IMMEDIATELY, so the alert starts from "everything currently matching is old news".
|
| 445 |
-
# Deferring this to the first write hook would mean the next edit announces the whole view.
|
| 446 |
-
outcome = _evaluate(session, rec)
|
| 447 |
-
return {"alert": {**rec, "seeded": True}, "first": outcome}
|
| 448 |
-
|
| 449 |
-
|
| 450 |
-
def _require_filtered_view(session: Session, topic: str, view_id: str):
|
| 451 |
-
"""400 unless `view_id` exists on `topic` AND actually narrows something.
|
| 452 |
-
|
| 453 |
-
β AN ALERT ON AN UNFILTERED VIEW IS SILENTLY INCAPABLE OF ALERTING, which is worse than one
|
| 454 |
-
that is refused. `filter_eval` treats an inactive tree as "no narrowing, every row shows"
|
| 455 |
-
(`visible_pids`'s own rule), so such an alert seeds with the entire table and can never see an
|
| 456 |
-
entrant again β there is nothing left to enter. The owner's words are *"when a Record gets
|
| 457 |
-
into that Filter's criteria"*: no criteria, no alert, and said at creation rather than
|
| 458 |
-
discovered by never being notified.
|
| 459 |
-
|
| 460 |
-
`is_rule_active` is the SAME activeness predicate the engine and the column tints use β a
|
| 461 |
-
half-typed rule is not a filter, and this must agree with what actually narrows or it would
|
| 462 |
-
accept a view whose one rule the engine then ignores.
|
| 463 |
-
|
| 464 |
-
ββ WAVE 32 Β· T22 (owner item 17) β THIS FUNCTION WAS THE ERROR. Two defects, stacked, and
|
| 465 |
-
the second one hid the first.
|
| 466 |
-
|
| 467 |
-
(1) **IT ASKED FOR EVERY ROW OF A TABLE IT NEVER LOOKS AT.** The only thing read below is
|
| 468 |
-
`g["views"]`. `ut_assembly` defaults `with_rows=True`, so creating an alert on a
|
| 469 |
-
read-through grid built the whole pool β and `scoped_pool` refuses that with
|
| 470 |
-
`409 window_required` over 963,783 rows, exactly as it is supposed to. `with_rows=False`
|
| 471 |
-
(W31-T20's flag, built for precisely this) answers the same question with `scoped_pids`,
|
| 472 |
-
runs the SAME `_defn_or_refuse` wall, and does not refuse. **That is D-184's create half,
|
| 473 |
-
closed** β an alert on a read-through grid can now be made at all.
|
| 474 |
-
(2) **A BLANKET `except Exception` TURNED EVERY NAMED REFUSAL INTO A 503.** `HTTPException`
|
| 475 |
-
is an `Exception`, so `404 unknown_table`, `403 forbidden`, `409 window_required` and
|
| 476 |
-
`503 store_not_ready` β four refusals that each say what is wrong β were all replaced by
|
| 477 |
-
*"the table is unavailable β try again in a moment"*. β AND THAT SENTENCE NEVER REACHED
|
| 478 |
-
A USER EITHER: `alertsApi.errorMessage` discards the text of any status β₯ 500 by design
|
| 479 |
-
(a 5xx body is the server's internals), substituting *"Something went wrong on our
|
| 480 |
-
side."* β which is the owner's screenshot, word for word. A knowable cause returned as a
|
| 481 |
-
5xx is invisible by construction, so re-wording the 503 could never have fixed this.
|
| 482 |
-
β The except is narrowed, not deleted: an UNEXPECTED failure is still a 503, because that is
|
| 483 |
-
honest. What it may no longer do is catch a refusal that already knows its own name.
|
| 484 |
-
"""
|
| 485 |
-
from fastapi import HTTPException
|
| 486 |
-
|
| 487 |
-
from harness import filter_eval
|
| 488 |
-
|
| 489 |
-
try:
|
| 490 |
-
if topic.startswith("ut_"):
|
| 491 |
-
from routes_tables import ut_assembly
|
| 492 |
-
# β `consume_corrections=False` β the customer branch has always passed it and this
|
| 493 |
-
# one inherited a default nobody re-read. Creating an alert must not eat the one-shot
|
| 494 |
-
# field-name correction acks belonging to the `/workspace` refresh that exists to show
|
| 495 |
-
# them to the person who made the edit. Same defect `_evaluate`'s header records.
|
| 496 |
-
g = ut_assembly(session, topic,
|
| 497 |
-
storage_key=f"{session.tenant}:{topic}:{session.uname}",
|
| 498 |
-
consume_corrections=False, with_rows=False)
|
| 499 |
-
else:
|
| 500 |
-
from routes_customers import grid_assembly
|
| 501 |
-
g = grid_assembly(session, scope=topic, consume_corrections=False)
|
| 502 |
-
except HTTPException:
|
| 503 |
-
raise # it already names its own cause
|
| 504 |
-
except Exception as e: # noqa: BLE001
|
| 505 |
-
# Genuinely unexpected. Still a 503, and now it carries the exception TYPE β without it,
|
| 506 |
-
# the one path that reaches this branch is also the one path with nothing to debug from.
|
| 507 |
-
raise err(503, "unavailable",
|
| 508 |
-
f"the table could not be read ({type(e).__name__}) β try again in a moment")
|
| 509 |
-
view = _view_by_id(g, view_id)
|
| 510 |
-
if not isinstance(view, dict):
|
| 511 |
-
raise err(404, "no_view", "that view does not exist on this table")
|
| 512 |
-
nodes, _conj = filter_eval.tree_parts((view.get("config") or view).get("filters") or [])
|
| 513 |
-
|
| 514 |
-
# ββ WAVE 33 Β· T29 β **CORRECTION: THE BLOCK BELOW IS TRUE ABOUT THE CODE AND FALSE ABOUT
|
| 515 |
-
# PRODUCTION, AND IT MUST BE READ SECOND.** It claims the missing-argument `TypeError` "IS
|
| 516 |
-
# owner item 17" β the owner's *"Something went wrong"*. It was not, and it could not have
|
| 517 |
-
# been: at `cbcf005`, the build the owner was using, the dict-read on `views` sat ~10 lines
|
| 518 |
-
# ABOVE this call and raised `AttributeError` on EVERY request, so the walk never reached the
|
| 519 |
-
# leaf and the arity bug was unreachable. `_view_by_id`'s own header records that fix.
|
| 520 |
-
#
|
| 521 |
-
# β WHY THE STALE PARAGRAPH STAYS RATHER THAN GETTING DELETED: the arity bug was real, the
|
| 522 |
-
# fix was right, and the three reasons it hid are the most transferable thing in this file.
|
| 523 |
-
# What was wrong is only its CLAIM TO BE THE CAUSE. Two comment blocks in one function each
|
| 524 |
-
# naming themselves as the origin of the same screenshot are mutually exclusive, and the next
|
| 525 |
-
# reader believes whichever they meet first β which is why this correction sits above rather
|
| 526 |
-
# than below. Caught by a verifier that read the SHIPPED file at the deployed commit instead
|
| 527 |
-
# of the working tree. [[grep-output-is-not-source]]
|
| 528 |
-
#
|
| 529 |
-
# ββ WAVE 32 Β· T22 β **THE CALL BELOW WAS MISSING AN ARGUMENT** (and wave 32 believed, wrongly,
|
| 530 |
-
# that this was owner item 17 β see the correction directly above).
|
| 531 |
-
#
|
| 532 |
-
# `is_rule_active(rule, columns)` takes TWO parameters (`harness/filter_sql.py`; every other
|
| 533 |
-
# caller in the repo passes both). This one passed ONE, so the moment the walk reached a LEAF
|
| 534 |
-
# rule it raised `TypeError: is_rule_active() missing 1 required positional argument`.
|
| 535 |
-
#
|
| 536 |
-
# β READ WHAT THAT MEANS BEFORE FIXING ANYTHING ELSE: the walk only reaches a leaf when the
|
| 537 |
-
# view HAS a condition β and a view with a condition is the only kind an alert is allowed on.
|
| 538 |
-
# A view with no filters yields an empty `nodes`, so `_any_active` returns False without ever
|
| 539 |
-
# calling this, and the reader gets the honest 400 `no_filter`. **So the only path that
|
| 540 |
-
# worked was the refusal path: "Alert me about new records" had never once created an alert
|
| 541 |
-
# on a filtered view.** β And the raise lands OUTSIDE the `try` above, so it was not even the
|
| 542 |
-
# 503 β it was a bare FastAPI 500, which `alertsApi.errorMessage` renders as *"Something went
|
| 543 |
-
# wrong on our side. Try again in a moment."*, the owner's screenshot word for word.
|
| 544 |
-
#
|
| 545 |
-
# β THREE THINGS HID IT, and they are worth more than the fix. (1) Python does not check
|
| 546 |
-
# arity until the line RUNS, and this line runs only on the success path of a feature whose
|
| 547 |
-
# every test exercised its refusals. (2) The `no_filter` 400 above it is a real, correct,
|
| 548 |
-
# well-tested refusal, so the door looked alive. (3) `verify_alerts.py` asserts the refusal
|
| 549 |
-
# (`no_filter` reaches the user) and the transport β never a creation. A gate can be green,
|
| 550 |
-
# thorough and honest about everything except the one path the feature exists for.
|
| 551 |
-
#
|
| 552 |
-
# `_columns_map` is the DEFINITION of fields -> the membership set `is_rule_active` looks a
|
| 553 |
-
# column up in; building a second dict here would be a second answer to one question, which
|
| 554 |
-
# is this wave's other headline defect in a different file. Its leading underscore is a real
|
| 555 |
-
# smell and is BOOKED (PENDING, mailbox/C.md) rather than worked around.
|
| 556 |
-
columns = filter_eval._columns_map(g.get("fields") or [])
|
| 557 |
-
|
| 558 |
-
def _any_active(ns):
|
| 559 |
-
for n in ns or ():
|
| 560 |
-
if isinstance(n, dict) and isinstance(n.get("children"), list):
|
| 561 |
-
if _any_active(n["children"]):
|
| 562 |
-
return True
|
| 563 |
-
elif filter_eval.is_rule_active(n, columns):
|
| 564 |
-
return True
|
| 565 |
-
return False
|
| 566 |
-
|
| 567 |
-
if not _any_active(nodes):
|
| 568 |
-
raise err(400, "no_filter",
|
| 569 |
-
"this view has no active filter, so no record can ever ENTER it β add a "
|
| 570 |
-
"condition to the view first, then create the alert")
|
| 571 |
-
|
| 572 |
-
|
| 573 |
-
@router.delete("/alerts/{alert_id}")
|
| 574 |
-
def delete_alert(alert_id: str, session: Session = Depends(require_session)):
|
| 575 |
-
rec = next((r for r in alerts.list_alerts(st=session.runtime)
|
| 576 |
-
if str(r.get("id")) == str(alert_id)), None)
|
| 577 |
-
if rec is None:
|
| 578 |
-
raise err(404, "no_alert", "that alert does not exist")
|
| 579 |
-
if str(rec.get("owner")) != str(session.uname) and not session.admin:
|
| 580 |
-
raise err(403, "not_yours", "only the alert's owner (or an administrator) can delete it")
|
| 581 |
-
alerts.delete(alert_id, st=session.runtime)
|
| 582 |
-
return {"ok": True}
|
| 583 |
-
|
| 584 |
-
|
| 585 |
-
@router.post("/alerts/{alert_id}/run")
|
| 586 |
-
def run_alert(alert_id: str, session: Session = Depends(require_session)):
|
| 587 |
-
rec = next((r for r in alerts.list_alerts(user=session.uname, is_admin=session.admin,
|
| 588 |
-
st=session.runtime)
|
| 589 |
-
if str(r.get("id")) == str(alert_id)), None)
|
| 590 |
-
if rec is None:
|
| 591 |
-
raise err(404, "no_alert", "that alert does not exist")
|
| 592 |
-
return _evaluate(session, rec)
|
| 593 |
-
|
| 594 |
-
|
| 595 |
-
@router.get("/notifications")
|
| 596 |
-
def notifications(session: Session = Depends(require_session)):
|
| 597 |
-
"""The inbox β RE-EVALUATED on read, which is a deliberate design choice.
|
| 598 |
-
|
| 599 |
-
β A-S1-2 RESOLVED THE OTHER WAY, and the reason is structural rather than a shortcut. The
|
| 600 |
-
plan was a push hook: the automation engine calls `after_write` when it lands rows. But
|
| 601 |
-
`run_async` runs on a BACKGROUND THREAD with no `Session` in scope, and an alert must be
|
| 602 |
-
evaluated as its OWNER (see `_evaluate`) β so a push hook would have to mint a session inside
|
| 603 |
-
a worker thread from a tenant runtime, which is exactly the kind of ad-hoc identity
|
| 604 |
-
construction that leaks scope.
|
| 605 |
-
|
| 606 |
-
Pulling on read has none of that: the caller IS a session, the assemblies are already
|
| 607 |
-
scope-cached, and the user cannot observe the difference β an inbox is only ever read by
|
| 608 |
-
someone opening it. The cost is that a notification is minted when you LOOK rather than when
|
| 609 |
-
the row landed, so the `at` stamp is detection time, not arrival time.
|
| 610 |
-
|
| 611 |
-
`after_write` stays exported for the day the engine can hand over a real identity.
|
| 612 |
-
|
| 613 |
-
ββ W31-T24 β ONE ASSEMBLY PER (TOPIC, OWNER), NOT ONE PER ALERT.
|
| 614 |
-
β MEASURED FIRST, AND THE MEASUREMENT CORRECTS AN EARLIER READING OF IT. This route is
|
| 615 |
-
**20 ms in-process and 3,280 ms live** on tenant #0 β but tenant #0 has **ZERO alerts**
|
| 616 |
-
(censused 2026-08-12), so the 20 ms is an EMPTY LOOP and says nothing at all about what the
|
| 617 |
-
re-evaluation costs. The live 3,280 ms is the two store reads either side of that loop. So the
|
| 618 |
-
body below is not slow today; it is UNEXERCISED, and every alert a tenant creates adds a whole
|
| 619 |
-
grid assembly to an inbox poll. The memo turns O(alerts) into O(distinct topic Γ owner), which
|
| 620 |
-
is the difference between "fine" and "three seconds per alert" the day somebody uses the
|
| 621 |
-
feature. β Making the read cheap by evaluating LESS is the obvious wrong fix and is not what
|
| 622 |
-
this does: every alert is still evaluated, against the same rows, in the same order.
|
| 623 |
-
"""
|
| 624 |
-
assemblies = {}
|
| 625 |
-
for rec in alerts.list_alerts(user=session.uname, is_admin=False, st=session.runtime):
|
| 626 |
-
try:
|
| 627 |
-
_evaluate(session, rec, assemblies=assemblies)
|
| 628 |
-
except Exception: # noqa: BLE001
|
| 629 |
-
continue # one bad alert must not empty the pane
|
| 630 |
-
# β W32-T20 (C3): every item leaves through `inbox_view`, so a notification queued before
|
| 631 |
-
# this wave carries a `target` too. See `notification_view`'s header for why it is derived.
|
| 632 |
-
return inbox_view(alerts.inbox(session.uname, st=session.runtime))
|
| 633 |
-
|
| 634 |
-
|
| 635 |
-
@router.post("/notifications/read")
|
| 636 |
-
def read_notifications(body: dict = Body(default=None),
|
| 637 |
-
session: Session = Depends(require_session)):
|
| 638 |
-
body = body or {}
|
| 639 |
-
ids = body.get("ids")
|
| 640 |
-
if ids is not None and not isinstance(ids, list):
|
| 641 |
-
raise err(400, "bad_ids", "ids must be a list, or null to mark every notification")
|
| 642 |
-
# β THE SAME ENRICHMENT ON BOTH DOORS. `mark_read` returns a fresh inbox, and the Inbox
|
| 643 |
-
# module re-renders from it β an un-enriched answer here would strip `target` off every row
|
| 644 |
-
# the moment somebody marked one read, i.e. the feature would work until first use.
|
| 645 |
-
return inbox_view(alerts.mark_read(session.uname, ids, read=bool(body.get("read", True)),
|
| 646 |
-
st=session.runtime))
|
| 647 |
-
|
| 648 |
-
|
| 649 |
-
def after_write(session: Session, topic_key: str):
|
| 650 |
-
"""THE WRITE HOOK β call after a write that could change what a view matches.
|
| 651 |
-
|
| 652 |
-
Exported as a plain function (not a route) so `core.grid_events`' callers and S2's automation
|
| 653 |
-
upserts reach it the same way. It never raises: an alert evaluation failing must not fail the
|
| 654 |
-
edit that triggered it.
|
| 655 |
-
|
| 656 |
-
β W31-T24 β it shares `/notifications`' memo shape for the same reason: a write that changes
|
| 657 |
-
one view can trip several alerts on the SAME topic, and each would otherwise rebuild the table.
|
| 658 |
-
β STILL ZERO PRODUCTION CALLERS (W31-T24 confirmed it; the route docstring above says why the
|
| 659 |
-
push hook was resolved the other way). Booked rather than wired: minting a session inside the
|
| 660 |
-
engine's worker thread is the ad-hoc identity construction this file exists to avoid.
|
| 661 |
-
"""
|
| 662 |
-
try:
|
| 663 |
-
assemblies = {}
|
| 664 |
-
return alerts.after_write(topic_key, st=session.runtime,
|
| 665 |
-
runner=lambda rec: _evaluate(session, rec,
|
| 666 |
-
assemblies=assemblies))
|
| 667 |
-
except Exception: # noqa: BLE001
|
| 668 |
-
return {"evaluated": 0}
|
|
|
|
| 1 |
+
"""routes_alerts.py β the Alerts module (wave 20, owner item 25, contract C-ALERT).
|
| 2 |
+
|
| 3 |
+
GET /api/v1/alerts -> {alerts:[...]}
|
| 4 |
+
POST /api/v1/alerts <- {viewId, topic, label?}
|
| 5 |
+
DELETE /api/v1/alerts/{alert_id}
|
| 6 |
+
POST /api/v1/alerts/{alert_id}/run -> evaluate now (the pane's manual refresh)
|
| 7 |
+
GET /api/v1/notifications -> {unread, items:[...]}
|
| 8 |
+
POST /api/v1/notifications/read <- {ids:[...]|null, read?:bool}
|
| 9 |
+
|
| 10 |
+
The semantics β an alert is a view plus a remembered matched set, a notification is a NEW
|
| 11 |
+
ENTRANT, and the first evaluation seeds silently β live in `core.alerts` with the reasoning.
|
| 12 |
+
This file owns the two things a route must: WHO may do it, and HOW the view gets evaluated.
|
| 13 |
+
|
| 14 |
+
β **THE EVALUATION RUNS AS THE ALERT'S OWNER, NOT AS THE CALLER.** `_run_alert` builds the pool
|
| 15 |
+
for `rec['owner']`, never for whoever tripped the write hook. Any other choice leaks: a
|
| 16 |
+
full-access admin editing a cell would otherwise evaluate a BU-scoped user's alert over the whole
|
| 17 |
+
book, and the notification would name customers that user may not see β a permission leak wearing
|
| 18 |
+
a notification's clothes. The owner's own scope is the only correct basis for their alert.
|
| 19 |
+
|
| 20 |
+
β **AN ALERT IS NOT A SECOND READ PATH.** It resolves rows through the same
|
| 21 |
+
`routes_customers.grid_assembly` / `routes_tables.ut_assembly` the grid uses, so a row that an
|
| 22 |
+
alert can see is by construction a row its owner could open. Re-implementing the filter here
|
| 23 |
+
would be a second definition of "matches", and those two would drift.
|
| 24 |
+
"""
|
| 25 |
+
import re
|
| 26 |
+
|
| 27 |
+
from fastapi import APIRouter, Body, Depends
|
| 28 |
+
|
| 29 |
+
import core.alerts as alerts
|
| 30 |
+
from deps import Session, err, require_session
|
| 31 |
+
|
| 32 |
+
router = APIRouter(prefix="/api/v1")
|
| 33 |
+
|
| 34 |
+
#: The alert-bearing surfaces. `ut_` tables are admitted by prefix, like everywhere else.
|
| 35 |
+
_TOPICS = ("customer", "product")
|
| 36 |
+
|
| 37 |
+
# ββ ββ WAVE 32 Β· T20 Β· CONTRACT C3 β THE INBOX SHAPE, DERIVED ON READ ββββββββββββββββββββββββ
|
| 38 |
+
#
|
| 39 |
+
# `GET /notifications` gains `subject`, `kind` and `target` per item (`read` was always there).
|
| 40 |
+
#
|
| 41 |
+
# β DERIVED, NEVER STORED, AND THAT IS THE WHOLE OF WHY THIS WAVE EXISTS. Stamping the three
|
| 42 |
+
# keys onto the record at write time would give them to notifications minted AFTER the deploy and
|
| 43 |
+
# to nothing else β every notification already sitting in every tenant's inbox would open nothing,
|
| 44 |
+
# and the feature would be correct in the source and absent from the product
|
| 45 |
+
# ([[a-migration-that-runs-on-the-next-write]], D-201). A read-side derivation reaches a
|
| 46 |
+
# notification queued last month. It also keeps the store shape out of `core/alerts.py`, which is
|
| 47 |
+
# another lane's file this wave β but that is the convenience, not the reason.
|
| 48 |
+
#
|
| 49 |
+
# β TWO PRODUCERS WRITE TWO SHAPES into one inbox, and the vocabulary below is what tells them
|
| 50 |
+
# apart. `_queue` (a record ENTERED a watched view) sets `topic`+`viewId`. `notify()` sets
|
| 51 |
+
# `topic='automation'` and puts the producer's key in `alertId`, leaving `viewId` empty. Deciding
|
| 52 |
+
# here means the client branches on ONE field instead of re-deriving the same split.
|
| 53 |
+
#
|
| 54 |
+
# β **D-101 IS CLOSED HERE, BY SUBTRACTION.** There was a THIRD shape β `kind='automation_review'`
|
| 55 |
+
# + `autoId`, a card arriving at a review stage β and its producer `notify_review` was deleted by
|
| 56 |
+
# W27/R3 with the review lanes. `automation_engine.py`'s own tombstone (search `notify_review`)
|
| 57 |
+
# records the 2026-08-12 sweep: **no `.py` file anywhere produces one**, while the client branch,
|
| 58 |
+
# its route and three gate legs stayed fully alive. D-101's exit condition is *"the client review
|
| 59 |
+
# branch is deleted in the same change as any remaining residue, OR `notify_review` gains its real
|
| 60 |
+
# caller"* β the residue is zero, so the branch goes. It is not carried into the Inbox: a stored
|
| 61 |
+
# review notification (if any survives in a tenant from the wave-23 era) derives as an ordinary
|
| 62 |
+
# `alert` with no target, i.e. an honest unclickable row, which is correct β the board it pointed
|
| 63 |
+
# at was deleted two waves ago.
|
| 64 |
+
|
| 65 |
+
#: C3's `kind` vocabulary. Plain strings on the wire β the client must never union over them
|
| 66 |
+
#: (alertsModel's wave-9 law: a client union turns "the server grew a kind" into a dropped row).
|
| 67 |
+
NOTIF_KIND_ALERT = "alert"
|
| 68 |
+
NOTIF_KIND_AUTOMATION = "automation"
|
| 69 |
+
NOTIF_KIND_SHARE = "share"
|
| 70 |
+
|
| 71 |
+
#: C3's `target.module` vocabulary, and the automation sub-selection.
|
| 72 |
+
TARGET_MODULE_DATABASE = "database"
|
| 73 |
+
TARGET_MODULE_AUTOMATION = "automation"
|
| 74 |
+
TARGET_TAB_RUNS = "runs"
|
| 75 |
+
|
| 76 |
+
#: The topic `notify()` carries for a SHARE (W32-T28 writes it; nothing does yet, and a kind with
|
| 77 |
+
#: no producer is a string that reads as a feature β the reason this constant is named here and
|
| 78 |
+
#: cited from `routes_shares` rather than typed twice).
|
| 79 |
+
SHARE_TOPIC = "share"
|
| 80 |
+
|
| 81 |
+
#: `core.alerts.notify`'s default topic for a run outcome. Mirrors `inboxModel.AUTOMATION_TOPIC`.
|
| 82 |
+
AUTOMATION_TOPIC = "automation"
|
| 83 |
+
|
| 84 |
+
_UT_TOPIC = re.compile(r"ut_[A-Za-z0-9_]+\Z")
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def route_for_topic(topic):
|
| 88 |
+
"""A grid SCOPE key -> the registry route that renders it, or None.
|
| 89 |
+
|
| 90 |
+
β THE SAME TABLE AS `alertsModel.routeForTopic`, and the parity is GATED
|
| 91 |
+
(`verify_alerts.py`'s vocabulary scan) rather than trusted. The two built-ins are the only
|
| 92 |
+
pair that differ β the registry names the surface (`customer_data`) while the grid names the
|
| 93 |
+
scope (`customer`) β so a topic passed through as a route sends every click to a page that
|
| 94 |
+
does not exist. `None` for anything else: a target this product cannot resolve must be ABSENT
|
| 95 |
+
rather than plausible, because an absent target renders as a row that does not pretend to be
|
| 96 |
+
clickable, and a wrong one renders as a click that silently goes nowhere.
|
| 97 |
+
"""
|
| 98 |
+
t = str(topic or "").strip()
|
| 99 |
+
if t == "customer":
|
| 100 |
+
return "customer_data"
|
| 101 |
+
if t == "product":
|
| 102 |
+
return "product_data"
|
| 103 |
+
if _UT_TOPIC.match(t):
|
| 104 |
+
return t
|
| 105 |
+
return None
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def _refusal_code(exc):
|
| 109 |
+
"""The `error.code` an `HTTPException` raised by `deps.err()` carries, or `""`.
|
| 110 |
+
|
| 111 |
+
β W32-T22. Four refusals travel up the assembly chain β `unknown_table` (404), `forbidden`
|
| 112 |
+
(403), `window_required` (409) and `store_not_ready` (503) β and each already names its own
|
| 113 |
+
cause. Anything that reduces all four to one word is throwing away the only information the
|
| 114 |
+
reader could have acted on. Returns `""` for a plain exception, so a caller can tell
|
| 115 |
+
"refused, and here is why" apart from "broke, and we do not know why".
|
| 116 |
+
"""
|
| 117 |
+
detail = getattr(exc, "detail", None)
|
| 118 |
+
if isinstance(detail, dict):
|
| 119 |
+
inner = detail.get("error")
|
| 120 |
+
if isinstance(inner, dict):
|
| 121 |
+
return str(inner.get("code") or "")
|
| 122 |
+
return ""
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def notification_view(item):
|
| 126 |
+
"""One STORED notification -> the shape the Inbox renders. PURE, and total.
|
| 127 |
+
|
| 128 |
+
Never raises and never drops a row: an item it cannot classify comes back as an `alert` with
|
| 129 |
+
no `target`, which the client renders as an unclickable row rather than hiding. An inbox that
|
| 130 |
+
silently omits what it does not understand is the one failure a reader cannot detect.
|
| 131 |
+
"""
|
| 132 |
+
if not isinstance(item, dict):
|
| 133 |
+
return item
|
| 134 |
+
topic = str(item.get("topic") or "").strip()
|
| 135 |
+
alert_id = str(item.get("alertId") or "").strip()
|
| 136 |
+
|
| 137 |
+
# β THE ID TEST IS HALF OF EVERY BRANCH, and it is the load-bearing half. A row whose topic
|
| 138 |
+
# says `automation` but whose producer key never arrived (a truncated payload, a server
|
| 139 |
+
# mid-deploy) would otherwise be handed a target naming NOTHING β a click that appears to work
|
| 140 |
+
# and silently does not, which is this repo's most-repeated failure shape. Failing the test
|
| 141 |
+
# drops it to the `alert` branch, where `route_for_topic` refuses out loud by answering None.
|
| 142 |
+
if topic == AUTOMATION_TOPIC and alert_id:
|
| 143 |
+
kind = NOTIF_KIND_AUTOMATION
|
| 144 |
+
target = {"module": TARGET_MODULE_AUTOMATION, "id": alert_id, "tab": TARGET_TAB_RUNS}
|
| 145 |
+
elif topic == SHARE_TOPIC and alert_id:
|
| 146 |
+
# β W32-T28: the sharer writes `key=<the ROUTE to open>` and, for a shared VIEW,
|
| 147 |
+
# `row_id=<the view to select>`.
|
| 148 |
+
#
|
| 149 |
+
# β `key` IS ALREADY A ROUTE, NOT A RAW OBJECT ID, and the first version of this got it
|
| 150 |
+
# wrong in a way worth recording: a shared VIEW put the VIEW's id in `alertId`, so the
|
| 151 |
+
# target read `{module: "database", id: "view_42"}` β an instruction to open a database
|
| 152 |
+
# called `view_42`. It looked right in the payload and would have opened nothing. The
|
| 153 |
+
# producer resolves the object to its topic and hands over the route; this branch only
|
| 154 |
+
# shapes what it is given.
|
| 155 |
+
kind = NOTIF_KIND_SHARE
|
| 156 |
+
row_id = str(item.get("rowId") or "").strip()
|
| 157 |
+
target = {"module": TARGET_MODULE_DATABASE, "id": alert_id,
|
| 158 |
+
**({"tab": row_id} if row_id else {})}
|
| 159 |
+
else:
|
| 160 |
+
kind = NOTIF_KIND_ALERT
|
| 161 |
+
route = route_for_topic(topic)
|
| 162 |
+
view_id = str(item.get("viewId") or "").strip()
|
| 163 |
+
target = None if route is None else (
|
| 164 |
+
{"module": TARGET_MODULE_DATABASE, "id": route,
|
| 165 |
+
**({"tab": view_id} if view_id else {})})
|
| 166 |
+
|
| 167 |
+
# The email split: `subject` is the HEADER (what this is about β the alert, the automation,
|
| 168 |
+
# the database), `label` stays the BODY (what happened β the record that entered, the run
|
| 169 |
+
# summary). They were one field, which is why a notification read as a sentence with no
|
| 170 |
+
# sender and the pane could not be laid out like mail.
|
| 171 |
+
subject = str(item.get("alertLabel") or "").strip() or str(item.get("label") or "").strip()
|
| 172 |
+
# β `kind` is OVERWRITTEN, not merged. There was one stored value (`automation_review`) and it
|
| 173 |
+
# is D-101's dead one; leaving it through would give the client two vocabularies for one
|
| 174 |
+
# question, which is the defect this wave's item 6 is about in a different file.
|
| 175 |
+
# ββ W33-T28 (owner: "the Inbox reads like email") β THE SENDER, WHICH DID NOT EXIST.
|
| 176 |
+
#
|
| 177 |
+
# β A `verifier` reading the finished wave-32 surface found that the row's sender POSITION was
|
| 178 |
+
# occupied by `kindLabel(n.kind)` β the literals "Alert" / "Automation" / "Shared with you" β
|
| 179 |
+
# i.e. a CATEGORY standing where a who belongs, and no sender field anywhere on the wire, in
|
| 180 |
+
# the model or in the markup. Mail has a from. This is it.
|
| 181 |
+
#
|
| 182 |
+
# β IT IS DERIVED HERE, NOT STORED, FOR EVERY KIND BUT ONE β and the exception is the point.
|
| 183 |
+
# An alert firing and an automation landing rows have no person behind them; their honest
|
| 184 |
+
# sender is the machine that did it, named as the thing the reader recognises. A SHARE has a
|
| 185 |
+
# real person, and only the producer knows who: `routes_shares.py` writes it as `actor` and
|
| 186 |
+
# this reads it back. β It is NOT parsed out of the body prose ("<name> shared this with
|
| 187 |
+
# you") β a sender recovered by regexing a sentence breaks the first time the sentence is
|
| 188 |
+
# reworded, and it would break silently, in the header.
|
| 189 |
+
#
|
| 190 |
+
# β FALLS BACK, NEVER BLANK. A share queued BEFORE `actor` existed has none, and a row with an
|
| 191 |
+
# empty from column reads as a broken inbox rather than as an old notification.
|
| 192 |
+
actor = str(item.get("actor") or "").strip()
|
| 193 |
+
if kind == NOTIF_KIND_SHARE:
|
| 194 |
+
sender = actor or "A teammate"
|
| 195 |
+
elif kind == NOTIF_KIND_AUTOMATION:
|
| 196 |
+
# β "Agents", not "Automation" (W34-T40, corrected at QA 2026-08-17). The client's
|
| 197 |
+
# `senderOf` already falls back to `AGENTS_MODULE_LABEL` β but `if (sent) return sent`
|
| 198 |
+
# runs FIRST, so this server literal won and every actor-less automation notification
|
| 199 |
+
# showed the retired module name in the inbox's From column.
|
| 200 |
+
sender = actor or "Agents"
|
| 201 |
+
else:
|
| 202 |
+
sender = actor or "Alerts"
|
| 203 |
+
out = {**item, "read": bool(item.get("read")), "kind": kind,
|
| 204 |
+
"subject": subject or "Notification", "sender": sender}
|
| 205 |
+
if target is not None:
|
| 206 |
+
out["target"] = target
|
| 207 |
+
return out
|
| 208 |
+
|
| 209 |
+
|
| 210 |
+
def inbox_view(box):
|
| 211 |
+
"""`core.alerts.inbox()`'s answer, with every item put through {@link notification_view}.
|
| 212 |
+
|
| 213 |
+
β `unread` IS NOT RECOUNTED. It is the ACCOUNT's number and `items` is one page of it; a
|
| 214 |
+
recount here would make the badge a function of whatever this page happened to include, which
|
| 215 |
+
is the exact defect `alertsModel.parseInbox`'s own header records from the other side.
|
| 216 |
+
"""
|
| 217 |
+
if not isinstance(box, dict):
|
| 218 |
+
return box
|
| 219 |
+
items = box.get("items")
|
| 220 |
+
if not isinstance(items, list):
|
| 221 |
+
return box
|
| 222 |
+
# ββ W33-T28 / D-208 β THE SERVER'S CLOCK RIDES WITH THE PAGE, and it is what lets the client
|
| 223 |
+
# render a mail-shaped stamp ("09:41" today, "Aug 12" beyond) instead of `2026-08-13 09:41`.
|
| 224 |
+
#
|
| 225 |
+
# β THE CLIENT MUST NOT READ ITS OWN CLOCK, which is D-208's exit condition word for word and
|
| 226 |
+
# is why this key exists rather than a `new Date()` in the browser. `at` is sent as UTC WITH
|
| 227 |
+
# its offset (D-18) precisely so every reader sees the same instant; deciding "is this today?"
|
| 228 |
+
# against a browser clock would re-introduce the drift the offset exists to remove β a reader a
|
| 229 |
+
# day ahead being told an event happened tomorrow [[date-window-vocabulary]]. Both operands
|
| 230 |
+
# now come from the same machine.
|
| 231 |
+
# β Same funnel as the enrichment, so the read door and the mark-read door cannot disagree β
|
| 232 |
+
# the note two lines up records what happened last time only one of them was enriched.
|
| 233 |
+
return {**box, "now": alerts._now_iso(),
|
| 234 |
+
"items": [notification_view(n) for n in items]}
|
| 235 |
+
|
| 236 |
+
|
| 237 |
+
def _view_by_id(g, view_id):
|
| 238 |
+
"""One saved view out of an assembly, by id. `None` when there is no such view.
|
| 239 |
+
|
| 240 |
+
ββ W33-T29 (owner: *"Alert me about new records"* answering "Something went wrong") β THIS
|
| 241 |
+
FUNCTION EXISTS BECAUSE TWO CALL SITES BOTH WROTE `(g.get("views") or {}).get(view_id)`, AND
|
| 242 |
+
`g["views"] IS A LIST`. `aios_grid.views_from_defs` returns `[{...}]`, `workspace_wire` passes
|
| 243 |
+
it straight out and both `ut_assembly` and `grid_assembly` return it unchanged β so `.get` on
|
| 244 |
+
it raises `AttributeError`, and `views_from_defs` always returns at least one element, so the
|
| 245 |
+
`or {}` never fires. **It raised on EVERY call, on every topic, since wave 20.**
|
| 246 |
+
|
| 247 |
+
β AND THE TWO SITES FAILED DIFFERENTLY, WHICH IS WHY ONLY ONE WAS EVER REPORTED. In
|
| 248 |
+
`_require_filtered_view` the raise lands ABOVE the handler's own `try`, so it leaves as a bare
|
| 249 |
+
FastAPI 500 and the client's `errorMessage` turns any 5xx into *"Something went wrong on our
|
| 250 |
+
side"* β the exact sentence the owner reported (D-107's shape, again: an attribute error above
|
| 251 |
+
the guard arrives as plain text rather than as our envelope). In `_evaluate` the identical
|
| 252 |
+
line is swallowed by `/notifications`' `except Exception: continue`, so **every stored
|
| 253 |
+
view-alert was silently dropped from the Inbox** and nobody had anything to report at all.
|
| 254 |
+
One expression, one loud symptom and one silent one.
|
| 255 |
+
|
| 256 |
+
β SO IT IS A FUNCTION, NOT TWO FIXED LINES. Two copies of "find the view" is what let one site
|
| 257 |
+
be discussed for three waves while its twin went unnoticed [[one-question-two-normalizers]].
|
| 258 |
+
|
| 259 |
+
β It accepts a dict too, and that is not defensive noise: `verify_alerts`' door fixture was
|
| 260 |
+
keyed `{id: view}` β which is precisely why the gate was green while production raised on
|
| 261 |
+
every call. The fixture is moving to the production shape in this same change, and tolerating
|
| 262 |
+
both here means a caller that legitimately holds one cannot resurrect the bug.
|
| 263 |
+
"""
|
| 264 |
+
want = str(view_id or "")
|
| 265 |
+
if not want:
|
| 266 |
+
return None
|
| 267 |
+
views = (g or {}).get("views")
|
| 268 |
+
if isinstance(views, dict):
|
| 269 |
+
found = views.get(want)
|
| 270 |
+
return found if isinstance(found, dict) else None
|
| 271 |
+
if not isinstance(views, list):
|
| 272 |
+
return None
|
| 273 |
+
for v in views:
|
| 274 |
+
if isinstance(v, dict) and str(v.get("id") or "") == want:
|
| 275 |
+
return v
|
| 276 |
+
return None
|
| 277 |
+
|
| 278 |
+
|
| 279 |
+
def _topic_or_400(raw):
|
| 280 |
+
topic = str(raw or "").strip().lower()
|
| 281 |
+
if topic.startswith("ut_") or topic in _TOPICS:
|
| 282 |
+
return topic
|
| 283 |
+
raise err(400, "bad_topic", f"topic must be one of {', '.join(_TOPICS)} or a ut_ table")
|
| 284 |
+
|
| 285 |
+
|
| 286 |
+
def _owner_session(session: Session, owner: str):
|
| 287 |
+
"""A `Session` for the alert's OWNER (see the module note on why the owner, not the caller).
|
| 288 |
+
|
| 289 |
+
β `Session` exposes `uname`/`admin` as PROPERTIES derived from `user`, not as fields β so an
|
| 290 |
+
owner session is built by swapping the `user` RECORD and letting both derive themselves. An
|
| 291 |
+
earlier version passed `uname=`/`admin=` to the constructor, which would have raised on the
|
| 292 |
+
first write hook of the wave; the properties are the single definition of who a session is,
|
| 293 |
+
and going around them is how a session with an admin flag and a non-admin record exists.
|
| 294 |
+
|
| 295 |
+
Returns None when the owner is gone or deactivated β their alerts then stop evaluating rather
|
| 296 |
+
than evaluating as somebody else, which is the fail-closed direction.
|
| 297 |
+
"""
|
| 298 |
+
import core.users as users
|
| 299 |
+
|
| 300 |
+
if str(owner) == str(session.uname):
|
| 301 |
+
return session
|
| 302 |
+
rec = (users.registry() or {}).get(str(owner))
|
| 303 |
+
if not isinstance(rec, dict) or not rec.get("active", True):
|
| 304 |
+
return None
|
| 305 |
+
# `_public` is THE definition of what a session may know about its own account (never a hash
|
| 306 |
+
# or a salt) β the same one `routes_auth` uses. Building the dict by hand here would be a
|
| 307 |
+
# second definition, and the one that leaks is always the copy.
|
| 308 |
+
return Session(tenant=session.tenant, user=users._public(str(owner), rec),
|
| 309 |
+
claims=session.claims, runtime=session.runtime)
|
| 310 |
+
|
| 311 |
+
|
| 312 |
+
def _evaluate(session: Session, rec: dict, assemblies=None):
|
| 313 |
+
"""Resolve `rec`'s view over its topic AS THE ALERT'S OWNER, then fold the result in.
|
| 314 |
+
|
| 315 |
+
ββ W31-T24 β `assemblies` IS A PER-REQUEST MEMO, KEYED `(topic, owner)`, and it is the whole
|
| 316 |
+
of this ticket's server half. `/notifications` re-evaluates EVERY alert inline on read and each
|
| 317 |
+
one built a FULL assembly β the pool, the workspace, `rows_from_pool` over every row. Two
|
| 318 |
+
alerts on one view built that table twice; ten built it ten times. Nothing dedupes them,
|
| 319 |
+
because each `_evaluate` was a closed call.
|
| 320 |
+
β `(topic, owner)` and not `topic`: the assembly is built as the alert's OWNER (see the module
|
| 321 |
+
note β evaluating a BU-scoped user's alert on a full-access admin's pool is a permission leak
|
| 322 |
+
wearing a notification's clothes), so two owners on one topic are two DIFFERENT tables and
|
| 323 |
+
must never share an entry. Getting that key wrong is the one way this optimisation could leak.
|
| 324 |
+
β Passing nothing keeps the old behaviour exactly, which is what the create/run doors want:
|
| 325 |
+
they evaluate ONE alert and a memo for a single call is pure overhead.
|
| 326 |
+
"""
|
| 327 |
+
import aios_grid
|
| 328 |
+
from harness import filter_eval
|
| 329 |
+
|
| 330 |
+
owner_sess = _owner_session(session, rec.get("owner"))
|
| 331 |
+
if owner_sess is None:
|
| 332 |
+
return {"skipped": "owner_unavailable"}
|
| 333 |
+
topic = str(rec.get("topic") or "")
|
| 334 |
+
memo_key = (topic, str(owner_sess.uname))
|
| 335 |
+
g = assemblies.get(memo_key) if isinstance(assemblies, dict) else None
|
| 336 |
+
if g is None:
|
| 337 |
+
try:
|
| 338 |
+
if topic.startswith("ut_"):
|
| 339 |
+
from routes_tables import ut_assembly
|
| 340 |
+
# β `consume_corrections=False`, and the default was a REAL BUG, not a tidy-up.
|
| 341 |
+
# `ut_assembly` defaults it True, so every `/notifications` read CONSUMED the
|
| 342 |
+
# one-shot field-name correction acks for every `ut_` topic that has an alert β
|
| 343 |
+
# taking them from the `/workspace` refresh that exists to show them to the person
|
| 344 |
+
# who made the edit. The customer branch below has always passed False; this one
|
| 345 |
+
# inherited a default nobody re-read. An inbox poll must never consume a one-shot.
|
| 346 |
+
g = ut_assembly(owner_sess, topic,
|
| 347 |
+
storage_key=f"{owner_sess.tenant}:{topic}:{owner_sess.uname}",
|
| 348 |
+
consume_corrections=False)
|
| 349 |
+
else:
|
| 350 |
+
from routes_customers import grid_assembly
|
| 351 |
+
g = grid_assembly(owner_sess, scope=topic, consume_corrections=False)
|
| 352 |
+
except Exception as e: # noqa: BLE001
|
| 353 |
+
# β W32-T22 β SKIPPING IS FINE HERE; SKIPPING ANONYMOUSLY IS NOT. This one must not
|
| 354 |
+
# raise (one bad alert cannot empty an inbox), so unlike `_require_filtered_view` it
|
| 355 |
+
# keeps a blanket catch β but it now reports the refusal's OWN code where there is
|
| 356 |
+
# one. `type(e).__name__` said `HTTPException` for four different causes, and
|
| 357 |
+
# `lastError` is the only place a user ever learns why an alert stopped firing.
|
| 358 |
+
#
|
| 359 |
+
# β `with_rows=True` STAYS on this path, deliberately: unlike the create door, an
|
| 360 |
+
# evaluation genuinely needs the rows to run the filter over. So an alert on a
|
| 361 |
+
# read-through grid is created (T22) and then skips at evaluation with
|
| 362 |
+
# `window_required` naming why β which is D-184's remaining half, and it is a
|
| 363 |
+
# SENTENCE now rather than silence.
|
| 364 |
+
return {"skipped": _refusal_code(e) or "unavailable", "detail": type(e).__name__}
|
| 365 |
+
if isinstance(assemblies, dict):
|
| 366 |
+
assemblies[memo_key] = g
|
| 367 |
+
|
| 368 |
+
view = _view_by_id(g, rec.get("viewId"))
|
| 369 |
+
if not isinstance(view, dict):
|
| 370 |
+
# Deleted, or un-shared out from under the alert. Say so on the RECORD rather than
|
| 371 |
+
# deleting the alert: an alert that silently vanishes is indistinguishable from one that
|
| 372 |
+
# never fires, and the user cannot debug what is not there.
|
| 373 |
+
return {"skipped": "view_missing"}
|
| 374 |
+
|
| 375 |
+
# The SAME row build the grid and `/customers` use β `rows_from_pool` is what puts derived
|
| 376 |
+
# and overlay values on a row. Evaluating a filter against raw pool dicts would silently
|
| 377 |
+
# never match any condition on a user-created or measure column.
|
| 378 |
+
#
|
| 379 |
+
# ββ AND "THE SAME ROW BUILD" WAS NOT TRUE, WHICH MADE EVERY ALERT ON A `ut_*` DATABASE BLIND
|
| 380 |
+
# TO IMPORTED DATA. Found by a verifier driving one real assembly through both paths.
|
| 381 |
+
#
|
| 382 |
+
# `routes_tables.table_rows` β the grid the person is looking at β merges the DEFINITION rows
|
| 383 |
+
# underneath the overlay ("base first, overlay wins"; that merge is itself the fix for owner
|
| 384 |
+
# item 3, *"it all got reseted"*). `_evaluate` is a second copy of that read and never got it:
|
| 385 |
+
# it handed `ws['overlays']` to `rows_from_pool` raw, so for a `ut_*` table every base cell
|
| 386 |
+
# evaluated as BLANK. Measured on one assembly, same view, same rows:
|
| 387 |
+
# rows_src state='unpaid' / 'paid'
|
| 388 |
+
# _evaluate saw state='' / '' β every base cell blank
|
| 389 |
+
# the GRID saw state='unpaid' / 'paid'
|
| 390 |
+
# so `state eq unpaid` matched NOTHING while the view showed one row, and `state isEmpty`
|
| 391 |
+
# matched EVERYTHING while the view showed none. **The alert did not merely miss rows β it
|
| 392 |
+
# inverted.** End to end: a row whose value arrived by import, automation, paste or the create
|
| 393 |
+
# door never fired; only a value typed as a hand EDIT did.
|
| 394 |
+
#
|
| 395 |
+
# β Scope, so nobody widens the fix past its cause: materialised `ut_*` tables are hit;
|
| 396 |
+
# `customer`/`product` are not (their fields are `source: "odoo"` and read off `rows_src`);
|
| 397 |
+
# `ut_odoo_*` never reaches here (`with_rows=True` refuses first and returns
|
| 398 |
+
# `skipped: window_required`).
|
| 399 |
+
# β ORDER IS LOAD-BEARING AND IS THE GRID'S: base underneath, overlay ON TOP. Inverting it
|
| 400 |
+
# would let a stale definition value shadow an edit the user has just made β the same defect
|
| 401 |
+
# `table_rows`' own note records, arriving from the other side.
|
| 402 |
+
_ov = (g.get("ws") or {}).get("overlays") or {}
|
| 403 |
+
_merged = {}
|
| 404 |
+
for _r in g["rows_src"]:
|
| 405 |
+
_pid = str(_r.get("pid"))
|
| 406 |
+
_cells = {k: v for k, v in _r.items() if k != "pid"}
|
| 407 |
+
_o = _ov.get(_pid)
|
| 408 |
+
if isinstance(_o, dict):
|
| 409 |
+
_cells.update(_o)
|
| 410 |
+
_merged[_pid] = _cells
|
| 411 |
+
rows = aios_grid.rows_from_pool(g["rows_src"], g["fields"], _merged,
|
| 412 |
+
derived=g.get("derived"))
|
| 413 |
+
config = view.get("config") or view
|
| 414 |
+
ctx = filter_eval.EvalCtx(
|
| 415 |
+
cohort_sets={str(k): {str(p) for p in (v.get("memberPids") or ())}
|
| 416 |
+
for k, v in (g.get("lists") or {}).items() if isinstance(v, dict)},
|
| 417 |
+
measure_sets=g.get("measure_sets") or {},
|
| 418 |
+
today=g.get("today"))
|
| 419 |
+
pids = filter_eval.visible_pids(config.get("filters") or [], rows, g["fields"], ctx,
|
| 420 |
+
member_pids=config.get("memberPids"))
|
| 421 |
+
labels = {str(r.get("pid")): str(r.get("name") or r.get("pid")) for r in rows}
|
| 422 |
+
return alerts.evaluate(rec.get("id"), [str(p) for p in pids],
|
| 423 |
+
labels=labels, partial=False, st=session.runtime)
|
| 424 |
+
|
| 425 |
+
|
| 426 |
+
@router.get("/alerts")
|
| 427 |
+
def list_alerts(session: Session = Depends(require_session)):
|
| 428 |
+
return {"alerts": alerts.list_alerts(user=session.uname, is_admin=session.admin,
|
| 429 |
+
st=session.runtime)}
|
| 430 |
+
|
| 431 |
+
|
| 432 |
+
@router.post("/alerts")
|
| 433 |
+
def create_alert(body: dict = Body(default=None), session: Session = Depends(require_session)):
|
| 434 |
+
body = body or {}
|
| 435 |
+
view_id = str(body.get("viewId") or "").strip()
|
| 436 |
+
if not view_id:
|
| 437 |
+
raise err(400, "bad_view", "an alert needs the id of the view it watches")
|
| 438 |
+
topic = _topic_or_400(body.get("topic"))
|
| 439 |
+
_require_filtered_view(session, topic, view_id)
|
| 440 |
+
import uuid
|
| 441 |
+
aid = f"al_{uuid.uuid4().hex[:12]}"
|
| 442 |
+
rec = alerts.create(aid, view_id=view_id, topic=topic, owner=session.uname,
|
| 443 |
+
label=body.get("label") or "", st=session.runtime)
|
| 444 |
+
# SEED IMMEDIATELY, so the alert starts from "everything currently matching is old news".
|
| 445 |
+
# Deferring this to the first write hook would mean the next edit announces the whole view.
|
| 446 |
+
outcome = _evaluate(session, rec)
|
| 447 |
+
return {"alert": {**rec, "seeded": True}, "first": outcome}
|
| 448 |
+
|
| 449 |
+
|
| 450 |
+
def _require_filtered_view(session: Session, topic: str, view_id: str):
|
| 451 |
+
"""400 unless `view_id` exists on `topic` AND actually narrows something.
|
| 452 |
+
|
| 453 |
+
β AN ALERT ON AN UNFILTERED VIEW IS SILENTLY INCAPABLE OF ALERTING, which is worse than one
|
| 454 |
+
that is refused. `filter_eval` treats an inactive tree as "no narrowing, every row shows"
|
| 455 |
+
(`visible_pids`'s own rule), so such an alert seeds with the entire table and can never see an
|
| 456 |
+
entrant again β there is nothing left to enter. The owner's words are *"when a Record gets
|
| 457 |
+
into that Filter's criteria"*: no criteria, no alert, and said at creation rather than
|
| 458 |
+
discovered by never being notified.
|
| 459 |
+
|
| 460 |
+
`is_rule_active` is the SAME activeness predicate the engine and the column tints use β a
|
| 461 |
+
half-typed rule is not a filter, and this must agree with what actually narrows or it would
|
| 462 |
+
accept a view whose one rule the engine then ignores.
|
| 463 |
+
|
| 464 |
+
ββ WAVE 32 Β· T22 (owner item 17) β THIS FUNCTION WAS THE ERROR. Two defects, stacked, and
|
| 465 |
+
the second one hid the first.
|
| 466 |
+
|
| 467 |
+
(1) **IT ASKED FOR EVERY ROW OF A TABLE IT NEVER LOOKS AT.** The only thing read below is
|
| 468 |
+
`g["views"]`. `ut_assembly` defaults `with_rows=True`, so creating an alert on a
|
| 469 |
+
read-through grid built the whole pool β and `scoped_pool` refuses that with
|
| 470 |
+
`409 window_required` over 963,783 rows, exactly as it is supposed to. `with_rows=False`
|
| 471 |
+
(W31-T20's flag, built for precisely this) answers the same question with `scoped_pids`,
|
| 472 |
+
runs the SAME `_defn_or_refuse` wall, and does not refuse. **That is D-184's create half,
|
| 473 |
+
closed** β an alert on a read-through grid can now be made at all.
|
| 474 |
+
(2) **A BLANKET `except Exception` TURNED EVERY NAMED REFUSAL INTO A 503.** `HTTPException`
|
| 475 |
+
is an `Exception`, so `404 unknown_table`, `403 forbidden`, `409 window_required` and
|
| 476 |
+
`503 store_not_ready` β four refusals that each say what is wrong β were all replaced by
|
| 477 |
+
*"the table is unavailable β try again in a moment"*. β AND THAT SENTENCE NEVER REACHED
|
| 478 |
+
A USER EITHER: `alertsApi.errorMessage` discards the text of any status β₯ 500 by design
|
| 479 |
+
(a 5xx body is the server's internals), substituting *"Something went wrong on our
|
| 480 |
+
side."* β which is the owner's screenshot, word for word. A knowable cause returned as a
|
| 481 |
+
5xx is invisible by construction, so re-wording the 503 could never have fixed this.
|
| 482 |
+
β The except is narrowed, not deleted: an UNEXPECTED failure is still a 503, because that is
|
| 483 |
+
honest. What it may no longer do is catch a refusal that already knows its own name.
|
| 484 |
+
"""
|
| 485 |
+
from fastapi import HTTPException
|
| 486 |
+
|
| 487 |
+
from harness import filter_eval
|
| 488 |
+
|
| 489 |
+
try:
|
| 490 |
+
if topic.startswith("ut_"):
|
| 491 |
+
from routes_tables import ut_assembly
|
| 492 |
+
# β `consume_corrections=False` β the customer branch has always passed it and this
|
| 493 |
+
# one inherited a default nobody re-read. Creating an alert must not eat the one-shot
|
| 494 |
+
# field-name correction acks belonging to the `/workspace` refresh that exists to show
|
| 495 |
+
# them to the person who made the edit. Same defect `_evaluate`'s header records.
|
| 496 |
+
g = ut_assembly(session, topic,
|
| 497 |
+
storage_key=f"{session.tenant}:{topic}:{session.uname}",
|
| 498 |
+
consume_corrections=False, with_rows=False)
|
| 499 |
+
else:
|
| 500 |
+
from routes_customers import grid_assembly
|
| 501 |
+
g = grid_assembly(session, scope=topic, consume_corrections=False)
|
| 502 |
+
except HTTPException:
|
| 503 |
+
raise # it already names its own cause
|
| 504 |
+
except Exception as e: # noqa: BLE001
|
| 505 |
+
# Genuinely unexpected. Still a 503, and now it carries the exception TYPE β without it,
|
| 506 |
+
# the one path that reaches this branch is also the one path with nothing to debug from.
|
| 507 |
+
raise err(503, "unavailable",
|
| 508 |
+
f"the table could not be read ({type(e).__name__}) β try again in a moment")
|
| 509 |
+
view = _view_by_id(g, view_id)
|
| 510 |
+
if not isinstance(view, dict):
|
| 511 |
+
raise err(404, "no_view", "that view does not exist on this table")
|
| 512 |
+
nodes, _conj = filter_eval.tree_parts((view.get("config") or view).get("filters") or [])
|
| 513 |
+
|
| 514 |
+
# ββ WAVE 33 Β· T29 β **CORRECTION: THE BLOCK BELOW IS TRUE ABOUT THE CODE AND FALSE ABOUT
|
| 515 |
+
# PRODUCTION, AND IT MUST BE READ SECOND.** It claims the missing-argument `TypeError` "IS
|
| 516 |
+
# owner item 17" β the owner's *"Something went wrong"*. It was not, and it could not have
|
| 517 |
+
# been: at `cbcf005`, the build the owner was using, the dict-read on `views` sat ~10 lines
|
| 518 |
+
# ABOVE this call and raised `AttributeError` on EVERY request, so the walk never reached the
|
| 519 |
+
# leaf and the arity bug was unreachable. `_view_by_id`'s own header records that fix.
|
| 520 |
+
#
|
| 521 |
+
# β WHY THE STALE PARAGRAPH STAYS RATHER THAN GETTING DELETED: the arity bug was real, the
|
| 522 |
+
# fix was right, and the three reasons it hid are the most transferable thing in this file.
|
| 523 |
+
# What was wrong is only its CLAIM TO BE THE CAUSE. Two comment blocks in one function each
|
| 524 |
+
# naming themselves as the origin of the same screenshot are mutually exclusive, and the next
|
| 525 |
+
# reader believes whichever they meet first β which is why this correction sits above rather
|
| 526 |
+
# than below. Caught by a verifier that read the SHIPPED file at the deployed commit instead
|
| 527 |
+
# of the working tree. [[grep-output-is-not-source]]
|
| 528 |
+
#
|
| 529 |
+
# ββ WAVE 32 Β· T22 β **THE CALL BELOW WAS MISSING AN ARGUMENT** (and wave 32 believed, wrongly,
|
| 530 |
+
# that this was owner item 17 β see the correction directly above).
|
| 531 |
+
#
|
| 532 |
+
# `is_rule_active(rule, columns)` takes TWO parameters (`harness/filter_sql.py`; every other
|
| 533 |
+
# caller in the repo passes both). This one passed ONE, so the moment the walk reached a LEAF
|
| 534 |
+
# rule it raised `TypeError: is_rule_active() missing 1 required positional argument`.
|
| 535 |
+
#
|
| 536 |
+
# β READ WHAT THAT MEANS BEFORE FIXING ANYTHING ELSE: the walk only reaches a leaf when the
|
| 537 |
+
# view HAS a condition β and a view with a condition is the only kind an alert is allowed on.
|
| 538 |
+
# A view with no filters yields an empty `nodes`, so `_any_active` returns False without ever
|
| 539 |
+
# calling this, and the reader gets the honest 400 `no_filter`. **So the only path that
|
| 540 |
+
# worked was the refusal path: "Alert me about new records" had never once created an alert
|
| 541 |
+
# on a filtered view.** β And the raise lands OUTSIDE the `try` above, so it was not even the
|
| 542 |
+
# 503 β it was a bare FastAPI 500, which `alertsApi.errorMessage` renders as *"Something went
|
| 543 |
+
# wrong on our side. Try again in a moment."*, the owner's screenshot word for word.
|
| 544 |
+
#
|
| 545 |
+
# β THREE THINGS HID IT, and they are worth more than the fix. (1) Python does not check
|
| 546 |
+
# arity until the line RUNS, and this line runs only on the success path of a feature whose
|
| 547 |
+
# every test exercised its refusals. (2) The `no_filter` 400 above it is a real, correct,
|
| 548 |
+
# well-tested refusal, so the door looked alive. (3) `verify_alerts.py` asserts the refusal
|
| 549 |
+
# (`no_filter` reaches the user) and the transport β never a creation. A gate can be green,
|
| 550 |
+
# thorough and honest about everything except the one path the feature exists for.
|
| 551 |
+
#
|
| 552 |
+
# `_columns_map` is the DEFINITION of fields -> the membership set `is_rule_active` looks a
|
| 553 |
+
# column up in; building a second dict here would be a second answer to one question, which
|
| 554 |
+
# is this wave's other headline defect in a different file. Its leading underscore is a real
|
| 555 |
+
# smell and is BOOKED (PENDING, mailbox/C.md) rather than worked around.
|
| 556 |
+
columns = filter_eval._columns_map(g.get("fields") or [])
|
| 557 |
+
|
| 558 |
+
def _any_active(ns):
|
| 559 |
+
for n in ns or ():
|
| 560 |
+
if isinstance(n, dict) and isinstance(n.get("children"), list):
|
| 561 |
+
if _any_active(n["children"]):
|
| 562 |
+
return True
|
| 563 |
+
elif filter_eval.is_rule_active(n, columns):
|
| 564 |
+
return True
|
| 565 |
+
return False
|
| 566 |
+
|
| 567 |
+
if not _any_active(nodes):
|
| 568 |
+
raise err(400, "no_filter",
|
| 569 |
+
"this view has no active filter, so no record can ever ENTER it β add a "
|
| 570 |
+
"condition to the view first, then create the alert")
|
| 571 |
+
|
| 572 |
+
|
| 573 |
+
@router.delete("/alerts/{alert_id}")
|
| 574 |
+
def delete_alert(alert_id: str, session: Session = Depends(require_session)):
|
| 575 |
+
rec = next((r for r in alerts.list_alerts(st=session.runtime)
|
| 576 |
+
if str(r.get("id")) == str(alert_id)), None)
|
| 577 |
+
if rec is None:
|
| 578 |
+
raise err(404, "no_alert", "that alert does not exist")
|
| 579 |
+
if str(rec.get("owner")) != str(session.uname) and not session.admin:
|
| 580 |
+
raise err(403, "not_yours", "only the alert's owner (or an administrator) can delete it")
|
| 581 |
+
alerts.delete(alert_id, st=session.runtime)
|
| 582 |
+
return {"ok": True}
|
| 583 |
+
|
| 584 |
+
|
| 585 |
+
@router.post("/alerts/{alert_id}/run")
|
| 586 |
+
def run_alert(alert_id: str, session: Session = Depends(require_session)):
|
| 587 |
+
rec = next((r for r in alerts.list_alerts(user=session.uname, is_admin=session.admin,
|
| 588 |
+
st=session.runtime)
|
| 589 |
+
if str(r.get("id")) == str(alert_id)), None)
|
| 590 |
+
if rec is None:
|
| 591 |
+
raise err(404, "no_alert", "that alert does not exist")
|
| 592 |
+
return _evaluate(session, rec)
|
| 593 |
+
|
| 594 |
+
|
| 595 |
+
@router.get("/notifications")
|
| 596 |
+
def notifications(session: Session = Depends(require_session)):
|
| 597 |
+
"""The inbox β RE-EVALUATED on read, which is a deliberate design choice.
|
| 598 |
+
|
| 599 |
+
β A-S1-2 RESOLVED THE OTHER WAY, and the reason is structural rather than a shortcut. The
|
| 600 |
+
plan was a push hook: the automation engine calls `after_write` when it lands rows. But
|
| 601 |
+
`run_async` runs on a BACKGROUND THREAD with no `Session` in scope, and an alert must be
|
| 602 |
+
evaluated as its OWNER (see `_evaluate`) β so a push hook would have to mint a session inside
|
| 603 |
+
a worker thread from a tenant runtime, which is exactly the kind of ad-hoc identity
|
| 604 |
+
construction that leaks scope.
|
| 605 |
+
|
| 606 |
+
Pulling on read has none of that: the caller IS a session, the assemblies are already
|
| 607 |
+
scope-cached, and the user cannot observe the difference β an inbox is only ever read by
|
| 608 |
+
someone opening it. The cost is that a notification is minted when you LOOK rather than when
|
| 609 |
+
the row landed, so the `at` stamp is detection time, not arrival time.
|
| 610 |
+
|
| 611 |
+
`after_write` stays exported for the day the engine can hand over a real identity.
|
| 612 |
+
|
| 613 |
+
ββ W31-T24 β ONE ASSEMBLY PER (TOPIC, OWNER), NOT ONE PER ALERT.
|
| 614 |
+
β MEASURED FIRST, AND THE MEASUREMENT CORRECTS AN EARLIER READING OF IT. This route is
|
| 615 |
+
**20 ms in-process and 3,280 ms live** on tenant #0 β but tenant #0 has **ZERO alerts**
|
| 616 |
+
(censused 2026-08-12), so the 20 ms is an EMPTY LOOP and says nothing at all about what the
|
| 617 |
+
re-evaluation costs. The live 3,280 ms is the two store reads either side of that loop. So the
|
| 618 |
+
body below is not slow today; it is UNEXERCISED, and every alert a tenant creates adds a whole
|
| 619 |
+
grid assembly to an inbox poll. The memo turns O(alerts) into O(distinct topic Γ owner), which
|
| 620 |
+
is the difference between "fine" and "three seconds per alert" the day somebody uses the
|
| 621 |
+
feature. β Making the read cheap by evaluating LESS is the obvious wrong fix and is not what
|
| 622 |
+
this does: every alert is still evaluated, against the same rows, in the same order.
|
| 623 |
+
"""
|
| 624 |
+
assemblies = {}
|
| 625 |
+
for rec in alerts.list_alerts(user=session.uname, is_admin=False, st=session.runtime):
|
| 626 |
+
try:
|
| 627 |
+
_evaluate(session, rec, assemblies=assemblies)
|
| 628 |
+
except Exception: # noqa: BLE001
|
| 629 |
+
continue # one bad alert must not empty the pane
|
| 630 |
+
# β W32-T20 (C3): every item leaves through `inbox_view`, so a notification queued before
|
| 631 |
+
# this wave carries a `target` too. See `notification_view`'s header for why it is derived.
|
| 632 |
+
return inbox_view(alerts.inbox(session.uname, st=session.runtime))
|
| 633 |
+
|
| 634 |
+
|
| 635 |
+
@router.post("/notifications/read")
|
| 636 |
+
def read_notifications(body: dict = Body(default=None),
|
| 637 |
+
session: Session = Depends(require_session)):
|
| 638 |
+
body = body or {}
|
| 639 |
+
ids = body.get("ids")
|
| 640 |
+
if ids is not None and not isinstance(ids, list):
|
| 641 |
+
raise err(400, "bad_ids", "ids must be a list, or null to mark every notification")
|
| 642 |
+
# β THE SAME ENRICHMENT ON BOTH DOORS. `mark_read` returns a fresh inbox, and the Inbox
|
| 643 |
+
# module re-renders from it β an un-enriched answer here would strip `target` off every row
|
| 644 |
+
# the moment somebody marked one read, i.e. the feature would work until first use.
|
| 645 |
+
return inbox_view(alerts.mark_read(session.uname, ids, read=bool(body.get("read", True)),
|
| 646 |
+
st=session.runtime))
|
| 647 |
+
|
| 648 |
+
|
| 649 |
+
def after_write(session: Session, topic_key: str):
|
| 650 |
+
"""THE WRITE HOOK β call after a write that could change what a view matches.
|
| 651 |
+
|
| 652 |
+
Exported as a plain function (not a route) so `core.grid_events`' callers and S2's automation
|
| 653 |
+
upserts reach it the same way. It never raises: an alert evaluation failing must not fail the
|
| 654 |
+
edit that triggered it.
|
| 655 |
+
|
| 656 |
+
β W31-T24 β it shares `/notifications`' memo shape for the same reason: a write that changes
|
| 657 |
+
one view can trip several alerts on the SAME topic, and each would otherwise rebuild the table.
|
| 658 |
+
β STILL ZERO PRODUCTION CALLERS (W31-T24 confirmed it; the route docstring above says why the
|
| 659 |
+
push hook was resolved the other way). Booked rather than wired: minting a session inside the
|
| 660 |
+
engine's worker thread is the ad-hoc identity construction this file exists to avoid.
|
| 661 |
+
"""
|
| 662 |
+
try:
|
| 663 |
+
assemblies = {}
|
| 664 |
+
return alerts.after_write(topic_key, st=session.runtime,
|
| 665 |
+
runner=lambda rec: _evaluate(session, rec,
|
| 666 |
+
assemblies=assemblies))
|
| 667 |
+
except Exception: # noqa: BLE001
|
| 668 |
+
return {"evaluated": 0}
|
api/routes_automation.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
api/routes_grid.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
api/routes_keychain.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
api/routes_nav.py
CHANGED
|
@@ -489,8 +489,15 @@ def nav(session: Session = Depends(require_session)):
|
|
| 489 |
# the moment this import failed. A sentinel that can equal real data is not a sentinel.
|
| 490 |
_ut_locked_mode = str(user_tables.AUTOMATION_RECORD_MODE)
|
| 491 |
_lent = user_tables.lend(session.runtime, **{user_tables.STORE_KEY: _ut_defs})
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 492 |
for e in user_tables.nav_entries(viewer=session.uname, is_admin=session.admin,
|
| 493 |
st=_lent):
|
|
|
|
|
|
|
| 494 |
pages.append({"key": e["key"], "label": e["label"],
|
| 495 |
"source": e.get("source") or "Blank", "chrome": "main"})
|
| 496 |
except Exception:
|
|
|
|
| 489 |
# the moment this import failed. A sentinel that can equal real data is not a sentinel.
|
| 490 |
_ut_locked_mode = str(user_tables.AUTOMATION_RECORD_MODE)
|
| 491 |
_lent = user_tables.lend(session.runtime, **{user_tables.STORE_KEY: _ut_defs})
|
| 492 |
+
# β AND THE EDITOR'S DENY REACHES THIS HALF TOO. `nav_entries` is `may_open`-filtered
|
| 493 |
+
# β creator, admin or a share β which knows nothing about the access toggle in
|
| 494 |
+
# Manage user. An administrator unticking a shared database left it on the rail.
|
| 495 |
+
# `nav_may_open` composes the two and cannot WIDEN past `may_open` on a `ut_*` key.
|
| 496 |
+
import core.perm_scope as _perm_scope
|
| 497 |
for e in user_tables.nav_entries(viewer=session.uname, is_admin=session.admin,
|
| 498 |
st=_lent):
|
| 499 |
+
if not _perm_scope.nav_may_open(session.user, e["key"], st=_lent):
|
| 500 |
+
continue
|
| 501 |
pages.append({"key": e["key"], "label": e["label"],
|
| 502 |
"source": e.get("source") or "Blank", "chrome": "main"})
|
| 503 |
except Exception:
|
api/routes_oauth.py
CHANGED
|
@@ -1,114 +1,114 @@
|
|
| 1 |
-
"""routes_oauth.py β the OAuth connector surface (wave 22, contract C5 + A2/A3 / R12).
|
| 2 |
-
|
| 3 |
-
Thin over `oauth_connect`, the way `routes_automation` is thin over the engine: sessions,
|
| 4 |
-
shapes and status codes here; every decision that could be wrong lives in the module a gate
|
| 5 |
-
can drive without a server. GENERIC over `{provider}` (C5-A2): the routes read the registry,
|
| 6 |
-
so the day a second provider lands here is the day nothing in this file changes.
|
| 7 |
-
|
| 8 |
-
MOUNTED FROM `routes_automation` (not `main.py`): this wave's ownership fence gives no session
|
| 9 |
-
`main.py`, and `routes_automation` is already included there β so this router rides inside it
|
| 10 |
-
(`/api/v1` + `/oauth/...`). Lifting the include into `main.py` later is a two-line change that
|
| 11 |
-
alters no path.
|
| 12 |
-
|
| 13 |
-
β THE TWO REDIRECT LAWS (A3): `/{provider}/start` answers **302 to the provider's consent
|
| 14 |
-
screen** β it is a top-level navigation the client reaches by `<a href>`, never JSON. The
|
| 15 |
-
callback 302s BACK to the return path the `state` carried (relative-only, sanitised by
|
| 16 |
-
`oauth_connect.safe_next`), so the user lands where they left β connected or not, whatever
|
| 17 |
-
went wrong rides in the query string; a dead-end error page where the app used to be reads as
|
| 18 |
-
"the product broke", not "the connect failed".
|
| 19 |
-
"""
|
| 20 |
-
import os
|
| 21 |
-
|
| 22 |
-
from fastapi import APIRouter, Depends, Request
|
| 23 |
-
from fastapi.responses import RedirectResponse
|
| 24 |
-
|
| 25 |
-
import oauth_connect
|
| 26 |
-
from deps import Session, err, require_session
|
| 27 |
-
|
| 28 |
-
router = APIRouter(prefix="/oauth")
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
def _redirect_uri(request: Request, provider: str) -> str:
|
| 32 |
-
"""The redirect URI this deployment registers at the provider β env-pinned when the
|
| 33 |
-
container sits behind a proxy that rewrites the scheme (the HF Space), else derived from
|
| 34 |
-
the request. MUST match a console-registered URI verbatim, so it is computed in exactly
|
| 35 |
-
one place.
|
| 36 |
-
|
| 37 |
-
β WAVE 29 (R4): `deploy_web.py` now PUSHES `AIOS_PUBLIC_BASE` on every deploy, defaulted to
|
| 38 |
-
the same URL as `APP_BASE_URL`, so the pinned branch is the one that runs in production and
|
| 39 |
-
the request-derived fallback below is effectively dev-only.
|
| 40 |
-
β THAT MAKES THIS FUNCTION A CUSTOM-DOMAIN COUPLING, not merely a scheme fix. Whatever host
|
| 41 |
-
this returns is where the provider sends the user BACK, and the session cookie is host-only
|
| 42 |
-
(`aios_session.py:114-117`, no `domain=`) β so a callback base that disagrees with the host
|
| 43 |
-
the user actually browsed plants the session on the wrong hostname and they return logged
|
| 44 |
-
out. Moving the app to a new hostname means moving this value AND re-registering the
|
| 45 |
-
resulting URI in the provider console; one without the other fails closed.
|
| 46 |
-
Runbook: `.claude/wiki/research/loopable-domain-runbook.md`."""
|
| 47 |
-
base = (os.environ.get("AIOS_PUBLIC_BASE") or "").strip().rstrip("/")
|
| 48 |
-
if not base:
|
| 49 |
-
base = f"{request.url.scheme}://{request.url.netloc}"
|
| 50 |
-
return f"{base}/api/v1/oauth/{provider}/callback"
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
@router.get("/status")
|
| 54 |
-
def oauth_status(session: Session = Depends(require_session)):
|
| 55 |
-
"""C5's status shape for the SESSION user, one entry per registry provider:
|
| 56 |
-
`{google: {connected, email, reconnect, configured}}` today. The bit the email trigger's
|
| 57 |
-
`ready` reads through."""
|
| 58 |
-
return oauth_connect.status(session.runtime, session.uname)
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
def _offered_or_404(provider: str):
|
| 62 |
-
"""ββ W32-T14 / OWNER ITEM 12 / R6 β A PROVIDER THE PRODUCT DOES NOT OFFER HAS NO DOOR.
|
| 63 |
-
|
| 64 |
-
The owner pasted the failure this replaces: clicking Connect on the Google card answered
|
| 65 |
-
`503 oauth_unavailable` as raw JSON. R6's fix is not a nicer error β it is that the flow is
|
| 66 |
-
not offered at all, so the honest status is the one for a URL that does not exist. `404`
|
| 67 |
-
rather than `503`, deliberately: a 503 says *"come back later"* about a door that is not
|
| 68 |
-
coming back until somebody pays for CASA verification (D-45, ~$540β1,800/yr).
|
| 69 |
-
|
| 70 |
-
β Every door in this router goes through it, START included, because the JSON the owner saw
|
| 71 |
-
came from the start route and a guard on one leg is a guard on one leg.
|
| 72 |
-
"""
|
| 73 |
-
if not oauth_connect.offered(provider):
|
| 74 |
-
raise err(404, "unknown_provider", f"{provider!r} is not a connectable provider")
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
@router.get("/{provider}/start")
|
| 78 |
-
def oauth_start(provider: str, request: Request, next: str = "",
|
| 79 |
-
session: Session = Depends(require_session)):
|
| 80 |
-
"""302 to the provider's consent screen (A3 β a navigation, never JSON). `?next=` is the
|
| 81 |
-
RELATIVE path the callback returns the browser to; it rides inside the single-use state,
|
| 82 |
-
sanitised, so the round trip cannot be steered off-origin."""
|
| 83 |
-
_offered_or_404(provider)
|
| 84 |
-
url, problem = oauth_connect.start(provider, session.uname,
|
| 85 |
-
_redirect_uri(request, provider), next_path=next)
|
| 86 |
-
if problem:
|
| 87 |
-
raise err(503 if "not configured" in problem else 404, "oauth_unavailable", problem)
|
| 88 |
-
return RedirectResponse(url, status_code=302)
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
@router.get("/{provider}/callback")
|
| 92 |
-
def oauth_callback(provider: str, request: Request,
|
| 93 |
-
session: Session = Depends(require_session),
|
| 94 |
-
state: str = "", code: str = "", error: str = ""):
|
| 95 |
-
"""The provider's redirect target. Exchanges the code, stores the per-user slot, and sends
|
| 96 |
-
the browser back to the state's return path β connected or not (see module header)."""
|
| 97 |
-
_offered_or_404(provider)
|
| 98 |
-
if error:
|
| 99 |
-
home = "/#/"
|
| 100 |
-
return RedirectResponse(f"{home}?oauthError={error[:80]}", status_code=302)
|
| 101 |
-
email, home, problem = oauth_connect.callback(session.runtime, session.uname, state, code)
|
| 102 |
-
sep = "&" if "?" in home else "?"
|
| 103 |
-
if problem:
|
| 104 |
-
return RedirectResponse(f"{home}{sep}oauthError=connect_failed", status_code=302)
|
| 105 |
-
return RedirectResponse(f"{home}{sep}connected={provider}", status_code=302)
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
@router.post("/{provider}/disconnect")
|
| 109 |
-
def oauth_disconnect(provider: str, session: Session = Depends(require_session)):
|
| 110 |
-
_offered_or_404(provider)
|
| 111 |
-
if oauth_connect.provider_def(provider) is None:
|
| 112 |
-
raise err(404, "unknown_provider", f"{provider!r} is not a connectable provider")
|
| 113 |
-
oauth_connect.disconnect(session.runtime, session.uname, provider)
|
| 114 |
-
return {"disconnected": provider}
|
|
|
|
| 1 |
+
"""routes_oauth.py β the OAuth connector surface (wave 22, contract C5 + A2/A3 / R12).
|
| 2 |
+
|
| 3 |
+
Thin over `oauth_connect`, the way `routes_automation` is thin over the engine: sessions,
|
| 4 |
+
shapes and status codes here; every decision that could be wrong lives in the module a gate
|
| 5 |
+
can drive without a server. GENERIC over `{provider}` (C5-A2): the routes read the registry,
|
| 6 |
+
so the day a second provider lands here is the day nothing in this file changes.
|
| 7 |
+
|
| 8 |
+
MOUNTED FROM `routes_automation` (not `main.py`): this wave's ownership fence gives no session
|
| 9 |
+
`main.py`, and `routes_automation` is already included there β so this router rides inside it
|
| 10 |
+
(`/api/v1` + `/oauth/...`). Lifting the include into `main.py` later is a two-line change that
|
| 11 |
+
alters no path.
|
| 12 |
+
|
| 13 |
+
β THE TWO REDIRECT LAWS (A3): `/{provider}/start` answers **302 to the provider's consent
|
| 14 |
+
screen** β it is a top-level navigation the client reaches by `<a href>`, never JSON. The
|
| 15 |
+
callback 302s BACK to the return path the `state` carried (relative-only, sanitised by
|
| 16 |
+
`oauth_connect.safe_next`), so the user lands where they left β connected or not, whatever
|
| 17 |
+
went wrong rides in the query string; a dead-end error page where the app used to be reads as
|
| 18 |
+
"the product broke", not "the connect failed".
|
| 19 |
+
"""
|
| 20 |
+
import os
|
| 21 |
+
|
| 22 |
+
from fastapi import APIRouter, Depends, Request
|
| 23 |
+
from fastapi.responses import RedirectResponse
|
| 24 |
+
|
| 25 |
+
import oauth_connect
|
| 26 |
+
from deps import Session, err, require_session
|
| 27 |
+
|
| 28 |
+
router = APIRouter(prefix="/oauth")
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def _redirect_uri(request: Request, provider: str) -> str:
|
| 32 |
+
"""The redirect URI this deployment registers at the provider β env-pinned when the
|
| 33 |
+
container sits behind a proxy that rewrites the scheme (the HF Space), else derived from
|
| 34 |
+
the request. MUST match a console-registered URI verbatim, so it is computed in exactly
|
| 35 |
+
one place.
|
| 36 |
+
|
| 37 |
+
β WAVE 29 (R4): `deploy_web.py` now PUSHES `AIOS_PUBLIC_BASE` on every deploy, defaulted to
|
| 38 |
+
the same URL as `APP_BASE_URL`, so the pinned branch is the one that runs in production and
|
| 39 |
+
the request-derived fallback below is effectively dev-only.
|
| 40 |
+
β THAT MAKES THIS FUNCTION A CUSTOM-DOMAIN COUPLING, not merely a scheme fix. Whatever host
|
| 41 |
+
this returns is where the provider sends the user BACK, and the session cookie is host-only
|
| 42 |
+
(`aios_session.py:114-117`, no `domain=`) β so a callback base that disagrees with the host
|
| 43 |
+
the user actually browsed plants the session on the wrong hostname and they return logged
|
| 44 |
+
out. Moving the app to a new hostname means moving this value AND re-registering the
|
| 45 |
+
resulting URI in the provider console; one without the other fails closed.
|
| 46 |
+
Runbook: `.claude/wiki/research/loopable-domain-runbook.md`."""
|
| 47 |
+
base = (os.environ.get("AIOS_PUBLIC_BASE") or "").strip().rstrip("/")
|
| 48 |
+
if not base:
|
| 49 |
+
base = f"{request.url.scheme}://{request.url.netloc}"
|
| 50 |
+
return f"{base}/api/v1/oauth/{provider}/callback"
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
@router.get("/status")
|
| 54 |
+
def oauth_status(session: Session = Depends(require_session)):
|
| 55 |
+
"""C5's status shape for the SESSION user, one entry per registry provider:
|
| 56 |
+
`{google: {connected, email, reconnect, configured}}` today. The bit the email trigger's
|
| 57 |
+
`ready` reads through."""
|
| 58 |
+
return oauth_connect.status(session.runtime, session.uname)
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def _offered_or_404(provider: str):
|
| 62 |
+
"""ββ W32-T14 / OWNER ITEM 12 / R6 β A PROVIDER THE PRODUCT DOES NOT OFFER HAS NO DOOR.
|
| 63 |
+
|
| 64 |
+
The owner pasted the failure this replaces: clicking Connect on the Google card answered
|
| 65 |
+
`503 oauth_unavailable` as raw JSON. R6's fix is not a nicer error β it is that the flow is
|
| 66 |
+
not offered at all, so the honest status is the one for a URL that does not exist. `404`
|
| 67 |
+
rather than `503`, deliberately: a 503 says *"come back later"* about a door that is not
|
| 68 |
+
coming back until somebody pays for CASA verification (D-45, ~$540β1,800/yr).
|
| 69 |
+
|
| 70 |
+
β Every door in this router goes through it, START included, because the JSON the owner saw
|
| 71 |
+
came from the start route and a guard on one leg is a guard on one leg.
|
| 72 |
+
"""
|
| 73 |
+
if not oauth_connect.offered(provider):
|
| 74 |
+
raise err(404, "unknown_provider", f"{provider!r} is not a connectable provider")
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
@router.get("/{provider}/start")
|
| 78 |
+
def oauth_start(provider: str, request: Request, next: str = "",
|
| 79 |
+
session: Session = Depends(require_session)):
|
| 80 |
+
"""302 to the provider's consent screen (A3 β a navigation, never JSON). `?next=` is the
|
| 81 |
+
RELATIVE path the callback returns the browser to; it rides inside the single-use state,
|
| 82 |
+
sanitised, so the round trip cannot be steered off-origin."""
|
| 83 |
+
_offered_or_404(provider)
|
| 84 |
+
url, problem = oauth_connect.start(provider, session.uname,
|
| 85 |
+
_redirect_uri(request, provider), next_path=next)
|
| 86 |
+
if problem:
|
| 87 |
+
raise err(503 if "not configured" in problem else 404, "oauth_unavailable", problem)
|
| 88 |
+
return RedirectResponse(url, status_code=302)
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
@router.get("/{provider}/callback")
|
| 92 |
+
def oauth_callback(provider: str, request: Request,
|
| 93 |
+
session: Session = Depends(require_session),
|
| 94 |
+
state: str = "", code: str = "", error: str = ""):
|
| 95 |
+
"""The provider's redirect target. Exchanges the code, stores the per-user slot, and sends
|
| 96 |
+
the browser back to the state's return path β connected or not (see module header)."""
|
| 97 |
+
_offered_or_404(provider)
|
| 98 |
+
if error:
|
| 99 |
+
home = "/#/"
|
| 100 |
+
return RedirectResponse(f"{home}?oauthError={error[:80]}", status_code=302)
|
| 101 |
+
email, home, problem = oauth_connect.callback(session.runtime, session.uname, state, code)
|
| 102 |
+
sep = "&" if "?" in home else "?"
|
| 103 |
+
if problem:
|
| 104 |
+
return RedirectResponse(f"{home}{sep}oauthError=connect_failed", status_code=302)
|
| 105 |
+
return RedirectResponse(f"{home}{sep}connected={provider}", status_code=302)
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
@router.post("/{provider}/disconnect")
|
| 109 |
+
def oauth_disconnect(provider: str, session: Session = Depends(require_session)):
|
| 110 |
+
_offered_or_404(provider)
|
| 111 |
+
if oauth_connect.provider_def(provider) is None:
|
| 112 |
+
raise err(404, "unknown_provider", f"{provider!r} is not a connectable provider")
|
| 113 |
+
oauth_connect.disconnect(session.runtime, session.uname, provider)
|
| 114 |
+
return {"disconnected": provider}
|
api/routes_publish.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
api/routes_query.py
CHANGED
|
@@ -18,9 +18,17 @@ import uuid
|
|
| 18 |
from fastapi import APIRouter, Body, Depends, Response
|
| 19 |
|
| 20 |
from deps import (Session, assistant_read_scope, assistant_source_status, err,
|
| 21 |
-
require_session)
|
| 22 |
-
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
MAX_QUESTION = 500
|
| 26 |
MAX_VISIBLE = 24
|
|
|
|
| 18 |
from fastapi import APIRouter, Body, Depends, Response
|
| 19 |
|
| 20 |
from deps import (Session, assistant_read_scope, assistant_source_status, err,
|
| 21 |
+
module_gate, require_session)
|
| 22 |
+
|
| 23 |
+
# ββ GATED AS A ROUTER, BECAUSE UNTIL 2026-08-18 THESE ROUTES HAD NO MODULE GATE AT
|
| 24 |
+
# ALL β every door here was bare `require_session`, so "no Assistant" only ever meant "no
|
| 25 |
+
# button": the rail hid the surface while the API answered anyone signed in. The moment Manage
|
| 26 |
+
# user grew an Assistant toggle (owner ruling, same day), an unenforced deny would be a control
|
| 27 |
+
# that lies β the class this repo keeps paying for. One dependency on the router covers
|
| 28 |
+
# every route in the file, present and future, instead of fifteen signatures each one edit away
|
| 29 |
+
# from being the hole. `automation`'s routes have carried module_gate since wave 20; this brings
|
| 30 |
+
# the OTHER surface to the same rule, through the same `nav_may_open` the rail reads.
|
| 31 |
+
router = APIRouter(prefix="/api/v1", dependencies=[Depends(module_gate("analyst"))])
|
| 32 |
|
| 33 |
MAX_QUESTION = 500
|
| 34 |
MAX_VISIBLE = 24
|
api/routes_records.py
CHANGED
|
@@ -1,211 +1,211 @@
|
|
| 1 |
-
"""Record detail routes: durable comments, scoped to the caller's book β ON EVERY DATABASE.
|
| 2 |
-
|
| 3 |
-
β WAVE 19 (owner item 12). This file used to be the CUSTOMER record's comment routes with a
|
| 4 |
-
customer-shaped wall bolted to the module import line: `_in_book` asked
|
| 5 |
-
`routes_customers.allowed_pids` whatever surface the browser was on. Opening a PRODUCT record and
|
| 6 |
-
typing a comment therefore asked the customer book about a CRC32 hash of a SKU code, and the
|
| 7 |
-
panel answered "that customer is not in your book" β the owner's report. The dangerous half is
|
| 8 |
-
the one nobody sees: a hash that collides with a real partner id passes the wall, and the comment
|
| 9 |
-
is filed against somebody's customer where the whole team can read it.
|
| 10 |
-
|
| 11 |
-
THE SHAPE NOW: `?scope=` names the database (the same vocabulary `/workspace?scope=` and the
|
| 12 |
-
events route's `scopeKey` already speak), and `_pool_or_refuse` resolves BOTH halves of the wall
|
| 13 |
-
per scope β the GRANT and the ROW SET β by asking that topic's own route, never by re-deriving
|
| 14 |
-
one here:
|
| 15 |
-
|
| 16 |
-
customer / cohort `routes_customers.allowed_pids` behind the `customer_data` grant
|
| 17 |
-
product `routes_products.scoped_pool` behind the `product_data` grant
|
| 18 |
-
ut_<slug> `routes_tables.scoped_pids`, whose `_defn_or_refuse` IS the wall
|
| 19 |
-
(404 unknown / 403 not yours β a user table has no module grant).
|
| 20 |
-
β W33-T03/D-183: `scoped_pIDs`, not `scoped_pOOL` β the pool builds every
|
| 21 |
-
ROW to derive a pid set this module discards, and RAISES 409 on a
|
| 22 |
-
read-through grid past one window, which is why the record drawer painted
|
| 23 |
-
an error page on `ut_odoo_gl_lines`.
|
| 24 |
-
|
| 25 |
-
β THE PATH KEEPS ITS `/customers/` SEGMENT. It is the shipped URL and `verify_api.py`'s E1a
|
| 26 |
-
section pins it; the scope now travels beside it explicitly. A nicer noun is not worth churning
|
| 27 |
-
another session's gate mid-wave β the WALL is the query parameter, not the word.
|
| 28 |
-
|
| 29 |
-
β NO DEFAULT BEYOND THE LEGACY ONE. An absent `scope` means `customer`, which is what every
|
| 30 |
-
shipped client sent and what keeps the old callers byte-identical; an UNRECOGNISED scope is a
|
| 31 |
-
400, never a silent fallback to the customer book (`routes_grid._scope_or_400`'s rule, and for
|
| 32 |
-
the same reason: a typo served as `customer` answers a question nobody asked).
|
| 33 |
-
"""
|
| 34 |
-
from fastapi import APIRouter, Body, Depends, Query
|
| 35 |
-
|
| 36 |
-
from deps import Session, err, require_session
|
| 37 |
-
|
| 38 |
-
router = APIRouter(prefix="/api/v1")
|
| 39 |
-
|
| 40 |
-
#: The customer topic's two names β one book, two surfaces (the Cohort page is the customer table
|
| 41 |
-
#: over hand-curated sets). Mirrors `modules.cohort.LEGACY_SCOPES` / `core.record_comments`.
|
| 42 |
-
_CUSTOMER_SCOPES = ("", "customer", "cohort")
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
def _scope_or_400(raw):
|
| 46 |
-
scope = str(raw or "customer").strip().lower()
|
| 47 |
-
if scope in _CUSTOMER_SCOPES or scope == "product" or scope.startswith("ut_"):
|
| 48 |
-
return "customer" if scope in _CUSTOMER_SCOPES else scope
|
| 49 |
-
raise err(400, "bad_scope",
|
| 50 |
-
"scope must be customer, cohort, product or a ut_ database β refusing to guess")
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
def _pool_or_refuse(session: Session, scope: str):
|
| 54 |
-
"""The pids this session may attach comments to ON THIS DATABASE β grant wall included.
|
| 55 |
-
|
| 56 |
-
Returns **`(pids, unbounded)`** β a 2-tuple on EVERY branch. Raises the topic's own 403/404/503,
|
| 57 |
-
so a caller who may not open the surface never learns anything about the row they asked about.
|
| 58 |
-
|
| 59 |
-
β `unbounded` is TRUE only when the row set could not be ENUMERATED (a read-through grid past
|
| 60 |
-
one window), never when it is merely EMPTY. Those are opposite answers and `scoped_pids` returns
|
| 61 |
-
`frozenset()` for both β see the branch below.
|
| 62 |
-
β THE SHAPE IS A CONTRACT EVEN THOUGH THIS FUNCTION IS PRIVATE, and it has two consumers that
|
| 63 |
-
do not travel together: `_in_book` here, and a NEGATIVE CONTROL in `aios-web/api/verify_scopes.py`
|
| 64 |
-
that REPLACES this function with its own lambda. A gate's test double is a caller
|
| 65 |
-
([[test-double-patched-by-a-name-list]]); when this signature moved, that double kept returning a
|
| 66 |
-
bare frozenset and the section died on `ValueError: too many values to unpack` β no tally, no
|
| 67 |
-
failing name. Change the shape here and that double changes with it.
|
| 68 |
-
"""
|
| 69 |
-
if scope == "product":
|
| 70 |
-
from routes_products import MODULE as PRODUCT_MODULE, scoped_pool
|
| 71 |
-
|
| 72 |
-
session.require(PRODUCT_MODULE)
|
| 73 |
-
pids, _team, _rows, _fields = scoped_pool(session)
|
| 74 |
-
# β `(pids, unbounded)` on EVERY branch. This one returned a bare frozenset for ten minutes
|
| 75 |
-
# after the `ut_` branch grew its second element, and `_in_book`'s unpack would have raised
|
| 76 |
-
# a `TypeError` β a 500 on every product comment β while both other branches worked. A
|
| 77 |
-
# return shape is a contract even when the function is private.
|
| 78 |
-
return (pids, False)
|
| 79 |
-
if scope.startswith("ut_"):
|
| 80 |
-
# No module grant exists for a user table β `_defn_or_refuse` inside `scoped_pids` IS
|
| 81 |
-
# the wall (creator or admin, fail-closed), and it answers 404 before 403 exactly as the
|
| 82 |
-
# rows routes do.
|
| 83 |
-
#
|
| 84 |
-
# ββ W33-T03 / D-183 β `scoped_pids`, NOT `scoped_pool`, AND THAT ONE WORD IS THE BUG.
|
| 85 |
-
# `scoped_pool` builds every ROW to derive a pid set this function then throws away, and on
|
| 86 |
-
# a read-through grid larger than one window it RAISES `409 window_required`. So opening the
|
| 87 |
-
# record drawer on `ut_odoo_gl_lines` (975,137 rows) painted an error page β for a panel
|
| 88 |
-
# that renders comments about ONE row it already has. `scoped_pids` answers the identical
|
| 89 |
-
# question (its docstring: *"the pid set is IDENTICAL, not merely equivalent"*) and takes
|
| 90 |
-
# W31-T20's `limits` OUT-PARAMETER instead of raising, which is the same shape `/workspace`
|
| 91 |
-
# used to become openable on those two grids.
|
| 92 |
-
from routes_tables import scoped_pids
|
| 93 |
-
|
| 94 |
-
limits = []
|
| 95 |
-
pids, _fields, _defn = scoped_pids(session, scope, limits=limits)
|
| 96 |
-
# β AN EMPTY PID SET AND AN UNRESOLVABLE ONE ARE OPPOSITE ANSWERS, and collapsing them is
|
| 97 |
-
# how a fail-closed default becomes a lie. `scoped_pids` returns `frozenset()` BOTH for a
|
| 98 |
-
# database with no rows and for a read-through grid too big to enumerate β it distinguishes
|
| 99 |
-
# them by APPENDING R6's sentence to `limits`. Without this branch the drawer would move
|
| 100 |
-
# from a 409 error page to a 403 "not in your book" on a row the user is looking at, which
|
| 101 |
-
# is the same defect wearing a politer message ([[empty-answer-vs-unfinished-answer]]).
|
| 102 |
-
# β ADMITTING HERE IS NOT A WIDENING, and the ruling is wave 27 / D-72: on a `ut_*` database
|
| 103 |
-
# THE TENANT IS THE UNIT β `scoped_pool` itself carries "no per-row owner filter; the
|
| 104 |
-
# table-level wall is the WHOLE wall". `_defn_or_refuse` has already run inside
|
| 105 |
-
# `scoped_pids` and answered 404/403. The pid set was only ever an existence check.
|
| 106 |
-
return (pids, bool(limits))
|
| 107 |
-
from routes_customers import MODULE as CUSTOMER_MODULE, allowed_pids
|
| 108 |
-
|
| 109 |
-
session.require(CUSTOMER_MODULE)
|
| 110 |
-
return (frozenset(allowed_pids(session)), False)
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
def _in_book(pid, session, scope):
|
| 114 |
-
pids, unbounded = _pool_or_refuse(session, scope)
|
| 115 |
-
if unbounded:
|
| 116 |
-
# The table wall passed and the row set is larger than this process will enumerate. Said
|
| 117 |
-
# out loud rather than silently admitting: R6's second sentence is that a limit which
|
| 118 |
-
# cannot be removed gets REPORTED, and this is the one place the report has no envelope to
|
| 119 |
-
# ride in.
|
| 120 |
-
print(f"[records] {scope}: pid membership unresolved (read-through beyond one window) β "
|
| 121 |
-
f"admitting on the table wall alone, per D-72")
|
| 122 |
-
return
|
| 123 |
-
if pid not in pids:
|
| 124 |
-
# 403, not 404: the record may exist, but this session may not inspect it.
|
| 125 |
-
raise err(403, "out_of_scope", "that record is not in your book")
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
def _unavailable():
|
| 129 |
-
return err(
|
| 130 |
-
503,
|
| 131 |
-
"store_unavailable",
|
| 132 |
-
"record comments are temporarily unavailable β no change was saved",
|
| 133 |
-
)
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
# β WAVE 21 (D-17): the CANONICAL path is /records/{pid}/comments β comments hang off a RECORD
|
| 137 |
-
# in whatever topic `?scope=` names, and the customer-flavoured noun was wave-19 residue (the
|
| 138 |
-
# wall was always the query param). The old path stays as an ALIAS because the shipped client
|
| 139 |
-
# still calls it; verify_api pins the canonical path AND that the alias answers, so removing
|
| 140 |
-
# the alias later is a decision, never an accident.
|
| 141 |
-
@router.get("/records/{pid}/comments")
|
| 142 |
-
@router.get("/customers/{pid}/comments")
|
| 143 |
-
def comments(pid: int, scope: str = Query(default="customer"),
|
| 144 |
-
session: Session = Depends(require_session)):
|
| 145 |
-
from core import record_comments
|
| 146 |
-
|
| 147 |
-
scope = _scope_or_400(scope)
|
| 148 |
-
_in_book(pid, session, scope)
|
| 149 |
-
try:
|
| 150 |
-
rows = record_comments.list_comments(session.runtime, pid, scope=scope)
|
| 151 |
-
except record_comments.CommentsUnavailable:
|
| 152 |
-
raise _unavailable()
|
| 153 |
-
return {"comments": rows}
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
@router.post("/records/{pid}/comments", status_code=201)
|
| 157 |
-
@router.post("/customers/{pid}/comments", status_code=201)
|
| 158 |
-
def create_comment(
|
| 159 |
-
pid: int,
|
| 160 |
-
body: dict = Body(default=None),
|
| 161 |
-
scope: str = Query(default="customer"),
|
| 162 |
-
session: Session = Depends(require_session),
|
| 163 |
-
):
|
| 164 |
-
from core import record_comments
|
| 165 |
-
|
| 166 |
-
scope = _scope_or_400(scope)
|
| 167 |
-
_in_book(pid, session, scope)
|
| 168 |
-
try:
|
| 169 |
-
comment = record_comments.add_comment(
|
| 170 |
-
session.runtime,
|
| 171 |
-
pid,
|
| 172 |
-
(body or {}).get("body"),
|
| 173 |
-
session.uname,
|
| 174 |
-
session.user.get("name") or session.uname,
|
| 175 |
-
scope=scope,
|
| 176 |
-
)
|
| 177 |
-
except ValueError as exc:
|
| 178 |
-
raise err(400, "bad_comment", str(exc))
|
| 179 |
-
except record_comments.CommentsUnavailable:
|
| 180 |
-
raise _unavailable()
|
| 181 |
-
return {"comment": comment}
|
| 182 |
-
|
| 183 |
-
|
| 184 |
-
@router.delete("/records/{pid}/comments/{comment_id}")
|
| 185 |
-
@router.delete("/customers/{pid}/comments/{comment_id}")
|
| 186 |
-
def remove_comment(
|
| 187 |
-
pid: int,
|
| 188 |
-
comment_id: str,
|
| 189 |
-
scope: str = Query(default="customer"),
|
| 190 |
-
session: Session = Depends(require_session),
|
| 191 |
-
):
|
| 192 |
-
from core import record_comments
|
| 193 |
-
|
| 194 |
-
scope = _scope_or_400(scope)
|
| 195 |
-
_in_book(pid, session, scope)
|
| 196 |
-
try:
|
| 197 |
-
deleted = record_comments.delete_comment(
|
| 198 |
-
session.runtime,
|
| 199 |
-
pid,
|
| 200 |
-
comment_id,
|
| 201 |
-
session.uname,
|
| 202 |
-
admin=session.admin,
|
| 203 |
-
scope=scope,
|
| 204 |
-
)
|
| 205 |
-
except record_comments.CommentForbidden:
|
| 206 |
-
raise err(403, "comment_forbidden", "only the author may delete this comment")
|
| 207 |
-
except record_comments.CommentsUnavailable:
|
| 208 |
-
raise _unavailable()
|
| 209 |
-
if not deleted:
|
| 210 |
-
raise err(404, "comment_not_found", "that comment no longer exists")
|
| 211 |
-
return {"ok": True, "id": comment_id}
|
|
|
|
| 1 |
+
"""Record detail routes: durable comments, scoped to the caller's book β ON EVERY DATABASE.
|
| 2 |
+
|
| 3 |
+
β WAVE 19 (owner item 12). This file used to be the CUSTOMER record's comment routes with a
|
| 4 |
+
customer-shaped wall bolted to the module import line: `_in_book` asked
|
| 5 |
+
`routes_customers.allowed_pids` whatever surface the browser was on. Opening a PRODUCT record and
|
| 6 |
+
typing a comment therefore asked the customer book about a CRC32 hash of a SKU code, and the
|
| 7 |
+
panel answered "that customer is not in your book" β the owner's report. The dangerous half is
|
| 8 |
+
the one nobody sees: a hash that collides with a real partner id passes the wall, and the comment
|
| 9 |
+
is filed against somebody's customer where the whole team can read it.
|
| 10 |
+
|
| 11 |
+
THE SHAPE NOW: `?scope=` names the database (the same vocabulary `/workspace?scope=` and the
|
| 12 |
+
events route's `scopeKey` already speak), and `_pool_or_refuse` resolves BOTH halves of the wall
|
| 13 |
+
per scope β the GRANT and the ROW SET β by asking that topic's own route, never by re-deriving
|
| 14 |
+
one here:
|
| 15 |
+
|
| 16 |
+
customer / cohort `routes_customers.allowed_pids` behind the `customer_data` grant
|
| 17 |
+
product `routes_products.scoped_pool` behind the `product_data` grant
|
| 18 |
+
ut_<slug> `routes_tables.scoped_pids`, whose `_defn_or_refuse` IS the wall
|
| 19 |
+
(404 unknown / 403 not yours β a user table has no module grant).
|
| 20 |
+
β W33-T03/D-183: `scoped_pIDs`, not `scoped_pOOL` β the pool builds every
|
| 21 |
+
ROW to derive a pid set this module discards, and RAISES 409 on a
|
| 22 |
+
read-through grid past one window, which is why the record drawer painted
|
| 23 |
+
an error page on `ut_odoo_gl_lines`.
|
| 24 |
+
|
| 25 |
+
β THE PATH KEEPS ITS `/customers/` SEGMENT. It is the shipped URL and `verify_api.py`'s E1a
|
| 26 |
+
section pins it; the scope now travels beside it explicitly. A nicer noun is not worth churning
|
| 27 |
+
another session's gate mid-wave β the WALL is the query parameter, not the word.
|
| 28 |
+
|
| 29 |
+
β NO DEFAULT BEYOND THE LEGACY ONE. An absent `scope` means `customer`, which is what every
|
| 30 |
+
shipped client sent and what keeps the old callers byte-identical; an UNRECOGNISED scope is a
|
| 31 |
+
400, never a silent fallback to the customer book (`routes_grid._scope_or_400`'s rule, and for
|
| 32 |
+
the same reason: a typo served as `customer` answers a question nobody asked).
|
| 33 |
+
"""
|
| 34 |
+
from fastapi import APIRouter, Body, Depends, Query
|
| 35 |
+
|
| 36 |
+
from deps import Session, err, require_session
|
| 37 |
+
|
| 38 |
+
router = APIRouter(prefix="/api/v1")
|
| 39 |
+
|
| 40 |
+
#: The customer topic's two names β one book, two surfaces (the Cohort page is the customer table
|
| 41 |
+
#: over hand-curated sets). Mirrors `modules.cohort.LEGACY_SCOPES` / `core.record_comments`.
|
| 42 |
+
_CUSTOMER_SCOPES = ("", "customer", "cohort")
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def _scope_or_400(raw):
|
| 46 |
+
scope = str(raw or "customer").strip().lower()
|
| 47 |
+
if scope in _CUSTOMER_SCOPES or scope == "product" or scope.startswith("ut_"):
|
| 48 |
+
return "customer" if scope in _CUSTOMER_SCOPES else scope
|
| 49 |
+
raise err(400, "bad_scope",
|
| 50 |
+
"scope must be customer, cohort, product or a ut_ database β refusing to guess")
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def _pool_or_refuse(session: Session, scope: str):
|
| 54 |
+
"""The pids this session may attach comments to ON THIS DATABASE β grant wall included.
|
| 55 |
+
|
| 56 |
+
Returns **`(pids, unbounded)`** β a 2-tuple on EVERY branch. Raises the topic's own 403/404/503,
|
| 57 |
+
so a caller who may not open the surface never learns anything about the row they asked about.
|
| 58 |
+
|
| 59 |
+
β `unbounded` is TRUE only when the row set could not be ENUMERATED (a read-through grid past
|
| 60 |
+
one window), never when it is merely EMPTY. Those are opposite answers and `scoped_pids` returns
|
| 61 |
+
`frozenset()` for both β see the branch below.
|
| 62 |
+
β THE SHAPE IS A CONTRACT EVEN THOUGH THIS FUNCTION IS PRIVATE, and it has two consumers that
|
| 63 |
+
do not travel together: `_in_book` here, and a NEGATIVE CONTROL in `aios-web/api/verify_scopes.py`
|
| 64 |
+
that REPLACES this function with its own lambda. A gate's test double is a caller
|
| 65 |
+
([[test-double-patched-by-a-name-list]]); when this signature moved, that double kept returning a
|
| 66 |
+
bare frozenset and the section died on `ValueError: too many values to unpack` β no tally, no
|
| 67 |
+
failing name. Change the shape here and that double changes with it.
|
| 68 |
+
"""
|
| 69 |
+
if scope == "product":
|
| 70 |
+
from routes_products import MODULE as PRODUCT_MODULE, scoped_pool
|
| 71 |
+
|
| 72 |
+
session.require(PRODUCT_MODULE)
|
| 73 |
+
pids, _team, _rows, _fields = scoped_pool(session)
|
| 74 |
+
# β `(pids, unbounded)` on EVERY branch. This one returned a bare frozenset for ten minutes
|
| 75 |
+
# after the `ut_` branch grew its second element, and `_in_book`'s unpack would have raised
|
| 76 |
+
# a `TypeError` β a 500 on every product comment β while both other branches worked. A
|
| 77 |
+
# return shape is a contract even when the function is private.
|
| 78 |
+
return (pids, False)
|
| 79 |
+
if scope.startswith("ut_"):
|
| 80 |
+
# No module grant exists for a user table β `_defn_or_refuse` inside `scoped_pids` IS
|
| 81 |
+
# the wall (creator or admin, fail-closed), and it answers 404 before 403 exactly as the
|
| 82 |
+
# rows routes do.
|
| 83 |
+
#
|
| 84 |
+
# ββ W33-T03 / D-183 β `scoped_pids`, NOT `scoped_pool`, AND THAT ONE WORD IS THE BUG.
|
| 85 |
+
# `scoped_pool` builds every ROW to derive a pid set this function then throws away, and on
|
| 86 |
+
# a read-through grid larger than one window it RAISES `409 window_required`. So opening the
|
| 87 |
+
# record drawer on `ut_odoo_gl_lines` (975,137 rows) painted an error page β for a panel
|
| 88 |
+
# that renders comments about ONE row it already has. `scoped_pids` answers the identical
|
| 89 |
+
# question (its docstring: *"the pid set is IDENTICAL, not merely equivalent"*) and takes
|
| 90 |
+
# W31-T20's `limits` OUT-PARAMETER instead of raising, which is the same shape `/workspace`
|
| 91 |
+
# used to become openable on those two grids.
|
| 92 |
+
from routes_tables import scoped_pids
|
| 93 |
+
|
| 94 |
+
limits = []
|
| 95 |
+
pids, _fields, _defn = scoped_pids(session, scope, limits=limits)
|
| 96 |
+
# β AN EMPTY PID SET AND AN UNRESOLVABLE ONE ARE OPPOSITE ANSWERS, and collapsing them is
|
| 97 |
+
# how a fail-closed default becomes a lie. `scoped_pids` returns `frozenset()` BOTH for a
|
| 98 |
+
# database with no rows and for a read-through grid too big to enumerate β it distinguishes
|
| 99 |
+
# them by APPENDING R6's sentence to `limits`. Without this branch the drawer would move
|
| 100 |
+
# from a 409 error page to a 403 "not in your book" on a row the user is looking at, which
|
| 101 |
+
# is the same defect wearing a politer message ([[empty-answer-vs-unfinished-answer]]).
|
| 102 |
+
# β ADMITTING HERE IS NOT A WIDENING, and the ruling is wave 27 / D-72: on a `ut_*` database
|
| 103 |
+
# THE TENANT IS THE UNIT β `scoped_pool` itself carries "no per-row owner filter; the
|
| 104 |
+
# table-level wall is the WHOLE wall". `_defn_or_refuse` has already run inside
|
| 105 |
+
# `scoped_pids` and answered 404/403. The pid set was only ever an existence check.
|
| 106 |
+
return (pids, bool(limits))
|
| 107 |
+
from routes_customers import MODULE as CUSTOMER_MODULE, allowed_pids
|
| 108 |
+
|
| 109 |
+
session.require(CUSTOMER_MODULE)
|
| 110 |
+
return (frozenset(allowed_pids(session)), False)
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def _in_book(pid, session, scope):
|
| 114 |
+
pids, unbounded = _pool_or_refuse(session, scope)
|
| 115 |
+
if unbounded:
|
| 116 |
+
# The table wall passed and the row set is larger than this process will enumerate. Said
|
| 117 |
+
# out loud rather than silently admitting: R6's second sentence is that a limit which
|
| 118 |
+
# cannot be removed gets REPORTED, and this is the one place the report has no envelope to
|
| 119 |
+
# ride in.
|
| 120 |
+
print(f"[records] {scope}: pid membership unresolved (read-through beyond one window) β "
|
| 121 |
+
f"admitting on the table wall alone, per D-72")
|
| 122 |
+
return
|
| 123 |
+
if pid not in pids:
|
| 124 |
+
# 403, not 404: the record may exist, but this session may not inspect it.
|
| 125 |
+
raise err(403, "out_of_scope", "that record is not in your book")
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
def _unavailable():
|
| 129 |
+
return err(
|
| 130 |
+
503,
|
| 131 |
+
"store_unavailable",
|
| 132 |
+
"record comments are temporarily unavailable β no change was saved",
|
| 133 |
+
)
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
# β WAVE 21 (D-17): the CANONICAL path is /records/{pid}/comments β comments hang off a RECORD
|
| 137 |
+
# in whatever topic `?scope=` names, and the customer-flavoured noun was wave-19 residue (the
|
| 138 |
+
# wall was always the query param). The old path stays as an ALIAS because the shipped client
|
| 139 |
+
# still calls it; verify_api pins the canonical path AND that the alias answers, so removing
|
| 140 |
+
# the alias later is a decision, never an accident.
|
| 141 |
+
@router.get("/records/{pid}/comments")
|
| 142 |
+
@router.get("/customers/{pid}/comments")
|
| 143 |
+
def comments(pid: int, scope: str = Query(default="customer"),
|
| 144 |
+
session: Session = Depends(require_session)):
|
| 145 |
+
from core import record_comments
|
| 146 |
+
|
| 147 |
+
scope = _scope_or_400(scope)
|
| 148 |
+
_in_book(pid, session, scope)
|
| 149 |
+
try:
|
| 150 |
+
rows = record_comments.list_comments(session.runtime, pid, scope=scope)
|
| 151 |
+
except record_comments.CommentsUnavailable:
|
| 152 |
+
raise _unavailable()
|
| 153 |
+
return {"comments": rows}
|
| 154 |
+
|
| 155 |
+
|
| 156 |
+
@router.post("/records/{pid}/comments", status_code=201)
|
| 157 |
+
@router.post("/customers/{pid}/comments", status_code=201)
|
| 158 |
+
def create_comment(
|
| 159 |
+
pid: int,
|
| 160 |
+
body: dict = Body(default=None),
|
| 161 |
+
scope: str = Query(default="customer"),
|
| 162 |
+
session: Session = Depends(require_session),
|
| 163 |
+
):
|
| 164 |
+
from core import record_comments
|
| 165 |
+
|
| 166 |
+
scope = _scope_or_400(scope)
|
| 167 |
+
_in_book(pid, session, scope)
|
| 168 |
+
try:
|
| 169 |
+
comment = record_comments.add_comment(
|
| 170 |
+
session.runtime,
|
| 171 |
+
pid,
|
| 172 |
+
(body or {}).get("body"),
|
| 173 |
+
session.uname,
|
| 174 |
+
session.user.get("name") or session.uname,
|
| 175 |
+
scope=scope,
|
| 176 |
+
)
|
| 177 |
+
except ValueError as exc:
|
| 178 |
+
raise err(400, "bad_comment", str(exc))
|
| 179 |
+
except record_comments.CommentsUnavailable:
|
| 180 |
+
raise _unavailable()
|
| 181 |
+
return {"comment": comment}
|
| 182 |
+
|
| 183 |
+
|
| 184 |
+
@router.delete("/records/{pid}/comments/{comment_id}")
|
| 185 |
+
@router.delete("/customers/{pid}/comments/{comment_id}")
|
| 186 |
+
def remove_comment(
|
| 187 |
+
pid: int,
|
| 188 |
+
comment_id: str,
|
| 189 |
+
scope: str = Query(default="customer"),
|
| 190 |
+
session: Session = Depends(require_session),
|
| 191 |
+
):
|
| 192 |
+
from core import record_comments
|
| 193 |
+
|
| 194 |
+
scope = _scope_or_400(scope)
|
| 195 |
+
_in_book(pid, session, scope)
|
| 196 |
+
try:
|
| 197 |
+
deleted = record_comments.delete_comment(
|
| 198 |
+
session.runtime,
|
| 199 |
+
pid,
|
| 200 |
+
comment_id,
|
| 201 |
+
session.uname,
|
| 202 |
+
admin=session.admin,
|
| 203 |
+
scope=scope,
|
| 204 |
+
)
|
| 205 |
+
except record_comments.CommentForbidden:
|
| 206 |
+
raise err(403, "comment_forbidden", "only the author may delete this comment")
|
| 207 |
+
except record_comments.CommentsUnavailable:
|
| 208 |
+
raise _unavailable()
|
| 209 |
+
if not deleted:
|
| 210 |
+
raise err(404, "comment_not_found", "that comment no longer exists")
|
| 211 |
+
return {"ok": True, "id": comment_id}
|
api/routes_shares.py
CHANGED
|
@@ -1,393 +1,393 @@
|
|
| 1 |
-
"""routes_shares.py β the manage-access surface (wave 20, owner ruling R10, contract C-SHARE).
|
| 2 |
-
|
| 3 |
-
GET /api/v1/share/{kind}/{oid} -> {owner, entries:[{user,role}], mayAdminister, people}
|
| 4 |
-
PUT /api/v1/share/{kind}/{oid} <- {entries:[{user,role}]} (REPLACES the set)
|
| 5 |
-
GET /api/v1/share/mine -> {view:[id], folder:[id], database:[id]}
|
| 6 |
-
|
| 7 |
-
`kind` β view | folder | database. Roles are `view` | `edit` β the same two words the view rail
|
| 8 |
-
already speaks, now extended to folders and databases so there is ONE vocabulary in the UI
|
| 9 |
-
(R10: "the same picker views use").
|
| 10 |
-
|
| 11 |
-
β **RE-SHARING IS THE OWNER'S, AND THAT IS ENFORCED HERE, NOT IN THE CLIENT.** `PUT` requires
|
| 12 |
-
`shares.may_administer` (owner or admin). A collaborator with `edit` may change an object's
|
| 13 |
-
CONTENT and may not change who else can reach it β otherwise anyone you shared a view with could
|
| 14 |
-
widen it to everyone, or grant themselves ownership and lock you out. The client greys the editor
|
| 15 |
-
for non-administrators; that is a courtesy, and this check is the wall.
|
| 16 |
-
|
| 17 |
-
β **THE GRANT NEVER WIDENS PAST THE MODULE WALL β ON A GOVERNED MODULE.** `*` ("everyone") means
|
| 18 |
-
every account that can already open the surface: `require_session` plus the topic's own gate run
|
| 19 |
-
first, and for `customer_data` / `product_data` the receiver's own row scope and hidden-field
|
| 20 |
-
closure run BEFORE any foreign view is merged. Sharing there can only narrow-or-equal the set that
|
| 21 |
-
could already reach the data ([[aios-permissioning]]).
|
| 22 |
-
|
| 23 |
-
ββ **AND THAT SENTENCE IS FALSE FOR `kind='database'`, WHICH IS WHY IT NOW SAYS "ON A GOVERNED
|
| 24 |
-
MODULE" (W32-T26, audit S-8).** `routes_admin._PERM_MODULES` is `("customer_data","product_data")`
|
| 25 |
-
and `_clean_perms` **400s** on anything else, so **no row filter and no hidden field can even be
|
| 26 |
-
DECLARED for a `ut_*` database** β `routes_tables.py` makes zero `perm_scope` calls and passes
|
| 27 |
-
`hidden_keys=frozenset()`. There is no module wall behind a user table for a grant to be bounded
|
| 28 |
-
by: **this registry IS the wall.** So a `database` grant is ALL-OR-NOTHING β every row, every
|
| 29 |
-
column β and an `*` database grant admits every account in the tenant to all of it.
|
| 30 |
-
That is a real capability, deliberately kept; what was wrong was a docstring promising a second
|
| 31 |
-
wall that does not exist for this kind. Scoping user tables is booked, not done
|
| 32 |
-
(`waves/wave32/sharing-audit.md` S-8).
|
| 33 |
-
|
| 34 |
-
β **TWO SYSTEMS ANSWER "IS THIS SHARED", AND THEY ARE NOT THE SAME ONE (audit S-4).** THIS
|
| 35 |
-
registry decides who appears in *"Shared with me"* and who may re-share. **`table_store.is_shared`
|
| 36 |
-
β the view's own `permissions` β is what actually decides who may OPEN a view.** A grant here
|
| 37 |
-
whose object is invisible under that one is a row in a list that opens a refusal, which is what
|
| 38 |
-
made item 18 worth auditing. `_entries_or_400` closes the common cause (a name nobody has), but
|
| 39 |
-
the two vocabularies are still two.
|
| 40 |
-
"""
|
| 41 |
-
from fastapi import APIRouter, Body, Depends
|
| 42 |
-
|
| 43 |
-
import core.shares as shares
|
| 44 |
-
import core.users as users
|
| 45 |
-
from deps import Session, err, require_session
|
| 46 |
-
# β W32-T28 (C3) β the SHARE notification's topic word, imported from the module that CLASSIFIES
|
| 47 |
-
# it (`routes_alerts.notification_view`) rather than typed again here. The producer and the
|
| 48 |
-
# reader agreeing about one string is the whole difference between an Inbox row that opens the
|
| 49 |
-
# shared database and one that is quietly unclickable.
|
| 50 |
-
from routes_alerts import SHARE_TOPIC as _SHARE_TOPIC
|
| 51 |
-
|
| 52 |
-
router = APIRouter(prefix="/api/v1")
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
def _kind_or_400(raw):
|
| 56 |
-
try:
|
| 57 |
-
return shares._check_kind(raw)
|
| 58 |
-
except ValueError as e:
|
| 59 |
-
raise err(400, "bad_kind", str(e))
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
# ββ ββ WAVE 32 Β· T26 (owner item 18, ruling R12) β THE WALL THIS FILE SAID IT HAD βββββββββββββ
|
| 63 |
-
#
|
| 64 |
-
# `put_share`'s comment used to justify the first-claim rule with *"reaching this route at all
|
| 65 |
-
# means passing the surface's own wall"*. **There was no such wall.** `kind` and `oid` are free
|
| 66 |
-
# strings off the URL and the only dependency was `require_session`, so any signed-in account
|
| 67 |
-
# could `PUT` a grant on an id it had never seen. Because the 403 sat behind `if rec["owner"]`,
|
| 68 |
-
# an object with no grant record skipped the check entirely and the caller was stamped OWNER β
|
| 69 |
-
# sticky, so **the real creator was then refused on their own view, permanently.** Driven, not
|
| 70 |
-
# argued: `waves/wave32/sharing-audit.md` S-1 carries the four-step transcript.
|
| 71 |
-
#
|
| 72 |
-
# β AND IT WAS SILENT ON BOTH SIDES. The claimant does not even see the object in their own
|
| 73 |
-
# "Shared with me" (`shared_with` excludes what you own), so nothing appears anywhere until the
|
| 74 |
-
# victim next opens the dialog.
|
| 75 |
-
|
| 76 |
-
#: The built-in grid topics. A view or folder lives in `{topic}_table_workspace`, and the share
|
| 77 |
-
#: route is not told which topic β so resolving one means asking each.
|
| 78 |
-
_BUILTIN_TOPICS = ("customer", "product")
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
def _topics(session):
|
| 82 |
-
"""Every topic whose workspace could hold a view or folder for this tenant.
|
| 83 |
-
|
| 84 |
-
β `all_defs`, never `all_tables` β the latter is the whole 28.6 MB row payload (~703 ms on
|
| 85 |
-
tenant #0) to answer a question about KEYS (D-185).
|
| 86 |
-
"""
|
| 87 |
-
try:
|
| 88 |
-
import core.user_tables as ut
|
| 89 |
-
return (*_BUILTIN_TOPICS, *(ut.all_defs(st=session.runtime) or {}))
|
| 90 |
-
except Exception: # noqa: BLE001
|
| 91 |
-
return _BUILTIN_TOPICS
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
def _owns_object(session, kind, oid):
|
| 95 |
-
"""May this caller CLAIM an object that has no grant record yet β i.e. do they own it?
|
| 96 |
-
|
| 97 |
-
β THIS GUARDS THE CLAIM, NOT THE READ, AND THAT IS DELIBERATE. Resolving a view means asking
|
| 98 |
-
each topic's workspace in turn, which is N store reads; making every share call pay that
|
| 99 |
-
would put a loop on a route the manage-access dialog opens. The dangerous path is the one
|
| 100 |
-
where a caller is about to be stamped OWNER of something nobody owns β so the resolution runs
|
| 101 |
-
exactly there, and the common path (a record exists, `may_administer` decides) is untouched.
|
| 102 |
-
"""
|
| 103 |
-
if session.admin:
|
| 104 |
-
return True
|
| 105 |
-
if kind == "database":
|
| 106 |
-
# β `may_open` is THE resolver for a user table (its own docstring says so) and already
|
| 107 |
-
# admits creator, admin, or a `database` grantee. Re-implementing "who owns a table"
|
| 108 |
-
# here would be the second definition this wave keeps finding.
|
| 109 |
-
try:
|
| 110 |
-
import core.user_tables as ut
|
| 111 |
-
return bool(ut.may_open(oid, session.uname, is_admin=session.admin,
|
| 112 |
-
st=session.runtime))
|
| 113 |
-
except Exception: # noqa: BLE001
|
| 114 |
-
return False
|
| 115 |
-
try:
|
| 116 |
-
import core.table_store as table_store
|
| 117 |
-
except Exception: # noqa: BLE001
|
| 118 |
-
return False
|
| 119 |
-
for topic in _topics(session):
|
| 120 |
-
try:
|
| 121 |
-
ops = table_store.make(f"{topic}_table_workspace", st=session.runtime)
|
| 122 |
-
hit = ops.find_view(oid) if kind == "view" else ops.find_folder(oid)
|
| 123 |
-
except Exception: # noqa: BLE001
|
| 124 |
-
continue
|
| 125 |
-
if hit:
|
| 126 |
-
# `find_view`/`find_folder` answer `(owner_username, β¦)`. The claim belongs to the
|
| 127 |
-
# person whose personal stratum holds it β anybody else reaching this line is
|
| 128 |
-
# exactly the case S-1 describes.
|
| 129 |
-
return str(hit[0]) == str(session.uname)
|
| 130 |
-
return False
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
def _can_see_object(session, kind, oid):
|
| 134 |
-
"""May this caller READ an object's grant list β i.e. can they reach the object at all?
|
| 135 |
-
|
| 136 |
-
ββ THIS IS DELIBERATELY WIDER THAN {@link _owns_object}, AND CONFLATING THE TWO IS A
|
| 137 |
-
REGRESSION I SHIPPED AND CAUGHT. The first version of T26 guarded BOTH doors with the
|
| 138 |
-
ownership test, which reads sensibly and is wrong for the read, because **`find_view` searches
|
| 139 |
-
PERSONAL STRATA ONLY** (its own docstring says so). So a view living in alice's stratum with
|
| 140 |
-
`permissions.edit = "collaborative"` and no grant record yet β a view bob **can open and edit
|
| 141 |
-
in the grid** β answered `404` when bob opened its manage-access dialog. Measured before
|
| 142 |
-
fixing: `table_store._may_see(view, "bob") is True` while `GET /share/view/vc` said
|
| 143 |
-
`404 no_object`.
|
| 144 |
-
β THAT IS THE AUDIT'S OWN S-4 BITING THE AUDIT'S OWN FIX: two systems answer "is this shared",
|
| 145 |
-
and the wall consulted the grant registry (system A) plus stratum ownership, never the view's
|
| 146 |
-
`permissions` (system B) β which is the one that actually decides who may OPEN it.
|
| 147 |
-
β And it hides the ANSWER, not just the editor. `ViewSidebar`'s Share row is deliberately not
|
| 148 |
-
gated on edit rights because *"hiding the row from everyone else would hide the ANSWER too β
|
| 149 |
-
'who has this?' is a fair question for anyone the view was shared with"*. A 404 there tells a
|
| 150 |
-
legitimate collaborator their view does not exist.
|
| 151 |
-
|
| 152 |
-
β THE CLAIM KEEPS THE NARROW TEST. Being able to SEE an object must not let you become its
|
| 153 |
-
owner β that is S-1, and widening this predicate onto `put_share` would re-open it.
|
| 154 |
-
"""
|
| 155 |
-
if _owns_object(session, kind, oid):
|
| 156 |
-
return True
|
| 157 |
-
if kind != "view":
|
| 158 |
-
# A folder carries no per-object visibility flag of its own, and a database's `may_open`
|
| 159 |
-
# (inside `_owns_object`) already admits grantees. Nothing wider to ask.
|
| 160 |
-
return False
|
| 161 |
-
try:
|
| 162 |
-
import core.table_store as table_store
|
| 163 |
-
for topic in _topics(session):
|
| 164 |
-
hit = table_store.make(f"{topic}_table_workspace", st=session.runtime).find_view(oid)
|
| 165 |
-
if hit:
|
| 166 |
-
return bool(table_store._may_see(hit[1] if len(hit) > 1 else {},
|
| 167 |
-
session.uname, is_admin=session.admin))
|
| 168 |
-
except Exception: # noqa: BLE001
|
| 169 |
-
return False
|
| 170 |
-
return False
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
def _entries_or_400(session, entries):
|
| 174 |
-
"""Validate a grant list against the tenant's REAL, ACTIVE accounts β and refuse BY NAME.
|
| 175 |
-
|
| 176 |
-
β `core.shares._clean_entries` silently drops junk, and its docstring argues that correctly:
|
| 177 |
-
a UI mid-save must not lose the whole list to one malformed row. **But it validates the SHAPE
|
| 178 |
-
of a string and the role word β never that the user EXISTS, is ACTIVE, or is in this tenant**,
|
| 179 |
-
so a typo'd name is stored, reported as a successful save, and never reaches anybody. The
|
| 180 |
-
sharer believes the person has access. That is item 18's plain reading.
|
| 181 |
-
β The correct population is computed THREE FUNCTIONS BELOW and served to the picker
|
| 182 |
-
(`_people`). One route, two populations, and the write door was the permissive one.
|
| 183 |
-
β `*` (everyone) is not a user and is admitted deliberately β it is R10's vocabulary for
|
| 184 |
-
"every account that can already open the surface".
|
| 185 |
-
"""
|
| 186 |
-
known = {p["username"].strip().lower() for p in _people(session.tenant)}
|
| 187 |
-
unknown = []
|
| 188 |
-
for e in entries or ():
|
| 189 |
-
if not isinstance(e, dict):
|
| 190 |
-
continue
|
| 191 |
-
user = str(e.get("user") or "").strip().lower()
|
| 192 |
-
if user and user != shares.EVERYONE and user not in known:
|
| 193 |
-
unknown.append(user)
|
| 194 |
-
if unknown:
|
| 195 |
-
raise err(400, "unknown_people",
|
| 196 |
-
"no active account in this workspace is named "
|
| 197 |
-
+ ", ".join(sorted(set(unknown)))
|
| 198 |
-
+ " β nothing was shared. Pick people from the list rather than typing a name.")
|
| 199 |
-
|
| 200 |
-
|
| 201 |
-
@router.get("/share/mine")
|
| 202 |
-
def my_shares(session: Session = Depends(require_session)):
|
| 203 |
-
"""Everything shared WITH me, by kind β the "Shared with me" rail section (R10).
|
| 204 |
-
|
| 205 |
-
Registered before `/share/{kind}/{oid}` so the literal path wins the match; FastAPI resolves
|
| 206 |
-
in declaration order and `mine` would otherwise be read as a `kind`, answering 400 for a URL
|
| 207 |
-
that is not malformed at all.
|
| 208 |
-
"""
|
| 209 |
-
return shares.shared_with(session.uname, st=session.runtime)
|
| 210 |
-
|
| 211 |
-
|
| 212 |
-
@router.get("/share/{kind}/{oid}")
|
| 213 |
-
def get_share(kind: str, oid: str, session: Session = Depends(require_session)):
|
| 214 |
-
kind = _kind_or_400(kind)
|
| 215 |
-
rec = shares.grants(kind, oid, st=session.runtime)
|
| 216 |
-
role = shares.role_for(kind, oid, session.uname, is_admin=session.admin, st=session.runtime)
|
| 217 |
-
may_admin = shares.may_administer(kind, oid, session.uname, is_admin=session.admin,
|
| 218 |
-
st=session.runtime)
|
| 219 |
-
# β W32-T26 (audit S-3) β A STRANGER LEARNS NOTHING. This route used to answer for ANY id:
|
| 220 |
-
# who owns it, everyone it is granted to, and the tenant's whole usernameβname directory β
|
| 221 |
-
# to any signed-in session, about objects it cannot open. Now a caller with no role on an
|
| 222 |
-
# object must prove they can reach it, and gets a 404 otherwise: the same answer a
|
| 223 |
-
# non-existent id gives, so the route cannot be used to probe which ids are real.
|
| 224 |
-
# β `role is None` is the cheap pre-test, so the N-topic resolution below runs only for a
|
| 225 |
-
# caller who has no relationship with the object at all.
|
| 226 |
-
if role is None and not _can_see_object(session, kind, oid):
|
| 227 |
-
raise err(404, "no_object", "no such item, or it is not shared with this account")
|
| 228 |
-
return {
|
| 229 |
-
**rec,
|
| 230 |
-
"role": role,
|
| 231 |
-
"mayAdminister": may_admin,
|
| 232 |
-
# β WAVE 21 (C1 identity fix): grant entries BIND on USERNAMES, so the picker must carry
|
| 233 |
-
# them. `assignable_people` serves bare display names because `user`-kind CELLS store
|
| 234 |
-
# display names β that list's shape cannot change without migrating cell values β so
|
| 235 |
-
# this route serves objects of its own. Existing grants that were written as lowercased
|
| 236 |
-
# display names are normalised by the wave-21 cleanup script.
|
| 237 |
-
# β W32-T26 (audit S-3) β the roster is the EDITOR's data, so it rides only for a caller
|
| 238 |
-
# who may open the editor. A read-only grantee gets the grant list (their fair question is
|
| 239 |
-
# "who else has this?") and not a directory of every account in the workspace.
|
| 240 |
-
"people": _people(session.tenant) if may_admin else [],
|
| 241 |
-
}
|
| 242 |
-
|
| 243 |
-
|
| 244 |
-
def _people(tenant):
|
| 245 |
-
"""[{username, name}] for this tenant β same population as `assignable_people`, with the
|
| 246 |
-
BINDING identity alongside the display one."""
|
| 247 |
-
try:
|
| 248 |
-
reg = users.registry() or {}
|
| 249 |
-
except Exception:
|
| 250 |
-
return []
|
| 251 |
-
want = str(tenant or '').strip().lower()
|
| 252 |
-
out = []
|
| 253 |
-
for uname, u in reg.items():
|
| 254 |
-
if not isinstance(u, dict) or u.get('active') is False:
|
| 255 |
-
continue
|
| 256 |
-
if want and str(u.get('tenant') or 'royal-imports').strip().lower() != want:
|
| 257 |
-
continue
|
| 258 |
-
out.append({"username": str(uname), "name": str(u.get('name') or uname)})
|
| 259 |
-
return sorted(out, key=lambda p: p["name"].lower())
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
@router.put("/share/{kind}/{oid}")
|
| 263 |
-
def put_share(kind: str, oid: str, body: dict = Body(default=None),
|
| 264 |
-
session: Session = Depends(require_session)):
|
| 265 |
-
kind = _kind_or_400(kind)
|
| 266 |
-
body = body or {}
|
| 267 |
-
rec = shares.grants(kind, oid, st=session.runtime)
|
| 268 |
-
# An object with NO grant record yet has no owner β the first person to share it claims it.
|
| 269 |
-
# That is safe because reaching this route at all means passing the surface's own wall, and
|
| 270 |
-
# the alternative (refusing until somebody seeds an owner) would make a brand-new folder
|
| 271 |
-
# unshareable by the person who just made it.
|
| 272 |
-
if rec["owner"]:
|
| 273 |
-
if not shares.may_administer(kind, oid, session.uname, is_admin=session.admin,
|
| 274 |
-
st=session.runtime):
|
| 275 |
-
raise err(403, "not_owner",
|
| 276 |
-
"only the owner of this item (or an administrator) can change who it is "
|
| 277 |
-
"shared with")
|
| 278 |
-
# ββ W32-T26 (audit S-1) β THE CLAIM NOW HAS A PRECONDITION. An object with no grant record
|
| 279 |
-
# is still claimed by the first person to share it β that rule is right, and refusing until
|
| 280 |
-
# somebody seeds an owner would make a brand-new folder unshareable by the person who just
|
| 281 |
-
# made it. What was missing is the half the old comment ASSERTED and the code never did: the
|
| 282 |
-
# claimant has to be able to reach the object. Without this, any signed-in account could
|
| 283 |
-
# stamp itself owner of an id it had never seen and lock the real creator out for good.
|
| 284 |
-
elif not _owns_object(session, kind, oid):
|
| 285 |
-
raise err(404, "no_object", "no such item, or it is not shared with this account")
|
| 286 |
-
entries = body.get("entries")
|
| 287 |
-
if not isinstance(entries, list):
|
| 288 |
-
raise err(400, "bad_entries",
|
| 289 |
-
"entries must be a list of {user, role} β send [] to un-share, which is how "
|
| 290 |
-
"revoking is expressed")
|
| 291 |
-
_entries_or_400(session, entries)
|
| 292 |
-
out = shares.set_grants(kind, oid, entries, owner=rec["owner"] or session.uname,
|
| 293 |
-
st=session.runtime)
|
| 294 |
-
_notify_new_grantees(session, kind, oid, before=rec["entries"], after=out.get("entries") or [])
|
| 295 |
-
return out
|
| 296 |
-
|
| 297 |
-
|
| 298 |
-
def _notify_new_grantees(session, kind, oid, before, after):
|
| 299 |
-
"""ββ W32-T28 (owner item 18's last clause, contract C3) β tell the RECEIVER, in their Inbox.
|
| 300 |
-
|
| 301 |
-
Owner item 18 ends *"being shared a database notifies the receiver"*. Until now sharing was
|
| 302 |
-
silent: the grant landed in a rail section the receiver had to notice on their own, which is
|
| 303 |
-
why "I shared it with you" and "I never saw it" were both true.
|
| 304 |
-
|
| 305 |
-
β WRITTEN ON THE SHARE, NEVER POLLED. `/notifications` re-evaluates view-ALERTS on read
|
| 306 |
-
because an alert is a live question about rows; a share is an EVENT that happened once, and
|
| 307 |
-
polling for it would mean re-deriving "was this new?" on every inbox open β the diff below
|
| 308 |
-
only exists here, at the moment the set changes.
|
| 309 |
-
|
| 310 |
-
β ONLY THE NEWLY ADDED. `PUT` REPLACES the whole entry set (revoking is expressed by absence),
|
| 311 |
-
so every save re-sends everyone who was already there. Diffing against `before` is what stops
|
| 312 |
-
a rename or a role change from ringing the bell for people whose access did not change.
|
| 313 |
-
β `*` IS NOT NOTIFIED: there is no user to name, and minting one notification per account in
|
| 314 |
-
the tenant on a single click is a broadcast nobody asked for. The rail still shows it.
|
| 315 |
-
β IT NEVER RAISES. A notification that fails must not fail the share that triggered it β the
|
| 316 |
-
grant is the user's actual intent, and `core.alerts.notify` writes with `flush='async'`.
|
| 317 |
-
"""
|
| 318 |
-
try:
|
| 319 |
-
was = {e.get("user") for e in (before or ()) if isinstance(e, dict)}
|
| 320 |
-
fresh = [str(e.get("user")) for e in (after or ())
|
| 321 |
-
if isinstance(e, dict) and e.get("user") not in was
|
| 322 |
-
and e.get("user") != shares.EVERYONE]
|
| 323 |
-
if not fresh:
|
| 324 |
-
return
|
| 325 |
-
import core.alerts as alerts
|
| 326 |
-
|
| 327 |
-
label, route, view_id = _object_ref(session, kind, oid)
|
| 328 |
-
if not route:
|
| 329 |
-
# β NO ROUTE, NO NOTIFICATION β the receiver would get a row that opens nothing, and
|
| 330 |
-
# `notification_view` would have to invent a target. Silence is the honest answer
|
| 331 |
-
# here; the rail still shows the grant under "Shared with me".
|
| 332 |
-
return
|
| 333 |
-
sharer = str(session.user.get("name") or session.uname)
|
| 334 |
-
for user in fresh:
|
| 335 |
-
# β THE SHAPE IS `routes_alerts.notification_view`'s SHARE BRANCH, and the two must
|
| 336 |
-
# agree or the Inbox row is unclickable: `topic` selects the branch and `key` becomes
|
| 337 |
-
# `alertId`, which that branch reads as the id to open. Both constants are IMPORTED
|
| 338 |
-
# from there rather than typed again β one vocabulary, one owner.
|
| 339 |
-
# ββ W33-T28 (`ASK C-14`, answered) β `actor` IS THE SENDER, AND IT IS THE ONLY WAY
|
| 340 |
-
# THE INBOX CAN NAME ONE. An alert and an automation have no person behind them and
|
| 341 |
-
# are honestly named by their machine; a SHARE has a real person, and only this call
|
| 342 |
-
# site knows who. β It is passed as its OWN field rather than recovered from the
|
| 343 |
-
# `detail` prose below: a sender parsed out of "<name> shared this with you" breaks
|
| 344 |
-
# the first time the sentence is reworded, silently, in the header
|
| 345 |
-
# [[grep-output-is-not-source]]. The prose stays as the body; this is the From.
|
| 346 |
-
alerts.notify(user, label, topic=_SHARE_TOPIC, key=route, row_id=view_id,
|
| 347 |
-
detail=f"{sharer} shared this with you", actor=sharer,
|
| 348 |
-
st=session.runtime)
|
| 349 |
-
except Exception: # noqa: BLE001
|
| 350 |
-
return
|
| 351 |
-
|
| 352 |
-
|
| 353 |
-
def _object_ref(session, kind, oid):
|
| 354 |
-
"""`(label, route, view_id)` β what to CALL the shared thing, and where it OPENS.
|
| 355 |
-
|
| 356 |
-
β THE ROUTE IS RESOLVED HERE, NOT SHAPED IN THE CONSUMER, AND THE FIRST VERSION GOT IT
|
| 357 |
-
WRONG: it put the raw `oid` in the notification's key, so a shared VIEW produced
|
| 358 |
-
`target: {module: "database", id: "view_42"}` β an instruction to open a database named
|
| 359 |
-
`view_42`. It read perfectly in the payload and would have opened nothing. **A view is not
|
| 360 |
-
addressable on its own; it is a SELECTION inside a topic's grid**, so the pair is what has to
|
| 361 |
-
travel. Caught by looking at the notification the driver actually produced, not by reading
|
| 362 |
-
the code back.
|
| 363 |
-
|
| 364 |
-
β `label` never falls back to a raw id. A notification headed `ut_leads_3f2a` tells the
|
| 365 |
-
receiver nothing they can act on, and the id is already in the target.
|
| 366 |
-
β An unresolvable object answers `route=None`, and the caller then sends NOTHING rather than
|
| 367 |
-
a row that opens nowhere.
|
| 368 |
-
"""
|
| 369 |
-
try:
|
| 370 |
-
if kind == "database":
|
| 371 |
-
import core.user_tables as ut
|
| 372 |
-
defn = (ut.all_defs(st=session.runtime) or {}).get(str(oid)) or {}
|
| 373 |
-
# A user table IS its own route key in both vocabularies (`route_for_topic`).
|
| 374 |
-
return (str(defn.get("label") or "").strip() or "A database", str(oid), "")
|
| 375 |
-
import core.table_store as table_store
|
| 376 |
-
from routes_alerts import route_for_topic
|
| 377 |
-
for topic in _topics(session):
|
| 378 |
-
ops = table_store.make(f"{topic}_table_workspace", st=session.runtime)
|
| 379 |
-
hit = ops.find_view(oid) if kind == "view" else ops.find_folder(oid)
|
| 380 |
-
if not hit:
|
| 381 |
-
continue
|
| 382 |
-
route = route_for_topic(topic)
|
| 383 |
-
if not route:
|
| 384 |
-
break
|
| 385 |
-
row = hit[1] if len(hit) > 1 else {}
|
| 386 |
-
name = str((row or {}).get("name") or "").strip()
|
| 387 |
-
# β Only a VIEW carries a selection. A folder is a rail grouping, so the target opens
|
| 388 |
-
# the grid and stops there rather than naming a view the receiver did not get.
|
| 389 |
-
return (name or ("A view" if kind == "view" else "A folder"),
|
| 390 |
-
route, str(oid) if kind == "view" else "")
|
| 391 |
-
except Exception: # noqa: BLE001
|
| 392 |
-
pass
|
| 393 |
-
return ({"view": "A view", "folder": "A folder"}.get(kind, "An item"), None, "")
|
|
|
|
| 1 |
+
"""routes_shares.py β the manage-access surface (wave 20, owner ruling R10, contract C-SHARE).
|
| 2 |
+
|
| 3 |
+
GET /api/v1/share/{kind}/{oid} -> {owner, entries:[{user,role}], mayAdminister, people}
|
| 4 |
+
PUT /api/v1/share/{kind}/{oid} <- {entries:[{user,role}]} (REPLACES the set)
|
| 5 |
+
GET /api/v1/share/mine -> {view:[id], folder:[id], database:[id]}
|
| 6 |
+
|
| 7 |
+
`kind` β view | folder | database. Roles are `view` | `edit` β the same two words the view rail
|
| 8 |
+
already speaks, now extended to folders and databases so there is ONE vocabulary in the UI
|
| 9 |
+
(R10: "the same picker views use").
|
| 10 |
+
|
| 11 |
+
β **RE-SHARING IS THE OWNER'S, AND THAT IS ENFORCED HERE, NOT IN THE CLIENT.** `PUT` requires
|
| 12 |
+
`shares.may_administer` (owner or admin). A collaborator with `edit` may change an object's
|
| 13 |
+
CONTENT and may not change who else can reach it β otherwise anyone you shared a view with could
|
| 14 |
+
widen it to everyone, or grant themselves ownership and lock you out. The client greys the editor
|
| 15 |
+
for non-administrators; that is a courtesy, and this check is the wall.
|
| 16 |
+
|
| 17 |
+
β **THE GRANT NEVER WIDENS PAST THE MODULE WALL β ON A GOVERNED MODULE.** `*` ("everyone") means
|
| 18 |
+
every account that can already open the surface: `require_session` plus the topic's own gate run
|
| 19 |
+
first, and for `customer_data` / `product_data` the receiver's own row scope and hidden-field
|
| 20 |
+
closure run BEFORE any foreign view is merged. Sharing there can only narrow-or-equal the set that
|
| 21 |
+
could already reach the data ([[aios-permissioning]]).
|
| 22 |
+
|
| 23 |
+
ββ **AND THAT SENTENCE IS FALSE FOR `kind='database'`, WHICH IS WHY IT NOW SAYS "ON A GOVERNED
|
| 24 |
+
MODULE" (W32-T26, audit S-8).** `routes_admin._PERM_MODULES` is `("customer_data","product_data")`
|
| 25 |
+
and `_clean_perms` **400s** on anything else, so **no row filter and no hidden field can even be
|
| 26 |
+
DECLARED for a `ut_*` database** β `routes_tables.py` makes zero `perm_scope` calls and passes
|
| 27 |
+
`hidden_keys=frozenset()`. There is no module wall behind a user table for a grant to be bounded
|
| 28 |
+
by: **this registry IS the wall.** So a `database` grant is ALL-OR-NOTHING β every row, every
|
| 29 |
+
column β and an `*` database grant admits every account in the tenant to all of it.
|
| 30 |
+
That is a real capability, deliberately kept; what was wrong was a docstring promising a second
|
| 31 |
+
wall that does not exist for this kind. Scoping user tables is booked, not done
|
| 32 |
+
(`waves/wave32/sharing-audit.md` S-8).
|
| 33 |
+
|
| 34 |
+
β **TWO SYSTEMS ANSWER "IS THIS SHARED", AND THEY ARE NOT THE SAME ONE (audit S-4).** THIS
|
| 35 |
+
registry decides who appears in *"Shared with me"* and who may re-share. **`table_store.is_shared`
|
| 36 |
+
β the view's own `permissions` β is what actually decides who may OPEN a view.** A grant here
|
| 37 |
+
whose object is invisible under that one is a row in a list that opens a refusal, which is what
|
| 38 |
+
made item 18 worth auditing. `_entries_or_400` closes the common cause (a name nobody has), but
|
| 39 |
+
the two vocabularies are still two.
|
| 40 |
+
"""
|
| 41 |
+
from fastapi import APIRouter, Body, Depends
|
| 42 |
+
|
| 43 |
+
import core.shares as shares
|
| 44 |
+
import core.users as users
|
| 45 |
+
from deps import Session, err, require_session
|
| 46 |
+
# β W32-T28 (C3) β the SHARE notification's topic word, imported from the module that CLASSIFIES
|
| 47 |
+
# it (`routes_alerts.notification_view`) rather than typed again here. The producer and the
|
| 48 |
+
# reader agreeing about one string is the whole difference between an Inbox row that opens the
|
| 49 |
+
# shared database and one that is quietly unclickable.
|
| 50 |
+
from routes_alerts import SHARE_TOPIC as _SHARE_TOPIC
|
| 51 |
+
|
| 52 |
+
router = APIRouter(prefix="/api/v1")
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
def _kind_or_400(raw):
|
| 56 |
+
try:
|
| 57 |
+
return shares._check_kind(raw)
|
| 58 |
+
except ValueError as e:
|
| 59 |
+
raise err(400, "bad_kind", str(e))
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
# ββ ββ WAVE 32 Β· T26 (owner item 18, ruling R12) β THE WALL THIS FILE SAID IT HAD βββββββββββββ
|
| 63 |
+
#
|
| 64 |
+
# `put_share`'s comment used to justify the first-claim rule with *"reaching this route at all
|
| 65 |
+
# means passing the surface's own wall"*. **There was no such wall.** `kind` and `oid` are free
|
| 66 |
+
# strings off the URL and the only dependency was `require_session`, so any signed-in account
|
| 67 |
+
# could `PUT` a grant on an id it had never seen. Because the 403 sat behind `if rec["owner"]`,
|
| 68 |
+
# an object with no grant record skipped the check entirely and the caller was stamped OWNER β
|
| 69 |
+
# sticky, so **the real creator was then refused on their own view, permanently.** Driven, not
|
| 70 |
+
# argued: `waves/wave32/sharing-audit.md` S-1 carries the four-step transcript.
|
| 71 |
+
#
|
| 72 |
+
# β AND IT WAS SILENT ON BOTH SIDES. The claimant does not even see the object in their own
|
| 73 |
+
# "Shared with me" (`shared_with` excludes what you own), so nothing appears anywhere until the
|
| 74 |
+
# victim next opens the dialog.
|
| 75 |
+
|
| 76 |
+
#: The built-in grid topics. A view or folder lives in `{topic}_table_workspace`, and the share
|
| 77 |
+
#: route is not told which topic β so resolving one means asking each.
|
| 78 |
+
_BUILTIN_TOPICS = ("customer", "product")
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
def _topics(session):
|
| 82 |
+
"""Every topic whose workspace could hold a view or folder for this tenant.
|
| 83 |
+
|
| 84 |
+
β `all_defs`, never `all_tables` β the latter is the whole 28.6 MB row payload (~703 ms on
|
| 85 |
+
tenant #0) to answer a question about KEYS (D-185).
|
| 86 |
+
"""
|
| 87 |
+
try:
|
| 88 |
+
import core.user_tables as ut
|
| 89 |
+
return (*_BUILTIN_TOPICS, *(ut.all_defs(st=session.runtime) or {}))
|
| 90 |
+
except Exception: # noqa: BLE001
|
| 91 |
+
return _BUILTIN_TOPICS
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def _owns_object(session, kind, oid):
|
| 95 |
+
"""May this caller CLAIM an object that has no grant record yet β i.e. do they own it?
|
| 96 |
+
|
| 97 |
+
β THIS GUARDS THE CLAIM, NOT THE READ, AND THAT IS DELIBERATE. Resolving a view means asking
|
| 98 |
+
each topic's workspace in turn, which is N store reads; making every share call pay that
|
| 99 |
+
would put a loop on a route the manage-access dialog opens. The dangerous path is the one
|
| 100 |
+
where a caller is about to be stamped OWNER of something nobody owns β so the resolution runs
|
| 101 |
+
exactly there, and the common path (a record exists, `may_administer` decides) is untouched.
|
| 102 |
+
"""
|
| 103 |
+
if session.admin:
|
| 104 |
+
return True
|
| 105 |
+
if kind == "database":
|
| 106 |
+
# β `may_open` is THE resolver for a user table (its own docstring says so) and already
|
| 107 |
+
# admits creator, admin, or a `database` grantee. Re-implementing "who owns a table"
|
| 108 |
+
# here would be the second definition this wave keeps finding.
|
| 109 |
+
try:
|
| 110 |
+
import core.user_tables as ut
|
| 111 |
+
return bool(ut.may_open(oid, session.uname, is_admin=session.admin,
|
| 112 |
+
st=session.runtime))
|
| 113 |
+
except Exception: # noqa: BLE001
|
| 114 |
+
return False
|
| 115 |
+
try:
|
| 116 |
+
import core.table_store as table_store
|
| 117 |
+
except Exception: # noqa: BLE001
|
| 118 |
+
return False
|
| 119 |
+
for topic in _topics(session):
|
| 120 |
+
try:
|
| 121 |
+
ops = table_store.make(f"{topic}_table_workspace", st=session.runtime)
|
| 122 |
+
hit = ops.find_view(oid) if kind == "view" else ops.find_folder(oid)
|
| 123 |
+
except Exception: # noqa: BLE001
|
| 124 |
+
continue
|
| 125 |
+
if hit:
|
| 126 |
+
# `find_view`/`find_folder` answer `(owner_username, β¦)`. The claim belongs to the
|
| 127 |
+
# person whose personal stratum holds it β anybody else reaching this line is
|
| 128 |
+
# exactly the case S-1 describes.
|
| 129 |
+
return str(hit[0]) == str(session.uname)
|
| 130 |
+
return False
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
def _can_see_object(session, kind, oid):
|
| 134 |
+
"""May this caller READ an object's grant list β i.e. can they reach the object at all?
|
| 135 |
+
|
| 136 |
+
ββ THIS IS DELIBERATELY WIDER THAN {@link _owns_object}, AND CONFLATING THE TWO IS A
|
| 137 |
+
REGRESSION I SHIPPED AND CAUGHT. The first version of T26 guarded BOTH doors with the
|
| 138 |
+
ownership test, which reads sensibly and is wrong for the read, because **`find_view` searches
|
| 139 |
+
PERSONAL STRATA ONLY** (its own docstring says so). So a view living in alice's stratum with
|
| 140 |
+
`permissions.edit = "collaborative"` and no grant record yet β a view bob **can open and edit
|
| 141 |
+
in the grid** β answered `404` when bob opened its manage-access dialog. Measured before
|
| 142 |
+
fixing: `table_store._may_see(view, "bob") is True` while `GET /share/view/vc` said
|
| 143 |
+
`404 no_object`.
|
| 144 |
+
β THAT IS THE AUDIT'S OWN S-4 BITING THE AUDIT'S OWN FIX: two systems answer "is this shared",
|
| 145 |
+
and the wall consulted the grant registry (system A) plus stratum ownership, never the view's
|
| 146 |
+
`permissions` (system B) β which is the one that actually decides who may OPEN it.
|
| 147 |
+
β And it hides the ANSWER, not just the editor. `ViewSidebar`'s Share row is deliberately not
|
| 148 |
+
gated on edit rights because *"hiding the row from everyone else would hide the ANSWER too β
|
| 149 |
+
'who has this?' is a fair question for anyone the view was shared with"*. A 404 there tells a
|
| 150 |
+
legitimate collaborator their view does not exist.
|
| 151 |
+
|
| 152 |
+
β THE CLAIM KEEPS THE NARROW TEST. Being able to SEE an object must not let you become its
|
| 153 |
+
owner β that is S-1, and widening this predicate onto `put_share` would re-open it.
|
| 154 |
+
"""
|
| 155 |
+
if _owns_object(session, kind, oid):
|
| 156 |
+
return True
|
| 157 |
+
if kind != "view":
|
| 158 |
+
# A folder carries no per-object visibility flag of its own, and a database's `may_open`
|
| 159 |
+
# (inside `_owns_object`) already admits grantees. Nothing wider to ask.
|
| 160 |
+
return False
|
| 161 |
+
try:
|
| 162 |
+
import core.table_store as table_store
|
| 163 |
+
for topic in _topics(session):
|
| 164 |
+
hit = table_store.make(f"{topic}_table_workspace", st=session.runtime).find_view(oid)
|
| 165 |
+
if hit:
|
| 166 |
+
return bool(table_store._may_see(hit[1] if len(hit) > 1 else {},
|
| 167 |
+
session.uname, is_admin=session.admin))
|
| 168 |
+
except Exception: # noqa: BLE001
|
| 169 |
+
return False
|
| 170 |
+
return False
|
| 171 |
+
|
| 172 |
+
|
| 173 |
+
def _entries_or_400(session, entries):
|
| 174 |
+
"""Validate a grant list against the tenant's REAL, ACTIVE accounts β and refuse BY NAME.
|
| 175 |
+
|
| 176 |
+
β `core.shares._clean_entries` silently drops junk, and its docstring argues that correctly:
|
| 177 |
+
a UI mid-save must not lose the whole list to one malformed row. **But it validates the SHAPE
|
| 178 |
+
of a string and the role word β never that the user EXISTS, is ACTIVE, or is in this tenant**,
|
| 179 |
+
so a typo'd name is stored, reported as a successful save, and never reaches anybody. The
|
| 180 |
+
sharer believes the person has access. That is item 18's plain reading.
|
| 181 |
+
β The correct population is computed THREE FUNCTIONS BELOW and served to the picker
|
| 182 |
+
(`_people`). One route, two populations, and the write door was the permissive one.
|
| 183 |
+
β `*` (everyone) is not a user and is admitted deliberately β it is R10's vocabulary for
|
| 184 |
+
"every account that can already open the surface".
|
| 185 |
+
"""
|
| 186 |
+
known = {p["username"].strip().lower() for p in _people(session.tenant)}
|
| 187 |
+
unknown = []
|
| 188 |
+
for e in entries or ():
|
| 189 |
+
if not isinstance(e, dict):
|
| 190 |
+
continue
|
| 191 |
+
user = str(e.get("user") or "").strip().lower()
|
| 192 |
+
if user and user != shares.EVERYONE and user not in known:
|
| 193 |
+
unknown.append(user)
|
| 194 |
+
if unknown:
|
| 195 |
+
raise err(400, "unknown_people",
|
| 196 |
+
"no active account in this workspace is named "
|
| 197 |
+
+ ", ".join(sorted(set(unknown)))
|
| 198 |
+
+ " β nothing was shared. Pick people from the list rather than typing a name.")
|
| 199 |
+
|
| 200 |
+
|
| 201 |
+
@router.get("/share/mine")
|
| 202 |
+
def my_shares(session: Session = Depends(require_session)):
|
| 203 |
+
"""Everything shared WITH me, by kind β the "Shared with me" rail section (R10).
|
| 204 |
+
|
| 205 |
+
Registered before `/share/{kind}/{oid}` so the literal path wins the match; FastAPI resolves
|
| 206 |
+
in declaration order and `mine` would otherwise be read as a `kind`, answering 400 for a URL
|
| 207 |
+
that is not malformed at all.
|
| 208 |
+
"""
|
| 209 |
+
return shares.shared_with(session.uname, st=session.runtime)
|
| 210 |
+
|
| 211 |
+
|
| 212 |
+
@router.get("/share/{kind}/{oid}")
|
| 213 |
+
def get_share(kind: str, oid: str, session: Session = Depends(require_session)):
|
| 214 |
+
kind = _kind_or_400(kind)
|
| 215 |
+
rec = shares.grants(kind, oid, st=session.runtime)
|
| 216 |
+
role = shares.role_for(kind, oid, session.uname, is_admin=session.admin, st=session.runtime)
|
| 217 |
+
may_admin = shares.may_administer(kind, oid, session.uname, is_admin=session.admin,
|
| 218 |
+
st=session.runtime)
|
| 219 |
+
# β W32-T26 (audit S-3) β A STRANGER LEARNS NOTHING. This route used to answer for ANY id:
|
| 220 |
+
# who owns it, everyone it is granted to, and the tenant's whole usernameβname directory β
|
| 221 |
+
# to any signed-in session, about objects it cannot open. Now a caller with no role on an
|
| 222 |
+
# object must prove they can reach it, and gets a 404 otherwise: the same answer a
|
| 223 |
+
# non-existent id gives, so the route cannot be used to probe which ids are real.
|
| 224 |
+
# β `role is None` is the cheap pre-test, so the N-topic resolution below runs only for a
|
| 225 |
+
# caller who has no relationship with the object at all.
|
| 226 |
+
if role is None and not _can_see_object(session, kind, oid):
|
| 227 |
+
raise err(404, "no_object", "no such item, or it is not shared with this account")
|
| 228 |
+
return {
|
| 229 |
+
**rec,
|
| 230 |
+
"role": role,
|
| 231 |
+
"mayAdminister": may_admin,
|
| 232 |
+
# β WAVE 21 (C1 identity fix): grant entries BIND on USERNAMES, so the picker must carry
|
| 233 |
+
# them. `assignable_people` serves bare display names because `user`-kind CELLS store
|
| 234 |
+
# display names β that list's shape cannot change without migrating cell values β so
|
| 235 |
+
# this route serves objects of its own. Existing grants that were written as lowercased
|
| 236 |
+
# display names are normalised by the wave-21 cleanup script.
|
| 237 |
+
# β W32-T26 (audit S-3) β the roster is the EDITOR's data, so it rides only for a caller
|
| 238 |
+
# who may open the editor. A read-only grantee gets the grant list (their fair question is
|
| 239 |
+
# "who else has this?") and not a directory of every account in the workspace.
|
| 240 |
+
"people": _people(session.tenant) if may_admin else [],
|
| 241 |
+
}
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
def _people(tenant):
|
| 245 |
+
"""[{username, name}] for this tenant β same population as `assignable_people`, with the
|
| 246 |
+
BINDING identity alongside the display one."""
|
| 247 |
+
try:
|
| 248 |
+
reg = users.registry() or {}
|
| 249 |
+
except Exception:
|
| 250 |
+
return []
|
| 251 |
+
want = str(tenant or '').strip().lower()
|
| 252 |
+
out = []
|
| 253 |
+
for uname, u in reg.items():
|
| 254 |
+
if not isinstance(u, dict) or u.get('active') is False:
|
| 255 |
+
continue
|
| 256 |
+
if want and str(u.get('tenant') or 'royal-imports').strip().lower() != want:
|
| 257 |
+
continue
|
| 258 |
+
out.append({"username": str(uname), "name": str(u.get('name') or uname)})
|
| 259 |
+
return sorted(out, key=lambda p: p["name"].lower())
|
| 260 |
+
|
| 261 |
+
|
| 262 |
+
@router.put("/share/{kind}/{oid}")
|
| 263 |
+
def put_share(kind: str, oid: str, body: dict = Body(default=None),
|
| 264 |
+
session: Session = Depends(require_session)):
|
| 265 |
+
kind = _kind_or_400(kind)
|
| 266 |
+
body = body or {}
|
| 267 |
+
rec = shares.grants(kind, oid, st=session.runtime)
|
| 268 |
+
# An object with NO grant record yet has no owner β the first person to share it claims it.
|
| 269 |
+
# That is safe because reaching this route at all means passing the surface's own wall, and
|
| 270 |
+
# the alternative (refusing until somebody seeds an owner) would make a brand-new folder
|
| 271 |
+
# unshareable by the person who just made it.
|
| 272 |
+
if rec["owner"]:
|
| 273 |
+
if not shares.may_administer(kind, oid, session.uname, is_admin=session.admin,
|
| 274 |
+
st=session.runtime):
|
| 275 |
+
raise err(403, "not_owner",
|
| 276 |
+
"only the owner of this item (or an administrator) can change who it is "
|
| 277 |
+
"shared with")
|
| 278 |
+
# ββ W32-T26 (audit S-1) β THE CLAIM NOW HAS A PRECONDITION. An object with no grant record
|
| 279 |
+
# is still claimed by the first person to share it β that rule is right, and refusing until
|
| 280 |
+
# somebody seeds an owner would make a brand-new folder unshareable by the person who just
|
| 281 |
+
# made it. What was missing is the half the old comment ASSERTED and the code never did: the
|
| 282 |
+
# claimant has to be able to reach the object. Without this, any signed-in account could
|
| 283 |
+
# stamp itself owner of an id it had never seen and lock the real creator out for good.
|
| 284 |
+
elif not _owns_object(session, kind, oid):
|
| 285 |
+
raise err(404, "no_object", "no such item, or it is not shared with this account")
|
| 286 |
+
entries = body.get("entries")
|
| 287 |
+
if not isinstance(entries, list):
|
| 288 |
+
raise err(400, "bad_entries",
|
| 289 |
+
"entries must be a list of {user, role} β send [] to un-share, which is how "
|
| 290 |
+
"revoking is expressed")
|
| 291 |
+
_entries_or_400(session, entries)
|
| 292 |
+
out = shares.set_grants(kind, oid, entries, owner=rec["owner"] or session.uname,
|
| 293 |
+
st=session.runtime)
|
| 294 |
+
_notify_new_grantees(session, kind, oid, before=rec["entries"], after=out.get("entries") or [])
|
| 295 |
+
return out
|
| 296 |
+
|
| 297 |
+
|
| 298 |
+
def _notify_new_grantees(session, kind, oid, before, after):
|
| 299 |
+
"""ββ W32-T28 (owner item 18's last clause, contract C3) β tell the RECEIVER, in their Inbox.
|
| 300 |
+
|
| 301 |
+
Owner item 18 ends *"being shared a database notifies the receiver"*. Until now sharing was
|
| 302 |
+
silent: the grant landed in a rail section the receiver had to notice on their own, which is
|
| 303 |
+
why "I shared it with you" and "I never saw it" were both true.
|
| 304 |
+
|
| 305 |
+
β WRITTEN ON THE SHARE, NEVER POLLED. `/notifications` re-evaluates view-ALERTS on read
|
| 306 |
+
because an alert is a live question about rows; a share is an EVENT that happened once, and
|
| 307 |
+
polling for it would mean re-deriving "was this new?" on every inbox open β the diff below
|
| 308 |
+
only exists here, at the moment the set changes.
|
| 309 |
+
|
| 310 |
+
β ONLY THE NEWLY ADDED. `PUT` REPLACES the whole entry set (revoking is expressed by absence),
|
| 311 |
+
so every save re-sends everyone who was already there. Diffing against `before` is what stops
|
| 312 |
+
a rename or a role change from ringing the bell for people whose access did not change.
|
| 313 |
+
β `*` IS NOT NOTIFIED: there is no user to name, and minting one notification per account in
|
| 314 |
+
the tenant on a single click is a broadcast nobody asked for. The rail still shows it.
|
| 315 |
+
β IT NEVER RAISES. A notification that fails must not fail the share that triggered it β the
|
| 316 |
+
grant is the user's actual intent, and `core.alerts.notify` writes with `flush='async'`.
|
| 317 |
+
"""
|
| 318 |
+
try:
|
| 319 |
+
was = {e.get("user") for e in (before or ()) if isinstance(e, dict)}
|
| 320 |
+
fresh = [str(e.get("user")) for e in (after or ())
|
| 321 |
+
if isinstance(e, dict) and e.get("user") not in was
|
| 322 |
+
and e.get("user") != shares.EVERYONE]
|
| 323 |
+
if not fresh:
|
| 324 |
+
return
|
| 325 |
+
import core.alerts as alerts
|
| 326 |
+
|
| 327 |
+
label, route, view_id = _object_ref(session, kind, oid)
|
| 328 |
+
if not route:
|
| 329 |
+
# β NO ROUTE, NO NOTIFICATION β the receiver would get a row that opens nothing, and
|
| 330 |
+
# `notification_view` would have to invent a target. Silence is the honest answer
|
| 331 |
+
# here; the rail still shows the grant under "Shared with me".
|
| 332 |
+
return
|
| 333 |
+
sharer = str(session.user.get("name") or session.uname)
|
| 334 |
+
for user in fresh:
|
| 335 |
+
# β THE SHAPE IS `routes_alerts.notification_view`'s SHARE BRANCH, and the two must
|
| 336 |
+
# agree or the Inbox row is unclickable: `topic` selects the branch and `key` becomes
|
| 337 |
+
# `alertId`, which that branch reads as the id to open. Both constants are IMPORTED
|
| 338 |
+
# from there rather than typed again β one vocabulary, one owner.
|
| 339 |
+
# ββ W33-T28 (`ASK C-14`, answered) β `actor` IS THE SENDER, AND IT IS THE ONLY WAY
|
| 340 |
+
# THE INBOX CAN NAME ONE. An alert and an automation have no person behind them and
|
| 341 |
+
# are honestly named by their machine; a SHARE has a real person, and only this call
|
| 342 |
+
# site knows who. β It is passed as its OWN field rather than recovered from the
|
| 343 |
+
# `detail` prose below: a sender parsed out of "<name> shared this with you" breaks
|
| 344 |
+
# the first time the sentence is reworded, silently, in the header
|
| 345 |
+
# [[grep-output-is-not-source]]. The prose stays as the body; this is the From.
|
| 346 |
+
alerts.notify(user, label, topic=_SHARE_TOPIC, key=route, row_id=view_id,
|
| 347 |
+
detail=f"{sharer} shared this with you", actor=sharer,
|
| 348 |
+
st=session.runtime)
|
| 349 |
+
except Exception: # noqa: BLE001
|
| 350 |
+
return
|
| 351 |
+
|
| 352 |
+
|
| 353 |
+
def _object_ref(session, kind, oid):
|
| 354 |
+
"""`(label, route, view_id)` β what to CALL the shared thing, and where it OPENS.
|
| 355 |
+
|
| 356 |
+
β THE ROUTE IS RESOLVED HERE, NOT SHAPED IN THE CONSUMER, AND THE FIRST VERSION GOT IT
|
| 357 |
+
WRONG: it put the raw `oid` in the notification's key, so a shared VIEW produced
|
| 358 |
+
`target: {module: "database", id: "view_42"}` β an instruction to open a database named
|
| 359 |
+
`view_42`. It read perfectly in the payload and would have opened nothing. **A view is not
|
| 360 |
+
addressable on its own; it is a SELECTION inside a topic's grid**, so the pair is what has to
|
| 361 |
+
travel. Caught by looking at the notification the driver actually produced, not by reading
|
| 362 |
+
the code back.
|
| 363 |
+
|
| 364 |
+
β `label` never falls back to a raw id. A notification headed `ut_leads_3f2a` tells the
|
| 365 |
+
receiver nothing they can act on, and the id is already in the target.
|
| 366 |
+
β An unresolvable object answers `route=None`, and the caller then sends NOTHING rather than
|
| 367 |
+
a row that opens nowhere.
|
| 368 |
+
"""
|
| 369 |
+
try:
|
| 370 |
+
if kind == "database":
|
| 371 |
+
import core.user_tables as ut
|
| 372 |
+
defn = (ut.all_defs(st=session.runtime) or {}).get(str(oid)) or {}
|
| 373 |
+
# A user table IS its own route key in both vocabularies (`route_for_topic`).
|
| 374 |
+
return (str(defn.get("label") or "").strip() or "A database", str(oid), "")
|
| 375 |
+
import core.table_store as table_store
|
| 376 |
+
from routes_alerts import route_for_topic
|
| 377 |
+
for topic in _topics(session):
|
| 378 |
+
ops = table_store.make(f"{topic}_table_workspace", st=session.runtime)
|
| 379 |
+
hit = ops.find_view(oid) if kind == "view" else ops.find_folder(oid)
|
| 380 |
+
if not hit:
|
| 381 |
+
continue
|
| 382 |
+
route = route_for_topic(topic)
|
| 383 |
+
if not route:
|
| 384 |
+
break
|
| 385 |
+
row = hit[1] if len(hit) > 1 else {}
|
| 386 |
+
name = str((row or {}).get("name") or "").strip()
|
| 387 |
+
# β Only a VIEW carries a selection. A folder is a rail grouping, so the target opens
|
| 388 |
+
# the grid and stops there rather than naming a view the receiver did not get.
|
| 389 |
+
return (name or ("A view" if kind == "view" else "A folder"),
|
| 390 |
+
route, str(oid) if kind == "view" else "")
|
| 391 |
+
except Exception: # noqa: BLE001
|
| 392 |
+
pass
|
| 393 |
+
return ({"view": "A view", "folder": "A folder"}.get(kind, "An item"), None, "")
|
api/routes_slack.py
CHANGED
|
@@ -1,772 +1,772 @@
|
|
| 1 |
-
"""routes_slack.py β MANAGE AGENT: a bot per Slack channel, walled by the engine that walls a
|
| 2 |
-
person. (wave 33, owner item 10 Β· `W33-T38` / `W33-T39` / `W33-T67`.)
|
| 3 |
-
|
| 4 |
-
Owner, verbatim (2026-08-14): *"Expand Slack we need to be able to configure an AI bot PER channel
|
| 5 |
-
whose permissions we can toggle based on Fields etc of the databases, exactly like how we toggle
|
| 6 |
-
permissioning per user."*
|
| 7 |
-
|
| 8 |
-
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 9 |
-
ββ D-84 SAID THIS WAS "adapted from OpenTag" AND THAT PREMISE IS FALSE (PRD amendment A1).
|
| 10 |
-
|
| 11 |
-
`PENDING.md` D-84 has named OpenTag as the model for per-Slack-channel permissioning since
|
| 12 |
-
2026-08-08. **OpenTag has no permission model to adapt.** `app/runtime-host.ts` is
|
| 13 |
-
`identifyUser: () => OPENTAG_SERVICE_USER` β ONE constant identity for every request β and its own
|
| 14 |
-
`sender-context.ts` says the Slack sender is *"informational only⦠not gating access."* Its
|
| 15 |
-
authorization ceiling is whole-MCP-server, decided once at process boot from env vars. It is a
|
| 16 |
-
live specimen of identity that LOOKS like a permission and is not.
|
| 17 |
-
|
| 18 |
-
`core/perm_scope.py` is already strictly more expressive: per database, per row, per FIELD, per
|
| 19 |
-
principal. So item 10 is OUR engine projected onto a CHANNEL principal β see `agent_principal`
|
| 20 |
-
below, which is the whole projection and is nine lines. Study:
|
| 21 |
-
`.claude/wiki/research/opentag-adoption.md` (licence gate: MIT, PASSED).
|
| 22 |
-
|
| 23 |
-
What OpenTag genuinely contributes is the WRITE-APPROVAL interceptor (`W33-T67`, `approval_card`
|
| 24 |
-
below) and nothing else. Its TRANSPORT is AVOID wholesale β hosted CopilotKit Intelligence, a
|
| 25 |
-
second Node process, a persistent outbound socket and each tenant's Slack credentials held by a
|
| 26 |
-
third party, against ONE process on the HF free tier. The door here is `routes_forms.py`'s proven
|
| 27 |
-
unauthenticated-public shape plus Slack's signing-secret check.
|
| 28 |
-
|
| 29 |
-
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 30 |
-
β A CHANNEL IS A PRINCIPAL, NOT A PERSON. The wall binds to the Slack conversation id, so every
|
| 31 |
-
human in that channel reads through the same rules. That is the honest reading of "a bot per
|
| 32 |
-
channel" and the pane says it out loud, because the alternative is an admin assuming per-person
|
| 33 |
-
scoping from a screen that looks exactly like the per-person one.
|
| 34 |
-
|
| 35 |
-
β AND THE IDENTITY IS THE CHANNEL ID, NEVER ITS NAME. A channel can be renamed; its id cannot.
|
| 36 |
-
A wall bound to `#sales` silently re-points the day somebody renames the channel.
|
| 37 |
-
"""
|
| 38 |
-
import hmac
|
| 39 |
-
import json
|
| 40 |
-
import os
|
| 41 |
-
import secrets
|
| 42 |
-
import time
|
| 43 |
-
from datetime import datetime, timezone
|
| 44 |
-
|
| 45 |
-
from fastapi import APIRouter, Body, Depends, Request
|
| 46 |
-
|
| 47 |
-
from deps import Session, err, require_session
|
| 48 |
-
from routes_admin import admin_gate
|
| 49 |
-
|
| 50 |
-
router = APIRouter(prefix="/api/v1")
|
| 51 |
-
|
| 52 |
-
#: The tenant's channel agents: `{id: record}`. A per-tenant bucket, so it rides
|
| 53 |
-
#: `runtime.store_key`'s prefix and never lands in tenant #0's namespace.
|
| 54 |
-
AGENTS_KEY = "slack_agents"
|
| 55 |
-
|
| 56 |
-
#: Pending mutations awaiting a human's click (`W33-T67`). Server-only, per tenant.
|
| 57 |
-
PENDING_KEY = "slack_pending_writes"
|
| 58 |
-
|
| 59 |
-
#: How long an approval card stays clickable. An approval is a statement about the world as it was
|
| 60 |
-
#: when the card was rendered; an hour later the values it names may no longer be the values it
|
| 61 |
-
#: would write. Expiring is the honest behaviour, and an expired card says so rather than 403ing.
|
| 62 |
-
APPROVAL_TTL_S = 15 * 60
|
| 63 |
-
|
| 64 |
-
#: Slack rejects a replayed request older than 5 minutes; so do we, before any signature work.
|
| 65 |
-
SLACK_MAX_SKEW_S = 60 * 5
|
| 66 |
-
|
| 67 |
-
MAX_BODY_BYTES = 64 * 1024
|
| 68 |
-
|
| 69 |
-
#: β ROW-CAPPED BECAUSE AN APPROVER WHO CANNOT READ THE CARD CANNOT APPROVE IT. Copied
|
| 70 |
-
#: deliberately from OpenTag's `agent/write_confirmation.py`, which caps for the same reason:
|
| 71 |
-
#: past a certain length Slack collapses a message behind "Show more", and a human clicking
|
| 72 |
-
#: Approve on a card whose tail they never saw is worse than no card at all.
|
| 73 |
-
APPROVAL_MAX_ROWS = 12
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
# ββ the tenant's agent records βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 77 |
-
def _agents(rt):
|
| 78 |
-
"""`{id: record}` for one tenant. `{}` on any failure β an unreadable bucket must degrade to
|
| 79 |
-
"this tenant has no agents", never to a 500 on the settings pane."""
|
| 80 |
-
try:
|
| 81 |
-
found = rt.get(AGENTS_KEY) or {}
|
| 82 |
-
except Exception: # noqa: BLE001
|
| 83 |
-
return {}
|
| 84 |
-
return found if isinstance(found, dict) else {}
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
def _clean_agent(raw, prior=None):
|
| 88 |
-
"""One stored record, built KEY BY KEY.
|
| 89 |
-
|
| 90 |
-
β Never `dict(raw)` and never `**raw`. This record is the SUBJECT of a permission decision;
|
| 91 |
-
a client-supplied key landing in it is a client-supplied input to `perm_scope`. The same rule
|
| 92 |
-
`routes_forms._public_form` follows on the way out, applied here on the way in.
|
| 93 |
-
"""
|
| 94 |
-
prior = prior if isinstance(prior, dict) else {}
|
| 95 |
-
out = {
|
| 96 |
-
"id": str(prior.get("id") or raw.get("id") or ""),
|
| 97 |
-
"channel": str(raw.get("channel") or prior.get("channel") or "").strip()[:64],
|
| 98 |
-
"channelName": str(raw.get("channelName") or prior.get("channelName") or "").strip()[:80],
|
| 99 |
-
"label": " ".join(str(raw.get("label") or prior.get("label") or "").split())[:80],
|
| 100 |
-
"active": bool(raw.get("active", prior.get("active", True))),
|
| 101 |
-
# β DEFAULT TRUE, AND `is not False` RATHER THAN TRUTHINESS. A record written before this
|
| 102 |
-
# key existed must read as "approval required": the one direction a default here is
|
| 103 |
-
# allowed to be wrong in is the safe one, because an unattended write cannot be undone by
|
| 104 |
-
# revoking the bot afterwards [[default-must-pass-its-own-guard]].
|
| 105 |
-
"writeApproval": raw.get("writeApproval", prior.get("writeApproval", True)) is not False,
|
| 106 |
-
# The permission block. Same shape, same validator, same engine as a user's.
|
| 107 |
-
"perms": prior.get("perms") if isinstance(prior.get("perms"), dict) else {},
|
| 108 |
-
"perms_v": int(prior.get("perms_v") or 0),
|
| 109 |
-
"createdBy": str(prior.get("createdBy") or raw.get("createdBy") or ""),
|
| 110 |
-
"createdAt": str(prior.get("createdAt") or raw.get("createdAt") or ""),
|
| 111 |
-
}
|
| 112 |
-
return out
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
def _put_agent(session, agent_id, mutate):
|
| 116 |
-
"""Read-modify-write ONE agent under the tenant's bucket, synchronously.
|
| 117 |
-
|
| 118 |
-
`flush="sync"` because the client re-reads the list immediately after every write here, and an
|
| 119 |
-
async flush would let the refetch win the race and paint the pre-write state β the
|
| 120 |
-
[[refetch-eats-its-own-write]] shape.
|
| 121 |
-
"""
|
| 122 |
-
def _set(cur):
|
| 123 |
-
cur = dict(cur or {})
|
| 124 |
-
rec = cur.get(agent_id) if isinstance(cur.get(agent_id), dict) else None
|
| 125 |
-
nxt = mutate(rec)
|
| 126 |
-
if nxt is None:
|
| 127 |
-
cur.pop(agent_id, None)
|
| 128 |
-
else:
|
| 129 |
-
cur[agent_id] = nxt
|
| 130 |
-
return cur
|
| 131 |
-
|
| 132 |
-
session.runtime.update(AGENTS_KEY, _set, flush="sync")
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
# ββ ββ THE PROJECTION β THE WHOLE OF "the same engine that permissions a user" βββββββββββββββ
|
| 136 |
-
def agent_principal(agent):
|
| 137 |
-
"""A channel agent, as a PRINCIPAL `core/perm_scope` already understands.
|
| 138 |
-
|
| 139 |
-
ββ THIS IS THE ENTIRE POINT OF THE TICKET AND IT IS NINE LINES. `perm_scope` takes a user
|
| 140 |
-
RECORD β a plain dict with `perms`, `perms_v` and `role` β not a User object and not a session.
|
| 141 |
-
So a channel agent carrying a `perms` block simply IS such a record, and `may_access`,
|
| 142 |
-
`visible_fields`, `hidden_keys` and `apply_row_scope` work on it unmodified. There is no second
|
| 143 |
-
engine, no parallel vocabulary, and nothing about a channel to keep in step with anything about
|
| 144 |
-
a person. That is what the owner's *"exactly like how we toggle permissioning per user"* means
|
| 145 |
-
when it is true in code rather than in appearance.
|
| 146 |
-
|
| 147 |
-
β `role` IS HARDCODED NON-ADMIN AND MUST STAY SO. `perm_scope.may_access` returns True
|
| 148 |
-
unconditionally for `role == 'admin'` β the break-glass clause that keeps the owner out of a
|
| 149 |
-
locked store. A bot has no such emergency and no hands: an agent record that could carry
|
| 150 |
-
`role: 'admin'` would be one stored key away from bypassing every wall on this page. The key is
|
| 151 |
-
not copied from the record; it is written here, every time.
|
| 152 |
-
|
| 153 |
-
β AND `perms_v` IS FORCED TO THE CURRENT VERSION, which makes an UNDECLARED database DENY
|
| 154 |
-
(C-PERM amendment 4). Without it a fresh agent would fall through to `perms.may_open`'s legacy
|
| 155 |
-
grant wall and read `modules: 'all'`-shaped defaults β i.e. a brand-new bot would start with
|
| 156 |
-
access to everything. Fail-closed from the first byte.
|
| 157 |
-
"""
|
| 158 |
-
import core.perm_scope as perm_scope
|
| 159 |
-
return {
|
| 160 |
-
"username": f"slack:{(agent or {}).get('channel') or '?'}",
|
| 161 |
-
"role": "agent",
|
| 162 |
-
"perms": (agent or {}).get("perms") or {},
|
| 163 |
-
"perms_v": perm_scope.PERMS_VERSION,
|
| 164 |
-
# No `bus`, no `agent`: a channel is not a business unit and not a salesperson. Absent
|
| 165 |
-
# rather than 'all' β `perms.allowed_bu_labels` narrows on absence, which is the direction
|
| 166 |
-
# this must fail in.
|
| 167 |
-
}
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
def agent_may_read(agent, module):
|
| 171 |
-
"""May this channel open `module` at all?"""
|
| 172 |
-
import core.perm_scope as perm_scope
|
| 173 |
-
if not (agent or {}).get("active", True):
|
| 174 |
-
return False
|
| 175 |
-
return bool(perm_scope.may_access(agent_principal(agent), module))
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
def agent_visible_fields(agent, fields, module):
|
| 179 |
-
"""`fields` minus this channel's hidden closure β the SAME transitive closure a person gets,
|
| 180 |
-
so a formula over a hidden column cannot leak it back through arithmetic."""
|
| 181 |
-
import core.perm_scope as perm_scope
|
| 182 |
-
return perm_scope.visible_fields(fields, agent_principal(agent), module)
|
| 183 |
-
|
| 184 |
-
|
| 185 |
-
def agent_rows(agent, rows, module, fields, ctx=None):
|
| 186 |
-
"""The rows this channel may receive β `permits()`, so an unanswerable permanent filter DENIES
|
| 187 |
-
rather than being ignored."""
|
| 188 |
-
import core.perm_scope as perm_scope
|
| 189 |
-
p = agent_principal(agent)
|
| 190 |
-
hide = perm_scope.hidden_keys(p, module, fields)
|
| 191 |
-
kept = perm_scope.apply_row_scope(rows, p, module, fields, ctx)
|
| 192 |
-
# Both wires, never one: the field list and the row payload are separate, and stripping only
|
| 193 |
-
# the first leaves the value sitting in the second where anyone can read it.
|
| 194 |
-
return [perm_scope.strip_row(r, hide) for r in kept]
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
# ββ the credential βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 198 |
-
def slack_creds(rt):
|
| 199 |
-
"""This TENANT's Slack credentials from its own keychain, or None.
|
| 200 |
-
|
| 201 |
-
β NO ENVIRONMENT FALLBACK, DELIBERATELY, AND THIS IS D-202's LESSON APPLIED BEFORE IT
|
| 202 |
-
HAPPENS AGAIN. Meta Ads shipped loading end-to-end and was still not a connector, because
|
| 203 |
-
`keychain.meta_creds` had no caller and the token came from the owner's `.env` β making it a
|
| 204 |
-
tenant-#0 FACT indistinguishable from a screenshot. `keychain.meta_creds`' own docstring spells
|
| 205 |
-
out the rule: handing the environment's credential to a tenant whose admin has not stored one
|
| 206 |
-
is the leak the resolver exists to prevent. So a tenant with no entry gets None and the pane
|
| 207 |
-
SAYS so.
|
| 208 |
-
|
| 209 |
-
β Reads through `list_entries` + `read_fields` (both public) rather than `keychain._first_creds`
|
| 210 |
-
(private, and in a file outside this lane's fence). Same deterministic rule: entries are
|
| 211 |
-
id-sorted, so "first" is stable across reads rather than dict-order luck.
|
| 212 |
-
"""
|
| 213 |
-
try:
|
| 214 |
-
import core.keychain as keychain
|
| 215 |
-
for row in keychain.list_entries(rt):
|
| 216 |
-
if str(row.get("type") or "") != "slack":
|
| 217 |
-
continue
|
| 218 |
-
fields = keychain.read_fields(rt, row.get("id"))
|
| 219 |
-
if isinstance(fields, dict) and fields.get("signing_secret"):
|
| 220 |
-
return fields
|
| 221 |
-
except Exception: # noqa: BLE001
|
| 222 |
-
# A locked or unreadable keychain reads as "not configured" for the PANE, which is honest;
|
| 223 |
-
# the signature check below fails closed regardless, so this cannot widen anything.
|
| 224 |
-
return None
|
| 225 |
-
return None
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
# ββ the authenticated doors (the Manage agent surface) βββββββββββββββββββββββββββββββββββββββ
|
| 229 |
-
def _summary(agent, modules):
|
| 230 |
-
"""One sentence per agent for the list row β computed here, because the alternative is one
|
| 231 |
-
round trip per row to fill one cell (the same reason `AdminUser.access` exists)."""
|
| 232 |
-
# β W36-T22 / C2 β every listed database is governed now; the `enforced` flag it used to
|
| 233 |
-
# filter on is DELETED, and `m.get("enforced")` would have gone silently falsy here and
|
| 234 |
-
# summarised every agent as "" (no databases at all).
|
| 235 |
-
governed = list(modules)
|
| 236 |
-
perms = agent.get("perms") or {}
|
| 237 |
-
open_n = sum(1 for m in governed if (perms.get(m["key"]) or {}).get("access"))
|
| 238 |
-
if not governed:
|
| 239 |
-
return ""
|
| 240 |
-
if open_n == 0:
|
| 241 |
-
return "No access"
|
| 242 |
-
restricted = sum(1 for m in governed
|
| 243 |
-
if (perms.get(m["key"]) or {}).get("access")
|
| 244 |
-
and ((perms.get(m["key"]) or {}).get("filter")
|
| 245 |
-
or (perms.get(m["key"]) or {}).get("hiddenFields")))
|
| 246 |
-
head = (f"All {open_n} database{'' if open_n == 1 else 's'}" if open_n == len(governed)
|
| 247 |
-
else f"{open_n} of {len(governed)} databases")
|
| 248 |
-
return f"{head}, {restricted} restricted" if restricted else head
|
| 249 |
-
|
| 250 |
-
|
| 251 |
-
@router.get("/agents")
|
| 252 |
-
def list_channel_agents(session: Session = Depends(admin_gate)):
|
| 253 |
-
"""This tenant's channel agents, plus whether Slack is reachable at all."""
|
| 254 |
-
import routes_admin
|
| 255 |
-
modules = routes_admin._perm_modules(session)
|
| 256 |
-
rows = []
|
| 257 |
-
for aid, a in sorted(_agents(session.runtime).items()):
|
| 258 |
-
if not isinstance(a, dict):
|
| 259 |
-
continue
|
| 260 |
-
rows.append({"id": aid, "channel": a.get("channel") or "",
|
| 261 |
-
"channelName": a.get("channelName") or "",
|
| 262 |
-
"label": a.get("label") or "", "active": a.get("active", True) is not False,
|
| 263 |
-
"writeApproval": a.get("writeApproval", True) is not False,
|
| 264 |
-
"summary": _summary(a, modules)})
|
| 265 |
-
return {"agents": rows, "configured": slack_creds(session.runtime) is not None}
|
| 266 |
-
|
| 267 |
-
|
| 268 |
-
@router.post("/agents")
|
| 269 |
-
def create_channel_agent(body: dict = Body(default=None),
|
| 270 |
-
session: Session = Depends(admin_gate)):
|
| 271 |
-
body = body if isinstance(body, dict) else {}
|
| 272 |
-
channel = str(body.get("channel") or "").strip()
|
| 273 |
-
if not channel:
|
| 274 |
-
raise err(400, "no_channel", "a Slack channel ID is required")
|
| 275 |
-
existing = _agents(session.runtime)
|
| 276 |
-
# β ONE AGENT PER CHANNEL. Two records for one conversation would mean two walls, and nothing
|
| 277 |
-
# anywhere decides which one applies β the [[one-question-two-normalizers]] shape, in the one
|
| 278 |
-
# place where the two answers are "may read" and "may not".
|
| 279 |
-
for a in existing.values():
|
| 280 |
-
if isinstance(a, dict) and str(a.get("channel") or "") == channel:
|
| 281 |
-
raise err(409, "channel_taken",
|
| 282 |
-
"that channel already has an agent β edit it instead of adding a second")
|
| 283 |
-
aid = secrets.token_urlsafe(9)
|
| 284 |
-
rec = _clean_agent(dict(body, id=aid, createdBy=session.uname,
|
| 285 |
-
createdAt=datetime.now(timezone.utc).isoformat(timespec="seconds")))
|
| 286 |
-
_put_agent(session, aid, lambda _prior: rec)
|
| 287 |
-
fresh = _agents(session.runtime).get(aid)
|
| 288 |
-
if not isinstance(fresh, dict):
|
| 289 |
-
# The store took the write and did not record it. A 200 here would tell an administrator
|
| 290 |
-
# the agent exists when it does not.
|
| 291 |
-
raise err(503, "store_unavailable", "the agent was NOT created")
|
| 292 |
-
return {"agent": {"id": aid, "channel": rec["channel"], "channelName": rec["channelName"],
|
| 293 |
-
"label": rec["label"], "active": rec["active"],
|
| 294 |
-
"writeApproval": rec["writeApproval"], "summary": "No access"}}
|
| 295 |
-
|
| 296 |
-
|
| 297 |
-
@router.get("/agents/{agent_id}")
|
| 298 |
-
def get_channel_agent(agent_id: str, session: Session = Depends(admin_gate)):
|
| 299 |
-
"""The agent PLUS the permission envelope β the SAME shape `get_perms` answers with.
|
| 300 |
-
|
| 301 |
-
β ONE PAYLOAD SHAPE FOR TWO ROOMS. `permsModel.parsePermsPayload` parses this, and
|
| 302 |
-
`settings/ModulePermsList` renders it, because they are literally the same client code. A
|
| 303 |
-
second envelope here would mean a second parser and a second set of defaults within a wave.
|
| 304 |
-
"""
|
| 305 |
-
import routes_admin
|
| 306 |
-
a = _agents(session.runtime).get(str(agent_id))
|
| 307 |
-
if not isinstance(a, dict):
|
| 308 |
-
raise err(404, "no_such_agent", "no agent with that id")
|
| 309 |
-
modules = routes_admin._perm_modules(session)
|
| 310 |
-
# ββ W36-T22 / C2 β EVERY listed database, `ut_*` included. β THIS IS WHY THE FLAG'S
|
| 311 |
-
# DELETION IS NOT A ONE-FILE CHANGE: `m.get("enforced")` on a row that no longer carries the
|
| 312 |
-
# key is None, so this list would have been EMPTY and the channel perms editor would have
|
| 313 |
-
# governed ZERO databases while looking entirely correct. A Slack channel is a principal of
|
| 314 |
-
# the SAME wall (`verify_perm_scope` section H) and gets the same catalogue.
|
| 315 |
-
governed = [m["key"] for m in modules]
|
| 316 |
-
stored = a.get("perms") or {}
|
| 317 |
-
principal = agent_principal(a)
|
| 318 |
-
import core.perm_scope as perm_scope
|
| 319 |
-
perms_out = {}
|
| 320 |
-
for k in governed:
|
| 321 |
-
e = stored.get(k)
|
| 322 |
-
# β W36-T22 β `may_read`, the same evaluator the read door uses, for the reason spelled
|
| 323 |
-
# out at `routes_admin.get_perms`. β It answers DIFFERENTLY here and correctly so: a
|
| 324 |
-
# channel agent is not a person with a share, so a `ut_*` key it has not been granted
|
| 325 |
-
# defaults CLOSED β which is the fail-closed direction for a bot, and the same answer the
|
| 326 |
-
# table routes would give it.
|
| 327 |
-
perms_out[k] = e if isinstance(e, dict) else {
|
| 328 |
-
"access": bool(perm_scope.may_read(principal, k, st=session.runtime)),
|
| 329 |
-
"filter": None, "hiddenFields": []}
|
| 330 |
-
return {"id": str(agent_id), "channel": a.get("channel") or "",
|
| 331 |
-
"channelName": a.get("channelName") or "", "label": a.get("label") or "",
|
| 332 |
-
"active": a.get("active", True) is not False,
|
| 333 |
-
"writeApproval": a.get("writeApproval", True) is not False,
|
| 334 |
-
"perms": perms_out,
|
| 335 |
-
# β ALWAYS THE CURRENT VERSION, because `agent_principal` forces it: the editor must
|
| 336 |
-
# render "undeclared = denied", not the legacy-grant reading it would show at 0.
|
| 337 |
-
"perms_v": perm_scope.PERMS_VERSION,
|
| 338 |
-
# A bot is NEVER an admin β see `agent_principal`. Sent so the editor never paints
|
| 339 |
-
# the "Everything (admin)" state for a principal that cannot have it.
|
| 340 |
-
"is_admin": False,
|
| 341 |
-
"modules": modules,
|
| 342 |
-
"fields_by_module": {k: routes_admin._module_fields(k, session=session)
|
| 343 |
-
for k in governed}}
|
| 344 |
-
|
| 345 |
-
|
| 346 |
-
@router.put("/agents/{agent_id}/perms")
|
| 347 |
-
def put_channel_agent_perms(agent_id: str, body: dict = Body(default=None),
|
| 348 |
-
session: Session = Depends(admin_gate)):
|
| 349 |
-
"""Replace this agent's permission block wholesale β the same validator a user's block gets."""
|
| 350 |
-
import routes_admin
|
| 351 |
-
body = body if isinstance(body, dict) else {}
|
| 352 |
-
if not isinstance(_agents(session.runtime).get(str(agent_id)), dict):
|
| 353 |
-
raise err(404, "no_such_agent", "no agent with that id")
|
| 354 |
-
if "perms" not in body:
|
| 355 |
-
raise err(400, "empty_patch", "no perms to save")
|
| 356 |
-
mods = routes_admin._perm_modules(session)
|
| 357 |
-
# β THE SAME `_clean_perms`, NOT A COPY OF IT. It refuses a filter this module cannot
|
| 358 |
-
# evaluate, refuses a hiddenFields key that names nothing, and refuses a BU condition the
|
| 359 |
-
# pushdown cannot read. Every one of those is exactly as true for a channel as for a person,
|
| 360 |
-
# and a second validator here is a second place for one of them to go missing.
|
| 361 |
-
# β CORRECTED W36-T22: this list used to end "refuses an unenforced `ut_*` key". That refusal
|
| 362 |
-
# is DELETED β W36-T21 armed the wall over every database, so there is no unenforced key left
|
| 363 |
-
# to refuse, and a comment naming a rule that no longer exists is how the next wave re-derives
|
| 364 |
-
# it ([[two-gates-can-assert-opposite-things]]).
|
| 365 |
-
cleaned = routes_admin._clean_perms(
|
| 366 |
-
body.get("perms"),
|
| 367 |
-
governed_keys={m["key"] for m in mods}, session=session) or {}
|
| 368 |
-
|
| 369 |
-
import core.perm_scope as perm_scope
|
| 370 |
-
|
| 371 |
-
def _mut(prior):
|
| 372 |
-
if not isinstance(prior, dict):
|
| 373 |
-
return None
|
| 374 |
-
nxt = dict(prior)
|
| 375 |
-
nxt["perms"] = cleaned
|
| 376 |
-
nxt["perms_v"] = perm_scope.PERMS_VERSION
|
| 377 |
-
return nxt
|
| 378 |
-
|
| 379 |
-
_put_agent(session, str(agent_id), _mut)
|
| 380 |
-
fresh = _agents(session.runtime).get(str(agent_id)) or {}
|
| 381 |
-
if (fresh.get("perms") or {}) != cleaned:
|
| 382 |
-
# The one direction this must never fail in: telling an administrator the wall is up when
|
| 383 |
-
# it is not.
|
| 384 |
-
raise err(503, "store_unavailable", "the permissions were not saved β the agent is UNCHANGED")
|
| 385 |
-
return {"id": str(agent_id), "perms": fresh.get("perms") or {},
|
| 386 |
-
"perms_v": int(fresh.get("perms_v") or 0)}
|
| 387 |
-
|
| 388 |
-
|
| 389 |
-
@router.patch("/agents/{agent_id}")
|
| 390 |
-
def patch_channel_agent(agent_id: str, body: dict = Body(default=None),
|
| 391 |
-
session: Session = Depends(admin_gate)):
|
| 392 |
-
body = body if isinstance(body, dict) else {}
|
| 393 |
-
if not isinstance(_agents(session.runtime).get(str(agent_id)), dict):
|
| 394 |
-
raise err(404, "no_such_agent", "no agent with that id")
|
| 395 |
-
#: β `perms` IS NOT PATCHABLE HERE. It has its own door with its own validator; accepting it
|
| 396 |
-
#: on the metadata route would be a second, unvalidated way to write the wall.
|
| 397 |
-
allowed = {k: v for k, v in body.items()
|
| 398 |
-
if k in ("label", "active", "writeApproval", "channelName")}
|
| 399 |
-
if not allowed:
|
| 400 |
-
raise err(400, "empty_patch", "nothing to change")
|
| 401 |
-
|
| 402 |
-
def _mut(prior):
|
| 403 |
-
return _clean_agent(allowed, prior=prior) if isinstance(prior, dict) else None
|
| 404 |
-
|
| 405 |
-
_put_agent(session, str(agent_id), _mut)
|
| 406 |
-
import routes_admin
|
| 407 |
-
a = _agents(session.runtime).get(str(agent_id)) or {}
|
| 408 |
-
return {"agent": {"id": str(agent_id), "channel": a.get("channel") or "",
|
| 409 |
-
"channelName": a.get("channelName") or "", "label": a.get("label") or "",
|
| 410 |
-
"active": a.get("active", True) is not False,
|
| 411 |
-
"writeApproval": a.get("writeApproval", True) is not False,
|
| 412 |
-
"summary": _summary(a, routes_admin._perm_modules(session))}}
|
| 413 |
-
|
| 414 |
-
|
| 415 |
-
@router.delete("/agents/{agent_id}")
|
| 416 |
-
def delete_channel_agent(agent_id: str, session: Session = Depends(admin_gate)):
|
| 417 |
-
if not isinstance(_agents(session.runtime).get(str(agent_id)), dict):
|
| 418 |
-
raise err(404, "no_such_agent", "no agent with that id")
|
| 419 |
-
_put_agent(session, str(agent_id), lambda _prior: None)
|
| 420 |
-
if isinstance(_agents(session.runtime).get(str(agent_id)), dict):
|
| 421 |
-
raise err(503, "store_unavailable", "the agent was NOT removed")
|
| 422 |
-
return {"ok": True}
|
| 423 |
-
|
| 424 |
-
|
| 425 |
-
# ββ β W33-T67: THE WRITE-APPROVAL GATE βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 426 |
-
def _is_empty(value):
|
| 427 |
-
"""Is this value ABSENT, as opposed to falsy?
|
| 428 |
-
|
| 429 |
-
β COPIED EXACTLY FROM OpenTag's `agent/write_confirmation.py`, and the exactness is the point:
|
| 430 |
-
`0` and `False` are VALUES a human must see on the card. Setting a price to 0 or a flag to
|
| 431 |
-
False is precisely the kind of write somebody needs to approve, and a naive `if not value`
|
| 432 |
-
drops both rows silently β the approver then reads a card that does not describe the write
|
| 433 |
-
they are approving. The MIT licence gate for this borrowing is recorded in
|
| 434 |
-
`.claude/wiki/research/opentag-adoption.md`.
|
| 435 |
-
"""
|
| 436 |
-
return value is None or (isinstance(value, str) and value.strip() == "")
|
| 437 |
-
|
| 438 |
-
|
| 439 |
-
def approval_card(action, values, agent=None):
|
| 440 |
-
"""The card a human reads before a bot writes. `{action, rows, truncated, note}`.
|
| 441 |
-
|
| 442 |
-
β ROW-CAPPED, and the cap is a SAFETY property rather than a layout one: past a certain
|
| 443 |
-
length Slack collapses a message behind "Show more", and **an approver who cannot read the
|
| 444 |
-
card cannot approve it.** When rows are dropped the card SAYS how many β a truncation nobody
|
| 445 |
-
was told about is the violation, not the truncation (wave-30 R6's second sentence).
|
| 446 |
-
"""
|
| 447 |
-
rows = [{"field": str(k), "value": v}
|
| 448 |
-
for k, v in (values or {}).items() if not _is_empty(v)]
|
| 449 |
-
rows.sort(key=lambda r: r["field"])
|
| 450 |
-
shown, dropped = rows[:APPROVAL_MAX_ROWS], max(0, len(rows) - APPROVAL_MAX_ROWS)
|
| 451 |
-
return {
|
| 452 |
-
"action": str(action or "")[:80],
|
| 453 |
-
"rows": shown,
|
| 454 |
-
"truncated": dropped,
|
| 455 |
-
"note": (f"{dropped} more field(s) not shown β open the record to review them before "
|
| 456 |
-
f"approving." if dropped else ""),
|
| 457 |
-
"agent": (agent or {}).get("label") or (agent or {}).get("channel") or "",
|
| 458 |
-
}
|
| 459 |
-
|
| 460 |
-
|
| 461 |
-
def intercept_write(agent, action, values, stash):
|
| 462 |
-
"""THE INTERCEPTOR. Returns `("commit", None)` or `("await_approval", card)`.
|
| 463 |
-
|
| 464 |
-
β The one thing worth taking from OpenTag: a MUTATING tool call is halted, rendered as a card
|
| 465 |
-
naming the action and its values, and committed only on an explicit human accept. Everything
|
| 466 |
-
else in that repo β the transport, the runtime, the identity model β is AVOID (amendment A1).
|
| 467 |
-
|
| 468 |
-
β FAIL-CLOSED ON A MISSING FLAG: `is not False`, so a record written before `writeApproval`
|
| 469 |
-
existed requires approval. The unsafe direction here is not recoverable β revoking a bot does
|
| 470 |
-
not un-write what it wrote.
|
| 471 |
-
"""
|
| 472 |
-
if (agent or {}).get("writeApproval", True) is not False:
|
| 473 |
-
card = approval_card(action, values, agent)
|
| 474 |
-
stash(card)
|
| 475 |
-
return "await_approval", card
|
| 476 |
-
return "commit", None
|
| 477 |
-
|
| 478 |
-
|
| 479 |
-
def _pending(rt):
|
| 480 |
-
try:
|
| 481 |
-
found = rt.get(PENDING_KEY) or {}
|
| 482 |
-
except Exception: # noqa: BLE001
|
| 483 |
-
return {}
|
| 484 |
-
return found if isinstance(found, dict) else {}
|
| 485 |
-
|
| 486 |
-
|
| 487 |
-
def stash_pending(rt, agent, action, values, now=None):
|
| 488 |
-
"""Store one awaiting-approval mutation and return its token.
|
| 489 |
-
|
| 490 |
-
The token is what rides the card's button, so it is `secrets.token_urlsafe` and not the
|
| 491 |
-
action's id: a guessable value on an unauthenticated door is a way to approve somebody else's
|
| 492 |
-
write. Server-only bucket, per tenant, exactly as `routes_forms.TOKENS_KEY` is.
|
| 493 |
-
"""
|
| 494 |
-
token = secrets.token_urlsafe(18)
|
| 495 |
-
row = {"at": float(now if now is not None else time.time()),
|
| 496 |
-
"channel": str((agent or {}).get("channel") or ""),
|
| 497 |
-
"action": str(action or "")[:80],
|
| 498 |
-
"values": values if isinstance(values, dict) else {}}
|
| 499 |
-
rt.update(PENDING_KEY, lambda cur: {**(cur or {}), token: row}, flush="sync")
|
| 500 |
-
return token
|
| 501 |
-
|
| 502 |
-
|
| 503 |
-
#: `{action kind: fn(rt, row) -> None}`. Registered by whatever can actually perform a mutation.
|
| 504 |
-
#:
|
| 505 |
-
#: ββ EMPTY IN PRODUCTION TODAY, AND SAYING SO IS THE POINT. `commit_pending` below is the ONE
|
| 506 |
-
#: place an approved write may be performed, so it is the place a negative control can bite β but
|
| 507 |
-
#: a gate is only as honest as what it guards. Until something registers here, an approved card
|
| 508 |
-
#: performs NOTHING and `commit_pending` returns `no_committer` rather than pretending. That is a
|
| 509 |
-
#: declared absence, not a silent one [[flag-shipped-without-its-writer]].
|
| 510 |
-
_COMMITTERS = {}
|
| 511 |
-
|
| 512 |
-
|
| 513 |
-
def register_committer(kind, fn):
|
| 514 |
-
"""Declare who may perform an approved mutation of `kind`. Idempotent by key."""
|
| 515 |
-
_COMMITTERS[str(kind)] = fn
|
| 516 |
-
|
| 517 |
-
|
| 518 |
-
def commit_pending(rt, token, approved, now=None):
|
| 519 |
-
"""Resolve ONE approval card. Returns one of
|
| 520 |
-
`expired` Β· `unknown` Β· `rejected` Β· `no_committer` Β· `committed` Β· `failed`.
|
| 521 |
-
|
| 522 |
-
ββ THIS IS THE WRITE GATE, AND IT IS A SEPARATE FUNCTION FROM THE ROUTE ON PURPOSE. T67's
|
| 523 |
-
`done-when` asks for an NC proving a REJECTED card writes nothing. A control aimed at the
|
| 524 |
-
route could not bite while the route had no write in it at all β it would stay green with the
|
| 525 |
-
guard deleted, because there would be nothing for the guard to be protecting anything from
|
| 526 |
-
([[gate-can-report-green-on-nothing]], and a verifier caught exactly that on this ticket's
|
| 527 |
-
first draft). Putting the commit behind ONE named seam gives the control a real subject: the
|
| 528 |
-
gate registers a recorder, and `rejected` must leave it untouched while `approve` must call it
|
| 529 |
-
exactly once.
|
| 530 |
-
|
| 531 |
-
β THE TOKEN IS CONSUMED BEFORE EITHER BRANCH, and before any committer runs. A card that
|
| 532 |
-
survives its own click is a replayable write, and "approve twice" must never mean "write
|
| 533 |
-
twice". The delete is `flush="sync"` for the same reason.
|
| 534 |
-
"""
|
| 535 |
-
now = float(now if now is not None else time.time())
|
| 536 |
-
row = _pending(rt).get(str(token))
|
| 537 |
-
if not isinstance(row, dict):
|
| 538 |
-
return "unknown"
|
| 539 |
-
# Consume FIRST β before the TTL verdict, before the approve/reject branch, before any write.
|
| 540 |
-
rt.update(PENDING_KEY,
|
| 541 |
-
lambda cur: {k: v for k, v in (cur or {}).items() if k != str(token)},
|
| 542 |
-
flush="sync")
|
| 543 |
-
if now - float(row.get("at") or 0) > APPROVAL_TTL_S:
|
| 544 |
-
return "expired"
|
| 545 |
-
if not approved:
|
| 546 |
-
# β NOTHING BELOW THIS LINE RUNS FOR A REJECTED CARD. The whole gate is this return.
|
| 547 |
-
return "rejected"
|
| 548 |
-
fn = _COMMITTERS.get(str(row.get("action") or ""))
|
| 549 |
-
if fn is None:
|
| 550 |
-
return "no_committer"
|
| 551 |
-
try:
|
| 552 |
-
fn(rt, row)
|
| 553 |
-
except Exception: # noqa: BLE001
|
| 554 |
-
# A committer that raises must not read as a commit. The card is already consumed, so the
|
| 555 |
-
# honest report is that it failed, and the human asks again.
|
| 556 |
-
return "failed"
|
| 557 |
-
return "committed"
|
| 558 |
-
|
| 559 |
-
|
| 560 |
-
# ββ the UNAUTHENTICATED Slack door βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 561 |
-
#: Sliding window, keyed on the SLACK TEAM, never the client IP.
|
| 562 |
-
#:
|
| 563 |
-
#: β WHY NOT THE IP, WHICH IS WHAT `routes_forms` DOES. Every Slack event arrives from Slack's own
|
| 564 |
-
#: infrastructure, so an IP window is one shared bucket for every workspace on the platform: one
|
| 565 |
-
#: busy tenant would spend the allowance for all of them, and the symptom would be another
|
| 566 |
-
#: tenant's bot going quiet. The team id is the closest thing to the actual noisy party.
|
| 567 |
-
#: β It is attacker-CONTROLLED before the signature is checked, so the window is applied AFTER
|
| 568 |
-
#: verification, never before β an unsigned request is refused on cost grounds anyway (a signature
|
| 569 |
-
#: check is a single HMAC).
|
| 570 |
-
RATE_WINDOW_S, RATE_PER_WINDOW = 60, 60
|
| 571 |
-
_HITS: dict = {}
|
| 572 |
-
|
| 573 |
-
|
| 574 |
-
def _rate_ok(key, now):
|
| 575 |
-
seen = [t for t in _HITS.get(key, ()) if now - t < RATE_WINDOW_S]
|
| 576 |
-
seen.append(now)
|
| 577 |
-
_HITS[key] = seen
|
| 578 |
-
if len(_HITS) > 4096:
|
| 579 |
-
for k in [k for k, v in _HITS.items() if not v or now - v[-1] > RATE_WINDOW_S]:
|
| 580 |
-
_HITS.pop(k, None)
|
| 581 |
-
return len(seen) <= RATE_PER_WINDOW
|
| 582 |
-
|
| 583 |
-
|
| 584 |
-
async def _raw_body(request):
|
| 585 |
-
"""The body as RAW BYTES, capped.
|
| 586 |
-
|
| 587 |
-
ββ RAW, NOT PARSED, AND THAT IS NOT A STYLE CHOICE. Slack signs the literal string
|
| 588 |
-
`v0:{timestamp}:{body}` byte for byte. `routes_forms._bounded_body` β the function this is
|
| 589 |
-
otherwise copied from β returns only the parsed dict and DISCARDS the bytes, so a verbatim
|
| 590 |
-
copy of it could not verify a Slack signature at all: re-serialising the dict changes
|
| 591 |
-
whitespace and key order and the HMAC no longer matches. The bug would look like "Slack
|
| 592 |
-
signatures are always invalid", which is indistinguishable from a wrong secret.
|
| 593 |
-
|
| 594 |
-
β And it streams rather than calling `request.body()`: FastAPI would otherwise have buffered
|
| 595 |
-
the whole body before the handler's first line, so a content-length check inside the handler
|
| 596 |
-
caps nothing (`routes_forms`' own scar, recorded in its docstring).
|
| 597 |
-
"""
|
| 598 |
-
size, chunks = 0, []
|
| 599 |
-
async for chunk in request.stream():
|
| 600 |
-
size += len(chunk)
|
| 601 |
-
if size > MAX_BODY_BYTES:
|
| 602 |
-
raise err(413, "body_too_large", "that request is too large")
|
| 603 |
-
chunks.append(chunk)
|
| 604 |
-
return b"".join(chunks)
|
| 605 |
-
|
| 606 |
-
|
| 607 |
-
def _refuse():
|
| 608 |
-
"""ONE refusal for every reason. No oracle.
|
| 609 |
-
|
| 610 |
-
A wrong signature, a stale timestamp, an unknown team, an unconfigured tenant and a channel
|
| 611 |
-
with no agent all answer identically β otherwise the door tells an unauthenticated caller
|
| 612 |
-
which tenants exist and which channels are configured, which is the question it was built to
|
| 613 |
-
not answer. `routes_forms._refuse` takes the same posture for the same reason.
|
| 614 |
-
"""
|
| 615 |
-
return err(403, "bad_request", "that request could not be verified")
|
| 616 |
-
|
| 617 |
-
|
| 618 |
-
def _verify(secret, timestamp, raw, signature, now):
|
| 619 |
-
"""Slack's v0 signature, with the replay window checked FIRST."""
|
| 620 |
-
try:
|
| 621 |
-
ts = int(str(timestamp or "0"))
|
| 622 |
-
except ValueError:
|
| 623 |
-
return False
|
| 624 |
-
if abs(now - ts) > SLACK_MAX_SKEW_S:
|
| 625 |
-
return False
|
| 626 |
-
base = b"v0:" + str(ts).encode("ascii") + b":" + (raw or b"")
|
| 627 |
-
import hashlib
|
| 628 |
-
expected = "v0=" + hmac.new(str(secret).encode("utf-8"), base, hashlib.sha256).hexdigest()
|
| 629 |
-
return hmac.compare_digest(expected, str(signature or ""))
|
| 630 |
-
|
| 631 |
-
|
| 632 |
-
def _resolve_tenant(raw, timestamp, signature, now):
|
| 633 |
-
"""`(slug, runtime, creds)` for the tenant whose signing secret verifies this request.
|
| 634 |
-
|
| 635 |
-
β THE TENANT IS DECIDED BY THE SIGNATURE, NEVER BY THE PAYLOAD. A body-supplied `team_id`
|
| 636 |
-
would let an unauthenticated caller name the tenant it wants to be β the exact widening
|
| 637 |
-
`deps._user_for`'s tenant-equality check closes on the authenticated side. So every configured
|
| 638 |
-
tenant's secret is tried and the one that VERIFIES names the tenant.
|
| 639 |
-
|
| 640 |
-
β NO EARLY `break` ON A FAILED CANDIDATE and no per-tenant error: the loop's cost must not
|
| 641 |
-
depend on which tenant matched. `routes_forms._resolve` walks tenants the same way, for the
|
| 642 |
-
same reason β a public door has no session to ask.
|
| 643 |
-
"""
|
| 644 |
-
from harness import runtime as _rt
|
| 645 |
-
hit = None
|
| 646 |
-
try:
|
| 647 |
-
slugs = list(_rt.known_tenants())
|
| 648 |
-
except Exception: # noqa: BLE001
|
| 649 |
-
return None, None, None
|
| 650 |
-
for slug in slugs:
|
| 651 |
-
try:
|
| 652 |
-
rt = _rt.get_runtime(slug)
|
| 653 |
-
except Exception: # noqa: BLE001
|
| 654 |
-
continue
|
| 655 |
-
creds = slack_creds(rt)
|
| 656 |
-
if not creds:
|
| 657 |
-
continue
|
| 658 |
-
if _verify(creds.get("signing_secret"), timestamp, raw, signature, now) and hit is None:
|
| 659 |
-
hit = (slug, rt, creds)
|
| 660 |
-
return hit if hit else (None, None, None)
|
| 661 |
-
|
| 662 |
-
|
| 663 |
-
@router.post("/slack/events")
|
| 664 |
-
async def slack_events(request: Request):
|
| 665 |
-
"""Slack's Events API. UNAUTHENTICATED BY CONSTRUCTION β no `Depends(require_session)`.
|
| 666 |
-
|
| 667 |
-
β Built like `routes_forms.py`'s public door and not like OpenTag's: one process, no vendor,
|
| 668 |
-
no second runtime, no persistent outbound socket, and this tenant's Slack secret never leaves
|
| 669 |
-
this deployment (amendment A1).
|
| 670 |
-
"""
|
| 671 |
-
now = time.time()
|
| 672 |
-
raw = await _raw_body(request)
|
| 673 |
-
ts = request.headers.get("x-slack-request-timestamp")
|
| 674 |
-
sig = request.headers.get("x-slack-signature")
|
| 675 |
-
slug, rt, _creds = _resolve_tenant(raw, ts, sig, now)
|
| 676 |
-
if not rt:
|
| 677 |
-
raise _refuse()
|
| 678 |
-
|
| 679 |
-
try:
|
| 680 |
-
payload = json.loads(raw or b"{}")
|
| 681 |
-
except ValueError:
|
| 682 |
-
raise _refuse()
|
| 683 |
-
if not isinstance(payload, dict):
|
| 684 |
-
raise _refuse()
|
| 685 |
-
|
| 686 |
-
# Slack's one-time endpoint handshake. Answered ONLY after the signature verified β an
|
| 687 |
-
# unsigned challenge echo would confirm the endpoint exists to anyone who probes it.
|
| 688 |
-
if payload.get("type") == "url_verification":
|
| 689 |
-
return {"challenge": str(payload.get("challenge") or "")[:512]}
|
| 690 |
-
|
| 691 |
-
team = str(payload.get("team_id") or slug)
|
| 692 |
-
if not _rate_ok(f"{slug}:{team}", now):
|
| 693 |
-
raise err(429, "too_many_requests", "too many requests β wait a moment and try again")
|
| 694 |
-
|
| 695 |
-
event = payload.get("event") if isinstance(payload.get("event"), dict) else {}
|
| 696 |
-
channel = str(event.get("channel") or "")
|
| 697 |
-
agent = None
|
| 698 |
-
for a in _agents(rt).values():
|
| 699 |
-
if isinstance(a, dict) and str(a.get("channel") or "") == channel:
|
| 700 |
-
agent = a
|
| 701 |
-
break
|
| 702 |
-
# β NO AGENT, OR AN INACTIVE ONE, IS SILENCE β a 200 with no action. Not a 403: Slack retries
|
| 703 |
-
# a non-2xx up to three times, so refusing here would turn "this channel is not configured"
|
| 704 |
-
# into three refusals per message, and the retry storm would be the only visible symptom.
|
| 705 |
-
# β And it must not say WHICH β a distinguishable answer tells an unauthenticated caller which
|
| 706 |
-
# channels this workspace has configured.
|
| 707 |
-
if not isinstance(agent, dict) or agent.get("active", True) is False:
|
| 708 |
-
return {"ok": True}
|
| 709 |
-
|
| 710 |
-
# ββ THE ANSWERING HALF IS NOT BUILT YET, AND SAYING SO IS THE POINT. What ships here is the
|
| 711 |
-
# DOOR and the WALL: a verified request, resolved to a tenant by signature, matched to a
|
| 712 |
-
# channel agent whose `perms` block is a `perm_scope` principal (`agent_principal`). Composing
|
| 713 |
-
# a reply means an LLM call on the cheap-first ladder plus the read path, and a half-built
|
| 714 |
-
# answerer that returns something plausible is worse than one that returns nothing.
|
| 715 |
-
# Booked rather than faked [[flag-shipped-without-its-writer]].
|
| 716 |
-
return {"ok": True}
|
| 717 |
-
|
| 718 |
-
|
| 719 |
-
@router.post("/slack/interact")
|
| 720 |
-
async def slack_interact(request: Request):
|
| 721 |
-
"""The approval card's button (`W33-T67`). UNAUTHENTICATED, signature-verified, same shape."""
|
| 722 |
-
now = time.time()
|
| 723 |
-
raw = await _raw_body(request)
|
| 724 |
-
slug, rt, _creds = _resolve_tenant(raw, request.headers.get("x-slack-request-timestamp"),
|
| 725 |
-
request.headers.get("x-slack-signature"), now)
|
| 726 |
-
if not rt:
|
| 727 |
-
raise _refuse()
|
| 728 |
-
# Slack posts interactions as `application/x-www-form-urlencoded` with a `payload=` field.
|
| 729 |
-
from urllib.parse import parse_qs
|
| 730 |
-
try:
|
| 731 |
-
form = parse_qs(raw.decode("utf-8"))
|
| 732 |
-
payload = json.loads((form.get("payload") or ["{}"])[0])
|
| 733 |
-
except Exception: # noqa: BLE001
|
| 734 |
-
raise _refuse()
|
| 735 |
-
if not isinstance(payload, dict):
|
| 736 |
-
raise _refuse()
|
| 737 |
-
if not _rate_ok(f"{slug}:interact", now):
|
| 738 |
-
raise err(429, "too_many_requests", "too many requests β wait a moment and try again")
|
| 739 |
-
|
| 740 |
-
actions = payload.get("actions") if isinstance(payload.get("actions"), list) else []
|
| 741 |
-
choice = (actions[0] if actions and isinstance(actions[0], dict) else {})
|
| 742 |
-
token = str(choice.get("value") or "")
|
| 743 |
-
# β ANYTHING THAT IS NOT LITERALLY "approve" IS A REJECT. Fail-closed on a malformed,
|
| 744 |
-
# truncated or unknown action id: the direction that can be wrong here without anyone
|
| 745 |
-
# noticing is the one that writes.
|
| 746 |
-
approved = str(choice.get("action_id") or "") == "approve"
|
| 747 |
-
# β THE ROUTE DOES NOT DECIDE β `commit_pending` does, and it is the ONE seam a negative
|
| 748 |
-
# control can aim at. Consumption, the TTL, the reject branch and the committer lookup all
|
| 749 |
-
# live there; this handler only turns its verdict into a sentence.
|
| 750 |
-
verdict = commit_pending(rt, token, approved, now=now)
|
| 751 |
-
return {"text": {
|
| 752 |
-
# An unknown, already-used or expired token is not an error: a person clicking a stale
|
| 753 |
-
# card should be told it is stale, not shown a failure.
|
| 754 |
-
"unknown": "That approval is no longer available. Ask the bot again.",
|
| 755 |
-
"expired": "That approval expired before it was answered. Ask the bot again.",
|
| 756 |
-
"rejected": "Rejected. Nothing was changed.",
|
| 757 |
-
# β HONEST, not reassuring: the answering half that would register a committer is booked,
|
| 758 |
-
# so an approved card today performs nothing and says exactly that rather than "Approved."
|
| 759 |
-
"no_committer": "Approved β but this workspace has nothing configured to carry it out yet.",
|
| 760 |
-
"failed": "That change could not be completed. Nothing was saved; ask the bot again.",
|
| 761 |
-
"committed": "Approved.",
|
| 762 |
-
}.get(verdict, "That approval is no longer available. Ask the bot again.")}
|
| 763 |
-
|
| 764 |
-
|
| 765 |
-
@router.get("/slack/health")
|
| 766 |
-
def slack_health(session: Session = Depends(require_session)):
|
| 767 |
-
"""Is Slack configured for THIS tenant? Session-gated, and it answers about one tenant only β
|
| 768 |
-
enumerating the others would answer a question about our customer list."""
|
| 769 |
-
creds = slack_creds(session.runtime)
|
| 770 |
-
return {"configured": creds is not None,
|
| 771 |
-
"agents": len(_agents(session.runtime)),
|
| 772 |
-
"hasBotToken": bool((creds or {}).get("bot_token"))}
|
|
|
|
| 1 |
+
"""routes_slack.py β MANAGE AGENT: a bot per Slack channel, walled by the engine that walls a
|
| 2 |
+
person. (wave 33, owner item 10 Β· `W33-T38` / `W33-T39` / `W33-T67`.)
|
| 3 |
+
|
| 4 |
+
Owner, verbatim (2026-08-14): *"Expand Slack we need to be able to configure an AI bot PER channel
|
| 5 |
+
whose permissions we can toggle based on Fields etc of the databases, exactly like how we toggle
|
| 6 |
+
permissioning per user."*
|
| 7 |
+
|
| 8 |
+
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 9 |
+
ββ D-84 SAID THIS WAS "adapted from OpenTag" AND THAT PREMISE IS FALSE (PRD amendment A1).
|
| 10 |
+
|
| 11 |
+
`PENDING.md` D-84 has named OpenTag as the model for per-Slack-channel permissioning since
|
| 12 |
+
2026-08-08. **OpenTag has no permission model to adapt.** `app/runtime-host.ts` is
|
| 13 |
+
`identifyUser: () => OPENTAG_SERVICE_USER` β ONE constant identity for every request β and its own
|
| 14 |
+
`sender-context.ts` says the Slack sender is *"informational only⦠not gating access."* Its
|
| 15 |
+
authorization ceiling is whole-MCP-server, decided once at process boot from env vars. It is a
|
| 16 |
+
live specimen of identity that LOOKS like a permission and is not.
|
| 17 |
+
|
| 18 |
+
`core/perm_scope.py` is already strictly more expressive: per database, per row, per FIELD, per
|
| 19 |
+
principal. So item 10 is OUR engine projected onto a CHANNEL principal β see `agent_principal`
|
| 20 |
+
below, which is the whole projection and is nine lines. Study:
|
| 21 |
+
`.claude/wiki/research/opentag-adoption.md` (licence gate: MIT, PASSED).
|
| 22 |
+
|
| 23 |
+
What OpenTag genuinely contributes is the WRITE-APPROVAL interceptor (`W33-T67`, `approval_card`
|
| 24 |
+
below) and nothing else. Its TRANSPORT is AVOID wholesale β hosted CopilotKit Intelligence, a
|
| 25 |
+
second Node process, a persistent outbound socket and each tenant's Slack credentials held by a
|
| 26 |
+
third party, against ONE process on the HF free tier. The door here is `routes_forms.py`'s proven
|
| 27 |
+
unauthenticated-public shape plus Slack's signing-secret check.
|
| 28 |
+
|
| 29 |
+
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 30 |
+
β A CHANNEL IS A PRINCIPAL, NOT A PERSON. The wall binds to the Slack conversation id, so every
|
| 31 |
+
human in that channel reads through the same rules. That is the honest reading of "a bot per
|
| 32 |
+
channel" and the pane says it out loud, because the alternative is an admin assuming per-person
|
| 33 |
+
scoping from a screen that looks exactly like the per-person one.
|
| 34 |
+
|
| 35 |
+
β AND THE IDENTITY IS THE CHANNEL ID, NEVER ITS NAME. A channel can be renamed; its id cannot.
|
| 36 |
+
A wall bound to `#sales` silently re-points the day somebody renames the channel.
|
| 37 |
+
"""
|
| 38 |
+
import hmac
|
| 39 |
+
import json
|
| 40 |
+
import os
|
| 41 |
+
import secrets
|
| 42 |
+
import time
|
| 43 |
+
from datetime import datetime, timezone
|
| 44 |
+
|
| 45 |
+
from fastapi import APIRouter, Body, Depends, Request
|
| 46 |
+
|
| 47 |
+
from deps import Session, err, require_session
|
| 48 |
+
from routes_admin import admin_gate
|
| 49 |
+
|
| 50 |
+
router = APIRouter(prefix="/api/v1")
|
| 51 |
+
|
| 52 |
+
#: The tenant's channel agents: `{id: record}`. A per-tenant bucket, so it rides
|
| 53 |
+
#: `runtime.store_key`'s prefix and never lands in tenant #0's namespace.
|
| 54 |
+
AGENTS_KEY = "slack_agents"
|
| 55 |
+
|
| 56 |
+
#: Pending mutations awaiting a human's click (`W33-T67`). Server-only, per tenant.
|
| 57 |
+
PENDING_KEY = "slack_pending_writes"
|
| 58 |
+
|
| 59 |
+
#: How long an approval card stays clickable. An approval is a statement about the world as it was
|
| 60 |
+
#: when the card was rendered; an hour later the values it names may no longer be the values it
|
| 61 |
+
#: would write. Expiring is the honest behaviour, and an expired card says so rather than 403ing.
|
| 62 |
+
APPROVAL_TTL_S = 15 * 60
|
| 63 |
+
|
| 64 |
+
#: Slack rejects a replayed request older than 5 minutes; so do we, before any signature work.
|
| 65 |
+
SLACK_MAX_SKEW_S = 60 * 5
|
| 66 |
+
|
| 67 |
+
MAX_BODY_BYTES = 64 * 1024
|
| 68 |
+
|
| 69 |
+
#: β ROW-CAPPED BECAUSE AN APPROVER WHO CANNOT READ THE CARD CANNOT APPROVE IT. Copied
|
| 70 |
+
#: deliberately from OpenTag's `agent/write_confirmation.py`, which caps for the same reason:
|
| 71 |
+
#: past a certain length Slack collapses a message behind "Show more", and a human clicking
|
| 72 |
+
#: Approve on a card whose tail they never saw is worse than no card at all.
|
| 73 |
+
APPROVAL_MAX_ROWS = 12
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
# ββ the tenant's agent records βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 77 |
+
def _agents(rt):
|
| 78 |
+
"""`{id: record}` for one tenant. `{}` on any failure β an unreadable bucket must degrade to
|
| 79 |
+
"this tenant has no agents", never to a 500 on the settings pane."""
|
| 80 |
+
try:
|
| 81 |
+
found = rt.get(AGENTS_KEY) or {}
|
| 82 |
+
except Exception: # noqa: BLE001
|
| 83 |
+
return {}
|
| 84 |
+
return found if isinstance(found, dict) else {}
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def _clean_agent(raw, prior=None):
|
| 88 |
+
"""One stored record, built KEY BY KEY.
|
| 89 |
+
|
| 90 |
+
β Never `dict(raw)` and never `**raw`. This record is the SUBJECT of a permission decision;
|
| 91 |
+
a client-supplied key landing in it is a client-supplied input to `perm_scope`. The same rule
|
| 92 |
+
`routes_forms._public_form` follows on the way out, applied here on the way in.
|
| 93 |
+
"""
|
| 94 |
+
prior = prior if isinstance(prior, dict) else {}
|
| 95 |
+
out = {
|
| 96 |
+
"id": str(prior.get("id") or raw.get("id") or ""),
|
| 97 |
+
"channel": str(raw.get("channel") or prior.get("channel") or "").strip()[:64],
|
| 98 |
+
"channelName": str(raw.get("channelName") or prior.get("channelName") or "").strip()[:80],
|
| 99 |
+
"label": " ".join(str(raw.get("label") or prior.get("label") or "").split())[:80],
|
| 100 |
+
"active": bool(raw.get("active", prior.get("active", True))),
|
| 101 |
+
# β DEFAULT TRUE, AND `is not False` RATHER THAN TRUTHINESS. A record written before this
|
| 102 |
+
# key existed must read as "approval required": the one direction a default here is
|
| 103 |
+
# allowed to be wrong in is the safe one, because an unattended write cannot be undone by
|
| 104 |
+
# revoking the bot afterwards [[default-must-pass-its-own-guard]].
|
| 105 |
+
"writeApproval": raw.get("writeApproval", prior.get("writeApproval", True)) is not False,
|
| 106 |
+
# The permission block. Same shape, same validator, same engine as a user's.
|
| 107 |
+
"perms": prior.get("perms") if isinstance(prior.get("perms"), dict) else {},
|
| 108 |
+
"perms_v": int(prior.get("perms_v") or 0),
|
| 109 |
+
"createdBy": str(prior.get("createdBy") or raw.get("createdBy") or ""),
|
| 110 |
+
"createdAt": str(prior.get("createdAt") or raw.get("createdAt") or ""),
|
| 111 |
+
}
|
| 112 |
+
return out
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
def _put_agent(session, agent_id, mutate):
|
| 116 |
+
"""Read-modify-write ONE agent under the tenant's bucket, synchronously.
|
| 117 |
+
|
| 118 |
+
`flush="sync"` because the client re-reads the list immediately after every write here, and an
|
| 119 |
+
async flush would let the refetch win the race and paint the pre-write state β the
|
| 120 |
+
[[refetch-eats-its-own-write]] shape.
|
| 121 |
+
"""
|
| 122 |
+
def _set(cur):
|
| 123 |
+
cur = dict(cur or {})
|
| 124 |
+
rec = cur.get(agent_id) if isinstance(cur.get(agent_id), dict) else None
|
| 125 |
+
nxt = mutate(rec)
|
| 126 |
+
if nxt is None:
|
| 127 |
+
cur.pop(agent_id, None)
|
| 128 |
+
else:
|
| 129 |
+
cur[agent_id] = nxt
|
| 130 |
+
return cur
|
| 131 |
+
|
| 132 |
+
session.runtime.update(AGENTS_KEY, _set, flush="sync")
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
# ββ ββ THE PROJECTION β THE WHOLE OF "the same engine that permissions a user" βββββββββββββββ
|
| 136 |
+
def agent_principal(agent):
|
| 137 |
+
"""A channel agent, as a PRINCIPAL `core/perm_scope` already understands.
|
| 138 |
+
|
| 139 |
+
ββ THIS IS THE ENTIRE POINT OF THE TICKET AND IT IS NINE LINES. `perm_scope` takes a user
|
| 140 |
+
RECORD β a plain dict with `perms`, `perms_v` and `role` β not a User object and not a session.
|
| 141 |
+
So a channel agent carrying a `perms` block simply IS such a record, and `may_access`,
|
| 142 |
+
`visible_fields`, `hidden_keys` and `apply_row_scope` work on it unmodified. There is no second
|
| 143 |
+
engine, no parallel vocabulary, and nothing about a channel to keep in step with anything about
|
| 144 |
+
a person. That is what the owner's *"exactly like how we toggle permissioning per user"* means
|
| 145 |
+
when it is true in code rather than in appearance.
|
| 146 |
+
|
| 147 |
+
β `role` IS HARDCODED NON-ADMIN AND MUST STAY SO. `perm_scope.may_access` returns True
|
| 148 |
+
unconditionally for `role == 'admin'` β the break-glass clause that keeps the owner out of a
|
| 149 |
+
locked store. A bot has no such emergency and no hands: an agent record that could carry
|
| 150 |
+
`role: 'admin'` would be one stored key away from bypassing every wall on this page. The key is
|
| 151 |
+
not copied from the record; it is written here, every time.
|
| 152 |
+
|
| 153 |
+
β AND `perms_v` IS FORCED TO THE CURRENT VERSION, which makes an UNDECLARED database DENY
|
| 154 |
+
(C-PERM amendment 4). Without it a fresh agent would fall through to `perms.may_open`'s legacy
|
| 155 |
+
grant wall and read `modules: 'all'`-shaped defaults β i.e. a brand-new bot would start with
|
| 156 |
+
access to everything. Fail-closed from the first byte.
|
| 157 |
+
"""
|
| 158 |
+
import core.perm_scope as perm_scope
|
| 159 |
+
return {
|
| 160 |
+
"username": f"slack:{(agent or {}).get('channel') or '?'}",
|
| 161 |
+
"role": "agent",
|
| 162 |
+
"perms": (agent or {}).get("perms") or {},
|
| 163 |
+
"perms_v": perm_scope.PERMS_VERSION,
|
| 164 |
+
# No `bus`, no `agent`: a channel is not a business unit and not a salesperson. Absent
|
| 165 |
+
# rather than 'all' β `perms.allowed_bu_labels` narrows on absence, which is the direction
|
| 166 |
+
# this must fail in.
|
| 167 |
+
}
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
def agent_may_read(agent, module):
|
| 171 |
+
"""May this channel open `module` at all?"""
|
| 172 |
+
import core.perm_scope as perm_scope
|
| 173 |
+
if not (agent or {}).get("active", True):
|
| 174 |
+
return False
|
| 175 |
+
return bool(perm_scope.may_access(agent_principal(agent), module))
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
def agent_visible_fields(agent, fields, module):
|
| 179 |
+
"""`fields` minus this channel's hidden closure β the SAME transitive closure a person gets,
|
| 180 |
+
so a formula over a hidden column cannot leak it back through arithmetic."""
|
| 181 |
+
import core.perm_scope as perm_scope
|
| 182 |
+
return perm_scope.visible_fields(fields, agent_principal(agent), module)
|
| 183 |
+
|
| 184 |
+
|
| 185 |
+
def agent_rows(agent, rows, module, fields, ctx=None):
|
| 186 |
+
"""The rows this channel may receive β `permits()`, so an unanswerable permanent filter DENIES
|
| 187 |
+
rather than being ignored."""
|
| 188 |
+
import core.perm_scope as perm_scope
|
| 189 |
+
p = agent_principal(agent)
|
| 190 |
+
hide = perm_scope.hidden_keys(p, module, fields)
|
| 191 |
+
kept = perm_scope.apply_row_scope(rows, p, module, fields, ctx)
|
| 192 |
+
# Both wires, never one: the field list and the row payload are separate, and stripping only
|
| 193 |
+
# the first leaves the value sitting in the second where anyone can read it.
|
| 194 |
+
return [perm_scope.strip_row(r, hide) for r in kept]
|
| 195 |
+
|
| 196 |
+
|
| 197 |
+
# ββ the credential βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 198 |
+
def slack_creds(rt):
|
| 199 |
+
"""This TENANT's Slack credentials from its own keychain, or None.
|
| 200 |
+
|
| 201 |
+
β NO ENVIRONMENT FALLBACK, DELIBERATELY, AND THIS IS D-202's LESSON APPLIED BEFORE IT
|
| 202 |
+
HAPPENS AGAIN. Meta Ads shipped loading end-to-end and was still not a connector, because
|
| 203 |
+
`keychain.meta_creds` had no caller and the token came from the owner's `.env` β making it a
|
| 204 |
+
tenant-#0 FACT indistinguishable from a screenshot. `keychain.meta_creds`' own docstring spells
|
| 205 |
+
out the rule: handing the environment's credential to a tenant whose admin has not stored one
|
| 206 |
+
is the leak the resolver exists to prevent. So a tenant with no entry gets None and the pane
|
| 207 |
+
SAYS so.
|
| 208 |
+
|
| 209 |
+
β Reads through `list_entries` + `read_fields` (both public) rather than `keychain._first_creds`
|
| 210 |
+
(private, and in a file outside this lane's fence). Same deterministic rule: entries are
|
| 211 |
+
id-sorted, so "first" is stable across reads rather than dict-order luck.
|
| 212 |
+
"""
|
| 213 |
+
try:
|
| 214 |
+
import core.keychain as keychain
|
| 215 |
+
for row in keychain.list_entries(rt):
|
| 216 |
+
if str(row.get("type") or "") != "slack":
|
| 217 |
+
continue
|
| 218 |
+
fields = keychain.read_fields(rt, row.get("id"))
|
| 219 |
+
if isinstance(fields, dict) and fields.get("signing_secret"):
|
| 220 |
+
return fields
|
| 221 |
+
except Exception: # noqa: BLE001
|
| 222 |
+
# A locked or unreadable keychain reads as "not configured" for the PANE, which is honest;
|
| 223 |
+
# the signature check below fails closed regardless, so this cannot widen anything.
|
| 224 |
+
return None
|
| 225 |
+
return None
|
| 226 |
+
|
| 227 |
+
|
| 228 |
+
# ββ the authenticated doors (the Manage agent surface) βββββββββββββββββββββββββββββββββββββββ
|
| 229 |
+
def _summary(agent, modules):
|
| 230 |
+
"""One sentence per agent for the list row β computed here, because the alternative is one
|
| 231 |
+
round trip per row to fill one cell (the same reason `AdminUser.access` exists)."""
|
| 232 |
+
# β W36-T22 / C2 β every listed database is governed now; the `enforced` flag it used to
|
| 233 |
+
# filter on is DELETED, and `m.get("enforced")` would have gone silently falsy here and
|
| 234 |
+
# summarised every agent as "" (no databases at all).
|
| 235 |
+
governed = list(modules)
|
| 236 |
+
perms = agent.get("perms") or {}
|
| 237 |
+
open_n = sum(1 for m in governed if (perms.get(m["key"]) or {}).get("access"))
|
| 238 |
+
if not governed:
|
| 239 |
+
return ""
|
| 240 |
+
if open_n == 0:
|
| 241 |
+
return "No access"
|
| 242 |
+
restricted = sum(1 for m in governed
|
| 243 |
+
if (perms.get(m["key"]) or {}).get("access")
|
| 244 |
+
and ((perms.get(m["key"]) or {}).get("filter")
|
| 245 |
+
or (perms.get(m["key"]) or {}).get("hiddenFields")))
|
| 246 |
+
head = (f"All {open_n} database{'' if open_n == 1 else 's'}" if open_n == len(governed)
|
| 247 |
+
else f"{open_n} of {len(governed)} databases")
|
| 248 |
+
return f"{head}, {restricted} restricted" if restricted else head
|
| 249 |
+
|
| 250 |
+
|
| 251 |
+
@router.get("/agents")
|
| 252 |
+
def list_channel_agents(session: Session = Depends(admin_gate)):
|
| 253 |
+
"""This tenant's channel agents, plus whether Slack is reachable at all."""
|
| 254 |
+
import routes_admin
|
| 255 |
+
modules = routes_admin._perm_modules(session)
|
| 256 |
+
rows = []
|
| 257 |
+
for aid, a in sorted(_agents(session.runtime).items()):
|
| 258 |
+
if not isinstance(a, dict):
|
| 259 |
+
continue
|
| 260 |
+
rows.append({"id": aid, "channel": a.get("channel") or "",
|
| 261 |
+
"channelName": a.get("channelName") or "",
|
| 262 |
+
"label": a.get("label") or "", "active": a.get("active", True) is not False,
|
| 263 |
+
"writeApproval": a.get("writeApproval", True) is not False,
|
| 264 |
+
"summary": _summary(a, modules)})
|
| 265 |
+
return {"agents": rows, "configured": slack_creds(session.runtime) is not None}
|
| 266 |
+
|
| 267 |
+
|
| 268 |
+
@router.post("/agents")
|
| 269 |
+
def create_channel_agent(body: dict = Body(default=None),
|
| 270 |
+
session: Session = Depends(admin_gate)):
|
| 271 |
+
body = body if isinstance(body, dict) else {}
|
| 272 |
+
channel = str(body.get("channel") or "").strip()
|
| 273 |
+
if not channel:
|
| 274 |
+
raise err(400, "no_channel", "a Slack channel ID is required")
|
| 275 |
+
existing = _agents(session.runtime)
|
| 276 |
+
# β ONE AGENT PER CHANNEL. Two records for one conversation would mean two walls, and nothing
|
| 277 |
+
# anywhere decides which one applies β the [[one-question-two-normalizers]] shape, in the one
|
| 278 |
+
# place where the two answers are "may read" and "may not".
|
| 279 |
+
for a in existing.values():
|
| 280 |
+
if isinstance(a, dict) and str(a.get("channel") or "") == channel:
|
| 281 |
+
raise err(409, "channel_taken",
|
| 282 |
+
"that channel already has an agent β edit it instead of adding a second")
|
| 283 |
+
aid = secrets.token_urlsafe(9)
|
| 284 |
+
rec = _clean_agent(dict(body, id=aid, createdBy=session.uname,
|
| 285 |
+
createdAt=datetime.now(timezone.utc).isoformat(timespec="seconds")))
|
| 286 |
+
_put_agent(session, aid, lambda _prior: rec)
|
| 287 |
+
fresh = _agents(session.runtime).get(aid)
|
| 288 |
+
if not isinstance(fresh, dict):
|
| 289 |
+
# The store took the write and did not record it. A 200 here would tell an administrator
|
| 290 |
+
# the agent exists when it does not.
|
| 291 |
+
raise err(503, "store_unavailable", "the agent was NOT created")
|
| 292 |
+
return {"agent": {"id": aid, "channel": rec["channel"], "channelName": rec["channelName"],
|
| 293 |
+
"label": rec["label"], "active": rec["active"],
|
| 294 |
+
"writeApproval": rec["writeApproval"], "summary": "No access"}}
|
| 295 |
+
|
| 296 |
+
|
| 297 |
+
@router.get("/agents/{agent_id}")
|
| 298 |
+
def get_channel_agent(agent_id: str, session: Session = Depends(admin_gate)):
|
| 299 |
+
"""The agent PLUS the permission envelope β the SAME shape `get_perms` answers with.
|
| 300 |
+
|
| 301 |
+
β ONE PAYLOAD SHAPE FOR TWO ROOMS. `permsModel.parsePermsPayload` parses this, and
|
| 302 |
+
`settings/ModulePermsList` renders it, because they are literally the same client code. A
|
| 303 |
+
second envelope here would mean a second parser and a second set of defaults within a wave.
|
| 304 |
+
"""
|
| 305 |
+
import routes_admin
|
| 306 |
+
a = _agents(session.runtime).get(str(agent_id))
|
| 307 |
+
if not isinstance(a, dict):
|
| 308 |
+
raise err(404, "no_such_agent", "no agent with that id")
|
| 309 |
+
modules = routes_admin._perm_modules(session)
|
| 310 |
+
# ββ W36-T22 / C2 β EVERY listed database, `ut_*` included. β THIS IS WHY THE FLAG'S
|
| 311 |
+
# DELETION IS NOT A ONE-FILE CHANGE: `m.get("enforced")` on a row that no longer carries the
|
| 312 |
+
# key is None, so this list would have been EMPTY and the channel perms editor would have
|
| 313 |
+
# governed ZERO databases while looking entirely correct. A Slack channel is a principal of
|
| 314 |
+
# the SAME wall (`verify_perm_scope` section H) and gets the same catalogue.
|
| 315 |
+
governed = [m["key"] for m in modules]
|
| 316 |
+
stored = a.get("perms") or {}
|
| 317 |
+
principal = agent_principal(a)
|
| 318 |
+
import core.perm_scope as perm_scope
|
| 319 |
+
perms_out = {}
|
| 320 |
+
for k in governed:
|
| 321 |
+
e = stored.get(k)
|
| 322 |
+
# β W36-T22 β `may_read`, the same evaluator the read door uses, for the reason spelled
|
| 323 |
+
# out at `routes_admin.get_perms`. β It answers DIFFERENTLY here and correctly so: a
|
| 324 |
+
# channel agent is not a person with a share, so a `ut_*` key it has not been granted
|
| 325 |
+
# defaults CLOSED β which is the fail-closed direction for a bot, and the same answer the
|
| 326 |
+
# table routes would give it.
|
| 327 |
+
perms_out[k] = e if isinstance(e, dict) else {
|
| 328 |
+
"access": bool(perm_scope.may_read(principal, k, st=session.runtime)),
|
| 329 |
+
"filter": None, "hiddenFields": []}
|
| 330 |
+
return {"id": str(agent_id), "channel": a.get("channel") or "",
|
| 331 |
+
"channelName": a.get("channelName") or "", "label": a.get("label") or "",
|
| 332 |
+
"active": a.get("active", True) is not False,
|
| 333 |
+
"writeApproval": a.get("writeApproval", True) is not False,
|
| 334 |
+
"perms": perms_out,
|
| 335 |
+
# β ALWAYS THE CURRENT VERSION, because `agent_principal` forces it: the editor must
|
| 336 |
+
# render "undeclared = denied", not the legacy-grant reading it would show at 0.
|
| 337 |
+
"perms_v": perm_scope.PERMS_VERSION,
|
| 338 |
+
# A bot is NEVER an admin β see `agent_principal`. Sent so the editor never paints
|
| 339 |
+
# the "Everything (admin)" state for a principal that cannot have it.
|
| 340 |
+
"is_admin": False,
|
| 341 |
+
"modules": modules,
|
| 342 |
+
"fields_by_module": {k: routes_admin._module_fields(k, session=session)
|
| 343 |
+
for k in governed}}
|
| 344 |
+
|
| 345 |
+
|
| 346 |
+
@router.put("/agents/{agent_id}/perms")
|
| 347 |
+
def put_channel_agent_perms(agent_id: str, body: dict = Body(default=None),
|
| 348 |
+
session: Session = Depends(admin_gate)):
|
| 349 |
+
"""Replace this agent's permission block wholesale β the same validator a user's block gets."""
|
| 350 |
+
import routes_admin
|
| 351 |
+
body = body if isinstance(body, dict) else {}
|
| 352 |
+
if not isinstance(_agents(session.runtime).get(str(agent_id)), dict):
|
| 353 |
+
raise err(404, "no_such_agent", "no agent with that id")
|
| 354 |
+
if "perms" not in body:
|
| 355 |
+
raise err(400, "empty_patch", "no perms to save")
|
| 356 |
+
mods = routes_admin._perm_modules(session)
|
| 357 |
+
# β THE SAME `_clean_perms`, NOT A COPY OF IT. It refuses a filter this module cannot
|
| 358 |
+
# evaluate, refuses a hiddenFields key that names nothing, and refuses a BU condition the
|
| 359 |
+
# pushdown cannot read. Every one of those is exactly as true for a channel as for a person,
|
| 360 |
+
# and a second validator here is a second place for one of them to go missing.
|
| 361 |
+
# β CORRECTED W36-T22: this list used to end "refuses an unenforced `ut_*` key". That refusal
|
| 362 |
+
# is DELETED β W36-T21 armed the wall over every database, so there is no unenforced key left
|
| 363 |
+
# to refuse, and a comment naming a rule that no longer exists is how the next wave re-derives
|
| 364 |
+
# it ([[two-gates-can-assert-opposite-things]]).
|
| 365 |
+
cleaned = routes_admin._clean_perms(
|
| 366 |
+
body.get("perms"),
|
| 367 |
+
governed_keys={m["key"] for m in mods}, session=session) or {}
|
| 368 |
+
|
| 369 |
+
import core.perm_scope as perm_scope
|
| 370 |
+
|
| 371 |
+
def _mut(prior):
|
| 372 |
+
if not isinstance(prior, dict):
|
| 373 |
+
return None
|
| 374 |
+
nxt = dict(prior)
|
| 375 |
+
nxt["perms"] = cleaned
|
| 376 |
+
nxt["perms_v"] = perm_scope.PERMS_VERSION
|
| 377 |
+
return nxt
|
| 378 |
+
|
| 379 |
+
_put_agent(session, str(agent_id), _mut)
|
| 380 |
+
fresh = _agents(session.runtime).get(str(agent_id)) or {}
|
| 381 |
+
if (fresh.get("perms") or {}) != cleaned:
|
| 382 |
+
# The one direction this must never fail in: telling an administrator the wall is up when
|
| 383 |
+
# it is not.
|
| 384 |
+
raise err(503, "store_unavailable", "the permissions were not saved β the agent is UNCHANGED")
|
| 385 |
+
return {"id": str(agent_id), "perms": fresh.get("perms") or {},
|
| 386 |
+
"perms_v": int(fresh.get("perms_v") or 0)}
|
| 387 |
+
|
| 388 |
+
|
| 389 |
+
@router.patch("/agents/{agent_id}")
|
| 390 |
+
def patch_channel_agent(agent_id: str, body: dict = Body(default=None),
|
| 391 |
+
session: Session = Depends(admin_gate)):
|
| 392 |
+
body = body if isinstance(body, dict) else {}
|
| 393 |
+
if not isinstance(_agents(session.runtime).get(str(agent_id)), dict):
|
| 394 |
+
raise err(404, "no_such_agent", "no agent with that id")
|
| 395 |
+
#: β `perms` IS NOT PATCHABLE HERE. It has its own door with its own validator; accepting it
|
| 396 |
+
#: on the metadata route would be a second, unvalidated way to write the wall.
|
| 397 |
+
allowed = {k: v for k, v in body.items()
|
| 398 |
+
if k in ("label", "active", "writeApproval", "channelName")}
|
| 399 |
+
if not allowed:
|
| 400 |
+
raise err(400, "empty_patch", "nothing to change")
|
| 401 |
+
|
| 402 |
+
def _mut(prior):
|
| 403 |
+
return _clean_agent(allowed, prior=prior) if isinstance(prior, dict) else None
|
| 404 |
+
|
| 405 |
+
_put_agent(session, str(agent_id), _mut)
|
| 406 |
+
import routes_admin
|
| 407 |
+
a = _agents(session.runtime).get(str(agent_id)) or {}
|
| 408 |
+
return {"agent": {"id": str(agent_id), "channel": a.get("channel") or "",
|
| 409 |
+
"channelName": a.get("channelName") or "", "label": a.get("label") or "",
|
| 410 |
+
"active": a.get("active", True) is not False,
|
| 411 |
+
"writeApproval": a.get("writeApproval", True) is not False,
|
| 412 |
+
"summary": _summary(a, routes_admin._perm_modules(session))}}
|
| 413 |
+
|
| 414 |
+
|
| 415 |
+
@router.delete("/agents/{agent_id}")
|
| 416 |
+
def delete_channel_agent(agent_id: str, session: Session = Depends(admin_gate)):
|
| 417 |
+
if not isinstance(_agents(session.runtime).get(str(agent_id)), dict):
|
| 418 |
+
raise err(404, "no_such_agent", "no agent with that id")
|
| 419 |
+
_put_agent(session, str(agent_id), lambda _prior: None)
|
| 420 |
+
if isinstance(_agents(session.runtime).get(str(agent_id)), dict):
|
| 421 |
+
raise err(503, "store_unavailable", "the agent was NOT removed")
|
| 422 |
+
return {"ok": True}
|
| 423 |
+
|
| 424 |
+
|
| 425 |
+
# ββ β W33-T67: THE WRITE-APPROVAL GATE βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 426 |
+
def _is_empty(value):
|
| 427 |
+
"""Is this value ABSENT, as opposed to falsy?
|
| 428 |
+
|
| 429 |
+
β COPIED EXACTLY FROM OpenTag's `agent/write_confirmation.py`, and the exactness is the point:
|
| 430 |
+
`0` and `False` are VALUES a human must see on the card. Setting a price to 0 or a flag to
|
| 431 |
+
False is precisely the kind of write somebody needs to approve, and a naive `if not value`
|
| 432 |
+
drops both rows silently β the approver then reads a card that does not describe the write
|
| 433 |
+
they are approving. The MIT licence gate for this borrowing is recorded in
|
| 434 |
+
`.claude/wiki/research/opentag-adoption.md`.
|
| 435 |
+
"""
|
| 436 |
+
return value is None or (isinstance(value, str) and value.strip() == "")
|
| 437 |
+
|
| 438 |
+
|
| 439 |
+
def approval_card(action, values, agent=None):
|
| 440 |
+
"""The card a human reads before a bot writes. `{action, rows, truncated, note}`.
|
| 441 |
+
|
| 442 |
+
β ROW-CAPPED, and the cap is a SAFETY property rather than a layout one: past a certain
|
| 443 |
+
length Slack collapses a message behind "Show more", and **an approver who cannot read the
|
| 444 |
+
card cannot approve it.** When rows are dropped the card SAYS how many β a truncation nobody
|
| 445 |
+
was told about is the violation, not the truncation (wave-30 R6's second sentence).
|
| 446 |
+
"""
|
| 447 |
+
rows = [{"field": str(k), "value": v}
|
| 448 |
+
for k, v in (values or {}).items() if not _is_empty(v)]
|
| 449 |
+
rows.sort(key=lambda r: r["field"])
|
| 450 |
+
shown, dropped = rows[:APPROVAL_MAX_ROWS], max(0, len(rows) - APPROVAL_MAX_ROWS)
|
| 451 |
+
return {
|
| 452 |
+
"action": str(action or "")[:80],
|
| 453 |
+
"rows": shown,
|
| 454 |
+
"truncated": dropped,
|
| 455 |
+
"note": (f"{dropped} more field(s) not shown β open the record to review them before "
|
| 456 |
+
f"approving." if dropped else ""),
|
| 457 |
+
"agent": (agent or {}).get("label") or (agent or {}).get("channel") or "",
|
| 458 |
+
}
|
| 459 |
+
|
| 460 |
+
|
| 461 |
+
def intercept_write(agent, action, values, stash):
|
| 462 |
+
"""THE INTERCEPTOR. Returns `("commit", None)` or `("await_approval", card)`.
|
| 463 |
+
|
| 464 |
+
β The one thing worth taking from OpenTag: a MUTATING tool call is halted, rendered as a card
|
| 465 |
+
naming the action and its values, and committed only on an explicit human accept. Everything
|
| 466 |
+
else in that repo β the transport, the runtime, the identity model β is AVOID (amendment A1).
|
| 467 |
+
|
| 468 |
+
β FAIL-CLOSED ON A MISSING FLAG: `is not False`, so a record written before `writeApproval`
|
| 469 |
+
existed requires approval. The unsafe direction here is not recoverable β revoking a bot does
|
| 470 |
+
not un-write what it wrote.
|
| 471 |
+
"""
|
| 472 |
+
if (agent or {}).get("writeApproval", True) is not False:
|
| 473 |
+
card = approval_card(action, values, agent)
|
| 474 |
+
stash(card)
|
| 475 |
+
return "await_approval", card
|
| 476 |
+
return "commit", None
|
| 477 |
+
|
| 478 |
+
|
| 479 |
+
def _pending(rt):
|
| 480 |
+
try:
|
| 481 |
+
found = rt.get(PENDING_KEY) or {}
|
| 482 |
+
except Exception: # noqa: BLE001
|
| 483 |
+
return {}
|
| 484 |
+
return found if isinstance(found, dict) else {}
|
| 485 |
+
|
| 486 |
+
|
| 487 |
+
def stash_pending(rt, agent, action, values, now=None):
|
| 488 |
+
"""Store one awaiting-approval mutation and return its token.
|
| 489 |
+
|
| 490 |
+
The token is what rides the card's button, so it is `secrets.token_urlsafe` and not the
|
| 491 |
+
action's id: a guessable value on an unauthenticated door is a way to approve somebody else's
|
| 492 |
+
write. Server-only bucket, per tenant, exactly as `routes_forms.TOKENS_KEY` is.
|
| 493 |
+
"""
|
| 494 |
+
token = secrets.token_urlsafe(18)
|
| 495 |
+
row = {"at": float(now if now is not None else time.time()),
|
| 496 |
+
"channel": str((agent or {}).get("channel") or ""),
|
| 497 |
+
"action": str(action or "")[:80],
|
| 498 |
+
"values": values if isinstance(values, dict) else {}}
|
| 499 |
+
rt.update(PENDING_KEY, lambda cur: {**(cur or {}), token: row}, flush="sync")
|
| 500 |
+
return token
|
| 501 |
+
|
| 502 |
+
|
| 503 |
+
#: `{action kind: fn(rt, row) -> None}`. Registered by whatever can actually perform a mutation.
|
| 504 |
+
#:
|
| 505 |
+
#: ββ EMPTY IN PRODUCTION TODAY, AND SAYING SO IS THE POINT. `commit_pending` below is the ONE
|
| 506 |
+
#: place an approved write may be performed, so it is the place a negative control can bite β but
|
| 507 |
+
#: a gate is only as honest as what it guards. Until something registers here, an approved card
|
| 508 |
+
#: performs NOTHING and `commit_pending` returns `no_committer` rather than pretending. That is a
|
| 509 |
+
#: declared absence, not a silent one [[flag-shipped-without-its-writer]].
|
| 510 |
+
_COMMITTERS = {}
|
| 511 |
+
|
| 512 |
+
|
| 513 |
+
def register_committer(kind, fn):
|
| 514 |
+
"""Declare who may perform an approved mutation of `kind`. Idempotent by key."""
|
| 515 |
+
_COMMITTERS[str(kind)] = fn
|
| 516 |
+
|
| 517 |
+
|
| 518 |
+
def commit_pending(rt, token, approved, now=None):
|
| 519 |
+
"""Resolve ONE approval card. Returns one of
|
| 520 |
+
`expired` Β· `unknown` Β· `rejected` Β· `no_committer` Β· `committed` Β· `failed`.
|
| 521 |
+
|
| 522 |
+
ββ THIS IS THE WRITE GATE, AND IT IS A SEPARATE FUNCTION FROM THE ROUTE ON PURPOSE. T67's
|
| 523 |
+
`done-when` asks for an NC proving a REJECTED card writes nothing. A control aimed at the
|
| 524 |
+
route could not bite while the route had no write in it at all β it would stay green with the
|
| 525 |
+
guard deleted, because there would be nothing for the guard to be protecting anything from
|
| 526 |
+
([[gate-can-report-green-on-nothing]], and a verifier caught exactly that on this ticket's
|
| 527 |
+
first draft). Putting the commit behind ONE named seam gives the control a real subject: the
|
| 528 |
+
gate registers a recorder, and `rejected` must leave it untouched while `approve` must call it
|
| 529 |
+
exactly once.
|
| 530 |
+
|
| 531 |
+
β THE TOKEN IS CONSUMED BEFORE EITHER BRANCH, and before any committer runs. A card that
|
| 532 |
+
survives its own click is a replayable write, and "approve twice" must never mean "write
|
| 533 |
+
twice". The delete is `flush="sync"` for the same reason.
|
| 534 |
+
"""
|
| 535 |
+
now = float(now if now is not None else time.time())
|
| 536 |
+
row = _pending(rt).get(str(token))
|
| 537 |
+
if not isinstance(row, dict):
|
| 538 |
+
return "unknown"
|
| 539 |
+
# Consume FIRST β before the TTL verdict, before the approve/reject branch, before any write.
|
| 540 |
+
rt.update(PENDING_KEY,
|
| 541 |
+
lambda cur: {k: v for k, v in (cur or {}).items() if k != str(token)},
|
| 542 |
+
flush="sync")
|
| 543 |
+
if now - float(row.get("at") or 0) > APPROVAL_TTL_S:
|
| 544 |
+
return "expired"
|
| 545 |
+
if not approved:
|
| 546 |
+
# β NOTHING BELOW THIS LINE RUNS FOR A REJECTED CARD. The whole gate is this return.
|
| 547 |
+
return "rejected"
|
| 548 |
+
fn = _COMMITTERS.get(str(row.get("action") or ""))
|
| 549 |
+
if fn is None:
|
| 550 |
+
return "no_committer"
|
| 551 |
+
try:
|
| 552 |
+
fn(rt, row)
|
| 553 |
+
except Exception: # noqa: BLE001
|
| 554 |
+
# A committer that raises must not read as a commit. The card is already consumed, so the
|
| 555 |
+
# honest report is that it failed, and the human asks again.
|
| 556 |
+
return "failed"
|
| 557 |
+
return "committed"
|
| 558 |
+
|
| 559 |
+
|
| 560 |
+
# ββ the UNAUTHENTICATED Slack door βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 561 |
+
#: Sliding window, keyed on the SLACK TEAM, never the client IP.
|
| 562 |
+
#:
|
| 563 |
+
#: β WHY NOT THE IP, WHICH IS WHAT `routes_forms` DOES. Every Slack event arrives from Slack's own
|
| 564 |
+
#: infrastructure, so an IP window is one shared bucket for every workspace on the platform: one
|
| 565 |
+
#: busy tenant would spend the allowance for all of them, and the symptom would be another
|
| 566 |
+
#: tenant's bot going quiet. The team id is the closest thing to the actual noisy party.
|
| 567 |
+
#: β It is attacker-CONTROLLED before the signature is checked, so the window is applied AFTER
|
| 568 |
+
#: verification, never before β an unsigned request is refused on cost grounds anyway (a signature
|
| 569 |
+
#: check is a single HMAC).
|
| 570 |
+
RATE_WINDOW_S, RATE_PER_WINDOW = 60, 60
|
| 571 |
+
_HITS: dict = {}
|
| 572 |
+
|
| 573 |
+
|
| 574 |
+
def _rate_ok(key, now):
|
| 575 |
+
seen = [t for t in _HITS.get(key, ()) if now - t < RATE_WINDOW_S]
|
| 576 |
+
seen.append(now)
|
| 577 |
+
_HITS[key] = seen
|
| 578 |
+
if len(_HITS) > 4096:
|
| 579 |
+
for k in [k for k, v in _HITS.items() if not v or now - v[-1] > RATE_WINDOW_S]:
|
| 580 |
+
_HITS.pop(k, None)
|
| 581 |
+
return len(seen) <= RATE_PER_WINDOW
|
| 582 |
+
|
| 583 |
+
|
| 584 |
+
async def _raw_body(request):
|
| 585 |
+
"""The body as RAW BYTES, capped.
|
| 586 |
+
|
| 587 |
+
ββ RAW, NOT PARSED, AND THAT IS NOT A STYLE CHOICE. Slack signs the literal string
|
| 588 |
+
`v0:{timestamp}:{body}` byte for byte. `routes_forms._bounded_body` β the function this is
|
| 589 |
+
otherwise copied from β returns only the parsed dict and DISCARDS the bytes, so a verbatim
|
| 590 |
+
copy of it could not verify a Slack signature at all: re-serialising the dict changes
|
| 591 |
+
whitespace and key order and the HMAC no longer matches. The bug would look like "Slack
|
| 592 |
+
signatures are always invalid", which is indistinguishable from a wrong secret.
|
| 593 |
+
|
| 594 |
+
β And it streams rather than calling `request.body()`: FastAPI would otherwise have buffered
|
| 595 |
+
the whole body before the handler's first line, so a content-length check inside the handler
|
| 596 |
+
caps nothing (`routes_forms`' own scar, recorded in its docstring).
|
| 597 |
+
"""
|
| 598 |
+
size, chunks = 0, []
|
| 599 |
+
async for chunk in request.stream():
|
| 600 |
+
size += len(chunk)
|
| 601 |
+
if size > MAX_BODY_BYTES:
|
| 602 |
+
raise err(413, "body_too_large", "that request is too large")
|
| 603 |
+
chunks.append(chunk)
|
| 604 |
+
return b"".join(chunks)
|
| 605 |
+
|
| 606 |
+
|
| 607 |
+
def _refuse():
|
| 608 |
+
"""ONE refusal for every reason. No oracle.
|
| 609 |
+
|
| 610 |
+
A wrong signature, a stale timestamp, an unknown team, an unconfigured tenant and a channel
|
| 611 |
+
with no agent all answer identically β otherwise the door tells an unauthenticated caller
|
| 612 |
+
which tenants exist and which channels are configured, which is the question it was built to
|
| 613 |
+
not answer. `routes_forms._refuse` takes the same posture for the same reason.
|
| 614 |
+
"""
|
| 615 |
+
return err(403, "bad_request", "that request could not be verified")
|
| 616 |
+
|
| 617 |
+
|
| 618 |
+
def _verify(secret, timestamp, raw, signature, now):
|
| 619 |
+
"""Slack's v0 signature, with the replay window checked FIRST."""
|
| 620 |
+
try:
|
| 621 |
+
ts = int(str(timestamp or "0"))
|
| 622 |
+
except ValueError:
|
| 623 |
+
return False
|
| 624 |
+
if abs(now - ts) > SLACK_MAX_SKEW_S:
|
| 625 |
+
return False
|
| 626 |
+
base = b"v0:" + str(ts).encode("ascii") + b":" + (raw or b"")
|
| 627 |
+
import hashlib
|
| 628 |
+
expected = "v0=" + hmac.new(str(secret).encode("utf-8"), base, hashlib.sha256).hexdigest()
|
| 629 |
+
return hmac.compare_digest(expected, str(signature or ""))
|
| 630 |
+
|
| 631 |
+
|
| 632 |
+
def _resolve_tenant(raw, timestamp, signature, now):
|
| 633 |
+
"""`(slug, runtime, creds)` for the tenant whose signing secret verifies this request.
|
| 634 |
+
|
| 635 |
+
β THE TENANT IS DECIDED BY THE SIGNATURE, NEVER BY THE PAYLOAD. A body-supplied `team_id`
|
| 636 |
+
would let an unauthenticated caller name the tenant it wants to be β the exact widening
|
| 637 |
+
`deps._user_for`'s tenant-equality check closes on the authenticated side. So every configured
|
| 638 |
+
tenant's secret is tried and the one that VERIFIES names the tenant.
|
| 639 |
+
|
| 640 |
+
β NO EARLY `break` ON A FAILED CANDIDATE and no per-tenant error: the loop's cost must not
|
| 641 |
+
depend on which tenant matched. `routes_forms._resolve` walks tenants the same way, for the
|
| 642 |
+
same reason β a public door has no session to ask.
|
| 643 |
+
"""
|
| 644 |
+
from harness import runtime as _rt
|
| 645 |
+
hit = None
|
| 646 |
+
try:
|
| 647 |
+
slugs = list(_rt.known_tenants())
|
| 648 |
+
except Exception: # noqa: BLE001
|
| 649 |
+
return None, None, None
|
| 650 |
+
for slug in slugs:
|
| 651 |
+
try:
|
| 652 |
+
rt = _rt.get_runtime(slug)
|
| 653 |
+
except Exception: # noqa: BLE001
|
| 654 |
+
continue
|
| 655 |
+
creds = slack_creds(rt)
|
| 656 |
+
if not creds:
|
| 657 |
+
continue
|
| 658 |
+
if _verify(creds.get("signing_secret"), timestamp, raw, signature, now) and hit is None:
|
| 659 |
+
hit = (slug, rt, creds)
|
| 660 |
+
return hit if hit else (None, None, None)
|
| 661 |
+
|
| 662 |
+
|
| 663 |
+
@router.post("/slack/events")
|
| 664 |
+
async def slack_events(request: Request):
|
| 665 |
+
"""Slack's Events API. UNAUTHENTICATED BY CONSTRUCTION β no `Depends(require_session)`.
|
| 666 |
+
|
| 667 |
+
β Built like `routes_forms.py`'s public door and not like OpenTag's: one process, no vendor,
|
| 668 |
+
no second runtime, no persistent outbound socket, and this tenant's Slack secret never leaves
|
| 669 |
+
this deployment (amendment A1).
|
| 670 |
+
"""
|
| 671 |
+
now = time.time()
|
| 672 |
+
raw = await _raw_body(request)
|
| 673 |
+
ts = request.headers.get("x-slack-request-timestamp")
|
| 674 |
+
sig = request.headers.get("x-slack-signature")
|
| 675 |
+
slug, rt, _creds = _resolve_tenant(raw, ts, sig, now)
|
| 676 |
+
if not rt:
|
| 677 |
+
raise _refuse()
|
| 678 |
+
|
| 679 |
+
try:
|
| 680 |
+
payload = json.loads(raw or b"{}")
|
| 681 |
+
except ValueError:
|
| 682 |
+
raise _refuse()
|
| 683 |
+
if not isinstance(payload, dict):
|
| 684 |
+
raise _refuse()
|
| 685 |
+
|
| 686 |
+
# Slack's one-time endpoint handshake. Answered ONLY after the signature verified β an
|
| 687 |
+
# unsigned challenge echo would confirm the endpoint exists to anyone who probes it.
|
| 688 |
+
if payload.get("type") == "url_verification":
|
| 689 |
+
return {"challenge": str(payload.get("challenge") or "")[:512]}
|
| 690 |
+
|
| 691 |
+
team = str(payload.get("team_id") or slug)
|
| 692 |
+
if not _rate_ok(f"{slug}:{team}", now):
|
| 693 |
+
raise err(429, "too_many_requests", "too many requests β wait a moment and try again")
|
| 694 |
+
|
| 695 |
+
event = payload.get("event") if isinstance(payload.get("event"), dict) else {}
|
| 696 |
+
channel = str(event.get("channel") or "")
|
| 697 |
+
agent = None
|
| 698 |
+
for a in _agents(rt).values():
|
| 699 |
+
if isinstance(a, dict) and str(a.get("channel") or "") == channel:
|
| 700 |
+
agent = a
|
| 701 |
+
break
|
| 702 |
+
# β NO AGENT, OR AN INACTIVE ONE, IS SILENCE β a 200 with no action. Not a 403: Slack retries
|
| 703 |
+
# a non-2xx up to three times, so refusing here would turn "this channel is not configured"
|
| 704 |
+
# into three refusals per message, and the retry storm would be the only visible symptom.
|
| 705 |
+
# β And it must not say WHICH β a distinguishable answer tells an unauthenticated caller which
|
| 706 |
+
# channels this workspace has configured.
|
| 707 |
+
if not isinstance(agent, dict) or agent.get("active", True) is False:
|
| 708 |
+
return {"ok": True}
|
| 709 |
+
|
| 710 |
+
# ββ THE ANSWERING HALF IS NOT BUILT YET, AND SAYING SO IS THE POINT. What ships here is the
|
| 711 |
+
# DOOR and the WALL: a verified request, resolved to a tenant by signature, matched to a
|
| 712 |
+
# channel agent whose `perms` block is a `perm_scope` principal (`agent_principal`). Composing
|
| 713 |
+
# a reply means an LLM call on the cheap-first ladder plus the read path, and a half-built
|
| 714 |
+
# answerer that returns something plausible is worse than one that returns nothing.
|
| 715 |
+
# Booked rather than faked [[flag-shipped-without-its-writer]].
|
| 716 |
+
return {"ok": True}
|
| 717 |
+
|
| 718 |
+
|
| 719 |
+
@router.post("/slack/interact")
|
| 720 |
+
async def slack_interact(request: Request):
|
| 721 |
+
"""The approval card's button (`W33-T67`). UNAUTHENTICATED, signature-verified, same shape."""
|
| 722 |
+
now = time.time()
|
| 723 |
+
raw = await _raw_body(request)
|
| 724 |
+
slug, rt, _creds = _resolve_tenant(raw, request.headers.get("x-slack-request-timestamp"),
|
| 725 |
+
request.headers.get("x-slack-signature"), now)
|
| 726 |
+
if not rt:
|
| 727 |
+
raise _refuse()
|
| 728 |
+
# Slack posts interactions as `application/x-www-form-urlencoded` with a `payload=` field.
|
| 729 |
+
from urllib.parse import parse_qs
|
| 730 |
+
try:
|
| 731 |
+
form = parse_qs(raw.decode("utf-8"))
|
| 732 |
+
payload = json.loads((form.get("payload") or ["{}"])[0])
|
| 733 |
+
except Exception: # noqa: BLE001
|
| 734 |
+
raise _refuse()
|
| 735 |
+
if not isinstance(payload, dict):
|
| 736 |
+
raise _refuse()
|
| 737 |
+
if not _rate_ok(f"{slug}:interact", now):
|
| 738 |
+
raise err(429, "too_many_requests", "too many requests β wait a moment and try again")
|
| 739 |
+
|
| 740 |
+
actions = payload.get("actions") if isinstance(payload.get("actions"), list) else []
|
| 741 |
+
choice = (actions[0] if actions and isinstance(actions[0], dict) else {})
|
| 742 |
+
token = str(choice.get("value") or "")
|
| 743 |
+
# β ANYTHING THAT IS NOT LITERALLY "approve" IS A REJECT. Fail-closed on a malformed,
|
| 744 |
+
# truncated or unknown action id: the direction that can be wrong here without anyone
|
| 745 |
+
# noticing is the one that writes.
|
| 746 |
+
approved = str(choice.get("action_id") or "") == "approve"
|
| 747 |
+
# β THE ROUTE DOES NOT DECIDE β `commit_pending` does, and it is the ONE seam a negative
|
| 748 |
+
# control can aim at. Consumption, the TTL, the reject branch and the committer lookup all
|
| 749 |
+
# live there; this handler only turns its verdict into a sentence.
|
| 750 |
+
verdict = commit_pending(rt, token, approved, now=now)
|
| 751 |
+
return {"text": {
|
| 752 |
+
# An unknown, already-used or expired token is not an error: a person clicking a stale
|
| 753 |
+
# card should be told it is stale, not shown a failure.
|
| 754 |
+
"unknown": "That approval is no longer available. Ask the bot again.",
|
| 755 |
+
"expired": "That approval expired before it was answered. Ask the bot again.",
|
| 756 |
+
"rejected": "Rejected. Nothing was changed.",
|
| 757 |
+
# β HONEST, not reassuring: the answering half that would register a committer is booked,
|
| 758 |
+
# so an approved card today performs nothing and says exactly that rather than "Approved."
|
| 759 |
+
"no_committer": "Approved β but this workspace has nothing configured to carry it out yet.",
|
| 760 |
+
"failed": "That change could not be completed. Nothing was saved; ask the bot again.",
|
| 761 |
+
"committed": "Approved.",
|
| 762 |
+
}.get(verdict, "That approval is no longer available. Ask the bot again.")}
|
| 763 |
+
|
| 764 |
+
|
| 765 |
+
@router.get("/slack/health")
|
| 766 |
+
def slack_health(session: Session = Depends(require_session)):
|
| 767 |
+
"""Is Slack configured for THIS tenant? Session-gated, and it answers about one tenant only β
|
| 768 |
+
enumerating the others would answer a question about our customer list."""
|
| 769 |
+
creds = slack_creds(session.runtime)
|
| 770 |
+
return {"configured": creds is not None,
|
| 771 |
+
"agents": len(_agents(session.runtime)),
|
| 772 |
+
"hasBotToken": bool((creds or {}).get("bot_token"))}
|
api/routes_statements.py
CHANGED
|
@@ -1,298 +1,298 @@
|
|
| 1 |
-
"""routes_statements.py β the statement-of-account sender, ported off Streamlit (EXIT-6).
|
| 2 |
-
|
| 3 |
-
β THIS IS THE ONE SANCTIONED ODOO WRITER IN THE ENTIRE SYSTEM. Everything else in AIOS is
|
| 4 |
-
read-only on Odoo by hard block. `modules/collections_send.py` owns a narrow client that whitelists
|
| 5 |
-
exactly `mail.mail create` (queueing an outbound email) and nothing else; these routes call it and
|
| 6 |
-
add no write of their own.
|
| 7 |
-
|
| 8 |
-
WHY IT EXISTS SEPARATELY FROM THE COLLECTIONS PAGE. Owner ruling wave-17 item 15 retired the
|
| 9 |
-
Collections *dashboard* β the worklist is the shared "Collections" view on the Customer grid, from
|
| 10 |
-
the same reconciled blocks. What that ruling explicitly left "untouched" is this send workflow, so
|
| 11 |
-
when `app.py` is deleted it is the ONLY live Streamlit-only feature, and it moves here rather than
|
| 12 |
-
dying with the host. Ported faithfully: same tiers, same filters, same template placeholders, same
|
| 13 |
-
preview, same test-send, same two-step confirm.
|
| 14 |
-
|
| 15 |
-
THE THREE GUARDRAILS, and where each is enforced:
|
| 16 |
-
1. SAFE_MODE (default ON) β in the DATA LAYER (`collections_send.queue_statement`), so no route,
|
| 17 |
-
payload or UI can bypass it. These routes only REPORT it; they never re-implement the check.
|
| 18 |
-
2. Admin only β `admin_gate` (role, fail-closed), mirroring the Streamlit `if not is_admin()`.
|
| 19 |
-
3. β TENANT β NEW HERE, and it did not exist in Streamlit because it could not. `cs_mod.Odoo()`
|
| 20 |
-
reads Odoo credentials from the ENVIRONMENT, which after the keychain cutover belongs to
|
| 21 |
-
TENANT #0 ALONE. On the multi-tenant API an unguarded route would let a nurilab or gtmlab
|
| 22 |
-
admin queue mail from Royal Imports' Odoo, as Royal Imports. `_royal_only` closes that; the
|
| 23 |
-
single-tenant Streamlit host never had the exposure, so this is a port that must ADD a wall
|
| 24 |
-
rather than copy one.
|
| 25 |
-
|
| 26 |
-
NOTHING SENDS ON A GET. The send route requires an explicit customer list in the body; there is no
|
| 27 |
-
"send all" parameter, deliberately β the confirm step is a product requirement, not a formality.
|
| 28 |
-
"""
|
| 29 |
-
from fastapi import APIRouter, Body, Depends
|
| 30 |
-
|
| 31 |
-
from deps import Session, err
|
| 32 |
-
from routes_admin import admin_gate
|
| 33 |
-
|
| 34 |
-
# W35-T35 (C8): the tenant predicate, held once. See `_royal_only` for why the import runs this
|
| 35 |
-
# way round. `main.py` already imports both modules, so this adds no load.
|
| 36 |
-
import automation_engine as engine
|
| 37 |
-
|
| 38 |
-
router = APIRouter(prefix="/api/v1")
|
| 39 |
-
|
| 40 |
-
#: Cache the Odoo follow-up pull briefly. The Streamlit page used `@st.cache_data(ttl=1800)`; the
|
| 41 |
-
#: list moves slowly (it is a dunning worklist, not a live feed) and the pull is a multi-model read.
|
| 42 |
-
_TTL = 1800
|
| 43 |
-
_cache = {"at": 0.0, "rows": None}
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
def _cs():
|
| 47 |
-
import modules.collections_send as cs
|
| 48 |
-
return cs
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
def _royal_only(session: Session) -> Session:
|
| 52 |
-
"""β See the module docstring, guardrail 3. The send client is env-credentialed, so it is
|
| 53 |
-
tenant #0's and only tenant #0's. Refuse for anyone else rather than send as the wrong company.
|
| 54 |
-
|
| 55 |
-
Keyed on the runtime, never on a request field: a tenant is a property of the SESSION.
|
| 56 |
-
|
| 57 |
-
ββ WAVE 35 Β· T35 / CONTRACT C8 β THE TEST ITSELF NOW LIVES IN ONE PLACE. Statements became an
|
| 58 |
-
agent step this wave, so the automation engine has to answer the same question at two more
|
| 59 |
-
doors (which tenants see the action, which tenants may store it). C8's words are "the predicate
|
| 60 |
-
is imported, never re-expressed", and this is the direction that import can run: the engine
|
| 61 |
-
imports no route module and no FastAPI, so reaching `_royal_only` FROM it would drag `deps` and
|
| 62 |
-
`routes_admin` into a route-free module. The engine holds the boolean; this holds the refusal.
|
| 63 |
-
β WHAT STAYS HERE IS THE HTTP SHAPE, and that is deliberate β a 404 rather than a 403, so the
|
| 64 |
-
surface is invisible rather than forbidden. The engine must not know about status codes."""
|
| 65 |
-
if not engine.is_statement_tenant(session.runtime):
|
| 66 |
-
raise err(404, "not_found", "statements are not configured for this workspace")
|
| 67 |
-
return session
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
def _gate(session: Session = Depends(admin_gate)) -> Session:
|
| 71 |
-
return _royal_only(session)
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
def _rows(force=False):
|
| 75 |
-
import time
|
| 76 |
-
cs = _cs()
|
| 77 |
-
if force or _cache["rows"] is None or (time.time() - _cache["at"]) > _TTL:
|
| 78 |
-
_cache["rows"] = cs.load_collection_list(cs.Odoo())
|
| 79 |
-
_cache["at"] = time.time()
|
| 80 |
-
return _cache["rows"], time.strftime("%Y-%m-%d %H:%M", time.localtime(_cache["at"]))
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
def _public(row):
|
| 84 |
-
"""Strip the internals the Streamlit grid also hid (`_`-prefixed + partner_id is kept, because
|
| 85 |
-
the client needs a stable row identity that is not the display name)."""
|
| 86 |
-
return {k: v for k, v in row.items() if not str(k).startswith("_")}
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
@router.get("/admin/statements")
|
| 90 |
-
def statements(refresh: int = 0, session: Session = Depends(_gate)):
|
| 91 |
-
"""The worklist + everything the sender UI needs to render itself honestly."""
|
| 92 |
-
cs = _cs()
|
| 93 |
-
try:
|
| 94 |
-
rows, loaded_at = _rows(force=bool(refresh))
|
| 95 |
-
except Exception as e:
|
| 96 |
-
raise err(502, "odoo_unavailable", f"could not load the collection list: {str(e)[:200]}")
|
| 97 |
-
return {
|
| 98 |
-
"rows": [_public(r) for r in rows],
|
| 99 |
-
"loadedAt": loaded_at,
|
| 100 |
-
# The guardrail is reported, never decided, here β the data layer owns it.
|
| 101 |
-
"safeMode": bool(cs.SAFE_MODE),
|
| 102 |
-
"safeRecipients": sorted(cs.SAFE_RECIPIENTS),
|
| 103 |
-
"sender": {"name": cs.SENDER_NAME, "email": cs.SENDER_EMAIL,
|
| 104 |
-
"replyTo": cs.REPLY_TO, "company": cs.COMPANY},
|
| 105 |
-
"templates": {"subject": cs.DEFAULT_SUBJECT, "intro": cs.DEFAULT_INTRO,
|
| 106 |
-
"footer": cs.DEFAULT_FOOTER},
|
| 107 |
-
# W35-T35: ONE literal, read from the engine, so the sender's worklist and a statements
|
| 108 |
-
# agent's tier filter cannot come to mean different things.
|
| 109 |
-
"tiers": list(engine.STATEMENT_TIERS),
|
| 110 |
-
}
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
def _find(rows, customer):
|
| 114 |
-
return next((r for r in rows if r.get("Customer") == customer), None)
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
@router.post("/admin/statements/preview")
|
| 118 |
-
def preview(body: dict = Body(default=None), session: Session = Depends(_gate)):
|
| 119 |
-
"""Render ONE customer's statement exactly as the send path would."""
|
| 120 |
-
cs = _cs()
|
| 121 |
-
body = body or {}
|
| 122 |
-
rows, _ = _rows()
|
| 123 |
-
row = _find(rows, body.get("customer"))
|
| 124 |
-
if row is None:
|
| 125 |
-
raise err(404, "not_found", "no such customer on the collection list")
|
| 126 |
-
t = body.get("templates") or {}
|
| 127 |
-
import datetime as dt
|
| 128 |
-
month = dt.date.today().strftime("%B %Y")
|
| 129 |
-
subject = (t.get("subject") or cs.DEFAULT_SUBJECT)
|
| 130 |
-
try:
|
| 131 |
-
subject = subject.format(customer=row["Customer"], company=cs.COMPANY, month=month)
|
| 132 |
-
except Exception: # noqa: BLE001
|
| 133 |
-
# An unknown placeholder is the user's typo, not a 500. Show the template verbatim so they
|
| 134 |
-
# can see what they typed rather than getting an opaque error.
|
| 135 |
-
# β W36-T42: THIS CAUGHT THREE OF THE WAYS `str.format` FAILS AND THERE ARE MORE.
|
| 136 |
-
# `{customer` is a ValueError, `{customer.x}` an AttributeError, `{customer:%Y}` a
|
| 137 |
-
# TypeError, and every one of them is the same user mistake this branch was written for.
|
| 138 |
-
# Naming a subset of the exception types turns a typo in the OTHER half into a 500 on the
|
| 139 |
-
# one screen somebody opens to avoid mailing 200 people the wrong thing.
|
| 140 |
-
pass
|
| 141 |
-
return {
|
| 142 |
-
"html": cs.render_statement_html(row, t.get("intro") or cs.DEFAULT_INTRO,
|
| 143 |
-
t.get("footer") or cs.DEFAULT_FOOTER),
|
| 144 |
-
"to": row.get("Email") or "",
|
| 145 |
-
"subject": subject,
|
| 146 |
-
}
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
@router.post("/admin/statements/send")
|
| 150 |
-
def send(body: dict = Body(default=None), session: Session = Depends(_gate)):
|
| 151 |
-
"""Queue statements. Returns per-customer outcomes β NEVER a bare count.
|
| 152 |
-
|
| 153 |
-
`overrideTo` is the test-send path: one customer, one address. SAFE_MODE still applies (the
|
| 154 |
-
data layer refuses an address outside the allow-list), which is why the route does not check it.
|
| 155 |
-
"""
|
| 156 |
-
cs = _cs()
|
| 157 |
-
body = body or {}
|
| 158 |
-
names = [str(n) for n in (body.get("customers") or []) if str(n).strip()]
|
| 159 |
-
if not names:
|
| 160 |
-
raise err(400, "bad_request", "name at least one customer")
|
| 161 |
-
# ββ W36-T42 / D-299 β A DECLARED TEST SEND WITH NO ADDRESS REFUSES. IT DOES NOT BECOME A
|
| 162 |
-
# REAL ONE. This read `(body.get("overrideTo") or "").strip() or None`, so a caller who sent
|
| 163 |
-
# `overrideTo: ""` (the field left blank, the state not yet typed into, a trimmed-away space)
|
| 164 |
-
# got a REAL statement mailed to the debtor's own address, reported back as `test: false`. On
|
| 165 |
-
# the one route in this product that is allowed to write to Odoo, the difference between a
|
| 166 |
-
# rehearsal and mailing a live customer was one empty string.
|
| 167 |
-
#
|
| 168 |
-
# β THE DISCRIMINATOR IS THE CALLER'S DECLARATION, NOT THE VALUE. A body with no `overrideTo`
|
| 169 |
-
# key and no `test` flag is the PRODUCTION send, and refusing that would delete the feature
|
| 170 |
-
# D-299 exists to preserve. What can be refused is a caller who SAID this is a test: the key
|
| 171 |
-
# being present, or `test: true`, is that statement, and an empty address beside it is the
|
| 172 |
-
# mistake. So both spellings of "absent" are covered: the key present and blank, and `test`
|
| 173 |
-
# asserted with no key at all.
|
| 174 |
-
#
|
| 175 |
-
# β THIS IS THE SECOND OF THREE WALLS AND THE ONLY ONE A PAYLOAD CANNOT ROUTE AROUND.
|
| 176 |
-
# `automationApi.testSendStatement` refuses an empty address before the request is built, and
|
| 177 |
-
# SAFE_MODE refuses an address outside the allow-list inside `queue_statement`. The client
|
| 178 |
-
# wall is bypassable by construction (anything can POST); this one is not.
|
| 179 |
-
declared_test = ("overrideTo" in body) or bool(body.get("test"))
|
| 180 |
-
override = str(body.get("overrideTo") or "").strip()
|
| 181 |
-
if declared_test and not override:
|
| 182 |
-
raise err(400, "no_override_address",
|
| 183 |
-
"a test send needs the address to send the test to. Without one this would "
|
| 184 |
-
"mail the customer's own address, which is the opposite of a test")
|
| 185 |
-
override = override or None
|
| 186 |
-
if override and len(names) != 1:
|
| 187 |
-
raise err(400, "bad_request", "a test send takes exactly one customer")
|
| 188 |
-
t = body.get("templates") or {}
|
| 189 |
-
rows, _ = _rows()
|
| 190 |
-
|
| 191 |
-
sent, failed, skipped = [], [], []
|
| 192 |
-
for name in names:
|
| 193 |
-
row = _find(rows, name)
|
| 194 |
-
if row is None:
|
| 195 |
-
failed.append({"customer": name, "error": "not on the current collection list"})
|
| 196 |
-
continue
|
| 197 |
-
if not override and not row.get("Email"):
|
| 198 |
-
# The Streamlit page warned and skipped these. Reporting them SEPARATELY from failures
|
| 199 |
-
# keeps "we could not" distinct from "there was nowhere to send".
|
| 200 |
-
skipped.append({"customer": name, "reason": "no email address on the customer record"})
|
| 201 |
-
continue
|
| 202 |
-
try:
|
| 203 |
-
mid = cs.queue_statement(cs.Odoo(), row, t.get("subject") or cs.DEFAULT_SUBJECT,
|
| 204 |
-
t.get("intro") or cs.DEFAULT_INTRO,
|
| 205 |
-
t.get("footer") or cs.DEFAULT_FOOTER, override_to=override)
|
| 206 |
-
sent.append({"customer": name, "to": override or row.get("Email"), "mailId": mid})
|
| 207 |
-
except Exception as e:
|
| 208 |
-
# SafeModeBlocked lands here too, and that is correct: to the caller a guardrail refusal
|
| 209 |
-
# and an Odoo error are both "this one did not go", each with its own honest message.
|
| 210 |
-
failed.append({"customer": name, "error": str(e)[:200]})
|
| 211 |
-
return {"sent": sent, "failed": failed, "skipped": skipped,
|
| 212 |
-
"safeMode": bool(cs.SAFE_MODE), "test": bool(override)}
|
| 213 |
-
|
| 214 |
-
|
| 215 |
-
# ---------------------------------------------------------------------------------------------
|
| 216 |
-
# WAVE 35 Β· T36 / CONTRACT C8 / OWNER RULING R10 β THE AGENT'S PARKED BATCH.
|
| 217 |
-
#
|
| 218 |
-
# ββ THE AGENT ASSEMBLES; A PERSON CLICKS SEND. `automation_engine.run_statements` renders a batch
|
| 219 |
-
# and parks it on the automation definition, and it imports no mail path at all. THIS is the only
|
| 220 |
-
# door that releases one, and it is deliberately built out of the parts already here:
|
| 221 |
-
# Β· `_gate` β `admin_gate` THEN `_royal_only`, byte-for-byte the dependency the other three
|
| 222 |
-
# endpoints use. Not a copy of the checks: the same object.
|
| 223 |
-
# Β· `cs.queue_statement` β the same call `send()` above makes, so SAFE_MODE's allow-list refusal
|
| 224 |
-
# happens in the DATA LAYER, where no route, payload or UI can reach around it.
|
| 225 |
-
# β Nothing about the guardrails is re-expressed here, which is what makes "unchanged" checkable.
|
| 226 |
-
#
|
| 227 |
-
# β WHY THE BATCH IS RE-RENDERED FROM THE LIVE WORKLIST RATHER THAN SENT AS STORED. The parked
|
| 228 |
-
# `html` is what a person APPROVED and is what the review screen shows; but a balance can move
|
| 229 |
-
# between the parking and the click, and mailing a figure we know to be stale is worse than mailing
|
| 230 |
-
# a fresh one. So the parked batch decides WHO and WITH WHAT WORDS, and the live row decides the
|
| 231 |
-
# NUMBERS β the same split `preview` already makes. A customer who has left the worklist entirely
|
| 232 |
-
# (they paid) is reported as skipped rather than invoiced.
|
| 233 |
-
|
| 234 |
-
@router.get("/admin/statements/agent/{auto_id}")
|
| 235 |
-
def agent_batch(auto_id: str, session: Session = Depends(_gate)):
|
| 236 |
-
"""What is parked and waiting for a click, for ONE agent. `null` when nothing is."""
|
| 237 |
-
defn = (engine.all_definitions(session.runtime) or {}).get(str(auto_id)) or {}
|
| 238 |
-
pending = defn.get("pendingStatements")
|
| 239 |
-
if not isinstance(pending, dict):
|
| 240 |
-
return {"pending": None}
|
| 241 |
-
# β The rendered `html` is NOT returned in the list β a 200-statement batch of rendered mail is
|
| 242 |
-
# megabytes, and the review screen needs the count and the names to decide. `preview` already
|
| 243 |
-
# renders ONE on demand.
|
| 244 |
-
return {"pending": {
|
| 245 |
-
"ts": pending.get("ts"), "count": int(pending.get("count") or 0),
|
| 246 |
-
"notes": list(pending.get("notes") or []),
|
| 247 |
-
"items": [{k: v for k, v in it.items() if k != "html"}
|
| 248 |
-
for it in (pending.get("items") or [])],
|
| 249 |
-
"safeMode": bool(_cs().SAFE_MODE)}}
|
| 250 |
-
|
| 251 |
-
|
| 252 |
-
@router.post("/admin/statements/agent/{auto_id}/send")
|
| 253 |
-
def agent_send(auto_id: str, body: dict = Body(default=None),
|
| 254 |
-
session: Session = Depends(_gate)):
|
| 255 |
-
"""Release a parked batch. THE CLICK R10 REQUIRES β nothing else in the system calls this.
|
| 256 |
-
|
| 257 |
-
β IT REFUSES AN EMPTY OR MISSING BATCH rather than answering 200 with nothing sent: a send
|
| 258 |
-
that reports success and mails nobody is the `view_upsert` failure mode (200 OK, zero writes)
|
| 259 |
-
arriving on the one route in this system that talks to a mail server.
|
| 260 |
-
"""
|
| 261 |
-
cs = _cs()
|
| 262 |
-
defn = (engine.all_definitions(session.runtime) or {}).get(str(auto_id)) or {}
|
| 263 |
-
pending = defn.get("pendingStatements")
|
| 264 |
-
items = list((pending or {}).get("items") or []) if isinstance(pending, dict) else []
|
| 265 |
-
if not items:
|
| 266 |
-
raise err(404, "no_batch", "there is nothing parked for this agent to send")
|
| 267 |
-
# β An explicit subset is allowed (a person unticking a customer on the review screen) but it
|
| 268 |
-
# may only NARROW the parked batch. A name that was never parked cannot be introduced by the
|
| 269 |
-
# payload, or the review stops being what authorised the send.
|
| 270 |
-
want = {str(n) for n in (body or {}).get("customers") or []}
|
| 271 |
-
if want:
|
| 272 |
-
items = [it for it in items if str(it.get("customer")) in want]
|
| 273 |
-
if not items:
|
| 274 |
-
raise err(400, "bad_request", "none of those customers are in the parked batch")
|
| 275 |
-
rows, _ = _rows()
|
| 276 |
-
sent, failed, skipped = [], [], []
|
| 277 |
-
for it in items:
|
| 278 |
-
name = str(it.get("customer") or "")
|
| 279 |
-
row = _find(rows, name)
|
| 280 |
-
if row is None:
|
| 281 |
-
skipped.append({"customer": name,
|
| 282 |
-
"reason": "no longer on the collection list, so nothing is owed"})
|
| 283 |
-
continue
|
| 284 |
-
try:
|
| 285 |
-
# β THE SAME DATA-LAYER CALL `send()` MAKES. SafeModeBlocked is raised INSIDE
|
| 286 |
-
# `queue_statement`, so the guardrail cannot be argued with from here.
|
| 287 |
-
mid = cs.queue_statement(cs.Odoo(), row,
|
| 288 |
-
str(it.get("subject") or "") or cs.DEFAULT_SUBJECT,
|
| 289 |
-
cs.DEFAULT_INTRO, cs.DEFAULT_FOOTER)
|
| 290 |
-
sent.append({"customer": name, "to": row.get("Email"), "mailId": mid})
|
| 291 |
-
except Exception as e: # noqa: BLE001
|
| 292 |
-
failed.append({"customer": name, "error": str(e)[:200]})
|
| 293 |
-
# β CLEARED ONLY WHEN NOTHING IS LEFT TO RETRY. A batch dropped while some of it failed would
|
| 294 |
-
# lose the list of who still needs a statement, and nobody would know to look.
|
| 295 |
-
if sent and not failed:
|
| 296 |
-
engine.clear_statements(session.runtime, auto_id)
|
| 297 |
-
return {"sent": sent, "failed": failed, "skipped": skipped,
|
| 298 |
-
"safeMode": bool(cs.SAFE_MODE), "cleared": bool(sent and not failed)}
|
|
|
|
| 1 |
+
"""routes_statements.py β the statement-of-account sender, ported off Streamlit (EXIT-6).
|
| 2 |
+
|
| 3 |
+
β THIS IS THE ONE SANCTIONED ODOO WRITER IN THE ENTIRE SYSTEM. Everything else in AIOS is
|
| 4 |
+
read-only on Odoo by hard block. `modules/collections_send.py` owns a narrow client that whitelists
|
| 5 |
+
exactly `mail.mail create` (queueing an outbound email) and nothing else; these routes call it and
|
| 6 |
+
add no write of their own.
|
| 7 |
+
|
| 8 |
+
WHY IT EXISTS SEPARATELY FROM THE COLLECTIONS PAGE. Owner ruling wave-17 item 15 retired the
|
| 9 |
+
Collections *dashboard* β the worklist is the shared "Collections" view on the Customer grid, from
|
| 10 |
+
the same reconciled blocks. What that ruling explicitly left "untouched" is this send workflow, so
|
| 11 |
+
when `app.py` is deleted it is the ONLY live Streamlit-only feature, and it moves here rather than
|
| 12 |
+
dying with the host. Ported faithfully: same tiers, same filters, same template placeholders, same
|
| 13 |
+
preview, same test-send, same two-step confirm.
|
| 14 |
+
|
| 15 |
+
THE THREE GUARDRAILS, and where each is enforced:
|
| 16 |
+
1. SAFE_MODE (default ON) β in the DATA LAYER (`collections_send.queue_statement`), so no route,
|
| 17 |
+
payload or UI can bypass it. These routes only REPORT it; they never re-implement the check.
|
| 18 |
+
2. Admin only β `admin_gate` (role, fail-closed), mirroring the Streamlit `if not is_admin()`.
|
| 19 |
+
3. β TENANT β NEW HERE, and it did not exist in Streamlit because it could not. `cs_mod.Odoo()`
|
| 20 |
+
reads Odoo credentials from the ENVIRONMENT, which after the keychain cutover belongs to
|
| 21 |
+
TENANT #0 ALONE. On the multi-tenant API an unguarded route would let a nurilab or gtmlab
|
| 22 |
+
admin queue mail from Royal Imports' Odoo, as Royal Imports. `_royal_only` closes that; the
|
| 23 |
+
single-tenant Streamlit host never had the exposure, so this is a port that must ADD a wall
|
| 24 |
+
rather than copy one.
|
| 25 |
+
|
| 26 |
+
NOTHING SENDS ON A GET. The send route requires an explicit customer list in the body; there is no
|
| 27 |
+
"send all" parameter, deliberately β the confirm step is a product requirement, not a formality.
|
| 28 |
+
"""
|
| 29 |
+
from fastapi import APIRouter, Body, Depends
|
| 30 |
+
|
| 31 |
+
from deps import Session, err
|
| 32 |
+
from routes_admin import admin_gate
|
| 33 |
+
|
| 34 |
+
# W35-T35 (C8): the tenant predicate, held once. See `_royal_only` for why the import runs this
|
| 35 |
+
# way round. `main.py` already imports both modules, so this adds no load.
|
| 36 |
+
import automation_engine as engine
|
| 37 |
+
|
| 38 |
+
router = APIRouter(prefix="/api/v1")
|
| 39 |
+
|
| 40 |
+
#: Cache the Odoo follow-up pull briefly. The Streamlit page used `@st.cache_data(ttl=1800)`; the
|
| 41 |
+
#: list moves slowly (it is a dunning worklist, not a live feed) and the pull is a multi-model read.
|
| 42 |
+
_TTL = 1800
|
| 43 |
+
_cache = {"at": 0.0, "rows": None}
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def _cs():
|
| 47 |
+
import modules.collections_send as cs
|
| 48 |
+
return cs
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def _royal_only(session: Session) -> Session:
|
| 52 |
+
"""β See the module docstring, guardrail 3. The send client is env-credentialed, so it is
|
| 53 |
+
tenant #0's and only tenant #0's. Refuse for anyone else rather than send as the wrong company.
|
| 54 |
+
|
| 55 |
+
Keyed on the runtime, never on a request field: a tenant is a property of the SESSION.
|
| 56 |
+
|
| 57 |
+
ββ WAVE 35 Β· T35 / CONTRACT C8 β THE TEST ITSELF NOW LIVES IN ONE PLACE. Statements became an
|
| 58 |
+
agent step this wave, so the automation engine has to answer the same question at two more
|
| 59 |
+
doors (which tenants see the action, which tenants may store it). C8's words are "the predicate
|
| 60 |
+
is imported, never re-expressed", and this is the direction that import can run: the engine
|
| 61 |
+
imports no route module and no FastAPI, so reaching `_royal_only` FROM it would drag `deps` and
|
| 62 |
+
`routes_admin` into a route-free module. The engine holds the boolean; this holds the refusal.
|
| 63 |
+
β WHAT STAYS HERE IS THE HTTP SHAPE, and that is deliberate β a 404 rather than a 403, so the
|
| 64 |
+
surface is invisible rather than forbidden. The engine must not know about status codes."""
|
| 65 |
+
if not engine.is_statement_tenant(session.runtime):
|
| 66 |
+
raise err(404, "not_found", "statements are not configured for this workspace")
|
| 67 |
+
return session
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def _gate(session: Session = Depends(admin_gate)) -> Session:
|
| 71 |
+
return _royal_only(session)
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
def _rows(force=False):
|
| 75 |
+
import time
|
| 76 |
+
cs = _cs()
|
| 77 |
+
if force or _cache["rows"] is None or (time.time() - _cache["at"]) > _TTL:
|
| 78 |
+
_cache["rows"] = cs.load_collection_list(cs.Odoo())
|
| 79 |
+
_cache["at"] = time.time()
|
| 80 |
+
return _cache["rows"], time.strftime("%Y-%m-%d %H:%M", time.localtime(_cache["at"]))
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def _public(row):
|
| 84 |
+
"""Strip the internals the Streamlit grid also hid (`_`-prefixed + partner_id is kept, because
|
| 85 |
+
the client needs a stable row identity that is not the display name)."""
|
| 86 |
+
return {k: v for k, v in row.items() if not str(k).startswith("_")}
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
@router.get("/admin/statements")
|
| 90 |
+
def statements(refresh: int = 0, session: Session = Depends(_gate)):
|
| 91 |
+
"""The worklist + everything the sender UI needs to render itself honestly."""
|
| 92 |
+
cs = _cs()
|
| 93 |
+
try:
|
| 94 |
+
rows, loaded_at = _rows(force=bool(refresh))
|
| 95 |
+
except Exception as e:
|
| 96 |
+
raise err(502, "odoo_unavailable", f"could not load the collection list: {str(e)[:200]}")
|
| 97 |
+
return {
|
| 98 |
+
"rows": [_public(r) for r in rows],
|
| 99 |
+
"loadedAt": loaded_at,
|
| 100 |
+
# The guardrail is reported, never decided, here β the data layer owns it.
|
| 101 |
+
"safeMode": bool(cs.SAFE_MODE),
|
| 102 |
+
"safeRecipients": sorted(cs.SAFE_RECIPIENTS),
|
| 103 |
+
"sender": {"name": cs.SENDER_NAME, "email": cs.SENDER_EMAIL,
|
| 104 |
+
"replyTo": cs.REPLY_TO, "company": cs.COMPANY},
|
| 105 |
+
"templates": {"subject": cs.DEFAULT_SUBJECT, "intro": cs.DEFAULT_INTRO,
|
| 106 |
+
"footer": cs.DEFAULT_FOOTER},
|
| 107 |
+
# W35-T35: ONE literal, read from the engine, so the sender's worklist and a statements
|
| 108 |
+
# agent's tier filter cannot come to mean different things.
|
| 109 |
+
"tiers": list(engine.STATEMENT_TIERS),
|
| 110 |
+
}
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def _find(rows, customer):
|
| 114 |
+
return next((r for r in rows if r.get("Customer") == customer), None)
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
@router.post("/admin/statements/preview")
|
| 118 |
+
def preview(body: dict = Body(default=None), session: Session = Depends(_gate)):
|
| 119 |
+
"""Render ONE customer's statement exactly as the send path would."""
|
| 120 |
+
cs = _cs()
|
| 121 |
+
body = body or {}
|
| 122 |
+
rows, _ = _rows()
|
| 123 |
+
row = _find(rows, body.get("customer"))
|
| 124 |
+
if row is None:
|
| 125 |
+
raise err(404, "not_found", "no such customer on the collection list")
|
| 126 |
+
t = body.get("templates") or {}
|
| 127 |
+
import datetime as dt
|
| 128 |
+
month = dt.date.today().strftime("%B %Y")
|
| 129 |
+
subject = (t.get("subject") or cs.DEFAULT_SUBJECT)
|
| 130 |
+
try:
|
| 131 |
+
subject = subject.format(customer=row["Customer"], company=cs.COMPANY, month=month)
|
| 132 |
+
except Exception: # noqa: BLE001
|
| 133 |
+
# An unknown placeholder is the user's typo, not a 500. Show the template verbatim so they
|
| 134 |
+
# can see what they typed rather than getting an opaque error.
|
| 135 |
+
# β W36-T42: THIS CAUGHT THREE OF THE WAYS `str.format` FAILS AND THERE ARE MORE.
|
| 136 |
+
# `{customer` is a ValueError, `{customer.x}` an AttributeError, `{customer:%Y}` a
|
| 137 |
+
# TypeError, and every one of them is the same user mistake this branch was written for.
|
| 138 |
+
# Naming a subset of the exception types turns a typo in the OTHER half into a 500 on the
|
| 139 |
+
# one screen somebody opens to avoid mailing 200 people the wrong thing.
|
| 140 |
+
pass
|
| 141 |
+
return {
|
| 142 |
+
"html": cs.render_statement_html(row, t.get("intro") or cs.DEFAULT_INTRO,
|
| 143 |
+
t.get("footer") or cs.DEFAULT_FOOTER),
|
| 144 |
+
"to": row.get("Email") or "",
|
| 145 |
+
"subject": subject,
|
| 146 |
+
}
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
@router.post("/admin/statements/send")
|
| 150 |
+
def send(body: dict = Body(default=None), session: Session = Depends(_gate)):
|
| 151 |
+
"""Queue statements. Returns per-customer outcomes β NEVER a bare count.
|
| 152 |
+
|
| 153 |
+
`overrideTo` is the test-send path: one customer, one address. SAFE_MODE still applies (the
|
| 154 |
+
data layer refuses an address outside the allow-list), which is why the route does not check it.
|
| 155 |
+
"""
|
| 156 |
+
cs = _cs()
|
| 157 |
+
body = body or {}
|
| 158 |
+
names = [str(n) for n in (body.get("customers") or []) if str(n).strip()]
|
| 159 |
+
if not names:
|
| 160 |
+
raise err(400, "bad_request", "name at least one customer")
|
| 161 |
+
# ββ W36-T42 / D-299 β A DECLARED TEST SEND WITH NO ADDRESS REFUSES. IT DOES NOT BECOME A
|
| 162 |
+
# REAL ONE. This read `(body.get("overrideTo") or "").strip() or None`, so a caller who sent
|
| 163 |
+
# `overrideTo: ""` (the field left blank, the state not yet typed into, a trimmed-away space)
|
| 164 |
+
# got a REAL statement mailed to the debtor's own address, reported back as `test: false`. On
|
| 165 |
+
# the one route in this product that is allowed to write to Odoo, the difference between a
|
| 166 |
+
# rehearsal and mailing a live customer was one empty string.
|
| 167 |
+
#
|
| 168 |
+
# β THE DISCRIMINATOR IS THE CALLER'S DECLARATION, NOT THE VALUE. A body with no `overrideTo`
|
| 169 |
+
# key and no `test` flag is the PRODUCTION send, and refusing that would delete the feature
|
| 170 |
+
# D-299 exists to preserve. What can be refused is a caller who SAID this is a test: the key
|
| 171 |
+
# being present, or `test: true`, is that statement, and an empty address beside it is the
|
| 172 |
+
# mistake. So both spellings of "absent" are covered: the key present and blank, and `test`
|
| 173 |
+
# asserted with no key at all.
|
| 174 |
+
#
|
| 175 |
+
# β THIS IS THE SECOND OF THREE WALLS AND THE ONLY ONE A PAYLOAD CANNOT ROUTE AROUND.
|
| 176 |
+
# `automationApi.testSendStatement` refuses an empty address before the request is built, and
|
| 177 |
+
# SAFE_MODE refuses an address outside the allow-list inside `queue_statement`. The client
|
| 178 |
+
# wall is bypassable by construction (anything can POST); this one is not.
|
| 179 |
+
declared_test = ("overrideTo" in body) or bool(body.get("test"))
|
| 180 |
+
override = str(body.get("overrideTo") or "").strip()
|
| 181 |
+
if declared_test and not override:
|
| 182 |
+
raise err(400, "no_override_address",
|
| 183 |
+
"a test send needs the address to send the test to. Without one this would "
|
| 184 |
+
"mail the customer's own address, which is the opposite of a test")
|
| 185 |
+
override = override or None
|
| 186 |
+
if override and len(names) != 1:
|
| 187 |
+
raise err(400, "bad_request", "a test send takes exactly one customer")
|
| 188 |
+
t = body.get("templates") or {}
|
| 189 |
+
rows, _ = _rows()
|
| 190 |
+
|
| 191 |
+
sent, failed, skipped = [], [], []
|
| 192 |
+
for name in names:
|
| 193 |
+
row = _find(rows, name)
|
| 194 |
+
if row is None:
|
| 195 |
+
failed.append({"customer": name, "error": "not on the current collection list"})
|
| 196 |
+
continue
|
| 197 |
+
if not override and not row.get("Email"):
|
| 198 |
+
# The Streamlit page warned and skipped these. Reporting them SEPARATELY from failures
|
| 199 |
+
# keeps "we could not" distinct from "there was nowhere to send".
|
| 200 |
+
skipped.append({"customer": name, "reason": "no email address on the customer record"})
|
| 201 |
+
continue
|
| 202 |
+
try:
|
| 203 |
+
mid = cs.queue_statement(cs.Odoo(), row, t.get("subject") or cs.DEFAULT_SUBJECT,
|
| 204 |
+
t.get("intro") or cs.DEFAULT_INTRO,
|
| 205 |
+
t.get("footer") or cs.DEFAULT_FOOTER, override_to=override)
|
| 206 |
+
sent.append({"customer": name, "to": override or row.get("Email"), "mailId": mid})
|
| 207 |
+
except Exception as e:
|
| 208 |
+
# SafeModeBlocked lands here too, and that is correct: to the caller a guardrail refusal
|
| 209 |
+
# and an Odoo error are both "this one did not go", each with its own honest message.
|
| 210 |
+
failed.append({"customer": name, "error": str(e)[:200]})
|
| 211 |
+
return {"sent": sent, "failed": failed, "skipped": skipped,
|
| 212 |
+
"safeMode": bool(cs.SAFE_MODE), "test": bool(override)}
|
| 213 |
+
|
| 214 |
+
|
| 215 |
+
# ---------------------------------------------------------------------------------------------
|
| 216 |
+
# WAVE 35 Β· T36 / CONTRACT C8 / OWNER RULING R10 β THE AGENT'S PARKED BATCH.
|
| 217 |
+
#
|
| 218 |
+
# ββ THE AGENT ASSEMBLES; A PERSON CLICKS SEND. `automation_engine.run_statements` renders a batch
|
| 219 |
+
# and parks it on the automation definition, and it imports no mail path at all. THIS is the only
|
| 220 |
+
# door that releases one, and it is deliberately built out of the parts already here:
|
| 221 |
+
# Β· `_gate` β `admin_gate` THEN `_royal_only`, byte-for-byte the dependency the other three
|
| 222 |
+
# endpoints use. Not a copy of the checks: the same object.
|
| 223 |
+
# Β· `cs.queue_statement` β the same call `send()` above makes, so SAFE_MODE's allow-list refusal
|
| 224 |
+
# happens in the DATA LAYER, where no route, payload or UI can reach around it.
|
| 225 |
+
# β Nothing about the guardrails is re-expressed here, which is what makes "unchanged" checkable.
|
| 226 |
+
#
|
| 227 |
+
# β WHY THE BATCH IS RE-RENDERED FROM THE LIVE WORKLIST RATHER THAN SENT AS STORED. The parked
|
| 228 |
+
# `html` is what a person APPROVED and is what the review screen shows; but a balance can move
|
| 229 |
+
# between the parking and the click, and mailing a figure we know to be stale is worse than mailing
|
| 230 |
+
# a fresh one. So the parked batch decides WHO and WITH WHAT WORDS, and the live row decides the
|
| 231 |
+
# NUMBERS β the same split `preview` already makes. A customer who has left the worklist entirely
|
| 232 |
+
# (they paid) is reported as skipped rather than invoiced.
|
| 233 |
+
|
| 234 |
+
@router.get("/admin/statements/agent/{auto_id}")
|
| 235 |
+
def agent_batch(auto_id: str, session: Session = Depends(_gate)):
|
| 236 |
+
"""What is parked and waiting for a click, for ONE agent. `null` when nothing is."""
|
| 237 |
+
defn = (engine.all_definitions(session.runtime) or {}).get(str(auto_id)) or {}
|
| 238 |
+
pending = defn.get("pendingStatements")
|
| 239 |
+
if not isinstance(pending, dict):
|
| 240 |
+
return {"pending": None}
|
| 241 |
+
# β The rendered `html` is NOT returned in the list β a 200-statement batch of rendered mail is
|
| 242 |
+
# megabytes, and the review screen needs the count and the names to decide. `preview` already
|
| 243 |
+
# renders ONE on demand.
|
| 244 |
+
return {"pending": {
|
| 245 |
+
"ts": pending.get("ts"), "count": int(pending.get("count") or 0),
|
| 246 |
+
"notes": list(pending.get("notes") or []),
|
| 247 |
+
"items": [{k: v for k, v in it.items() if k != "html"}
|
| 248 |
+
for it in (pending.get("items") or [])],
|
| 249 |
+
"safeMode": bool(_cs().SAFE_MODE)}}
|
| 250 |
+
|
| 251 |
+
|
| 252 |
+
@router.post("/admin/statements/agent/{auto_id}/send")
|
| 253 |
+
def agent_send(auto_id: str, body: dict = Body(default=None),
|
| 254 |
+
session: Session = Depends(_gate)):
|
| 255 |
+
"""Release a parked batch. THE CLICK R10 REQUIRES β nothing else in the system calls this.
|
| 256 |
+
|
| 257 |
+
β IT REFUSES AN EMPTY OR MISSING BATCH rather than answering 200 with nothing sent: a send
|
| 258 |
+
that reports success and mails nobody is the `view_upsert` failure mode (200 OK, zero writes)
|
| 259 |
+
arriving on the one route in this system that talks to a mail server.
|
| 260 |
+
"""
|
| 261 |
+
cs = _cs()
|
| 262 |
+
defn = (engine.all_definitions(session.runtime) or {}).get(str(auto_id)) or {}
|
| 263 |
+
pending = defn.get("pendingStatements")
|
| 264 |
+
items = list((pending or {}).get("items") or []) if isinstance(pending, dict) else []
|
| 265 |
+
if not items:
|
| 266 |
+
raise err(404, "no_batch", "there is nothing parked for this agent to send")
|
| 267 |
+
# β An explicit subset is allowed (a person unticking a customer on the review screen) but it
|
| 268 |
+
# may only NARROW the parked batch. A name that was never parked cannot be introduced by the
|
| 269 |
+
# payload, or the review stops being what authorised the send.
|
| 270 |
+
want = {str(n) for n in (body or {}).get("customers") or []}
|
| 271 |
+
if want:
|
| 272 |
+
items = [it for it in items if str(it.get("customer")) in want]
|
| 273 |
+
if not items:
|
| 274 |
+
raise err(400, "bad_request", "none of those customers are in the parked batch")
|
| 275 |
+
rows, _ = _rows()
|
| 276 |
+
sent, failed, skipped = [], [], []
|
| 277 |
+
for it in items:
|
| 278 |
+
name = str(it.get("customer") or "")
|
| 279 |
+
row = _find(rows, name)
|
| 280 |
+
if row is None:
|
| 281 |
+
skipped.append({"customer": name,
|
| 282 |
+
"reason": "no longer on the collection list, so nothing is owed"})
|
| 283 |
+
continue
|
| 284 |
+
try:
|
| 285 |
+
# β THE SAME DATA-LAYER CALL `send()` MAKES. SafeModeBlocked is raised INSIDE
|
| 286 |
+
# `queue_statement`, so the guardrail cannot be argued with from here.
|
| 287 |
+
mid = cs.queue_statement(cs.Odoo(), row,
|
| 288 |
+
str(it.get("subject") or "") or cs.DEFAULT_SUBJECT,
|
| 289 |
+
cs.DEFAULT_INTRO, cs.DEFAULT_FOOTER)
|
| 290 |
+
sent.append({"customer": name, "to": row.get("Email"), "mailId": mid})
|
| 291 |
+
except Exception as e: # noqa: BLE001
|
| 292 |
+
failed.append({"customer": name, "error": str(e)[:200]})
|
| 293 |
+
# β CLEARED ONLY WHEN NOTHING IS LEFT TO RETRY. A batch dropped while some of it failed would
|
| 294 |
+
# lose the list of who still needs a statement, and nobody would know to look.
|
| 295 |
+
if sent and not failed:
|
| 296 |
+
engine.clear_statements(session.runtime, auto_id)
|
| 297 |
+
return {"sent": sent, "failed": failed, "skipped": skipped,
|
| 298 |
+
"safeMode": bool(cs.SAFE_MODE), "cleared": bool(sent and not failed)}
|
api/routes_tables.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
api/routes_web_agent.py
CHANGED
|
@@ -1,84 +1,84 @@
|
|
| 1 |
-
"""routes_web_agent.py β the door that lets a person TEST the web agent (wave 31, R10 / D-51).
|
| 2 |
-
|
| 3 |
-
The owner's words are the whole reason this file exists: *"we already laid the foundation of this
|
| 4 |
-
but never test anything."* The capability is otherwise reachable only from inside an automation
|
| 5 |
-
run, which means the first person to discover it is broken is a customer at 3am.
|
| 6 |
-
|
| 7 |
-
GET /api/v1/web-agent/capability any session β can this deployment run a web step, and
|
| 8 |
-
if not, the SENTENCE saying why
|
| 9 |
-
POST /api/v1/web-agent/test ADMIN β run one real `web_read` and show what
|
| 10 |
-
came back, or the sentence
|
| 11 |
-
|
| 12 |
-
β NEITHER ROUTE IS THE SEAM. `automation_engine` calls `web_agent.run_step` directly (contract
|
| 13 |
-
C5); these are an operator surface over the same function, so a green test here and a red run
|
| 14 |
-
there cannot disagree about anything except the input.
|
| 15 |
-
|
| 16 |
-
β `POST /test` BLOCKS FOR ~10-30 s and COSTS A FRACTION OF A CENT. It is `def`, not `async def`,
|
| 17 |
-
so FastAPI runs it in the threadpool and one test cannot stall the event loop for everybody.
|
| 18 |
-
|
| 19 |
-
β ON THE URL IT WILL FETCH: the fetch happens inside an ephemeral HF Job on Hugging Face's
|
| 20 |
-
network, never from this server, so this is not a door into our own infrastructure. It is still
|
| 21 |
-
admin-gated, because it spends money and because D-51 Β§5's authorisation posture ("only systems
|
| 22 |
-
the tenant is authorised to use, at their instruction") is not something an ordinary member
|
| 23 |
-
should be able to commit the tenant to.
|
| 24 |
-
|
| 25 |
-
β MOUNTED, and the gate is what says so rather than this sentence: `main.py:80` imports it and
|
| 26 |
-
`main.py:316` includes it, and `verify_web_agent.py` asserts the route answers rather than trusting
|
| 27 |
-
either line. This paragraph read "NOT MOUNTED YET" for a whole wave after the mount landed, which
|
| 28 |
-
is the same class of stale claim as a green gate on a router nobody wired: three finished routers
|
| 29 |
-
once shipped 404-dead behind entirely green gates, and prose is not the control that stops it.
|
| 30 |
-
"""
|
| 31 |
-
from fastapi import APIRouter, Body, Depends
|
| 32 |
-
|
| 33 |
-
import web_agent
|
| 34 |
-
from deps import Session, require_session, err
|
| 35 |
-
from routes_admin import admin_gate
|
| 36 |
-
|
| 37 |
-
router = APIRouter(prefix="/api/v1")
|
| 38 |
-
|
| 39 |
-
MAX_URL = 2000
|
| 40 |
-
MAX_SELECTOR = 400
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
@router.get("/web-agent/capability")
|
| 44 |
-
def web_agent_capability(session: Session = Depends(require_session)):
|
| 45 |
-
"""Can a web step run here at all? Configuration, not liveness β see `web_agent.capability`.
|
| 46 |
-
|
| 47 |
-
Deliberately NARROW: it answers the question a UI needs ("may I offer this, and what do I say
|
| 48 |
-
if not") and withholds the deployment detail (namespace, which token key, the image) that
|
| 49 |
-
only an operator has any use for. Nothing here is ever a credential.
|
| 50 |
-
"""
|
| 51 |
-
cap = web_agent.capability()
|
| 52 |
-
return {"ready": bool(cap["ready"]), "reason": cap["reason"],
|
| 53 |
-
"runnableKinds": cap["runnableKinds"], "profile": cap["profile"]}
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
@router.post("/web-agent/test")
|
| 57 |
-
def web_agent_test(session: Session = Depends(admin_gate), body: dict = Body(...)):
|
| 58 |
-
"""Run ONE real `web_read` and report exactly what the seam returned.
|
| 59 |
-
|
| 60 |
-
The response mirrors the seam's own contract rather than flattening it: `ok` plus a `value`,
|
| 61 |
-
or `ok:false` plus the SENTENCE. A test surface that turns a named failure into "something
|
| 62 |
-
went wrong" would hide the one thing it exists to show.
|
| 63 |
-
"""
|
| 64 |
-
url = str(body.get("url") or "").strip()
|
| 65 |
-
selector = str(body.get("selector") or "").strip()
|
| 66 |
-
if len(url) > MAX_URL or len(selector) > MAX_SELECTOR:
|
| 67 |
-
raise err(400, "too_long", "the URL or selector is longer than this door accepts")
|
| 68 |
-
|
| 69 |
-
step = {"kind": "web_read", "id": "test", "url": url, "selector": selector,
|
| 70 |
-
"attr": (body.get("attr") or "text"), "all": bool(body.get("all")),
|
| 71 |
-
"timeoutMs": int(body.get("timeoutMs") or 20000)}
|
| 72 |
-
if body.get("waitFor"):
|
| 73 |
-
step["waitFor"] = str(body["waitFor"])
|
| 74 |
-
|
| 75 |
-
notes = []
|
| 76 |
-
result, error = web_agent.run_step(
|
| 77 |
-
step, {"tenant": session.tenant, "runId": f"test-{session.tenant}-{session.uname}",
|
| 78 |
-
"log": notes.append})
|
| 79 |
-
if error:
|
| 80 |
-
# 200 with `ok:false`, not a 4xx: the request was well-formed and the ANSWER is that the
|
| 81 |
-
# web step did not succeed. A 500 here would make an ordinary "the selector matched
|
| 82 |
-
# nothing" look like a server fault.
|
| 83 |
-
return {"ok": False, "error": error, "log": notes[-6:]}
|
| 84 |
-
return {"ok": True, "result": result, "log": notes[-6:]}
|
|
|
|
| 1 |
+
"""routes_web_agent.py β the door that lets a person TEST the web agent (wave 31, R10 / D-51).
|
| 2 |
+
|
| 3 |
+
The owner's words are the whole reason this file exists: *"we already laid the foundation of this
|
| 4 |
+
but never test anything."* The capability is otherwise reachable only from inside an automation
|
| 5 |
+
run, which means the first person to discover it is broken is a customer at 3am.
|
| 6 |
+
|
| 7 |
+
GET /api/v1/web-agent/capability any session β can this deployment run a web step, and
|
| 8 |
+
if not, the SENTENCE saying why
|
| 9 |
+
POST /api/v1/web-agent/test ADMIN β run one real `web_read` and show what
|
| 10 |
+
came back, or the sentence
|
| 11 |
+
|
| 12 |
+
β NEITHER ROUTE IS THE SEAM. `automation_engine` calls `web_agent.run_step` directly (contract
|
| 13 |
+
C5); these are an operator surface over the same function, so a green test here and a red run
|
| 14 |
+
there cannot disagree about anything except the input.
|
| 15 |
+
|
| 16 |
+
β `POST /test` BLOCKS FOR ~10-30 s and COSTS A FRACTION OF A CENT. It is `def`, not `async def`,
|
| 17 |
+
so FastAPI runs it in the threadpool and one test cannot stall the event loop for everybody.
|
| 18 |
+
|
| 19 |
+
β ON THE URL IT WILL FETCH: the fetch happens inside an ephemeral HF Job on Hugging Face's
|
| 20 |
+
network, never from this server, so this is not a door into our own infrastructure. It is still
|
| 21 |
+
admin-gated, because it spends money and because D-51 Β§5's authorisation posture ("only systems
|
| 22 |
+
the tenant is authorised to use, at their instruction") is not something an ordinary member
|
| 23 |
+
should be able to commit the tenant to.
|
| 24 |
+
|
| 25 |
+
β MOUNTED, and the gate is what says so rather than this sentence: `main.py:80` imports it and
|
| 26 |
+
`main.py:316` includes it, and `verify_web_agent.py` asserts the route answers rather than trusting
|
| 27 |
+
either line. This paragraph read "NOT MOUNTED YET" for a whole wave after the mount landed, which
|
| 28 |
+
is the same class of stale claim as a green gate on a router nobody wired: three finished routers
|
| 29 |
+
once shipped 404-dead behind entirely green gates, and prose is not the control that stops it.
|
| 30 |
+
"""
|
| 31 |
+
from fastapi import APIRouter, Body, Depends
|
| 32 |
+
|
| 33 |
+
import web_agent
|
| 34 |
+
from deps import Session, require_session, err
|
| 35 |
+
from routes_admin import admin_gate
|
| 36 |
+
|
| 37 |
+
router = APIRouter(prefix="/api/v1")
|
| 38 |
+
|
| 39 |
+
MAX_URL = 2000
|
| 40 |
+
MAX_SELECTOR = 400
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
@router.get("/web-agent/capability")
|
| 44 |
+
def web_agent_capability(session: Session = Depends(require_session)):
|
| 45 |
+
"""Can a web step run here at all? Configuration, not liveness β see `web_agent.capability`.
|
| 46 |
+
|
| 47 |
+
Deliberately NARROW: it answers the question a UI needs ("may I offer this, and what do I say
|
| 48 |
+
if not") and withholds the deployment detail (namespace, which token key, the image) that
|
| 49 |
+
only an operator has any use for. Nothing here is ever a credential.
|
| 50 |
+
"""
|
| 51 |
+
cap = web_agent.capability()
|
| 52 |
+
return {"ready": bool(cap["ready"]), "reason": cap["reason"],
|
| 53 |
+
"runnableKinds": cap["runnableKinds"], "profile": cap["profile"]}
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
@router.post("/web-agent/test")
|
| 57 |
+
def web_agent_test(session: Session = Depends(admin_gate), body: dict = Body(...)):
|
| 58 |
+
"""Run ONE real `web_read` and report exactly what the seam returned.
|
| 59 |
+
|
| 60 |
+
The response mirrors the seam's own contract rather than flattening it: `ok` plus a `value`,
|
| 61 |
+
or `ok:false` plus the SENTENCE. A test surface that turns a named failure into "something
|
| 62 |
+
went wrong" would hide the one thing it exists to show.
|
| 63 |
+
"""
|
| 64 |
+
url = str(body.get("url") or "").strip()
|
| 65 |
+
selector = str(body.get("selector") or "").strip()
|
| 66 |
+
if len(url) > MAX_URL or len(selector) > MAX_SELECTOR:
|
| 67 |
+
raise err(400, "too_long", "the URL or selector is longer than this door accepts")
|
| 68 |
+
|
| 69 |
+
step = {"kind": "web_read", "id": "test", "url": url, "selector": selector,
|
| 70 |
+
"attr": (body.get("attr") or "text"), "all": bool(body.get("all")),
|
| 71 |
+
"timeoutMs": int(body.get("timeoutMs") or 20000)}
|
| 72 |
+
if body.get("waitFor"):
|
| 73 |
+
step["waitFor"] = str(body["waitFor"])
|
| 74 |
+
|
| 75 |
+
notes = []
|
| 76 |
+
result, error = web_agent.run_step(
|
| 77 |
+
step, {"tenant": session.tenant, "runId": f"test-{session.tenant}-{session.uname}",
|
| 78 |
+
"log": notes.append})
|
| 79 |
+
if error:
|
| 80 |
+
# 200 with `ok:false`, not a 4xx: the request was well-formed and the ANSWER is that the
|
| 81 |
+
# web step did not succeed. A 500 here would make an ordinary "the selector matched
|
| 82 |
+
# nothing" look like a server fault.
|
| 83 |
+
return {"ok": False, "error": error, "log": notes[-6:]}
|
| 84 |
+
return {"ok": True, "result": result, "log": notes[-6:]}
|
platform/aios_grid.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
platform/core/perm_scope.py
CHANGED
|
@@ -87,6 +87,50 @@ def may_access(user, module):
|
|
| 87 |
return perms.may_open(user, module) # legacy record: the old grant wall
|
| 88 |
|
| 89 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
def assistant_entry(user, module):
|
| 91 |
"""The explicit database grant an Assistant snapshot may rely on, else ``None``.
|
| 92 |
|
|
|
|
| 87 |
return perms.may_open(user, module) # legacy record: the old grant wall
|
| 88 |
|
| 89 |
|
| 90 |
+
def nav_may_open(user, key, st=None):
|
| 91 |
+
"""May this account SEE `key` β in the nav, and at the route gate? The ADMISSION question.
|
| 92 |
+
|
| 93 |
+
ββ WHY THIS EXISTS: THE EDITOR'S DECISION REACHED THE READ DOOR AND NOTHING ELSE. Measured
|
| 94 |
+
on live 2026-08-18, on a real account. An administrator ticked *Odoo products* for Naomi in
|
| 95 |
+
Manage user and saved; the record stored `perms.product_data.access = True` and
|
| 96 |
+
`may_access()` agreed. **The database never appeared.** `perms.nav_pages` and
|
| 97 |
+
`deps.Session.require` both ask `perms.may_open` β the LEGACY `user['modules']` array β which
|
| 98 |
+
still read `['sales', 'customers', 'products']`, and `products` is the key of the ARCHIVED
|
| 99 |
+
*SKU* module, not of `product_data`. So `nav_pages` answered `['customer_data']` and the
|
| 100 |
+
route would have 403'd her even by URL. Two permission systems, the editor writing one and
|
| 101 |
+
every DOOR reading the other ([[two-permission-systems-one-armed]]).
|
| 102 |
+
|
| 103 |
+
β THE ASYMMETRY IS THE WHOLE DESIGN, and it is not the same rule twice:
|
| 104 |
+
|
| 105 |
+
1. an admin sees everything (break-glass, as everywhere else);
|
| 106 |
+
2. an EXPLICIT `access: false` DENIES, on any key β an administrator unticking a box must
|
| 107 |
+
take the row off the nav, and before this it did not;
|
| 108 |
+
3. a `ut_*` key with no explicit deny defers to `user_tables.may_open` β creator, admin or
|
| 109 |
+
a `core.shares` grant β and **an `access: true` entry may NEVER widen past it**. The
|
| 110 |
+
editor must not become a way to hand somebody another user's private table;
|
| 111 |
+
4. a REGISTRY TOPIC with an explicit entry takes that entry, grant included. Here the
|
| 112 |
+
editor IS the authority: `_clean_perms` writes an entry for every governed topic on
|
| 113 |
+
every save, so an entry means an administrator decided;
|
| 114 |
+
5. anything else β a module this editor does not govern β falls through to
|
| 115 |
+
`perms.may_open`, UNCHANGED.
|
| 116 |
+
|
| 117 |
+
β LEG 5 IS LOAD-BEARING AND IT IS WHY `may_access` COULD NOT SIMPLY BE CALLED HERE.
|
| 118 |
+
`may_access` reads migrated-and-undeclared as DENY, which is correct for a governed topic and
|
| 119 |
+
catastrophic for the nav: Naomi's block declares the ten governed keys and nothing else, so
|
| 120 |
+
`sales` would have gone from visible to denied β an outage dressed as a permission fix.
|
| 121 |
+
"""
|
| 122 |
+
if perms.is_admin(user):
|
| 123 |
+
return True
|
| 124 |
+
e = entry(user, key)
|
| 125 |
+
if e is not None and not bool(e.get('access', True)):
|
| 126 |
+
return False
|
| 127 |
+
if str(key or '').startswith(_ut().KEY_PREFIX):
|
| 128 |
+
return bool(_ut().may_open(key, (user or {}).get('username'), False, st=st))
|
| 129 |
+
if e is not None:
|
| 130 |
+
return bool(e.get('access', True))
|
| 131 |
+
return perms.may_open(user, key)
|
| 132 |
+
|
| 133 |
+
|
| 134 |
def assistant_entry(user, module):
|
| 135 |
"""The explicit database grant an Assistant snapshot may rely on, else ``None``.
|
| 136 |
|
platform/core/perms.py
CHANGED
|
@@ -189,7 +189,12 @@ def nav_pages(user):
|
|
| 189 |
# API no longer advertises it as a surface of the PRODUCT.
|
| 190 |
if m['key'] == 'settings':
|
| 191 |
continue
|
| 192 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 193 |
continue
|
| 194 |
row = {'key': m['key'], 'label': m['label'],
|
| 195 |
'chrome': 'main' if m.get('nav', True) else 'utility'}
|
|
@@ -314,6 +319,36 @@ def tenant_governable_modules(runtime, topics, ut_entries=()):
|
|
| 314 |
return out
|
| 315 |
|
| 316 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 317 |
#: β `enforced_module_keys` IS DELETED (W36-T22 / C2), NOT NEUTERED TO "every governable key".
|
| 318 |
#: It read `{m['key'] for m in modules if m.get('enforced')}`, so with the flag gone it would
|
| 319 |
#: answer the EMPTY SET and `_clean_perms` would refuse every module β and "fix" it by returning
|
|
|
|
| 189 |
# API no longer advertises it as a surface of the PRODUCT.
|
| 190 |
if m['key'] == 'settings':
|
| 191 |
continue
|
| 192 |
+
# β `nav_may_open`, NOT `may_open`: the permission editor's explicit decision is
|
| 193 |
+
# what an administrator just made, and until 2026-08-18 it reached the READ door
|
| 194 |
+
# and never this list β a database granted in Manage user, saved and confirmed,
|
| 195 |
+
# simply did not appear. Deferred import because `perm_scope` imports THIS module.
|
| 196 |
+
import core.perm_scope as _perm_scope
|
| 197 |
+
if not _perm_scope.nav_may_open(user, m['key']):
|
| 198 |
continue
|
| 199 |
row = {'key': m['key'], 'label': m['label'],
|
| 200 |
'chrome': 'main' if m.get('nav', True) else 'utility'}
|
|
|
|
| 319 |
return out
|
| 320 |
|
| 321 |
|
| 322 |
+
def governable_surface_modules():
|
| 323 |
+
"""The live APP SURFACES an account grant can toggle: `[{'key','label'}]`, registry order.
|
| 324 |
+
|
| 325 |
+
ββ OWNER RULING 2026-08-18, verbatim: *"I told you that everything needs to be
|
| 326 |
+
modular. And if there is a toggle for permissioning Assistant or Agent module view, it should
|
| 327 |
+
SHOW."* Until then the ONLY control over `analyst` and `automation` was the legacy
|
| 328 |
+
`user['modules']` array, which no screen has edited since the module editor was retired β
|
| 329 |
+
so three of tenant #0's four members lacked Assistant and Agents with no way to grant them
|
| 330 |
+
short of editing the store by hand.
|
| 331 |
+
|
| 332 |
+
β A SURFACE IS NOT A DATABASE AND THE EDITOR MUST KNOW THE DIFFERENCE. These rows have no
|
| 333 |
+
field schema, so the only rule that can exist for them is ACCESS β no row filter, no
|
| 334 |
+
hidden fields. `routes_admin._module_fields` answers `[]` for them (the editor's schemaless
|
| 335 |
+
rendering: a toggle and nothing else), and `_clean_perms`' field validation refuses any
|
| 336 |
+
filter or hiddenFields against an empty vocabulary, so a client cannot store a rule nothing
|
| 337 |
+
would apply.
|
| 338 |
+
|
| 339 |
+
β DERIVED, NEVER LISTED: every non-archived, non-`group_only` registry row with NO
|
| 340 |
+
`topic` that `nav_pages` would offer (not `validate_only`, not `api_surface: False`, not
|
| 341 |
+
`settings`). Today that is `analyst` (Assistant) and `automation` (Agents); a surface added
|
| 342 |
+
to the registry tomorrow is governable with no edit here β the same rule
|
| 343 |
+
`tenant_governable_modules` states for databases, applied to the other kind of row.
|
| 344 |
+
"""
|
| 345 |
+
return [{'key': m['key'], 'label': m['label']}
|
| 346 |
+
for m in registry.REGISTRY
|
| 347 |
+
if not m.get('archived') and not m.get('group_only') and not m.get('topic')
|
| 348 |
+
and not m.get('validate_only') and m.get('api_surface') is not False
|
| 349 |
+
and m['key'] != 'settings']
|
| 350 |
+
|
| 351 |
+
|
| 352 |
#: β `enforced_module_keys` IS DELETED (W36-T22 / C2), NOT NEUTERED TO "every governable key".
|
| 353 |
#: It read `{m['key'] for m in modules if m.get('enforced')}`, so with the flag gone it would
|
| 354 |
#: answer the EMPTY SET and `_clean_perms` would refuse every module β and "fix" it by returning
|
platform/core/registry.py
CHANGED
|
@@ -221,7 +221,7 @@ REGISTRY = [
|
|
| 221 |
'nav': False, # sidebar sentinel (account group) β owner IA 2026-07-12
|
| 222 |
'note': 'User scope settings: the Business Unit toggle (strict isolation) and the Data basis '
|
| 223 |
'toggle (Orders vs Invoiced) moved here from the sidebar.'},
|
| 224 |
-
{'key': 'analyst', 'label': '
|
| 225 |
'nav': False, # sidebar sentinel button; eval gate = harness/evals.py (run pre-ship, NOT in validate.py β live LLM cost)
|
| 226 |
'note': 'Ask the business a question in plain language: a small AI model calls governed tools over the semantic layer. Answers carry their query trace and drill links. AI-generated output (Art. 50 labeled).'},
|
| 227 |
]
|
|
|
|
| 221 |
'nav': False, # sidebar sentinel (account group) β owner IA 2026-07-12
|
| 222 |
'note': 'User scope settings: the Business Unit toggle (strict isolation) and the Data basis '
|
| 223 |
'toggle (Orders vs Invoiced) moved here from the sidebar.'},
|
| 224 |
+
{'key': 'analyst', 'label': 'Assistant', 'brand': False, 'hq': True, 'validate': False,
|
| 225 |
'nav': False, # sidebar sentinel button; eval gate = harness/evals.py (run pre-ship, NOT in validate.py β live LLM cost)
|
| 226 |
'note': 'Ask the business a question in plain language: a small AI model calls governed tools over the semantic layer. Answers carry their query trace and drill links. AI-generated output (Art. 50 labeled).'},
|
| 227 |
]
|
platform/core/script_sandbox.py
CHANGED
|
@@ -1,519 +1,519 @@
|
|
| 1 |
-
"""core/script_sandbox.py β WAVE 36 (R5 / R10, contract C1): running a tenant's OWN Python.
|
| 2 |
-
|
| 3 |
-
Owner item 6: *"Add code script as an interface (database View) so a user can build whatever they
|
| 4 |
-
want through the Agent chat interface."* Item 8: *"We need to really guardrail the reach of this
|
| 5 |
-
script. So let's really grill this down."* R10 ruled it SERVER-SIDE PYTHON after the trade was
|
| 6 |
-
stated, so this file is the guardrail, and one engine serves both items.
|
| 7 |
-
|
| 8 |
-
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 9 |
-
ββ THE ONE PARAGRAPH TO READ BEFORE CHANGING ANYTHING HERE.
|
| 10 |
-
|
| 11 |
-
In-process CPython cannot deliver two of this ticket's clauses. An AST allow-list plus a curated
|
| 12 |
-
namespace stops import, file, network and environment access β but it **cannot cap memory and
|
| 13 |
-
cannot interrupt a runaway loop**, because a `while True:` in the same interpreter is not a slow
|
| 14 |
-
request, it is the tenant's ONE FastAPI process gone. So the script runs in a **SUBPROCESS**:
|
| 15 |
-
`resource.setrlimit` for address space and CPU, a hard wall-clock kill from the parent, and the
|
| 16 |
-
allow-list inside. Neither half is sufficient; both are load-bearing.
|
| 17 |
-
|
| 18 |
-
β AND THE SUBPROCESS RECEIVES **ROWS, NEVER A STORE**. The parent calls C1's `scoped_table` under
|
| 19 |
-
the CALLING user's record and serialises the result; the child imports nothing from this repo and
|
| 20 |
-
holds no credential, no runtime and no store handle. Wiring W1 ("the sandbox has no second store
|
| 21 |
-
path") is then true by CONSTRUCTION rather than by discipline, and it is checkable: the child
|
| 22 |
-
reports its own `sys.modules`, and no `core.*` name may appear in it.
|
| 23 |
-
|
| 24 |
-
β NEVER A BLACKLIST. Every rule below is an ALLOW-LIST β a set of node types, a set of attribute
|
| 25 |
-
names, a dict of builtins. A blacklist of dangerous spellings is bypassable by construction, and
|
| 26 |
-
the bypass is usually one string method away (`"{0.__class__}".format(x)` performs its attribute
|
| 27 |
-
lookup inside `format`, so there is no `ast.Attribute` node to refuse).
|
| 28 |
-
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 29 |
-
|
| 30 |
-
The two layers, and they refuse DIFFERENT things on purpose:
|
| 31 |
-
|
| 32 |
-
1. `check_source()` β a pure function over source text. Refuses a construct the language offers
|
| 33 |
-
and this sandbox does not: `import`, `class`, `with`, `async`, `yield`, `global`, and every
|
| 34 |
-
attribute name outside `ALLOWED_ATTRS`.
|
| 35 |
-
2. `SANDBOX_BUILTINS` β the names that resolve at all. `__import__`, `open`, `eval`, `exec`,
|
| 36 |
-
`compile`, `getattr`, `globals`, `vars` and `type` are simply absent, so a source that gets
|
| 37 |
-
past layer 1 still finds nothing to call.
|
| 38 |
-
|
| 39 |
-
β THAT DUPLICATION IS DELIBERATE AND IT CHANGES HOW THE GATE MUST BE WRITTEN. `import os` is
|
| 40 |
-
refused twice, so a negative control that drops ONE layer sees the other refuse and reports
|
| 41 |
-
green β the shape that already cost this wave one missed control in `routes_agent_harness`. So
|
| 42 |
-
each layer is tested AT ITS OWN BOUNDARY: `check_source()` is called directly on source strings,
|
| 43 |
-
and `run()` is driven end to end. An NC drops one entry from one frozenset and the matching
|
| 44 |
-
boundary goes red.
|
| 45 |
-
"""
|
| 46 |
-
import ast
|
| 47 |
-
import json
|
| 48 |
-
import os
|
| 49 |
-
import subprocess
|
| 50 |
-
import sys
|
| 51 |
-
import tempfile
|
| 52 |
-
import time
|
| 53 |
-
from pathlib import Path
|
| 54 |
-
|
| 55 |
-
#: Wall clock, enforced by the PARENT with a kill. The one cap that works on every platform.
|
| 56 |
-
DEFAULT_TIMEOUT_S = 10.0
|
| 57 |
-
|
| 58 |
-
#: Address space for the child (`RLIMIT_AS`). POSIX only β see `run()`'s `caps` report.
|
| 59 |
-
DEFAULT_MEMORY_BYTES = 512 * 1024 * 1024
|
| 60 |
-
|
| 61 |
-
#: CPU seconds for the child (`RLIMIT_CPU`). POSIX only. Deliberately above the wall clock: the
|
| 62 |
-
#: wall-clock kill is the primary control and this is the backstop for a child that stops being
|
| 63 |
-
#: reachable. A CPU limit BELOW the timeout would make every slow script look like a CPU refusal.
|
| 64 |
-
DEFAULT_CPU_SECONDS = 15
|
| 65 |
-
|
| 66 |
-
#: What the script may print, in bytes. `print` is a curated builtin writing to a capped buffer,
|
| 67 |
-
#: and the child's real stdout goes to DEVNULL β so a script cannot fill a pipe, and anything
|
| 68 |
-
#: that escaped far enough to write to fd 1 has nowhere for it to land.
|
| 69 |
-
MAX_STDOUT_BYTES = 64 * 1024
|
| 70 |
-
|
| 71 |
-
#: The serialised ROW payload handed to the child. β A REFUSAL, NEVER A TRUNCATION (standing rule
|
| 72 |
-
#: 1): a short answer from a data tool is a wrong answer that looks right. Over this, `run()`
|
| 73 |
-
#: returns a named limit carrying its cause and a recommendation.
|
| 74 |
-
MAX_PAYLOAD_BYTES = 32 * 1024 * 1024
|
| 75 |
-
|
| 76 |
-
#: The emitted spec. A render spec is a description of a picture; one larger than this is data
|
| 77 |
-
#: pretending to be a description.
|
| 78 |
-
MAX_SPEC_BYTES = 2 * 1024 * 1024
|
| 79 |
-
|
| 80 |
-
MAX_SOURCE_BYTES = 128 * 1024
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
# ββββββ
|
| 84 |
-
#: Every `ast` node class a script may contain. β THE ABSENCES ARE THE POLICY: `Import` /
|
| 85 |
-
#: `ImportFrom` (no module reaches the script), `ClassDef` (a class body is a namespace with its
|
| 86 |
-
#: own scoping rules and buys a data script nothing), `With` (a context manager is `__enter__`
|
| 87 |
-
#: by another spelling), `Global` / `Nonlocal` (rebinding the sandbox's own names), and every
|
| 88 |
-
#: `Async*` / `Await` / `Yield` form (this engine is synchronous; a coroutine that is never
|
| 89 |
-
#: awaited is a silent no-op that looks like a working script).
|
| 90 |
-
ALLOWED_NODES = frozenset("""
|
| 91 |
-
Module Expr Assign AugAssign AnnAssign NamedExpr Return Pass Break Continue Delete Assert Raise
|
| 92 |
-
If For While Try TryStar ExceptHandler FunctionDef Lambda arguments arg keyword
|
| 93 |
-
BoolOp BinOp UnaryOp IfExp Dict Set List Tuple Starred Subscript Slice Compare Call Attribute Name
|
| 94 |
-
Constant JoinedStr FormattedValue ListComp SetComp DictComp GeneratorExp comprehension
|
| 95 |
-
Load Store Del
|
| 96 |
-
And Or Not Invert UAdd USub
|
| 97 |
-
Add Sub Mult Div FloorDiv Mod Pow LShift RShift BitOr BitXor BitAnd MatMult
|
| 98 |
-
Eq NotEq Lt LtE Gt GtE Is IsNot In NotIn
|
| 99 |
-
""".split())
|
| 100 |
-
|
| 101 |
-
#: Every attribute name a script may READ or CALL. ββ THIS IS THE LOAD-BEARING SET, and it is
|
| 102 |
-
#: an allow-list of NAMES rather than a refusal of dunders, because the interesting escapes are
|
| 103 |
-
#: ordinary-looking: `f.__globals__` on any function reaches the runner's own module namespace,
|
| 104 |
-
#: `e.__traceback__.tb_frame.f_globals` reaches it from an exception handler, and `().__class__`
|
| 105 |
-
#: reaches `object.__subclasses__`. None of those names is here, and neither is any name this
|
| 106 |
-
#: sandbox has not been asked for.
|
| 107 |
-
#: β `format` IS ABSENT DELIBERATELY. `"{0.__class__}".format(x)` performs the attribute lookup
|
| 108 |
-
#: INSIDE `str.format`, where no `ast.Attribute` node exists for layer 1 to see. f-strings are
|
| 109 |
-
#: fine β `f"{x.__class__}"` compiles to a real `Attribute` node and is refused.
|
| 110 |
-
ALLOWED_ATTRS = frozenset("""
|
| 111 |
-
append extend insert pop remove clear sort reverse copy count index
|
| 112 |
-
keys values items get setdefault update
|
| 113 |
-
add discard union intersection difference issubset issuperset
|
| 114 |
-
join split rsplit splitlines strip lstrip rstrip lower upper title capitalize casefold
|
| 115 |
-
replace startswith endswith find rfind zfill ljust rjust center partition removeprefix removesuffix
|
| 116 |
-
isdigit isalpha isalnum isspace isupper islower isnumeric
|
| 117 |
-
real imag numerator denominator
|
| 118 |
-
""".split())
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
class Refused(Exception):
|
| 122 |
-
"""A named refusal: `code` for a caller to branch on, `message` for a person to read."""
|
| 123 |
-
|
| 124 |
-
def __init__(self, code, message):
|
| 125 |
-
self.code, self.message = code, message
|
| 126 |
-
super().__init__(f"{code}: {message}")
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
def _attr_ok(name):
|
| 130 |
-
"""An attribute name passes only if it is on the list AND is not private.
|
| 131 |
-
|
| 132 |
-
β THE SECOND TEST IS NOT A BLACKLIST β it narrows an allow-list that already excludes every
|
| 133 |
-
private name. It is here so that adding a name to `ALLOWED_ATTRS` cannot open a dunder by
|
| 134 |
-
accident, which is the one edit a future reader is most likely to make in a hurry.
|
| 135 |
-
"""
|
| 136 |
-
return name in ALLOWED_ATTRS and not name.startswith("_")
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
def check_source(source):
|
| 140 |
-
"""LAYER 1. Return a `Refused` for source this sandbox will not run, or `None`.
|
| 141 |
-
|
| 142 |
-
β PURE, AND THAT IS WHAT MAKES IT TESTABLE AT ITS OWN BOUNDARY. It reads no file, spawns no
|
| 143 |
-
process and touches no store, so a gate can hand it a hundred hostile strings for free and an
|
| 144 |
-
NC can drop one entry from one frozenset and watch exactly this function change its answer.
|
| 145 |
-
"""
|
| 146 |
-
text = str(source or "")
|
| 147 |
-
if len(text.encode("utf-8", "replace")) > MAX_SOURCE_BYTES:
|
| 148 |
-
return Refused("source_too_long",
|
| 149 |
-
f"a script view is at most {MAX_SOURCE_BYTES // 1024} KB of source")
|
| 150 |
-
try:
|
| 151 |
-
tree = ast.parse(text)
|
| 152 |
-
except SyntaxError as exc:
|
| 153 |
-
return Refused("syntax", f"line {exc.lineno or 0}: {exc.msg}")
|
| 154 |
-
|
| 155 |
-
for node in ast.walk(tree):
|
| 156 |
-
kind = type(node).__name__
|
| 157 |
-
if kind not in ALLOWED_NODES:
|
| 158 |
-
return Refused("refused_construct",
|
| 159 |
-
f"line {getattr(node, 'lineno', 0)}: this sandbox does not run "
|
| 160 |
-
f"{_english(kind)}")
|
| 161 |
-
if isinstance(node, ast.Attribute) and not _attr_ok(node.attr):
|
| 162 |
-
return Refused("refused_attribute",
|
| 163 |
-
f"line {getattr(node, 'lineno', 0)}: the attribute "
|
| 164 |
-
f"'{node.attr}' is not available inside a script view")
|
| 165 |
-
# β A NAME may not be private either. `_` prefixed names are the runner's own, and a
|
| 166 |
-
# script that could bind one could shadow the machinery it runs on top of.
|
| 167 |
-
if isinstance(node, ast.Name) and node.id.startswith("_"):
|
| 168 |
-
return Refused("reserved_name",
|
| 169 |
-
f"line {getattr(node, 'lineno', 0)}: names starting with an "
|
| 170 |
-
f"underscore are reserved by the sandbox")
|
| 171 |
-
if isinstance(node, (ast.FunctionDef, ast.arg, ast.ExceptHandler)) and str(
|
| 172 |
-
getattr(node, "name", None) or getattr(node, "arg", "") or "").startswith("_"):
|
| 173 |
-
return Refused("reserved_name",
|
| 174 |
-
f"line {getattr(node, 'lineno', 0)}: names starting with an "
|
| 175 |
-
f"underscore are reserved by the sandbox")
|
| 176 |
-
if isinstance(node, ast.keyword) and str(node.arg or "").startswith("_"):
|
| 177 |
-
return Refused("reserved_name",
|
| 178 |
-
f"line {getattr(node, 'lineno', 0)}: keyword arguments starting with "
|
| 179 |
-
f"an underscore are reserved by the sandbox")
|
| 180 |
-
return None
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
_ENGLISH = {
|
| 184 |
-
"Import": "an import", "ImportFrom": "an import", "ClassDef": "a class definition",
|
| 185 |
-
"With": "a with block", "AsyncWith": "a with block", "AsyncFor": "an async loop",
|
| 186 |
-
"AsyncFunctionDef": "an async function", "Await": "await", "Yield": "yield",
|
| 187 |
-
"YieldFrom": "yield from", "Global": "a global statement", "Nonlocal": "a nonlocal statement",
|
| 188 |
-
"Match": "a match statement",
|
| 189 |
-
}
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
def _english(kind):
|
| 193 |
-
return _ENGLISH.get(kind, f"a {kind} expression")
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
# ββββββββββββββββββββββββββββββββββββββββββ LAYER 2 β the namespace, and the child program βββββ
|
| 197 |
-
#: The builtins a script may reach, BY NAME. Everything else is a `NameError` in the child.
|
| 198 |
-
#: β THE ABSENCES, again, are the policy: `__import__` `open` `eval` `exec` `compile` `input`
|
| 199 |
-
#: `getattr` `setattr` `delattr` `globals` `locals` `vars` `dir` `type` `super` `object` `help`
|
| 200 |
-
#: `exit` `breakpoint` `memoryview` `id`. Several are harmless on their own; each one is a step
|
| 201 |
-
#: on a published escape, and none has ever been asked for by a script that shapes rows.
|
| 202 |
-
#: β THE EXCEPTION CLASSES ARE HERE BECAUSE `try:` IS, and a `try` block whose `except` clause
|
| 203 |
-
#: cannot name what it catches is a construct that reads as supported and is not. They are safe
|
| 204 |
-
#: for the same reason everything else is: `Exception.__subclasses__` needs an attribute this
|
| 205 |
-
#: sandbox does not allow, so a class object in the namespace is a leaf, not a doorway.
|
| 206 |
-
SANDBOX_BUILTIN_NAMES = (
|
| 207 |
-
"abs all any bool bytes callable chr dict divmod enumerate filter float frozenset hash hex "
|
| 208 |
-
"int isinstance issubclass iter len list map max min next oct ord pow range repr reversed "
|
| 209 |
-
"round set slice sorted str sum tuple zip True False None "
|
| 210 |
-
"Exception ValueError TypeError KeyError IndexError ZeroDivisionError ArithmeticError "
|
| 211 |
-
"AttributeError StopIteration OverflowError"
|
| 212 |
-
).split()
|
| 213 |
-
|
| 214 |
-
#: The literal program the child runs. It is TEXT rather than a module because the child must
|
| 215 |
-
#: import nothing from this repo: a module would be found on `sys.path` and would drag `core`
|
| 216 |
-
#: with it, which is exactly the second store path W1 forbids.
|
| 217 |
-
#: β Every name in here is underscore-prefixed and layer 1 refuses a script from binding one, so
|
| 218 |
-
#: the runner's own machinery cannot be shadowed by the source it executes.
|
| 219 |
-
_RUNNER = r'''
|
| 220 |
-
import json as _json, os as _os, sys as _sys
|
| 221 |
-
|
| 222 |
-
_pay = _json.loads(open(_sys.argv[1], "r", encoding="utf-8").read())
|
| 223 |
-
_out = {"ok": False, "code": "not_run", "message": "the script did not run",
|
| 224 |
-
"stdout": "", "spec": None, "caps": {"wallClock": True, "memory": False, "cpu": False}}
|
| 225 |
-
|
| 226 |
-
# ββ the caps this platform can actually apply, reported either way (standing rule 1) ββββββββββ
|
| 227 |
-
try:
|
| 228 |
-
import resource as _res
|
| 229 |
-
_mem = int(_pay["memoryBytes"])
|
| 230 |
-
_res.setrlimit(_res.RLIMIT_AS, (_mem, _mem))
|
| 231 |
-
_out["caps"]["memory"] = True
|
| 232 |
-
_cpu = int(_pay["cpuSeconds"])
|
| 233 |
-
_res.setrlimit(_res.RLIMIT_CPU, (_cpu, _cpu))
|
| 234 |
-
_out["caps"]["cpu"] = True
|
| 235 |
-
except Exception:
|
| 236 |
-
# `resource` is POSIX only. The wall-clock kill in the parent still applies, and `caps` says
|
| 237 |
-
# which of the three held, never a silent partial.
|
| 238 |
-
pass
|
| 239 |
-
|
| 240 |
-
_printed = []
|
| 241 |
-
_spent = [0]
|
| 242 |
-
_LIMIT = int(_pay["maxStdout"])
|
| 243 |
-
|
| 244 |
-
|
| 245 |
-
def _print(*_a, **_k):
|
| 246 |
-
_text = (_k.get("sep") or " ").join(str(_x) for _x in _a) + (_k.get("end") or "\n")
|
| 247 |
-
_room = _LIMIT - _spent[0]
|
| 248 |
-
if _room > 0:
|
| 249 |
-
_printed.append(_text[:_room])
|
| 250 |
-
_spent[0] += len(_text)
|
| 251 |
-
|
| 252 |
-
|
| 253 |
-
class _Refusal(Exception):
|
| 254 |
-
"""The SANDBOX refusing, as distinct from the SCRIPT failing.
|
| 255 |
-
|
| 256 |
-
Without its own class these arrive as `ValueError`, indistinguishable from a `ValueError` the
|
| 257 |
-
script raised itself, and the answer then says "refused" about an ordinary bug in the tenant's
|
| 258 |
-
own code. Two different facts, two different codes.
|
| 259 |
-
"""
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
_emitted = []
|
| 263 |
-
|
| 264 |
-
|
| 265 |
-
def _emit(_spec):
|
| 266 |
-
if not isinstance(_spec, dict):
|
| 267 |
-
raise _Refusal("emit() takes a view spec, which is a dictionary")
|
| 268 |
-
if _emitted:
|
| 269 |
-
raise _Refusal("emit() was already called; a script view emits exactly one view")
|
| 270 |
-
_emitted.append(_spec)
|
| 271 |
-
|
| 272 |
-
|
| 273 |
-
_rows = _pay["rows"]
|
| 274 |
-
_fields = _pay["fields"]
|
| 275 |
-
_bound = _pay["table"]
|
| 276 |
-
|
| 277 |
-
|
| 278 |
-
def _scoped_table(_table=None):
|
| 279 |
-
if _table is not None and str(_table) != _bound:
|
| 280 |
-
raise _Refusal(
|
| 281 |
-
"this script view is bound to the database '" + _bound + "' and asked for '"
|
| 282 |
-
+ str(_table) + "'. A script view reads its own database only")
|
| 283 |
-
return [dict(_r) for _r in _rows]
|
| 284 |
-
|
| 285 |
-
|
| 286 |
-
def _scoped_fields():
|
| 287 |
-
return [dict(_f) for _f in _fields]
|
| 288 |
-
|
| 289 |
-
|
| 290 |
-
_ns = {"__builtins__": {_n: __builtins__[_n] if isinstance(__builtins__, dict)
|
| 291 |
-
else getattr(__builtins__, _n)
|
| 292 |
-
for _n in _pay["builtins"]}}
|
| 293 |
-
_ns["__builtins__"]["print"] = _print
|
| 294 |
-
_ns["print"] = _print
|
| 295 |
-
_ns["emit"] = _emit
|
| 296 |
-
_ns["scoped_table"] = _scoped_table
|
| 297 |
-
_ns["scoped_fields"] = _scoped_fields
|
| 298 |
-
_ns["table"] = _bound
|
| 299 |
-
|
| 300 |
-
try:
|
| 301 |
-
exec(compile(_pay["source"], "<script view>", "exec"), _ns)
|
| 302 |
-
if not _emitted:
|
| 303 |
-
_out.update(ok=False, code="no_view",
|
| 304 |
-
message="the script finished without calling emit(spec)")
|
| 305 |
-
else:
|
| 306 |
-
_out.update(ok=True, code="", message="", spec=_emitted[0])
|
| 307 |
-
except _Refusal as _e:
|
| 308 |
-
_out.update(ok=False, code="refused", message=str(_e)[:400])
|
| 309 |
-
except MemoryError:
|
| 310 |
-
_out.update(ok=False, code="memory",
|
| 311 |
-
message="the script used more memory than a script view is allowed")
|
| 312 |
-
except NameError as _e:
|
| 313 |
-
_out.update(ok=False, code="refused_name",
|
| 314 |
-
message=str(_e)[:200] + ". A script view may use only the names the sandbox "
|
| 315 |
-
"provides")
|
| 316 |
-
except BaseException as _e:
|
| 317 |
-
_out.update(ok=False, code="error",
|
| 318 |
-
message=type(_e).__name__ + ": " + str(_e)[:400])
|
| 319 |
-
|
| 320 |
-
_out["stdout"] = "".join(_printed)
|
| 321 |
-
_out["truncated"] = _spent[0] > _LIMIT
|
| 322 |
-
# β THE PROBE: what this child actually had. The PARENT strips it unless it was asked for, so a
|
| 323 |
-
# production run never carries it and a gate can still prove that no `core.*` module and no
|
| 324 |
-
# secret-shaped environment key was ever inside this process.
|
| 325 |
-
# β SNAPSHOTTED AFTER `exec`, and `os` is imported at the TOP so this list does not depend on
|
| 326 |
-
# dict-literal evaluation order. The first draft called `__import__("os")` inside this very
|
| 327 |
-
# expression, so whether `os` appeared depended on which value Python built first: a probe whose
|
| 328 |
-
# contents move with an unrelated edit is a probe a gate cannot assert against.
|
| 329 |
-
_out["probe"] = {"modules": sorted(_sys.modules), "env": sorted(_os.environ)}
|
| 330 |
-
open(_sys.argv[2], "w", encoding="utf-8").write(_json.dumps(_out, default=str))
|
| 331 |
-
'''
|
| 332 |
-
|
| 333 |
-
|
| 334 |
-
def _child_env():
|
| 335 |
-
"""The child's WHOLE environment. An allow-list of two keys, and neither is a credential.
|
| 336 |
-
|
| 337 |
-
β NOT `os.environ.copy()` MINUS SOMETHING. A subtractive environment ships every key nobody
|
| 338 |
-
thought to name: `HF_TOKEN`, `ODOO_PASSWORD`, `ANTHROPIC_API_KEY` and whatever the next
|
| 339 |
-
connector adds. The three names below are here because Python will not start on Windows
|
| 340 |
-
without them; on Linux this returns `{}` and the child runs with no environment at all.
|
| 341 |
-
"""
|
| 342 |
-
env = {}
|
| 343 |
-
for name in ("SystemRoot", "SYSTEMROOT", "WINDIR"):
|
| 344 |
-
if os.environ.get(name):
|
| 345 |
-
env[name] = os.environ[name]
|
| 346 |
-
return env
|
| 347 |
-
|
| 348 |
-
|
| 349 |
-
def run(source, rows, fields, table_key, *, timeout_s=DEFAULT_TIMEOUT_S,
|
| 350 |
-
memory_bytes=DEFAULT_MEMORY_BYTES, cpu_seconds=DEFAULT_CPU_SECONDS, probe=False):
|
| 351 |
-
"""Run ONE script over rows that are ALREADY scoped. Returns C3's envelope plus `caps`.
|
| 352 |
-
|
| 353 |
-
{ok, code, message, spec, stdout, truncated, ms, caps: {wallClock, memory, cpu}}
|
| 354 |
-
|
| 355 |
-
β THIS FUNCTION NEVER TOUCHES A STORE, AND THAT IS THE POINT: it takes rows. `run_view()`
|
| 356 |
-
below is the door that fetches them through C1; keeping the two apart is what lets a gate
|
| 357 |
-
drive the sandbox with no tenant, no runtime and no credential anywhere in the process.
|
| 358 |
-
|
| 359 |
-
β `caps` IS PART OF THE ANSWER, NOT DEBUG OUTPUT. On Windows `resource` does not exist, so
|
| 360 |
-
the memory and CPU limits are NOT applied and this says so. A caller that reports `ok:true`
|
| 361 |
-
without reading `caps` is claiming an enforcement that did not happen (standing rule 1).
|
| 362 |
-
"""
|
| 363 |
-
started = time.monotonic()
|
| 364 |
-
refusal = check_source(source)
|
| 365 |
-
if refusal is not None:
|
| 366 |
-
return _refusal(refusal.code, refusal.message, started)
|
| 367 |
-
|
| 368 |
-
payload = {"source": str(source or ""), "rows": rows, "fields": fields,
|
| 369 |
-
"table": str(table_key or ""), "builtins": SANDBOX_BUILTIN_NAMES,
|
| 370 |
-
"maxStdout": MAX_STDOUT_BYTES, "memoryBytes": int(memory_bytes),
|
| 371 |
-
"cpuSeconds": int(cpu_seconds)}
|
| 372 |
-
try:
|
| 373 |
-
blob = json.dumps(payload, default=str)
|
| 374 |
-
except (TypeError, ValueError) as exc:
|
| 375 |
-
return _refusal("bad_rows", f"these rows cannot be handed to a script ({exc})", started)
|
| 376 |
-
if len(blob.encode("utf-8", "replace")) > MAX_PAYLOAD_BYTES:
|
| 377 |
-
# β REPORTED, NOT TRUNCATED (standing rule 1's second sentence): cause and recommendation,
|
| 378 |
-
# in the words the owner asked for, rather than a quietly short answer.
|
| 379 |
-
return _refusal(
|
| 380 |
-
"payload_too_large",
|
| 381 |
-
f"this database's rows are larger than the {MAX_PAYLOAD_BYTES // (1024 * 1024)} MB a "
|
| 382 |
-
f"script view can be handed at once. Narrow the view with a filter, or raise the "
|
| 383 |
-
f"sandbox payload limit for this deployment", started)
|
| 384 |
-
|
| 385 |
-
with tempfile.TemporaryDirectory(prefix="aios-script-") as work:
|
| 386 |
-
pay_path = Path(work) / "payload.json"
|
| 387 |
-
res_path = Path(work) / "result.json"
|
| 388 |
-
pay_path.write_text(blob, encoding="utf-8")
|
| 389 |
-
# `-I` isolates the interpreter (no PYTHON* env, no user site), `-S` skips site-packages,
|
| 390 |
-
# and the program arrives on STDIN so there is no file for anything to import it as.
|
| 391 |
-
# β `-X utf8` AND AN EXPLICIT `encoding` ARE NOT TIDINESS. Without them this pipe is
|
| 392 |
-
# encoded with the parent's locale codec, which on this Windows box is cp1252: the runner
|
| 393 |
-
# text below cannot be represented in it and `subprocess.run` died with
|
| 394 |
-
# `UnicodeEncodeError` before the child ever started. A sandbox whose behaviour depends on
|
| 395 |
-
# the operator's locale is a sandbox with two behaviours. `-I` implies `-E`, so
|
| 396 |
-
# `PYTHONUTF8` in the environment could not have carried this, it has to be a flag.
|
| 397 |
-
argv = [sys.executable, "-I", "-S", "-X", "utf8", "-", str(pay_path), str(res_path)]
|
| 398 |
-
try:
|
| 399 |
-
done = subprocess.run(
|
| 400 |
-
argv, input=_RUNNER, text=True, encoding="utf-8", errors="replace",
|
| 401 |
-
cwd=work, env=_child_env(),
|
| 402 |
-
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=timeout_s)
|
| 403 |
-
except subprocess.TimeoutExpired:
|
| 404 |
-
return _refusal("timeout",
|
| 405 |
-
f"the script ran longer than {timeout_s:g} seconds and was stopped",
|
| 406 |
-
started)
|
| 407 |
-
except OSError as exc:
|
| 408 |
-
return _refusal("no_sandbox",
|
| 409 |
-
f"a script view could not be started on this deployment ({exc})",
|
| 410 |
-
started)
|
| 411 |
-
|
| 412 |
-
if not res_path.is_file():
|
| 413 |
-
# The child died without writing an answer: an rlimit signal, an OOM kill, or a crash.
|
| 414 |
-
# β NAMED BY ITS RETURN CODE rather than reported as a generic failure β a memory kill
|
| 415 |
-
# and a bug in this file must not read identically to an operator.
|
| 416 |
-
return _refusal(*_died(done), started)
|
| 417 |
-
try:
|
| 418 |
-
out = json.loads(res_path.read_text(encoding="utf-8"))
|
| 419 |
-
except (OSError, ValueError) as exc:
|
| 420 |
-
return _refusal("unreadable", f"the script's answer could not be read ({exc})",
|
| 421 |
-
started)
|
| 422 |
-
|
| 423 |
-
out["ms"] = int((time.monotonic() - started) * 1000)
|
| 424 |
-
if out.get("ok"):
|
| 425 |
-
spec_error = _check_spec(out.get("spec"))
|
| 426 |
-
if spec_error:
|
| 427 |
-
out.update(ok=False, code="bad_spec", message=spec_error, spec=None)
|
| 428 |
-
if not probe:
|
| 429 |
-
out.pop("probe", None)
|
| 430 |
-
return out
|
| 431 |
-
|
| 432 |
-
|
| 433 |
-
def _died(done):
|
| 434 |
-
"""`(code, message)` for a child that produced no answer."""
|
| 435 |
-
rc = done.returncode
|
| 436 |
-
tail = " ".join((done.stderr or "").split())[-300:]
|
| 437 |
-
if rc in (-9, 137):
|
| 438 |
-
return "memory", "the script was stopped for using too much memory"
|
| 439 |
-
if rc in (-24, 152):
|
| 440 |
-
return "timeout", "the script used more processor time than a script view is allowed"
|
| 441 |
-
return "crashed", f"the script view engine stopped without an answer{': ' + tail if tail else ''}"
|
| 442 |
-
|
| 443 |
-
|
| 444 |
-
def _refusal(code, message, started):
|
| 445 |
-
return {"ok": False, "code": code, "message": message, "spec": None, "stdout": "",
|
| 446 |
-
"truncated": False, "ms": int((time.monotonic() - started) * 1000),
|
| 447 |
-
"caps": {"wallClock": True, "memory": False, "cpu": False}}
|
| 448 |
-
|
| 449 |
-
|
| 450 |
-
def _check_spec(spec):
|
| 451 |
-
"""C3: a spec is a DESCRIPTION the client draws. Never HTML, never a script, never a URL.
|
| 452 |
-
|
| 453 |
-
β THE CHECK IS ON THE KEYS, NOT ON THE STRING CONTENTS. Scanning values for `<script>` is a
|
| 454 |
-
blacklist and would pass `<SCR` + `IPT>`; refusing a spec that carries an `html`, `script`,
|
| 455 |
-
`src` or `onclick` key refuses the SHAPE that would let a renderer be talked into executing
|
| 456 |
-
something. The vocabulary of legal `kind`s is the ROUTE's business (W36-T37) β this is the
|
| 457 |
-
floor every caller gets whether or not the route above it remembers.
|
| 458 |
-
"""
|
| 459 |
-
if not isinstance(spec, dict):
|
| 460 |
-
return "the script emitted something that is not a view spec"
|
| 461 |
-
try:
|
| 462 |
-
blob = json.dumps(spec)
|
| 463 |
-
except (TypeError, ValueError):
|
| 464 |
-
return "the emitted view spec is not something the client can be sent"
|
| 465 |
-
if len(blob.encode("utf-8", "replace")) > MAX_SPEC_BYTES:
|
| 466 |
-
return (f"the emitted view spec is over {MAX_SPEC_BYTES // (1024 * 1024)} MB. A view spec "
|
| 467 |
-
f"describes a picture; it is not where the rows go")
|
| 468 |
-
banned = {"html", "innerhtml", "script", "src", "srcdoc", "href", "style", "onclick", "onload"}
|
| 469 |
-
found = sorted(k for k in _keys_of(spec) if str(k).lower() in banned)
|
| 470 |
-
if found:
|
| 471 |
-
return (f"a view spec may not carry {', '.join(found)}. The client DRAWS a spec, so a "
|
| 472 |
-
f"markup or URL key would be a script by another name")
|
| 473 |
-
return None
|
| 474 |
-
|
| 475 |
-
|
| 476 |
-
def _keys_of(value, depth=0):
|
| 477 |
-
"""Every key anywhere in a nested spec. Bounded, so a deep structure cannot spin this."""
|
| 478 |
-
if depth > 12:
|
| 479 |
-
return
|
| 480 |
-
if isinstance(value, dict):
|
| 481 |
-
for key, sub in value.items():
|
| 482 |
-
yield key
|
| 483 |
-
yield from _keys_of(sub, depth + 1)
|
| 484 |
-
elif isinstance(value, (list, tuple)):
|
| 485 |
-
for sub in value:
|
| 486 |
-
yield from _keys_of(sub, depth + 1)
|
| 487 |
-
|
| 488 |
-
|
| 489 |
-
# ββββββββββββββββββββββββββββββββββββββββββββββ THE DOOR β C1 is the ONLY way to a row βββββββββ
|
| 490 |
-
def run_view(user, table_key, source, st=None, **kw):
|
| 491 |
-
"""Fetch through C1 under `user`'s scope, then run the script over what came back (R5).
|
| 492 |
-
|
| 493 |
-
ββ THE FETCH HAPPENS IN THE PARENT AND ONLY ROWS CROSS INTO THE CHILD. That is wiring W1
|
| 494 |
-
made structural: the child has no runtime to ask, no store handle to open and no credential
|
| 495 |
-
to use, so "a script cannot read what its caller cannot read" is not a rule anybody has to
|
| 496 |
-
keep β there is no second path for it to be broken through.
|
| 497 |
-
|
| 498 |
-
β C1'S THREE EXCEPTIONS ARE ANSWERED, NEVER SWALLOWED. `UnknownTable`, `Denied` and
|
| 499 |
-
`Unresolvable` mean three different things to a person; collapsing them into "no rows" is the
|
| 500 |
-
silent-empty answer C1 was written to make impossible. `Unresolvable.as_limit()` is handed
|
| 501 |
-
through in the words it was raised with β standing rule 1's second sentence, verbatim.
|
| 502 |
-
"""
|
| 503 |
-
import core.perm_scope as perm_scope
|
| 504 |
-
|
| 505 |
-
key = str(table_key or "")
|
| 506 |
-
try:
|
| 507 |
-
rows = perm_scope.scoped_table(user, key, st=st)
|
| 508 |
-
fields = perm_scope.scoped_fields(user, key, st=st)
|
| 509 |
-
except perm_scope.UnknownTable as exc:
|
| 510 |
-
return _refusal("unknown_table", str(exc) or f"there is no database '{key}'",
|
| 511 |
-
time.monotonic())
|
| 512 |
-
except perm_scope.Denied as exc:
|
| 513 |
-
return _refusal("denied", str(exc) or "this account may not read that database",
|
| 514 |
-
time.monotonic())
|
| 515 |
-
except perm_scope.Unresolvable as exc:
|
| 516 |
-
out = _refusal("unresolvable", str(exc), time.monotonic())
|
| 517 |
-
out["limit"] = exc.as_limit()
|
| 518 |
-
return out
|
| 519 |
-
return run(source, rows, fields, key, **kw)
|
|
|
|
| 1 |
+
"""core/script_sandbox.py β WAVE 36 (R5 / R10, contract C1): running a tenant's OWN Python.
|
| 2 |
+
|
| 3 |
+
Owner item 6: *"Add code script as an interface (database View) so a user can build whatever they
|
| 4 |
+
want through the Agent chat interface."* Item 8: *"We need to really guardrail the reach of this
|
| 5 |
+
script. So let's really grill this down."* R10 ruled it SERVER-SIDE PYTHON after the trade was
|
| 6 |
+
stated, so this file is the guardrail, and one engine serves both items.
|
| 7 |
+
|
| 8 |
+
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 9 |
+
ββ THE ONE PARAGRAPH TO READ BEFORE CHANGING ANYTHING HERE.
|
| 10 |
+
|
| 11 |
+
In-process CPython cannot deliver two of this ticket's clauses. An AST allow-list plus a curated
|
| 12 |
+
namespace stops import, file, network and environment access β but it **cannot cap memory and
|
| 13 |
+
cannot interrupt a runaway loop**, because a `while True:` in the same interpreter is not a slow
|
| 14 |
+
request, it is the tenant's ONE FastAPI process gone. So the script runs in a **SUBPROCESS**:
|
| 15 |
+
`resource.setrlimit` for address space and CPU, a hard wall-clock kill from the parent, and the
|
| 16 |
+
allow-list inside. Neither half is sufficient; both are load-bearing.
|
| 17 |
+
|
| 18 |
+
β AND THE SUBPROCESS RECEIVES **ROWS, NEVER A STORE**. The parent calls C1's `scoped_table` under
|
| 19 |
+
the CALLING user's record and serialises the result; the child imports nothing from this repo and
|
| 20 |
+
holds no credential, no runtime and no store handle. Wiring W1 ("the sandbox has no second store
|
| 21 |
+
path") is then true by CONSTRUCTION rather than by discipline, and it is checkable: the child
|
| 22 |
+
reports its own `sys.modules`, and no `core.*` name may appear in it.
|
| 23 |
+
|
| 24 |
+
β NEVER A BLACKLIST. Every rule below is an ALLOW-LIST β a set of node types, a set of attribute
|
| 25 |
+
names, a dict of builtins. A blacklist of dangerous spellings is bypassable by construction, and
|
| 26 |
+
the bypass is usually one string method away (`"{0.__class__}".format(x)` performs its attribute
|
| 27 |
+
lookup inside `format`, so there is no `ast.Attribute` node to refuse).
|
| 28 |
+
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 29 |
+
|
| 30 |
+
The two layers, and they refuse DIFFERENT things on purpose:
|
| 31 |
+
|
| 32 |
+
1. `check_source()` β a pure function over source text. Refuses a construct the language offers
|
| 33 |
+
and this sandbox does not: `import`, `class`, `with`, `async`, `yield`, `global`, and every
|
| 34 |
+
attribute name outside `ALLOWED_ATTRS`.
|
| 35 |
+
2. `SANDBOX_BUILTINS` β the names that resolve at all. `__import__`, `open`, `eval`, `exec`,
|
| 36 |
+
`compile`, `getattr`, `globals`, `vars` and `type` are simply absent, so a source that gets
|
| 37 |
+
past layer 1 still finds nothing to call.
|
| 38 |
+
|
| 39 |
+
β THAT DUPLICATION IS DELIBERATE AND IT CHANGES HOW THE GATE MUST BE WRITTEN. `import os` is
|
| 40 |
+
refused twice, so a negative control that drops ONE layer sees the other refuse and reports
|
| 41 |
+
green β the shape that already cost this wave one missed control in `routes_agent_harness`. So
|
| 42 |
+
each layer is tested AT ITS OWN BOUNDARY: `check_source()` is called directly on source strings,
|
| 43 |
+
and `run()` is driven end to end. An NC drops one entry from one frozenset and the matching
|
| 44 |
+
boundary goes red.
|
| 45 |
+
"""
|
| 46 |
+
import ast
|
| 47 |
+
import json
|
| 48 |
+
import os
|
| 49 |
+
import subprocess
|
| 50 |
+
import sys
|
| 51 |
+
import tempfile
|
| 52 |
+
import time
|
| 53 |
+
from pathlib import Path
|
| 54 |
+
|
| 55 |
+
#: Wall clock, enforced by the PARENT with a kill. The one cap that works on every platform.
|
| 56 |
+
DEFAULT_TIMEOUT_S = 10.0
|
| 57 |
+
|
| 58 |
+
#: Address space for the child (`RLIMIT_AS`). POSIX only β see `run()`'s `caps` report.
|
| 59 |
+
DEFAULT_MEMORY_BYTES = 512 * 1024 * 1024
|
| 60 |
+
|
| 61 |
+
#: CPU seconds for the child (`RLIMIT_CPU`). POSIX only. Deliberately above the wall clock: the
|
| 62 |
+
#: wall-clock kill is the primary control and this is the backstop for a child that stops being
|
| 63 |
+
#: reachable. A CPU limit BELOW the timeout would make every slow script look like a CPU refusal.
|
| 64 |
+
DEFAULT_CPU_SECONDS = 15
|
| 65 |
+
|
| 66 |
+
#: What the script may print, in bytes. `print` is a curated builtin writing to a capped buffer,
|
| 67 |
+
#: and the child's real stdout goes to DEVNULL β so a script cannot fill a pipe, and anything
|
| 68 |
+
#: that escaped far enough to write to fd 1 has nowhere for it to land.
|
| 69 |
+
MAX_STDOUT_BYTES = 64 * 1024
|
| 70 |
+
|
| 71 |
+
#: The serialised ROW payload handed to the child. β A REFUSAL, NEVER A TRUNCATION (standing rule
|
| 72 |
+
#: 1): a short answer from a data tool is a wrong answer that looks right. Over this, `run()`
|
| 73 |
+
#: returns a named limit carrying its cause and a recommendation.
|
| 74 |
+
MAX_PAYLOAD_BYTES = 32 * 1024 * 1024
|
| 75 |
+
|
| 76 |
+
#: The emitted spec. A render spec is a description of a picture; one larger than this is data
|
| 77 |
+
#: pretending to be a description.
|
| 78 |
+
MAX_SPEC_BYTES = 2 * 1024 * 1024
|
| 79 |
+
|
| 80 |
+
MAX_SOURCE_BYTES = 128 * 1024
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
# ββββββββββββββββββββββββββββββββββββββββββββββββββββββ LAYER 1 β the AST allow-list βββββββββββ
|
| 84 |
+
#: Every `ast` node class a script may contain. β THE ABSENCES ARE THE POLICY: `Import` /
|
| 85 |
+
#: `ImportFrom` (no module reaches the script), `ClassDef` (a class body is a namespace with its
|
| 86 |
+
#: own scoping rules and buys a data script nothing), `With` (a context manager is `__enter__`
|
| 87 |
+
#: by another spelling), `Global` / `Nonlocal` (rebinding the sandbox's own names), and every
|
| 88 |
+
#: `Async*` / `Await` / `Yield` form (this engine is synchronous; a coroutine that is never
|
| 89 |
+
#: awaited is a silent no-op that looks like a working script).
|
| 90 |
+
ALLOWED_NODES = frozenset("""
|
| 91 |
+
Module Expr Assign AugAssign AnnAssign NamedExpr Return Pass Break Continue Delete Assert Raise
|
| 92 |
+
If For While Try TryStar ExceptHandler FunctionDef Lambda arguments arg keyword
|
| 93 |
+
BoolOp BinOp UnaryOp IfExp Dict Set List Tuple Starred Subscript Slice Compare Call Attribute Name
|
| 94 |
+
Constant JoinedStr FormattedValue ListComp SetComp DictComp GeneratorExp comprehension
|
| 95 |
+
Load Store Del
|
| 96 |
+
And Or Not Invert UAdd USub
|
| 97 |
+
Add Sub Mult Div FloorDiv Mod Pow LShift RShift BitOr BitXor BitAnd MatMult
|
| 98 |
+
Eq NotEq Lt LtE Gt GtE Is IsNot In NotIn
|
| 99 |
+
""".split())
|
| 100 |
+
|
| 101 |
+
#: Every attribute name a script may READ or CALL. ββ THIS IS THE LOAD-BEARING SET, and it is
|
| 102 |
+
#: an allow-list of NAMES rather than a refusal of dunders, because the interesting escapes are
|
| 103 |
+
#: ordinary-looking: `f.__globals__` on any function reaches the runner's own module namespace,
|
| 104 |
+
#: `e.__traceback__.tb_frame.f_globals` reaches it from an exception handler, and `().__class__`
|
| 105 |
+
#: reaches `object.__subclasses__`. None of those names is here, and neither is any name this
|
| 106 |
+
#: sandbox has not been asked for.
|
| 107 |
+
#: β `format` IS ABSENT DELIBERATELY. `"{0.__class__}".format(x)` performs the attribute lookup
|
| 108 |
+
#: INSIDE `str.format`, where no `ast.Attribute` node exists for layer 1 to see. f-strings are
|
| 109 |
+
#: fine β `f"{x.__class__}"` compiles to a real `Attribute` node and is refused.
|
| 110 |
+
ALLOWED_ATTRS = frozenset("""
|
| 111 |
+
append extend insert pop remove clear sort reverse copy count index
|
| 112 |
+
keys values items get setdefault update
|
| 113 |
+
add discard union intersection difference issubset issuperset
|
| 114 |
+
join split rsplit splitlines strip lstrip rstrip lower upper title capitalize casefold
|
| 115 |
+
replace startswith endswith find rfind zfill ljust rjust center partition removeprefix removesuffix
|
| 116 |
+
isdigit isalpha isalnum isspace isupper islower isnumeric
|
| 117 |
+
real imag numerator denominator
|
| 118 |
+
""".split())
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
class Refused(Exception):
|
| 122 |
+
"""A named refusal: `code` for a caller to branch on, `message` for a person to read."""
|
| 123 |
+
|
| 124 |
+
def __init__(self, code, message):
|
| 125 |
+
self.code, self.message = code, message
|
| 126 |
+
super().__init__(f"{code}: {message}")
|
| 127 |
+
|
| 128 |
+
|
| 129 |
+
def _attr_ok(name):
|
| 130 |
+
"""An attribute name passes only if it is on the list AND is not private.
|
| 131 |
+
|
| 132 |
+
β THE SECOND TEST IS NOT A BLACKLIST β it narrows an allow-list that already excludes every
|
| 133 |
+
private name. It is here so that adding a name to `ALLOWED_ATTRS` cannot open a dunder by
|
| 134 |
+
accident, which is the one edit a future reader is most likely to make in a hurry.
|
| 135 |
+
"""
|
| 136 |
+
return name in ALLOWED_ATTRS and not name.startswith("_")
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
def check_source(source):
|
| 140 |
+
"""LAYER 1. Return a `Refused` for source this sandbox will not run, or `None`.
|
| 141 |
+
|
| 142 |
+
β PURE, AND THAT IS WHAT MAKES IT TESTABLE AT ITS OWN BOUNDARY. It reads no file, spawns no
|
| 143 |
+
process and touches no store, so a gate can hand it a hundred hostile strings for free and an
|
| 144 |
+
NC can drop one entry from one frozenset and watch exactly this function change its answer.
|
| 145 |
+
"""
|
| 146 |
+
text = str(source or "")
|
| 147 |
+
if len(text.encode("utf-8", "replace")) > MAX_SOURCE_BYTES:
|
| 148 |
+
return Refused("source_too_long",
|
| 149 |
+
f"a script view is at most {MAX_SOURCE_BYTES // 1024} KB of source")
|
| 150 |
+
try:
|
| 151 |
+
tree = ast.parse(text)
|
| 152 |
+
except SyntaxError as exc:
|
| 153 |
+
return Refused("syntax", f"line {exc.lineno or 0}: {exc.msg}")
|
| 154 |
+
|
| 155 |
+
for node in ast.walk(tree):
|
| 156 |
+
kind = type(node).__name__
|
| 157 |
+
if kind not in ALLOWED_NODES:
|
| 158 |
+
return Refused("refused_construct",
|
| 159 |
+
f"line {getattr(node, 'lineno', 0)}: this sandbox does not run "
|
| 160 |
+
f"{_english(kind)}")
|
| 161 |
+
if isinstance(node, ast.Attribute) and not _attr_ok(node.attr):
|
| 162 |
+
return Refused("refused_attribute",
|
| 163 |
+
f"line {getattr(node, 'lineno', 0)}: the attribute "
|
| 164 |
+
f"'{node.attr}' is not available inside a script view")
|
| 165 |
+
# β A NAME may not be private either. `_` prefixed names are the runner's own, and a
|
| 166 |
+
# script that could bind one could shadow the machinery it runs on top of.
|
| 167 |
+
if isinstance(node, ast.Name) and node.id.startswith("_"):
|
| 168 |
+
return Refused("reserved_name",
|
| 169 |
+
f"line {getattr(node, 'lineno', 0)}: names starting with an "
|
| 170 |
+
f"underscore are reserved by the sandbox")
|
| 171 |
+
if isinstance(node, (ast.FunctionDef, ast.arg, ast.ExceptHandler)) and str(
|
| 172 |
+
getattr(node, "name", None) or getattr(node, "arg", "") or "").startswith("_"):
|
| 173 |
+
return Refused("reserved_name",
|
| 174 |
+
f"line {getattr(node, 'lineno', 0)}: names starting with an "
|
| 175 |
+
f"underscore are reserved by the sandbox")
|
| 176 |
+
if isinstance(node, ast.keyword) and str(node.arg or "").startswith("_"):
|
| 177 |
+
return Refused("reserved_name",
|
| 178 |
+
f"line {getattr(node, 'lineno', 0)}: keyword arguments starting with "
|
| 179 |
+
f"an underscore are reserved by the sandbox")
|
| 180 |
+
return None
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
_ENGLISH = {
|
| 184 |
+
"Import": "an import", "ImportFrom": "an import", "ClassDef": "a class definition",
|
| 185 |
+
"With": "a with block", "AsyncWith": "a with block", "AsyncFor": "an async loop",
|
| 186 |
+
"AsyncFunctionDef": "an async function", "Await": "await", "Yield": "yield",
|
| 187 |
+
"YieldFrom": "yield from", "Global": "a global statement", "Nonlocal": "a nonlocal statement",
|
| 188 |
+
"Match": "a match statement",
|
| 189 |
+
}
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
def _english(kind):
|
| 193 |
+
return _ENGLISH.get(kind, f"a {kind} expression")
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
# ββββββββββββββββββββββββββββββββββββββββββ LAYER 2 β the namespace, and the child program βββββ
|
| 197 |
+
#: The builtins a script may reach, BY NAME. Everything else is a `NameError` in the child.
|
| 198 |
+
#: β THE ABSENCES, again, are the policy: `__import__` `open` `eval` `exec` `compile` `input`
|
| 199 |
+
#: `getattr` `setattr` `delattr` `globals` `locals` `vars` `dir` `type` `super` `object` `help`
|
| 200 |
+
#: `exit` `breakpoint` `memoryview` `id`. Several are harmless on their own; each one is a step
|
| 201 |
+
#: on a published escape, and none has ever been asked for by a script that shapes rows.
|
| 202 |
+
#: β THE EXCEPTION CLASSES ARE HERE BECAUSE `try:` IS, and a `try` block whose `except` clause
|
| 203 |
+
#: cannot name what it catches is a construct that reads as supported and is not. They are safe
|
| 204 |
+
#: for the same reason everything else is: `Exception.__subclasses__` needs an attribute this
|
| 205 |
+
#: sandbox does not allow, so a class object in the namespace is a leaf, not a doorway.
|
| 206 |
+
SANDBOX_BUILTIN_NAMES = (
|
| 207 |
+
"abs all any bool bytes callable chr dict divmod enumerate filter float frozenset hash hex "
|
| 208 |
+
"int isinstance issubclass iter len list map max min next oct ord pow range repr reversed "
|
| 209 |
+
"round set slice sorted str sum tuple zip True False None "
|
| 210 |
+
"Exception ValueError TypeError KeyError IndexError ZeroDivisionError ArithmeticError "
|
| 211 |
+
"AttributeError StopIteration OverflowError"
|
| 212 |
+
).split()
|
| 213 |
+
|
| 214 |
+
#: The literal program the child runs. It is TEXT rather than a module because the child must
|
| 215 |
+
#: import nothing from this repo: a module would be found on `sys.path` and would drag `core`
|
| 216 |
+
#: with it, which is exactly the second store path W1 forbids.
|
| 217 |
+
#: β Every name in here is underscore-prefixed and layer 1 refuses a script from binding one, so
|
| 218 |
+
#: the runner's own machinery cannot be shadowed by the source it executes.
|
| 219 |
+
_RUNNER = r'''
|
| 220 |
+
import json as _json, os as _os, sys as _sys
|
| 221 |
+
|
| 222 |
+
_pay = _json.loads(open(_sys.argv[1], "r", encoding="utf-8").read())
|
| 223 |
+
_out = {"ok": False, "code": "not_run", "message": "the script did not run",
|
| 224 |
+
"stdout": "", "spec": None, "caps": {"wallClock": True, "memory": False, "cpu": False}}
|
| 225 |
+
|
| 226 |
+
# ββ the caps this platform can actually apply, reported either way (standing rule 1) ββββββββββ
|
| 227 |
+
try:
|
| 228 |
+
import resource as _res
|
| 229 |
+
_mem = int(_pay["memoryBytes"])
|
| 230 |
+
_res.setrlimit(_res.RLIMIT_AS, (_mem, _mem))
|
| 231 |
+
_out["caps"]["memory"] = True
|
| 232 |
+
_cpu = int(_pay["cpuSeconds"])
|
| 233 |
+
_res.setrlimit(_res.RLIMIT_CPU, (_cpu, _cpu))
|
| 234 |
+
_out["caps"]["cpu"] = True
|
| 235 |
+
except Exception:
|
| 236 |
+
# `resource` is POSIX only. The wall-clock kill in the parent still applies, and `caps` says
|
| 237 |
+
# which of the three held, never a silent partial.
|
| 238 |
+
pass
|
| 239 |
+
|
| 240 |
+
_printed = []
|
| 241 |
+
_spent = [0]
|
| 242 |
+
_LIMIT = int(_pay["maxStdout"])
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
def _print(*_a, **_k):
|
| 246 |
+
_text = (_k.get("sep") or " ").join(str(_x) for _x in _a) + (_k.get("end") or "\n")
|
| 247 |
+
_room = _LIMIT - _spent[0]
|
| 248 |
+
if _room > 0:
|
| 249 |
+
_printed.append(_text[:_room])
|
| 250 |
+
_spent[0] += len(_text)
|
| 251 |
+
|
| 252 |
+
|
| 253 |
+
class _Refusal(Exception):
|
| 254 |
+
"""The SANDBOX refusing, as distinct from the SCRIPT failing.
|
| 255 |
+
|
| 256 |
+
Without its own class these arrive as `ValueError`, indistinguishable from a `ValueError` the
|
| 257 |
+
script raised itself, and the answer then says "refused" about an ordinary bug in the tenant's
|
| 258 |
+
own code. Two different facts, two different codes.
|
| 259 |
+
"""
|
| 260 |
+
|
| 261 |
+
|
| 262 |
+
_emitted = []
|
| 263 |
+
|
| 264 |
+
|
| 265 |
+
def _emit(_spec):
|
| 266 |
+
if not isinstance(_spec, dict):
|
| 267 |
+
raise _Refusal("emit() takes a view spec, which is a dictionary")
|
| 268 |
+
if _emitted:
|
| 269 |
+
raise _Refusal("emit() was already called; a script view emits exactly one view")
|
| 270 |
+
_emitted.append(_spec)
|
| 271 |
+
|
| 272 |
+
|
| 273 |
+
_rows = _pay["rows"]
|
| 274 |
+
_fields = _pay["fields"]
|
| 275 |
+
_bound = _pay["table"]
|
| 276 |
+
|
| 277 |
+
|
| 278 |
+
def _scoped_table(_table=None):
|
| 279 |
+
if _table is not None and str(_table) != _bound:
|
| 280 |
+
raise _Refusal(
|
| 281 |
+
"this script view is bound to the database '" + _bound + "' and asked for '"
|
| 282 |
+
+ str(_table) + "'. A script view reads its own database only")
|
| 283 |
+
return [dict(_r) for _r in _rows]
|
| 284 |
+
|
| 285 |
+
|
| 286 |
+
def _scoped_fields():
|
| 287 |
+
return [dict(_f) for _f in _fields]
|
| 288 |
+
|
| 289 |
+
|
| 290 |
+
_ns = {"__builtins__": {_n: __builtins__[_n] if isinstance(__builtins__, dict)
|
| 291 |
+
else getattr(__builtins__, _n)
|
| 292 |
+
for _n in _pay["builtins"]}}
|
| 293 |
+
_ns["__builtins__"]["print"] = _print
|
| 294 |
+
_ns["print"] = _print
|
| 295 |
+
_ns["emit"] = _emit
|
| 296 |
+
_ns["scoped_table"] = _scoped_table
|
| 297 |
+
_ns["scoped_fields"] = _scoped_fields
|
| 298 |
+
_ns["table"] = _bound
|
| 299 |
+
|
| 300 |
+
try:
|
| 301 |
+
exec(compile(_pay["source"], "<script view>", "exec"), _ns)
|
| 302 |
+
if not _emitted:
|
| 303 |
+
_out.update(ok=False, code="no_view",
|
| 304 |
+
message="the script finished without calling emit(spec)")
|
| 305 |
+
else:
|
| 306 |
+
_out.update(ok=True, code="", message="", spec=_emitted[0])
|
| 307 |
+
except _Refusal as _e:
|
| 308 |
+
_out.update(ok=False, code="refused", message=str(_e)[:400])
|
| 309 |
+
except MemoryError:
|
| 310 |
+
_out.update(ok=False, code="memory",
|
| 311 |
+
message="the script used more memory than a script view is allowed")
|
| 312 |
+
except NameError as _e:
|
| 313 |
+
_out.update(ok=False, code="refused_name",
|
| 314 |
+
message=str(_e)[:200] + ". A script view may use only the names the sandbox "
|
| 315 |
+
"provides")
|
| 316 |
+
except BaseException as _e:
|
| 317 |
+
_out.update(ok=False, code="error",
|
| 318 |
+
message=type(_e).__name__ + ": " + str(_e)[:400])
|
| 319 |
+
|
| 320 |
+
_out["stdout"] = "".join(_printed)
|
| 321 |
+
_out["truncated"] = _spent[0] > _LIMIT
|
| 322 |
+
# β THE PROBE: what this child actually had. The PARENT strips it unless it was asked for, so a
|
| 323 |
+
# production run never carries it and a gate can still prove that no `core.*` module and no
|
| 324 |
+
# secret-shaped environment key was ever inside this process.
|
| 325 |
+
# β SNAPSHOTTED AFTER `exec`, and `os` is imported at the TOP so this list does not depend on
|
| 326 |
+
# dict-literal evaluation order. The first draft called `__import__("os")` inside this very
|
| 327 |
+
# expression, so whether `os` appeared depended on which value Python built first: a probe whose
|
| 328 |
+
# contents move with an unrelated edit is a probe a gate cannot assert against.
|
| 329 |
+
_out["probe"] = {"modules": sorted(_sys.modules), "env": sorted(_os.environ)}
|
| 330 |
+
open(_sys.argv[2], "w", encoding="utf-8").write(_json.dumps(_out, default=str))
|
| 331 |
+
'''
|
| 332 |
+
|
| 333 |
+
|
| 334 |
+
def _child_env():
|
| 335 |
+
"""The child's WHOLE environment. An allow-list of two keys, and neither is a credential.
|
| 336 |
+
|
| 337 |
+
β NOT `os.environ.copy()` MINUS SOMETHING. A subtractive environment ships every key nobody
|
| 338 |
+
thought to name: `HF_TOKEN`, `ODOO_PASSWORD`, `ANTHROPIC_API_KEY` and whatever the next
|
| 339 |
+
connector adds. The three names below are here because Python will not start on Windows
|
| 340 |
+
without them; on Linux this returns `{}` and the child runs with no environment at all.
|
| 341 |
+
"""
|
| 342 |
+
env = {}
|
| 343 |
+
for name in ("SystemRoot", "SYSTEMROOT", "WINDIR"):
|
| 344 |
+
if os.environ.get(name):
|
| 345 |
+
env[name] = os.environ[name]
|
| 346 |
+
return env
|
| 347 |
+
|
| 348 |
+
|
| 349 |
+
def run(source, rows, fields, table_key, *, timeout_s=DEFAULT_TIMEOUT_S,
|
| 350 |
+
memory_bytes=DEFAULT_MEMORY_BYTES, cpu_seconds=DEFAULT_CPU_SECONDS, probe=False):
|
| 351 |
+
"""Run ONE script over rows that are ALREADY scoped. Returns C3's envelope plus `caps`.
|
| 352 |
+
|
| 353 |
+
{ok, code, message, spec, stdout, truncated, ms, caps: {wallClock, memory, cpu}}
|
| 354 |
+
|
| 355 |
+
β THIS FUNCTION NEVER TOUCHES A STORE, AND THAT IS THE POINT: it takes rows. `run_view()`
|
| 356 |
+
below is the door that fetches them through C1; keeping the two apart is what lets a gate
|
| 357 |
+
drive the sandbox with no tenant, no runtime and no credential anywhere in the process.
|
| 358 |
+
|
| 359 |
+
β `caps` IS PART OF THE ANSWER, NOT DEBUG OUTPUT. On Windows `resource` does not exist, so
|
| 360 |
+
the memory and CPU limits are NOT applied and this says so. A caller that reports `ok:true`
|
| 361 |
+
without reading `caps` is claiming an enforcement that did not happen (standing rule 1).
|
| 362 |
+
"""
|
| 363 |
+
started = time.monotonic()
|
| 364 |
+
refusal = check_source(source)
|
| 365 |
+
if refusal is not None:
|
| 366 |
+
return _refusal(refusal.code, refusal.message, started)
|
| 367 |
+
|
| 368 |
+
payload = {"source": str(source or ""), "rows": rows, "fields": fields,
|
| 369 |
+
"table": str(table_key or ""), "builtins": SANDBOX_BUILTIN_NAMES,
|
| 370 |
+
"maxStdout": MAX_STDOUT_BYTES, "memoryBytes": int(memory_bytes),
|
| 371 |
+
"cpuSeconds": int(cpu_seconds)}
|
| 372 |
+
try:
|
| 373 |
+
blob = json.dumps(payload, default=str)
|
| 374 |
+
except (TypeError, ValueError) as exc:
|
| 375 |
+
return _refusal("bad_rows", f"these rows cannot be handed to a script ({exc})", started)
|
| 376 |
+
if len(blob.encode("utf-8", "replace")) > MAX_PAYLOAD_BYTES:
|
| 377 |
+
# β REPORTED, NOT TRUNCATED (standing rule 1's second sentence): cause and recommendation,
|
| 378 |
+
# in the words the owner asked for, rather than a quietly short answer.
|
| 379 |
+
return _refusal(
|
| 380 |
+
"payload_too_large",
|
| 381 |
+
f"this database's rows are larger than the {MAX_PAYLOAD_BYTES // (1024 * 1024)} MB a "
|
| 382 |
+
f"script view can be handed at once. Narrow the view with a filter, or raise the "
|
| 383 |
+
f"sandbox payload limit for this deployment", started)
|
| 384 |
+
|
| 385 |
+
with tempfile.TemporaryDirectory(prefix="aios-script-") as work:
|
| 386 |
+
pay_path = Path(work) / "payload.json"
|
| 387 |
+
res_path = Path(work) / "result.json"
|
| 388 |
+
pay_path.write_text(blob, encoding="utf-8")
|
| 389 |
+
# `-I` isolates the interpreter (no PYTHON* env, no user site), `-S` skips site-packages,
|
| 390 |
+
# and the program arrives on STDIN so there is no file for anything to import it as.
|
| 391 |
+
# β `-X utf8` AND AN EXPLICIT `encoding` ARE NOT TIDINESS. Without them this pipe is
|
| 392 |
+
# encoded with the parent's locale codec, which on this Windows box is cp1252: the runner
|
| 393 |
+
# text below cannot be represented in it and `subprocess.run` died with
|
| 394 |
+
# `UnicodeEncodeError` before the child ever started. A sandbox whose behaviour depends on
|
| 395 |
+
# the operator's locale is a sandbox with two behaviours. `-I` implies `-E`, so
|
| 396 |
+
# `PYTHONUTF8` in the environment could not have carried this, it has to be a flag.
|
| 397 |
+
argv = [sys.executable, "-I", "-S", "-X", "utf8", "-", str(pay_path), str(res_path)]
|
| 398 |
+
try:
|
| 399 |
+
done = subprocess.run(
|
| 400 |
+
argv, input=_RUNNER, text=True, encoding="utf-8", errors="replace",
|
| 401 |
+
cwd=work, env=_child_env(),
|
| 402 |
+
stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=timeout_s)
|
| 403 |
+
except subprocess.TimeoutExpired:
|
| 404 |
+
return _refusal("timeout",
|
| 405 |
+
f"the script ran longer than {timeout_s:g} seconds and was stopped",
|
| 406 |
+
started)
|
| 407 |
+
except OSError as exc:
|
| 408 |
+
return _refusal("no_sandbox",
|
| 409 |
+
f"a script view could not be started on this deployment ({exc})",
|
| 410 |
+
started)
|
| 411 |
+
|
| 412 |
+
if not res_path.is_file():
|
| 413 |
+
# The child died without writing an answer: an rlimit signal, an OOM kill, or a crash.
|
| 414 |
+
# β NAMED BY ITS RETURN CODE rather than reported as a generic failure β a memory kill
|
| 415 |
+
# and a bug in this file must not read identically to an operator.
|
| 416 |
+
return _refusal(*_died(done), started)
|
| 417 |
+
try:
|
| 418 |
+
out = json.loads(res_path.read_text(encoding="utf-8"))
|
| 419 |
+
except (OSError, ValueError) as exc:
|
| 420 |
+
return _refusal("unreadable", f"the script's answer could not be read ({exc})",
|
| 421 |
+
started)
|
| 422 |
+
|
| 423 |
+
out["ms"] = int((time.monotonic() - started) * 1000)
|
| 424 |
+
if out.get("ok"):
|
| 425 |
+
spec_error = _check_spec(out.get("spec"))
|
| 426 |
+
if spec_error:
|
| 427 |
+
out.update(ok=False, code="bad_spec", message=spec_error, spec=None)
|
| 428 |
+
if not probe:
|
| 429 |
+
out.pop("probe", None)
|
| 430 |
+
return out
|
| 431 |
+
|
| 432 |
+
|
| 433 |
+
def _died(done):
|
| 434 |
+
"""`(code, message)` for a child that produced no answer."""
|
| 435 |
+
rc = done.returncode
|
| 436 |
+
tail = " ".join((done.stderr or "").split())[-300:]
|
| 437 |
+
if rc in (-9, 137):
|
| 438 |
+
return "memory", "the script was stopped for using too much memory"
|
| 439 |
+
if rc in (-24, 152):
|
| 440 |
+
return "timeout", "the script used more processor time than a script view is allowed"
|
| 441 |
+
return "crashed", f"the script view engine stopped without an answer{': ' + tail if tail else ''}"
|
| 442 |
+
|
| 443 |
+
|
| 444 |
+
def _refusal(code, message, started):
|
| 445 |
+
return {"ok": False, "code": code, "message": message, "spec": None, "stdout": "",
|
| 446 |
+
"truncated": False, "ms": int((time.monotonic() - started) * 1000),
|
| 447 |
+
"caps": {"wallClock": True, "memory": False, "cpu": False}}
|
| 448 |
+
|
| 449 |
+
|
| 450 |
+
def _check_spec(spec):
|
| 451 |
+
"""C3: a spec is a DESCRIPTION the client draws. Never HTML, never a script, never a URL.
|
| 452 |
+
|
| 453 |
+
β THE CHECK IS ON THE KEYS, NOT ON THE STRING CONTENTS. Scanning values for `<script>` is a
|
| 454 |
+
blacklist and would pass `<SCR` + `IPT>`; refusing a spec that carries an `html`, `script`,
|
| 455 |
+
`src` or `onclick` key refuses the SHAPE that would let a renderer be talked into executing
|
| 456 |
+
something. The vocabulary of legal `kind`s is the ROUTE's business (W36-T37) β this is the
|
| 457 |
+
floor every caller gets whether or not the route above it remembers.
|
| 458 |
+
"""
|
| 459 |
+
if not isinstance(spec, dict):
|
| 460 |
+
return "the script emitted something that is not a view spec"
|
| 461 |
+
try:
|
| 462 |
+
blob = json.dumps(spec)
|
| 463 |
+
except (TypeError, ValueError):
|
| 464 |
+
return "the emitted view spec is not something the client can be sent"
|
| 465 |
+
if len(blob.encode("utf-8", "replace")) > MAX_SPEC_BYTES:
|
| 466 |
+
return (f"the emitted view spec is over {MAX_SPEC_BYTES // (1024 * 1024)} MB. A view spec "
|
| 467 |
+
f"describes a picture; it is not where the rows go")
|
| 468 |
+
banned = {"html", "innerhtml", "script", "src", "srcdoc", "href", "style", "onclick", "onload"}
|
| 469 |
+
found = sorted(k for k in _keys_of(spec) if str(k).lower() in banned)
|
| 470 |
+
if found:
|
| 471 |
+
return (f"a view spec may not carry {', '.join(found)}. The client DRAWS a spec, so a "
|
| 472 |
+
f"markup or URL key would be a script by another name")
|
| 473 |
+
return None
|
| 474 |
+
|
| 475 |
+
|
| 476 |
+
def _keys_of(value, depth=0):
|
| 477 |
+
"""Every key anywhere in a nested spec. Bounded, so a deep structure cannot spin this."""
|
| 478 |
+
if depth > 12:
|
| 479 |
+
return
|
| 480 |
+
if isinstance(value, dict):
|
| 481 |
+
for key, sub in value.items():
|
| 482 |
+
yield key
|
| 483 |
+
yield from _keys_of(sub, depth + 1)
|
| 484 |
+
elif isinstance(value, (list, tuple)):
|
| 485 |
+
for sub in value:
|
| 486 |
+
yield from _keys_of(sub, depth + 1)
|
| 487 |
+
|
| 488 |
+
|
| 489 |
+
# ββββββββββββββββββββββββββββββββββββββββββββββ THE DOOR β C1 is the ONLY way to a row βββββββββ
|
| 490 |
+
def run_view(user, table_key, source, st=None, **kw):
|
| 491 |
+
"""Fetch through C1 under `user`'s scope, then run the script over what came back (R5).
|
| 492 |
+
|
| 493 |
+
ββ THE FETCH HAPPENS IN THE PARENT AND ONLY ROWS CROSS INTO THE CHILD. That is wiring W1
|
| 494 |
+
made structural: the child has no runtime to ask, no store handle to open and no credential
|
| 495 |
+
to use, so "a script cannot read what its caller cannot read" is not a rule anybody has to
|
| 496 |
+
keep β there is no second path for it to be broken through.
|
| 497 |
+
|
| 498 |
+
β C1'S THREE EXCEPTIONS ARE ANSWERED, NEVER SWALLOWED. `UnknownTable`, `Denied` and
|
| 499 |
+
`Unresolvable` mean three different things to a person; collapsing them into "no rows" is the
|
| 500 |
+
silent-empty answer C1 was written to make impossible. `Unresolvable.as_limit()` is handed
|
| 501 |
+
through in the words it was raised with β standing rule 1's second sentence, verbatim.
|
| 502 |
+
"""
|
| 503 |
+
import core.perm_scope as perm_scope
|
| 504 |
+
|
| 505 |
+
key = str(table_key or "")
|
| 506 |
+
try:
|
| 507 |
+
rows = perm_scope.scoped_table(user, key, st=st)
|
| 508 |
+
fields = perm_scope.scoped_fields(user, key, st=st)
|
| 509 |
+
except perm_scope.UnknownTable as exc:
|
| 510 |
+
return _refusal("unknown_table", str(exc) or f"there is no database '{key}'",
|
| 511 |
+
time.monotonic())
|
| 512 |
+
except perm_scope.Denied as exc:
|
| 513 |
+
return _refusal("denied", str(exc) or "this account may not read that database",
|
| 514 |
+
time.monotonic())
|
| 515 |
+
except perm_scope.Unresolvable as exc:
|
| 516 |
+
out = _refusal("unresolvable", str(exc), time.monotonic())
|
| 517 |
+
out["limit"] = exc.as_limit()
|
| 518 |
+
return out
|
| 519 |
+
return run(source, rows, fields, key, **kw)
|
platform/core/shares.py
CHANGED
|
@@ -1,254 +1,254 @@
|
|
| 1 |
-
"""core/shares.py β ONE grant registry for every shareable object (wave 20, owner ruling R10).
|
| 2 |
-
|
| 3 |
-
WHAT R10 ASKED FOR: folders and databases share with the **same two-role vocabulary views
|
| 4 |
-
already use** (specific users or everyone; role = view | edit), plus one manage-access editor
|
| 5 |
-
that can add or revoke people later, on any of the three.
|
| 6 |
-
|
| 7 |
-
WHY A REGISTRY RATHER THAN A FIELD ON EACH OBJECT. A view already carries its own `permissions`
|
| 8 |
-
(`core/table_store.py`) and that stays β moving it would rewrite every stored view for no gain.
|
| 9 |
-
But a FOLDER is a value inside one user's workspace blob and a DATABASE is a `user_tables`
|
| 10 |
-
definition; giving each its own grant field would put the same three-line permission decision in
|
| 11 |
-
three files owned by two sessions, which is how the three drift. One registry, one predicate,
|
| 12 |
-
three callers.
|
| 13 |
-
|
| 14 |
-
shares.set_grants(kind, oid, entries, owner=β¦, st=β¦) # replaces the whole grant set
|
| 15 |
-
shares.grants(kind, oid, st=β¦) # -> {'owner': str, 'entries': [...]}
|
| 16 |
-
shares.role_for(kind, oid, user, is_admin=β¦, st=β¦) # -> 'owner'|'edit'|'view'|None
|
| 17 |
-
shares.shared_with(user, kind=β¦, st=β¦) # -> [oid] this user was granted
|
| 18 |
-
|
| 19 |
-
THE ROLE VOCABULARY IS TWO WORDS AND THE DEFAULT IS THE NARROW ONE. `view` = may open and read;
|
| 20 |
-
`edit` = may also change the object's CONTENT. Neither ever means "may re-share": changing grants
|
| 21 |
-
is the OWNER's (or an admin's), which is `table_store._may_administer`'s existing rule promoted to
|
| 22 |
-
every kind. A collaborator who could rewrite grants could grant themselves sole ownership of
|
| 23 |
-
somebody else's object, or quietly widen a users-scoped share to everyone.
|
| 24 |
-
|
| 25 |
-
β AN UNREADABLE GRANT IS NO GRANT. Every path here fails closed β junk in the bucket, a missing
|
| 26 |
-
owner, an unknown role string all resolve to None rather than to a default that opens something.
|
| 27 |
-
[[aios-permissioning]]: no fail-open defaults, ever.
|
| 28 |
-
|
| 29 |
-
β THE BUCKET IS TENANT-SCOPED THROUGH `st`, like every other product-data write. Passing the
|
| 30 |
-
session's `TenantRuntime` is what keeps Nurilab's grants in Nurilab's store; the module default
|
| 31 |
-
(`core.store`) is tenant #0 and exists for the same reason it does everywhere else β the ~28
|
| 32 |
-
callers that predate multi-tenancy. (This is the D-5/D-16 residency shape, and this module does
|
| 33 |
-
NOT repeat their mistake: `st` is threaded from the first line rather than retrofitted.)
|
| 34 |
-
"""
|
| 35 |
-
import core.store as store
|
| 36 |
-
|
| 37 |
-
#: The store key. One bucket per tenant holds every kind's grants, because "what am I shared on"
|
| 38 |
-
#: is a question across kinds β the "Shared with me" folder (R10) is exactly that query, and
|
| 39 |
-
#: three separate buckets would make it three reads that can disagree about what a user can see.
|
| 40 |
-
SHARES_KEY = 'object_shares'
|
| 41 |
-
|
| 42 |
-
#: The shareable kinds. A CLOSED vocabulary: an unknown kind raises rather than creating a new
|
| 43 |
-
#: namespace by typo, which would silently grant nothing to nobody and read as "sharing is broken".
|
| 44 |
-
KINDS = ('view', 'folder', 'database')
|
| 45 |
-
|
| 46 |
-
#: `*` is "everyone who can already open the surface". It is NOT "every account on the platform".
|
| 47 |
-
#: Spelled as a single character so it can never collide with a username (usernames are lower-case
|
| 48 |
-
#: and non-empty by `core/users.py`, and are checked against this explicitly below).
|
| 49 |
-
#:
|
| 50 |
-
#: ββ **AND WHAT THAT MEANS DEPENDS ON THE KIND β THE LINE THIS NOTE USED TO CARRY WAS FALSE FOR
|
| 51 |
-
#: ONE OF THE THREE** (W33-T30, `waves/wave32/sharing-audit.md` S-8). It read *"the module/table
|
| 52 |
-
#: wall runs FIRST and this never widens past it"*, flatly, and the audit's own words for that are
|
| 53 |
-
#: *"the third docstring in this audit describing a check that is not on the path"*. Corrected
|
| 54 |
-
#: here rather than deleted, because the sentence is TRUE of two kinds and the difference is the
|
| 55 |
-
#: whole point:
|
| 56 |
-
#: * `kind='view'` / `kind='folder'` on a GOVERNED module (`customer_data`, `product_data`) β
|
| 57 |
-
#: the sentence holds. `require_session` plus the topic's own gate run first, and the
|
| 58 |
-
#: receiver's row scope and hidden-field closure are applied BEFORE any foreign view is
|
| 59 |
-
#: merged, so a grant can only narrow-or-equal what that account could already reach.
|
| 60 |
-
#: * `kind='database'` on a `ut_*` table β ββ **THE SENTENCE HOLDS HERE TOO NOW, AND THAT IS
|
| 61 |
-
#: W36-T21 / OWNER RULING R6 (audit S-8, CLOSED).** It did not until wave 36, and the reason
|
| 62 |
-
#: is worth keeping: `routes_admin._clean_perms` 400'd any key outside
|
| 63 |
-
#: `("customer_data", "product_data")`, so no row filter and no hidden field could even be
|
| 64 |
-
#: DECLARED for a user table, `routes_tables.py` made zero `perm_scope` calls, and it passed
|
| 65 |
-
#: `hidden_keys=frozenset()`. A `database` grant was therefore ALL-OR-NOTHING β every row,
|
| 66 |
-
#: every column β and this registry was the only wall behind it.
|
| 67 |
-
#:
|
| 68 |
-
#: β WHAT CHANGED: `perm_scope.scoped_table` (contract C1) is the ONE door to any database's
|
| 69 |
-
#: rows. `routes_tables` applies the permanent filter before `pids` is taken and the transitive
|
| 70 |
-
#: hidden-field closure after `workspace_wire`, on EVERY `ut_*` read β the same code that walls
|
| 71 |
-
#: `customer_data` β and a door that cannot apply them REFUSES rather than serving the lot. So a
|
| 72 |
-
#: `database` grant is once again bounded by a second wall: it decides WHETHER an account reaches
|
| 73 |
-
#: the database, and C1 decides WHICH rows and columns it then sees. An `*` grant still admits
|
| 74 |
-
#: every account in the tenant, and each of them still only sees what their own wall allows.
|
| 75 |
-
#:
|
| 76 |
-
#: β `routes_shares.py`'s module docstring carries the OLD sentence at the other door and is in
|
| 77 |
-
#: no wave-36 fence β the audit's own fix was "say so at both", so one of the two is now stale.
|
| 78 |
-
#: Booked in `mailbox/C.md` (C-14) rather than edited across a fence
|
| 79 |
-
#: [[two-gates-can-assert-opposite-things]].
|
| 80 |
-
EVERYONE = '*'
|
| 81 |
-
|
| 82 |
-
ROLES = ('view', 'edit')
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
def _st(st):
|
| 86 |
-
return st if st is not None else store
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
def _check_kind(kind):
|
| 90 |
-
k = str(kind or '').strip().lower()
|
| 91 |
-
if k not in KINDS:
|
| 92 |
-
raise ValueError(f'{kind!r} is not a shareable kind. Use one of {", ".join(KINDS)} β '
|
| 93 |
-
f'refusing to invent a namespace from a typo.')
|
| 94 |
-
return k
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
def _clean_entries(entries):
|
| 98 |
-
"""Normalise + REJECT junk, returning [{'user': str, 'role': 'view'|'edit'}].
|
| 99 |
-
|
| 100 |
-
Silently dropping a malformed entry is right here and wrong elsewhere: the caller is a UI
|
| 101 |
-
that just listed the people it is about to grant, so a rejected row must not abort the whole
|
| 102 |
-
save β but an entry with an unknown ROLE must not be stored as something else's default
|
| 103 |
-
either. Dropped, never coerced.
|
| 104 |
-
"""
|
| 105 |
-
out, seen = [], set()
|
| 106 |
-
for e in entries or ():
|
| 107 |
-
if not isinstance(e, dict):
|
| 108 |
-
continue
|
| 109 |
-
user = str(e.get('user') or '').strip().lower()
|
| 110 |
-
role = str(e.get('role') or '').strip().lower()
|
| 111 |
-
if not user or role not in ROLES or user in seen:
|
| 112 |
-
continue
|
| 113 |
-
seen.add(user)
|
| 114 |
-
out.append({'user': user, 'role': role})
|
| 115 |
-
return out
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
def grants(kind, oid, st=None):
|
| 119 |
-
"""`{'owner': str|None, 'entries': [{'user','role'}]}` β never raises on a junk bucket."""
|
| 120 |
-
kind = _check_kind(kind)
|
| 121 |
-
try:
|
| 122 |
-
bucket = (_st(st).get(SHARES_KEY) or {}).get(kind) or {}
|
| 123 |
-
rec = bucket.get(str(oid)) or {}
|
| 124 |
-
except Exception:
|
| 125 |
-
return {'owner': None, 'entries': []}
|
| 126 |
-
if not isinstance(rec, dict):
|
| 127 |
-
return {'owner': None, 'entries': []}
|
| 128 |
-
return {'owner': (str(rec.get('owner')).strip().lower() if rec.get('owner') else None),
|
| 129 |
-
'entries': _clean_entries(rec.get('entries'))}
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
def set_grants(kind, oid, entries, owner=None, st=None):
|
| 133 |
-
"""REPLACE the grant set for one object. Returns the stored record.
|
| 134 |
-
|
| 135 |
-
β REPLACE, NOT MERGE, and that is the contract the UI needs: revoking is expressed by an
|
| 136 |
-
entry's ABSENCE. A merge-only API cannot remove anybody without a second verb, and the
|
| 137 |
-
manage-access editor R10 asks for is exactly "here is the list now".
|
| 138 |
-
"""
|
| 139 |
-
kind = _check_kind(kind)
|
| 140 |
-
oid = str(oid)
|
| 141 |
-
clean = _clean_entries(entries)
|
| 142 |
-
owner_l = str(owner).strip().lower() if owner else None
|
| 143 |
-
|
| 144 |
-
def _apply(data):
|
| 145 |
-
by_kind = dict(data.get(kind) or {})
|
| 146 |
-
prior = by_kind.get(oid) if isinstance(by_kind.get(oid), dict) else {}
|
| 147 |
-
# The owner is STICKY: set once, and a later save that omits it must not orphan the
|
| 148 |
-
# object. An ownerless grant record cannot answer "who may re-share this", so every
|
| 149 |
-
# administer check would fail closed and the object would become unmanageable.
|
| 150 |
-
keep_owner = owner_l or (str(prior.get('owner')).strip().lower()
|
| 151 |
-
if prior.get('owner') else None)
|
| 152 |
-
if not clean and not keep_owner:
|
| 153 |
-
by_kind.pop(oid, None) # fully un-shared and unowned: leave no empty husk
|
| 154 |
-
else:
|
| 155 |
-
by_kind[oid] = {'owner': keep_owner, 'entries': clean}
|
| 156 |
-
data[kind] = by_kind
|
| 157 |
-
return data
|
| 158 |
-
|
| 159 |
-
_st(st).update(SHARES_KEY, _apply, flush='async')
|
| 160 |
-
return grants(kind, oid, st=st)
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
def role_for(kind, oid, user, is_admin=False, st=None):
|
| 164 |
-
"""`'owner'` | `'edit'` | `'view'` | `None` β the caller's effective role, fail-closed.
|
| 165 |
-
|
| 166 |
-
An ADMIN reads as `'owner'`: an admin who could not administer an object could not
|
| 167 |
-
administer the tenant either, which is `table_store._may_administer`'s existing rule and is
|
| 168 |
-
kept identical here so the two cannot disagree about the same view.
|
| 169 |
-
"""
|
| 170 |
-
user = str(user or '').strip().lower()
|
| 171 |
-
if not user:
|
| 172 |
-
return None
|
| 173 |
-
rec = grants(kind, oid, st=st)
|
| 174 |
-
if is_admin or (rec['owner'] and rec['owner'] == user):
|
| 175 |
-
return 'owner'
|
| 176 |
-
best = None
|
| 177 |
-
for e in rec['entries']:
|
| 178 |
-
if e['user'] == user or e['user'] == EVERYONE:
|
| 179 |
-
# The STRONGER of the two wins when both a personal and an everyone grant exist:
|
| 180 |
-
# naming somebody explicitly is how you RAISE them above the room, so an
|
| 181 |
-
# everyone-view + alice-edit pair must leave alice editing.
|
| 182 |
-
if e['role'] == 'edit':
|
| 183 |
-
return 'edit'
|
| 184 |
-
best = best or 'view'
|
| 185 |
-
return best
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
def may_see(kind, oid, user, is_admin=False, st=None):
|
| 189 |
-
return role_for(kind, oid, user, is_admin=is_admin, st=st) is not None
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
def may_edit(kind, oid, user, is_admin=False, st=None):
|
| 193 |
-
return role_for(kind, oid, user, is_admin=is_admin, st=st) in ('owner', 'edit')
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
def may_administer(kind, oid, user, is_admin=False, st=None):
|
| 197 |
-
"""Only the owner or an admin may change grants or delete. See the module note on why this
|
| 198 |
-
is deliberately narrower than `may_edit`."""
|
| 199 |
-
return role_for(kind, oid, user, is_admin=is_admin, st=st) == 'owner'
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
def shared_with(user, kind=None, st=None):
|
| 203 |
-
"""Every object id this user has been granted (excluding what they own).
|
| 204 |
-
|
| 205 |
-
This is the "Shared with me" query (R10). It EXCLUDES owned objects deliberately: a folder
|
| 206 |
-
you made is not something shared *with* you, and listing it there would make the system
|
| 207 |
-
folder a duplicate of the rail above it.
|
| 208 |
-
"""
|
| 209 |
-
user = str(user or '').strip().lower()
|
| 210 |
-
if not user:
|
| 211 |
-
return {}
|
| 212 |
-
try:
|
| 213 |
-
data = _st(st).get(SHARES_KEY) or {}
|
| 214 |
-
except Exception:
|
| 215 |
-
return {}
|
| 216 |
-
out = {}
|
| 217 |
-
for k in ([_check_kind(kind)] if kind else KINDS):
|
| 218 |
-
hits = []
|
| 219 |
-
for oid, rec in (data.get(k) or {}).items():
|
| 220 |
-
if not isinstance(rec, dict):
|
| 221 |
-
continue
|
| 222 |
-
owner = str(rec.get('owner') or '').strip().lower()
|
| 223 |
-
if owner == user:
|
| 224 |
-
continue
|
| 225 |
-
for e in _clean_entries(rec.get('entries')):
|
| 226 |
-
if e['user'] in (user, EVERYONE):
|
| 227 |
-
hits.append(str(oid))
|
| 228 |
-
break
|
| 229 |
-
out[k] = sorted(hits)
|
| 230 |
-
return out if kind is None else {_check_kind(kind): out[_check_kind(kind)]}
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
def drop_objects(pairs, st=None):
|
| 234 |
-
"""Remove whole grant RECORDS, owner husk included β wave 21, item 6a (C3).
|
| 235 |
-
|
| 236 |
-
A deleted object's grants must die with it: `shared_with` would otherwise serve ghost ids
|
| 237 |
-
into every receiver's "Shared with me" forever, and the ghost would 404 on open. One
|
| 238 |
-
transaction for the whole sweep β a table delete drops its database grant plus a view
|
| 239 |
-
grant per view that lived in its bucket."""
|
| 240 |
-
want = {}
|
| 241 |
-
for kind, oid in pairs or ():
|
| 242 |
-
want.setdefault(_check_kind(kind), set()).add(str(oid))
|
| 243 |
-
if not want:
|
| 244 |
-
return
|
| 245 |
-
|
| 246 |
-
def _apply(data):
|
| 247 |
-
for kind, oids in want.items():
|
| 248 |
-
by_kind = data.get(kind)
|
| 249 |
-
if isinstance(by_kind, dict):
|
| 250 |
-
for oid in oids:
|
| 251 |
-
by_kind.pop(oid, None)
|
| 252 |
-
return data
|
| 253 |
-
|
| 254 |
-
_st(st).update(SHARES_KEY, _apply, flush='async')
|
|
|
|
| 1 |
+
"""core/shares.py β ONE grant registry for every shareable object (wave 20, owner ruling R10).
|
| 2 |
+
|
| 3 |
+
WHAT R10 ASKED FOR: folders and databases share with the **same two-role vocabulary views
|
| 4 |
+
already use** (specific users or everyone; role = view | edit), plus one manage-access editor
|
| 5 |
+
that can add or revoke people later, on any of the three.
|
| 6 |
+
|
| 7 |
+
WHY A REGISTRY RATHER THAN A FIELD ON EACH OBJECT. A view already carries its own `permissions`
|
| 8 |
+
(`core/table_store.py`) and that stays β moving it would rewrite every stored view for no gain.
|
| 9 |
+
But a FOLDER is a value inside one user's workspace blob and a DATABASE is a `user_tables`
|
| 10 |
+
definition; giving each its own grant field would put the same three-line permission decision in
|
| 11 |
+
three files owned by two sessions, which is how the three drift. One registry, one predicate,
|
| 12 |
+
three callers.
|
| 13 |
+
|
| 14 |
+
shares.set_grants(kind, oid, entries, owner=β¦, st=β¦) # replaces the whole grant set
|
| 15 |
+
shares.grants(kind, oid, st=β¦) # -> {'owner': str, 'entries': [...]}
|
| 16 |
+
shares.role_for(kind, oid, user, is_admin=β¦, st=β¦) # -> 'owner'|'edit'|'view'|None
|
| 17 |
+
shares.shared_with(user, kind=β¦, st=β¦) # -> [oid] this user was granted
|
| 18 |
+
|
| 19 |
+
THE ROLE VOCABULARY IS TWO WORDS AND THE DEFAULT IS THE NARROW ONE. `view` = may open and read;
|
| 20 |
+
`edit` = may also change the object's CONTENT. Neither ever means "may re-share": changing grants
|
| 21 |
+
is the OWNER's (or an admin's), which is `table_store._may_administer`'s existing rule promoted to
|
| 22 |
+
every kind. A collaborator who could rewrite grants could grant themselves sole ownership of
|
| 23 |
+
somebody else's object, or quietly widen a users-scoped share to everyone.
|
| 24 |
+
|
| 25 |
+
β AN UNREADABLE GRANT IS NO GRANT. Every path here fails closed β junk in the bucket, a missing
|
| 26 |
+
owner, an unknown role string all resolve to None rather than to a default that opens something.
|
| 27 |
+
[[aios-permissioning]]: no fail-open defaults, ever.
|
| 28 |
+
|
| 29 |
+
β THE BUCKET IS TENANT-SCOPED THROUGH `st`, like every other product-data write. Passing the
|
| 30 |
+
session's `TenantRuntime` is what keeps Nurilab's grants in Nurilab's store; the module default
|
| 31 |
+
(`core.store`) is tenant #0 and exists for the same reason it does everywhere else β the ~28
|
| 32 |
+
callers that predate multi-tenancy. (This is the D-5/D-16 residency shape, and this module does
|
| 33 |
+
NOT repeat their mistake: `st` is threaded from the first line rather than retrofitted.)
|
| 34 |
+
"""
|
| 35 |
+
import core.store as store
|
| 36 |
+
|
| 37 |
+
#: The store key. One bucket per tenant holds every kind's grants, because "what am I shared on"
|
| 38 |
+
#: is a question across kinds β the "Shared with me" folder (R10) is exactly that query, and
|
| 39 |
+
#: three separate buckets would make it three reads that can disagree about what a user can see.
|
| 40 |
+
SHARES_KEY = 'object_shares'
|
| 41 |
+
|
| 42 |
+
#: The shareable kinds. A CLOSED vocabulary: an unknown kind raises rather than creating a new
|
| 43 |
+
#: namespace by typo, which would silently grant nothing to nobody and read as "sharing is broken".
|
| 44 |
+
KINDS = ('view', 'folder', 'database')
|
| 45 |
+
|
| 46 |
+
#: `*` is "everyone who can already open the surface". It is NOT "every account on the platform".
|
| 47 |
+
#: Spelled as a single character so it can never collide with a username (usernames are lower-case
|
| 48 |
+
#: and non-empty by `core/users.py`, and are checked against this explicitly below).
|
| 49 |
+
#:
|
| 50 |
+
#: ββ **AND WHAT THAT MEANS DEPENDS ON THE KIND β THE LINE THIS NOTE USED TO CARRY WAS FALSE FOR
|
| 51 |
+
#: ONE OF THE THREE** (W33-T30, `waves/wave32/sharing-audit.md` S-8). It read *"the module/table
|
| 52 |
+
#: wall runs FIRST and this never widens past it"*, flatly, and the audit's own words for that are
|
| 53 |
+
#: *"the third docstring in this audit describing a check that is not on the path"*. Corrected
|
| 54 |
+
#: here rather than deleted, because the sentence is TRUE of two kinds and the difference is the
|
| 55 |
+
#: whole point:
|
| 56 |
+
#: * `kind='view'` / `kind='folder'` on a GOVERNED module (`customer_data`, `product_data`) β
|
| 57 |
+
#: the sentence holds. `require_session` plus the topic's own gate run first, and the
|
| 58 |
+
#: receiver's row scope and hidden-field closure are applied BEFORE any foreign view is
|
| 59 |
+
#: merged, so a grant can only narrow-or-equal what that account could already reach.
|
| 60 |
+
#: * `kind='database'` on a `ut_*` table β ββ **THE SENTENCE HOLDS HERE TOO NOW, AND THAT IS
|
| 61 |
+
#: W36-T21 / OWNER RULING R6 (audit S-8, CLOSED).** It did not until wave 36, and the reason
|
| 62 |
+
#: is worth keeping: `routes_admin._clean_perms` 400'd any key outside
|
| 63 |
+
#: `("customer_data", "product_data")`, so no row filter and no hidden field could even be
|
| 64 |
+
#: DECLARED for a user table, `routes_tables.py` made zero `perm_scope` calls, and it passed
|
| 65 |
+
#: `hidden_keys=frozenset()`. A `database` grant was therefore ALL-OR-NOTHING β every row,
|
| 66 |
+
#: every column β and this registry was the only wall behind it.
|
| 67 |
+
#:
|
| 68 |
+
#: β WHAT CHANGED: `perm_scope.scoped_table` (contract C1) is the ONE door to any database's
|
| 69 |
+
#: rows. `routes_tables` applies the permanent filter before `pids` is taken and the transitive
|
| 70 |
+
#: hidden-field closure after `workspace_wire`, on EVERY `ut_*` read β the same code that walls
|
| 71 |
+
#: `customer_data` β and a door that cannot apply them REFUSES rather than serving the lot. So a
|
| 72 |
+
#: `database` grant is once again bounded by a second wall: it decides WHETHER an account reaches
|
| 73 |
+
#: the database, and C1 decides WHICH rows and columns it then sees. An `*` grant still admits
|
| 74 |
+
#: every account in the tenant, and each of them still only sees what their own wall allows.
|
| 75 |
+
#:
|
| 76 |
+
#: β `routes_shares.py`'s module docstring carries the OLD sentence at the other door and is in
|
| 77 |
+
#: no wave-36 fence β the audit's own fix was "say so at both", so one of the two is now stale.
|
| 78 |
+
#: Booked in `mailbox/C.md` (C-14) rather than edited across a fence
|
| 79 |
+
#: [[two-gates-can-assert-opposite-things]].
|
| 80 |
+
EVERYONE = '*'
|
| 81 |
+
|
| 82 |
+
ROLES = ('view', 'edit')
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def _st(st):
|
| 86 |
+
return st if st is not None else store
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def _check_kind(kind):
|
| 90 |
+
k = str(kind or '').strip().lower()
|
| 91 |
+
if k not in KINDS:
|
| 92 |
+
raise ValueError(f'{kind!r} is not a shareable kind. Use one of {", ".join(KINDS)} β '
|
| 93 |
+
f'refusing to invent a namespace from a typo.')
|
| 94 |
+
return k
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def _clean_entries(entries):
|
| 98 |
+
"""Normalise + REJECT junk, returning [{'user': str, 'role': 'view'|'edit'}].
|
| 99 |
+
|
| 100 |
+
Silently dropping a malformed entry is right here and wrong elsewhere: the caller is a UI
|
| 101 |
+
that just listed the people it is about to grant, so a rejected row must not abort the whole
|
| 102 |
+
save β but an entry with an unknown ROLE must not be stored as something else's default
|
| 103 |
+
either. Dropped, never coerced.
|
| 104 |
+
"""
|
| 105 |
+
out, seen = [], set()
|
| 106 |
+
for e in entries or ():
|
| 107 |
+
if not isinstance(e, dict):
|
| 108 |
+
continue
|
| 109 |
+
user = str(e.get('user') or '').strip().lower()
|
| 110 |
+
role = str(e.get('role') or '').strip().lower()
|
| 111 |
+
if not user or role not in ROLES or user in seen:
|
| 112 |
+
continue
|
| 113 |
+
seen.add(user)
|
| 114 |
+
out.append({'user': user, 'role': role})
|
| 115 |
+
return out
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def grants(kind, oid, st=None):
|
| 119 |
+
"""`{'owner': str|None, 'entries': [{'user','role'}]}` β never raises on a junk bucket."""
|
| 120 |
+
kind = _check_kind(kind)
|
| 121 |
+
try:
|
| 122 |
+
bucket = (_st(st).get(SHARES_KEY) or {}).get(kind) or {}
|
| 123 |
+
rec = bucket.get(str(oid)) or {}
|
| 124 |
+
except Exception:
|
| 125 |
+
return {'owner': None, 'entries': []}
|
| 126 |
+
if not isinstance(rec, dict):
|
| 127 |
+
return {'owner': None, 'entries': []}
|
| 128 |
+
return {'owner': (str(rec.get('owner')).strip().lower() if rec.get('owner') else None),
|
| 129 |
+
'entries': _clean_entries(rec.get('entries'))}
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def set_grants(kind, oid, entries, owner=None, st=None):
|
| 133 |
+
"""REPLACE the grant set for one object. Returns the stored record.
|
| 134 |
+
|
| 135 |
+
β REPLACE, NOT MERGE, and that is the contract the UI needs: revoking is expressed by an
|
| 136 |
+
entry's ABSENCE. A merge-only API cannot remove anybody without a second verb, and the
|
| 137 |
+
manage-access editor R10 asks for is exactly "here is the list now".
|
| 138 |
+
"""
|
| 139 |
+
kind = _check_kind(kind)
|
| 140 |
+
oid = str(oid)
|
| 141 |
+
clean = _clean_entries(entries)
|
| 142 |
+
owner_l = str(owner).strip().lower() if owner else None
|
| 143 |
+
|
| 144 |
+
def _apply(data):
|
| 145 |
+
by_kind = dict(data.get(kind) or {})
|
| 146 |
+
prior = by_kind.get(oid) if isinstance(by_kind.get(oid), dict) else {}
|
| 147 |
+
# The owner is STICKY: set once, and a later save that omits it must not orphan the
|
| 148 |
+
# object. An ownerless grant record cannot answer "who may re-share this", so every
|
| 149 |
+
# administer check would fail closed and the object would become unmanageable.
|
| 150 |
+
keep_owner = owner_l or (str(prior.get('owner')).strip().lower()
|
| 151 |
+
if prior.get('owner') else None)
|
| 152 |
+
if not clean and not keep_owner:
|
| 153 |
+
by_kind.pop(oid, None) # fully un-shared and unowned: leave no empty husk
|
| 154 |
+
else:
|
| 155 |
+
by_kind[oid] = {'owner': keep_owner, 'entries': clean}
|
| 156 |
+
data[kind] = by_kind
|
| 157 |
+
return data
|
| 158 |
+
|
| 159 |
+
_st(st).update(SHARES_KEY, _apply, flush='async')
|
| 160 |
+
return grants(kind, oid, st=st)
|
| 161 |
+
|
| 162 |
+
|
| 163 |
+
def role_for(kind, oid, user, is_admin=False, st=None):
|
| 164 |
+
"""`'owner'` | `'edit'` | `'view'` | `None` β the caller's effective role, fail-closed.
|
| 165 |
+
|
| 166 |
+
An ADMIN reads as `'owner'`: an admin who could not administer an object could not
|
| 167 |
+
administer the tenant either, which is `table_store._may_administer`'s existing rule and is
|
| 168 |
+
kept identical here so the two cannot disagree about the same view.
|
| 169 |
+
"""
|
| 170 |
+
user = str(user or '').strip().lower()
|
| 171 |
+
if not user:
|
| 172 |
+
return None
|
| 173 |
+
rec = grants(kind, oid, st=st)
|
| 174 |
+
if is_admin or (rec['owner'] and rec['owner'] == user):
|
| 175 |
+
return 'owner'
|
| 176 |
+
best = None
|
| 177 |
+
for e in rec['entries']:
|
| 178 |
+
if e['user'] == user or e['user'] == EVERYONE:
|
| 179 |
+
# The STRONGER of the two wins when both a personal and an everyone grant exist:
|
| 180 |
+
# naming somebody explicitly is how you RAISE them above the room, so an
|
| 181 |
+
# everyone-view + alice-edit pair must leave alice editing.
|
| 182 |
+
if e['role'] == 'edit':
|
| 183 |
+
return 'edit'
|
| 184 |
+
best = best or 'view'
|
| 185 |
+
return best
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def may_see(kind, oid, user, is_admin=False, st=None):
|
| 189 |
+
return role_for(kind, oid, user, is_admin=is_admin, st=st) is not None
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
def may_edit(kind, oid, user, is_admin=False, st=None):
|
| 193 |
+
return role_for(kind, oid, user, is_admin=is_admin, st=st) in ('owner', 'edit')
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
def may_administer(kind, oid, user, is_admin=False, st=None):
|
| 197 |
+
"""Only the owner or an admin may change grants or delete. See the module note on why this
|
| 198 |
+
is deliberately narrower than `may_edit`."""
|
| 199 |
+
return role_for(kind, oid, user, is_admin=is_admin, st=st) == 'owner'
|
| 200 |
+
|
| 201 |
+
|
| 202 |
+
def shared_with(user, kind=None, st=None):
|
| 203 |
+
"""Every object id this user has been granted (excluding what they own).
|
| 204 |
+
|
| 205 |
+
This is the "Shared with me" query (R10). It EXCLUDES owned objects deliberately: a folder
|
| 206 |
+
you made is not something shared *with* you, and listing it there would make the system
|
| 207 |
+
folder a duplicate of the rail above it.
|
| 208 |
+
"""
|
| 209 |
+
user = str(user or '').strip().lower()
|
| 210 |
+
if not user:
|
| 211 |
+
return {}
|
| 212 |
+
try:
|
| 213 |
+
data = _st(st).get(SHARES_KEY) or {}
|
| 214 |
+
except Exception:
|
| 215 |
+
return {}
|
| 216 |
+
out = {}
|
| 217 |
+
for k in ([_check_kind(kind)] if kind else KINDS):
|
| 218 |
+
hits = []
|
| 219 |
+
for oid, rec in (data.get(k) or {}).items():
|
| 220 |
+
if not isinstance(rec, dict):
|
| 221 |
+
continue
|
| 222 |
+
owner = str(rec.get('owner') or '').strip().lower()
|
| 223 |
+
if owner == user:
|
| 224 |
+
continue
|
| 225 |
+
for e in _clean_entries(rec.get('entries')):
|
| 226 |
+
if e['user'] in (user, EVERYONE):
|
| 227 |
+
hits.append(str(oid))
|
| 228 |
+
break
|
| 229 |
+
out[k] = sorted(hits)
|
| 230 |
+
return out if kind is None else {_check_kind(kind): out[_check_kind(kind)]}
|
| 231 |
+
|
| 232 |
+
|
| 233 |
+
def drop_objects(pairs, st=None):
|
| 234 |
+
"""Remove whole grant RECORDS, owner husk included β wave 21, item 6a (C3).
|
| 235 |
+
|
| 236 |
+
A deleted object's grants must die with it: `shared_with` would otherwise serve ghost ids
|
| 237 |
+
into every receiver's "Shared with me" forever, and the ghost would 404 on open. One
|
| 238 |
+
transaction for the whole sweep β a table delete drops its database grant plus a view
|
| 239 |
+
grant per view that lived in its bucket."""
|
| 240 |
+
want = {}
|
| 241 |
+
for kind, oid in pairs or ():
|
| 242 |
+
want.setdefault(_check_kind(kind), set()).add(str(oid))
|
| 243 |
+
if not want:
|
| 244 |
+
return
|
| 245 |
+
|
| 246 |
+
def _apply(data):
|
| 247 |
+
for kind, oids in want.items():
|
| 248 |
+
by_kind = data.get(kind)
|
| 249 |
+
if isinstance(by_kind, dict):
|
| 250 |
+
for oid in oids:
|
| 251 |
+
by_kind.pop(oid, None)
|
| 252 |
+
return data
|
| 253 |
+
|
| 254 |
+
_st(st).update(SHARES_KEY, _apply, flush='async')
|
platform/core/store.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
platform/core/table_store.py
CHANGED
|
@@ -1,746 +1,746 @@
|
|
| 1 |
-
"""The generic per-user TABLE WORKSPACE store β the persistence half of the table-page factory.
|
| 2 |
-
|
| 3 |
-
One durable store key holds one table OBJECT's per-user Airtable-style state:
|
| 4 |
-
|
| 5 |
-
{username: {'views': {view_id: SavedView},
|
| 6 |
-
'fields': {field_key: Field}, # notes + custom_ + measure_ strata
|
| 7 |
-
'overlays': {str(pid): {field_key: value}}}}
|
| 8 |
-
|
| 9 |
-
`make(table_key)` returns the six operations a table page's host loop needs, closed over that
|
| 10 |
-
key. The Customer table's ops (`modules/customer_data.py`, key 'customer_table_workspace') are
|
| 11 |
-
these exact functions β the logic MOVED here 2026-07-27 so that duplicating the Customer table
|
| 12 |
-
pattern to a new object is a registry row + a config, not a copy of the store plumbing
|
| 13 |
-
(owner directive: the table-page factory).
|
| 14 |
-
|
| 15 |
-
A LIST (membership/formula semantics) is deliberately a different store from a VIEW
|
| 16 |
-
(presentation/query state) β see modules/customer_data.py's customer_lists key.
|
| 17 |
-
"""
|
| 18 |
-
import core.store as store
|
| 19 |
-
|
| 20 |
-
#: Wave-9 I17 β the SHARED bucket. Views whose permissions make them visible to anyone but
|
| 21 |
-
#: their creator live here instead of in a personal workspace, under a key that cannot collide
|
| 22 |
-
#: with a username (usernames come from core/users.py and are never dunder-wrapped; `is_shared`
|
| 23 |
-
#: guards it anyway). ONE HOME PER VIEW, never both: a view moved to personal is REMOVED from
|
| 24 |
-
#: here, and a view shared is removed from its creator's workspace. Two homes would mean two
|
| 25 |
-
#: divergent copies the moment either was edited.
|
| 26 |
-
SHARED_KEY = '__shared__'
|
| 27 |
-
|
| 28 |
-
#: ββ W36-T25 β THE COLUMN SUMMARY IS THE DATABASE'S, NOT ONE ACCOUNT'S.
|
| 29 |
-
#:
|
| 30 |
-
#: Owner item 1, second half, verbatim: *"When i filter Avi on my computer sum works for the Sales
|
| 31 |
-
#: last 365 days column but not when my colleague filter it β The sum of the field is not showing
|
| 32 |
-
#: at the bottom even when i zoom out. It only works on my screen??"* Measured while scouting it:
|
| 33 |
-
#: `save_field` writes the WHOLE field payload into `data[username]`, so a column summary β which
|
| 34 |
-
#: is a statement about the COLUMN, identical for every reader by construction β was stored once
|
| 35 |
-
#: per account. Set it, and the totals row exists for you and for nobody else. `computeAggs` then
|
| 36 |
-
#: paints no totals row at all for the colleague (`showTotals` is false when no field carries an
|
| 37 |
-
#: `agg`), which is exactly "not showing at the bottom".
|
| 38 |
-
#:
|
| 39 |
-
#: β ONLY `agg` MOVES, AND THE LINE IS NOT ARBITRARY. Width, column order, the note, the display
|
| 40 |
-
#: format and every `custom_`/`measure_` definition stay per user, because each of those is a
|
| 41 |
-
#: statement about how ONE PERSON reads the column. "Sum this column" is a statement about what
|
| 42 |
-
#: the column MEANS, and two accounts disagreeing about it is the defect, not a preference.
|
| 43 |
-
#:
|
| 44 |
-
#: β IT LIVES IN THE `__shared__` MEMBER OF THIS SAME BUCKET rather than in
|
| 45 |
-
#: `core/shared_overlay.py`'s `<key>__shared` document, and the reason is transactional: a summary
|
| 46 |
-
#: is written by the same `save_field` call that writes the note beside it, and the two must land
|
| 47 |
-
#: or fail together. `__shared__` is already this store's tenant-wide member (shared VIEWS live
|
| 48 |
-
#: there) and is guarded against colliding with a username by `is_shared` and by `core/users.py`'s
|
| 49 |
-
#: never-dunder rule, so there is no second bucket, no second flush and no second failure mode.
|
| 50 |
-
SHARED_FIELD_KEYS = ('agg',)
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
def _may_see(view, viewer, is_admin=False):
|
| 54 |
-
"""Visibility for ONE shared view, fail-closed.
|
| 55 |
-
|
| 56 |
-
'collaborative' = everyone who can already open the module (the caller has gated that).
|
| 57 |
-
'users' = the named users, plus the creator, plus admins β an admin who could not
|
| 58 |
-
see a view could not administer it either.
|
| 59 |
-
Anything unrecognised returns False rather than defaulting open: an unreadable permission
|
| 60 |
-
must never widen access ([[aios-permissioning]] β no fail-open defaults).
|
| 61 |
-
"""
|
| 62 |
-
if not isinstance(view, dict):
|
| 63 |
-
return False
|
| 64 |
-
if view.get('createdBy') == viewer or is_admin:
|
| 65 |
-
return True
|
| 66 |
-
perms = view.get('permissions') or {}
|
| 67 |
-
edit = perms.get('edit')
|
| 68 |
-
if edit == 'collaborative':
|
| 69 |
-
return True
|
| 70 |
-
if edit == 'users':
|
| 71 |
-
return viewer in set(perms.get('users') or ())
|
| 72 |
-
return False # 'personal', absent, or junk
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
def _may_edit(view, viewer, is_admin=False):
|
| 76 |
-
"""Who may WRITE a shared view. Same set as visibility today β the owner's item asks 'who
|
| 77 |
-
can edit' and lists who 'can have access', i.e. seeing and editing are one grant. Kept as a
|
| 78 |
-
separate function so they can diverge (a future read-only share) without hunting callers."""
|
| 79 |
-
return _may_see(view, viewer, is_admin)
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
def _may_administer(view, viewer, is_admin=False):
|
| 83 |
-
"""Who may change a view's PERMISSIONS, or delete it: the creator or an admin ONLY.
|
| 84 |
-
|
| 85 |
-
Deliberately narrower than _may_edit. If a collaborator could rewrite `permissions` they
|
| 86 |
-
could grant themselves sole ownership of somebody else's view, or quietly widen a
|
| 87 |
-
users-scoped view to everyone β the classic privilege-escalation-by-edit hole.
|
| 88 |
-
"""
|
| 89 |
-
if not isinstance(view, dict):
|
| 90 |
-
return False
|
| 91 |
-
return bool(is_admin) or view.get('createdBy') == viewer
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
def is_shared(view):
|
| 95 |
-
"""A view belongs in the shared bucket when its permissions reach beyond its creator."""
|
| 96 |
-
return ((view or {}).get('permissions') or {}).get('edit') in ('collaborative', 'users')
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
def _shared_fields(data):
|
| 100 |
-
"""The tenant-wide field stratum of one workspace document β `{field_key: {'agg': β¦}}`.
|
| 101 |
-
|
| 102 |
-
β TOTAL AND FAIL-SOFT: a document written before W36-T25 has no `__shared__` member at all,
|
| 103 |
-
and one written by a future version may have something else in it. Either way this answers an
|
| 104 |
-
empty mapping rather than raising, because the caller is a READ that must still serve the
|
| 105 |
-
workspace β a column with no summary is the state every column was in yesterday.
|
| 106 |
-
"""
|
| 107 |
-
shared = (data or {}).get(SHARED_KEY)
|
| 108 |
-
fields = (shared or {}).get('fields') if isinstance(shared, dict) else None
|
| 109 |
-
return fields if isinstance(fields, dict) else {}
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
def split_shared(payload):
|
| 113 |
-
"""`(per_user, shared)` β one field payload divided at the strata boundary.
|
| 114 |
-
|
| 115 |
-
β W36-T25. `shared` carries only `SHARED_FIELD_KEYS` that are actually SET; `per_user` is the
|
| 116 |
-
payload without them. Split HERE rather than at each write door so the three doors that store
|
| 117 |
-
a field (`save_field`, `duplicate_field`, and the delete that must clear it) cannot come apart
|
| 118 |
-
about where a summary lives β which is the whole reason this module exists rather than being
|
| 119 |
-
copied per topic.
|
| 120 |
-
"""
|
| 121 |
-
payload = dict(payload or {})
|
| 122 |
-
shared = {}
|
| 123 |
-
for key in SHARED_FIELD_KEYS:
|
| 124 |
-
value = str(payload.pop(key, '') or '').strip()
|
| 125 |
-
if value:
|
| 126 |
-
shared[key] = value
|
| 127 |
-
return payload, shared
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
def source_override_is_empty(payload):
|
| 131 |
-
"""Does this stored definition of a SOURCE (non-custom) column carry any user state?
|
| 132 |
-
|
| 133 |
-
A cleared note on an immutable source field returns to the canonical schema instead of
|
| 134 |
-
leaving a meaningless override row. Custom fields remain even with an empty note β and so
|
| 135 |
-
does a PRESET field carrying a measure-window override (wave-2 item 8), a DISPLAY-format
|
| 136 |
-
override (wave-5 item 10), and, since W29-T83, a COLUMN SUMMARY.
|
| 137 |
-
|
| 138 |
-
β EVERY CLAUSE IS A SETTING A USER MADE, and each one omitted is a setting that silently
|
| 139 |
-
stops surviving a session. `agg` was missing: choosing Average on Customer's `Overdue days`
|
| 140 |
-
built an override whose only content was that summary, so this rule threw the whole row away
|
| 141 |
-
on write while the menu went on reading "Summary: Average" from the client's own optimistic
|
| 142 |
-
copy until the next login β a discarded WRITE wearing the face of a failed read
|
| 143 |
-
([[lost-write-looks-like-failed-read]]). Measured on `bac40c2`; a `ut_*` table, which stores
|
| 144 |
-
its definitions through another door entirely, kept it.
|
| 145 |
-
|
| 146 |
-
β ONE RULE, TWO CALLERS β here and `grid_events`' store-less fallback. Two copies of a
|
| 147 |
-
discard rule is how one of them keeps a write the other bins ([[one-evaluator-per-question]]).
|
| 148 |
-
β A CLEARED summary still drops the row, which is the intent: with nothing else set, the
|
| 149 |
-
column goes back to whatever the contract declares for it.
|
| 150 |
-
"""
|
| 151 |
-
payload = payload or {}
|
| 152 |
-
if payload.get('custom'):
|
| 153 |
-
return False
|
| 154 |
-
return (not str(payload.get('note') or '').strip()
|
| 155 |
-
and not isinstance(payload.get('measure'), dict)
|
| 156 |
-
and not isinstance(payload.get('format'), dict)
|
| 157 |
-
and not str(payload.get('agg') or '').strip())
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
def _unique_name(wanted, existing, *, fallback='Untitled', max_len=120):
|
| 161 |
-
"""Allocate one human-facing name inside a store.update transaction.
|
| 162 |
-
|
| 163 |
-
Keys/ids remain structural identity. Names compare case-insensitively after collapsing
|
| 164 |
-
whitespace, because those variants are indistinguishable in the UI. This helper belongs
|
| 165 |
-
in the store layer: allocating from a pre-write snapshot lets two concurrent requests both
|
| 166 |
-
choose the same free name before either write lands.
|
| 167 |
-
"""
|
| 168 |
-
limit = max(1, int(max_len))
|
| 169 |
-
|
| 170 |
-
def _clean(value):
|
| 171 |
-
return ' '.join(str(value or '').split())
|
| 172 |
-
|
| 173 |
-
base = (_clean(wanted) or _clean(fallback) or 'Untitled')[:limit].rstrip()
|
| 174 |
-
taken = {_clean(value).casefold() for value in existing if _clean(value)}
|
| 175 |
-
if base.casefold() not in taken:
|
| 176 |
-
return base
|
| 177 |
-
index = 2
|
| 178 |
-
while True:
|
| 179 |
-
suffix = f' {index}'
|
| 180 |
-
stem = base[:max(0, limit - len(suffix))].rstrip()
|
| 181 |
-
candidate = f'{stem}{suffix}' if stem else str(index)[-limit:]
|
| 182 |
-
if candidate.casefold() not in taken:
|
| 183 |
-
return candidate
|
| 184 |
-
index += 1
|
| 185 |
-
|
| 186 |
-
|
| 187 |
-
class TableStore:
|
| 188 |
-
"""The six store operations for one table object's workspace, closed over its store key.
|
| 189 |
-
|
| 190 |
-
`st` (wave 18, C3-UT) is the STORE HANDLE β anything exposing `get(name)` /
|
| 191 |
-
`update(name, fn, flush=)`. Default = `core.store` (tenant #0, every existing caller,
|
| 192 |
-
zero behaviour change). The API passes the session's `TenantRuntime`, whose accessors
|
| 193 |
-
apply the tenant prefix / repo binding β which is what makes a user table created by a
|
| 194 |
-
Nurilab admin land in Nurilab's store instead of Royal's.
|
| 195 |
-
"""
|
| 196 |
-
|
| 197 |
-
def __init__(self, table_key, st=None):
|
| 198 |
-
self.table_key = table_key
|
| 199 |
-
self._st = st if st is not None else store
|
| 200 |
-
|
| 201 |
-
@property
|
| 202 |
-
def st(self):
|
| 203 |
-
"""The bound store handle β for SIBLING registries (core/shares) that must read the
|
| 204 |
-
same tenant's buckets this workspace lives in (wave 21, C1)."""
|
| 205 |
-
return self._st
|
| 206 |
-
|
| 207 |
-
def find_view(self, view_id):
|
| 208 |
-
"""`(owner_username, view)` for a view living in ANY personal stratum, else None.
|
| 209 |
-
|
| 210 |
-
β Wave 21 (item 9, C1): the R10 grant registry names bare ids, so projecting a granted
|
| 211 |
-
view means locating the OWNER's record inside this topic's bucket. Personal strata
|
| 212 |
-
only β the `__shared__` bucket has its own read path (`shared_views`), and serving one
|
| 213 |
-
view from two finders is how two copies drift."""
|
| 214 |
-
vid = str(view_id or '').strip()
|
| 215 |
-
if not vid:
|
| 216 |
-
return None
|
| 217 |
-
try:
|
| 218 |
-
data = self._st.get(self.table_key) or {}
|
| 219 |
-
except Exception:
|
| 220 |
-
return None
|
| 221 |
-
for username, ws in data.items():
|
| 222 |
-
if username == SHARED_KEY or not isinstance(ws, dict):
|
| 223 |
-
continue
|
| 224 |
-
v = (ws.get('views') or {}).get(vid)
|
| 225 |
-
if isinstance(v, dict):
|
| 226 |
-
return str(username), dict(v)
|
| 227 |
-
return None
|
| 228 |
-
|
| 229 |
-
def find_folder(self, folder_id):
|
| 230 |
-
"""`(owner_username, folder_row, {view_id: view})` for a VIEWS folder living in any
|
| 231 |
-
personal stratum, else None. `find_view`'s sibling, and here for the same reason.
|
| 232 |
-
|
| 233 |
-
β D-37 (wave 20's R10 remainder, closed 2026-08-05): the grant registry accepts kind
|
| 234 |
-
`folder` and has since wave 20, but only the VIEW kind was ever projected β so "share
|
| 235 |
-
this folder with Karen" recorded a row, listed under Shared with me, and put nothing on
|
| 236 |
-
Karen's screen. Projecting a folder means two lookups the view path does not need: WHO
|
| 237 |
-
owns it, and WHICH views are filed in it. Folder membership lives in the owner's
|
| 238 |
-
`itemFolders` map (item id -> folder id), never on the view record, so the views are
|
| 239 |
-
found by asking that map rather than by reading a list off the folder.
|
| 240 |
-
|
| 241 |
-
Views only (`folders['views']`): the cohort surface has its own store and its own
|
| 242 |
-
sharing question, and answering both here would make one function mean two things.
|
| 243 |
-
"""
|
| 244 |
-
fid = str(folder_id or '').strip()
|
| 245 |
-
if not fid:
|
| 246 |
-
return None
|
| 247 |
-
try:
|
| 248 |
-
data = self._st.get(self.table_key) or {}
|
| 249 |
-
except Exception:
|
| 250 |
-
return None
|
| 251 |
-
for username, ws in data.items():
|
| 252 |
-
if username == SHARED_KEY or not isinstance(ws, dict):
|
| 253 |
-
continue
|
| 254 |
-
rows = (ws.get('folders') or {}).get('views') or []
|
| 255 |
-
hit = next((f for f in rows
|
| 256 |
-
if isinstance(f, dict) and str(f.get('id') or '') == fid), None)
|
| 257 |
-
if not hit:
|
| 258 |
-
continue
|
| 259 |
-
# β `itemFolders` IS KEYED BY SURFACE FIRST (`{'views': {itemId: folderId}, β¦}`) β
|
| 260 |
-
# reading item ids off the top level finds the surface names instead and matches
|
| 261 |
-
# nothing, so the projection silently returns an EMPTY folder and the feature looks
|
| 262 |
-
# exactly as broken as it was before the fix. Caught by this change's own gate,
|
| 263 |
-
# which is the entire argument for writing one.
|
| 264 |
-
placed = (ws.get('itemFolders') or {}).get('views') or {}
|
| 265 |
-
views = ws.get('views') or {}
|
| 266 |
-
inside = {str(vid): dict(v) for vid, v in views.items()
|
| 267 |
-
if isinstance(v, dict) and str(placed.get(str(vid)) or '') == fid}
|
| 268 |
-
return str(username), dict(hit), inside
|
| 269 |
-
return None
|
| 270 |
-
|
| 271 |
-
# ---------------------------------------------------------------- read
|
| 272 |
-
def workspace(self, username, consume_corrections=True):
|
| 273 |
-
"""One user's durable workspace: always the full three-strata shape."""
|
| 274 |
-
try:
|
| 275 |
-
data = self._st.get(self.table_key) or {}
|
| 276 |
-
ws = data.get(username, {}) or {}
|
| 277 |
-
# A collision acknowledgement is protocol state, not part of a field definition.
|
| 278 |
-
# Consume it with the first fresh workspace payload after the correcting write,
|
| 279 |
-
# then splice a bounded copy into that payload only. Keeping it out of `fields`
|
| 280 |
-
# prevents an old request id surviving forever and overriding a later rename.
|
| 281 |
-
corrections = {}
|
| 282 |
-
if consume_corrections and ws.get('fieldCorrections'):
|
| 283 |
-
def _take(current):
|
| 284 |
-
current_ws = current.get(username) or {}
|
| 285 |
-
pending = current_ws.get('fieldCorrections') or {}
|
| 286 |
-
corrections.update({
|
| 287 |
-
str(key)[:80]: dict(value)
|
| 288 |
-
for key, value in pending.items()
|
| 289 |
-
if isinstance(value, dict)
|
| 290 |
-
})
|
| 291 |
-
current_ws.pop('fieldCorrections', None)
|
| 292 |
-
return current
|
| 293 |
-
|
| 294 |
-
data = self._st.update(self.table_key, _take, flush='async')
|
| 295 |
-
ws = (data or {}).get(username, {}) or {}
|
| 296 |
-
except Exception:
|
| 297 |
-
data = {}
|
| 298 |
-
ws = {}
|
| 299 |
-
corrections = {}
|
| 300 |
-
fields = {
|
| 301 |
-
key: dict(value) if isinstance(value, dict) else value
|
| 302 |
-
for key, value in (ws.get('fields') or {}).items()
|
| 303 |
-
}
|
| 304 |
-
# ββ W36-T25 β THE TENANT-WIDE COLUMN SUMMARY, MERGED OVER THIS USER'S STRATUM.
|
| 305 |
-
# β IT MUST BE ABLE TO CREATE AN ENTRY, not only decorate one, and that is the whole
|
| 306 |
-
# reason this is a merge rather than a lookup: the colleague who never touched the column
|
| 307 |
-
# has NO per-user record for it, so a decorate-only pass would have left them with exactly
|
| 308 |
-
# the blank totals row the owner reported. `aios_grid.workspace_wire` reads `meta['agg']`
|
| 309 |
-
# off whatever is here, base column or custom one alike.
|
| 310 |
-
for key, shared in _shared_fields(data).items():
|
| 311 |
-
agg = str((shared or {}).get('agg') or '').strip()
|
| 312 |
-
if not agg:
|
| 313 |
-
continue
|
| 314 |
-
entry = fields.get(key)
|
| 315 |
-
fields[key] = {**entry, 'agg': agg} if isinstance(entry, dict) else {'agg': agg}
|
| 316 |
-
for key, ack in corrections.items():
|
| 317 |
-
field = fields.get(key)
|
| 318 |
-
accepted_label = str(ack.get('label') or '')[:120]
|
| 319 |
-
requested_label = str(ack.get('labelCorrectedFrom') or '')[:120]
|
| 320 |
-
correction_id = str(ack.get('labelCorrectionId') or '')[:180]
|
| 321 |
-
# A newer field write clears/replaces the pending ack in the SAME transaction.
|
| 322 |
-
# The label check is an extra belt against ever attaching a stale ack to a newer
|
| 323 |
-
# definition if a future store implementation weakens that ordering.
|
| 324 |
-
if (isinstance(field, dict) and accepted_label
|
| 325 |
-
and str(field.get('label') or '') == accepted_label
|
| 326 |
-
and requested_label and correction_id):
|
| 327 |
-
field['labelCorrectedFrom'] = requested_label
|
| 328 |
-
field['labelCorrectionId'] = correction_id
|
| 329 |
-
out = {
|
| 330 |
-
'views': dict(ws.get('views') or {}),
|
| 331 |
-
'fields': fields,
|
| 332 |
-
'overlays': dict(ws.get('overlays') or {}),
|
| 333 |
-
# wave-8 I11 (C4): folders over the saved views / cohorts sidebars. A FOURTH
|
| 334 |
-
# stratum rather than a key on each item β see aios_grid.clean_folders for why
|
| 335 |
-
# (a cohort lives in another store, and filing is an organising act, not part of
|
| 336 |
-
# what a view is). Absent for every workspace saved before this wave, which is
|
| 337 |
-
# exactly "no folders yet".
|
| 338 |
-
'folders': dict(ws.get('folders') or {}),
|
| 339 |
-
'itemFolders': dict(ws.get('itemFolders') or {}),
|
| 340 |
-
}
|
| 341 |
-
# 2026-07-31 (owner item 3): WHERE THE USER LEFT OFF survives a new browser. The
|
| 342 |
-
# client's localStorage copy wins when present; this is the server's answer for a
|
| 343 |
-
# fresh profile, which used to fall all the way to the system default view.
|
| 344 |
-
if ws.get('activeViewId'):
|
| 345 |
-
out['activeViewId'] = str(ws['activeViewId'])
|
| 346 |
-
# Wave 2026-08-02 (C-LAYOUT): the per-user record-detail field order. A fifth
|
| 347 |
-
# stratum, absent until the user first reorders β exactly "default order".
|
| 348 |
-
if isinstance(ws.get('recordLayout'), dict):
|
| 349 |
-
out['recordLayout'] = dict(ws['recordLayout'])
|
| 350 |
-
return out
|
| 351 |
-
|
| 352 |
-
# ---------------------------------------------------------------- write
|
| 353 |
-
def _update(self, username, change, shared=None):
|
| 354 |
-
"""Apply `change(ws)` to this user's stratum, and `shared(shared_fields)` to the
|
| 355 |
-
tenant-wide one, in ONE transaction.
|
| 356 |
-
|
| 357 |
-
β W36-T25 β `shared` IS A SECOND CALLBACK RATHER THAN A SECOND `update`, and that is the
|
| 358 |
-
whole reason it exists here instead of at the caller. `save_field` writes a note (per
|
| 359 |
-
user) and a column summary (tenant-wide) from ONE payload; two transactions would let the
|
| 360 |
-
summary land while the note did not, and the store's own commit is asynchronous, so the
|
| 361 |
-
window is real rather than theoretical. One `update`, one flush, one failure mode.
|
| 362 |
-
"""
|
| 363 |
-
def _up(data):
|
| 364 |
-
ws = data.setdefault(username, {})
|
| 365 |
-
ws.setdefault('views', {})
|
| 366 |
-
ws.setdefault('fields', {})
|
| 367 |
-
ws.setdefault('overlays', {})
|
| 368 |
-
ws.setdefault('folders', {})
|
| 369 |
-
ws.setdefault('itemFolders', {})
|
| 370 |
-
change(ws)
|
| 371 |
-
if shared is not None:
|
| 372 |
-
shared(data.setdefault(SHARED_KEY, {}).setdefault('fields', {}))
|
| 373 |
-
return data
|
| 374 |
-
# flush='async' (wave-7 W3): this is THE hot path β every autosaved filter tweak,
|
| 375 |
-
# column note and typed overlay cell lands here inside the component round-trip, and
|
| 376 |
-
# the historical synchronous hub commit cost seconds per edit. The mutation applies to
|
| 377 |
-
# the in-process cache (read-your-writes for every subsequent render); the hub write
|
| 378 |
-
# coalesces in the background. Registry/auth writes elsewhere stay flush='sync'.
|
| 379 |
-
return self._st.update(self.table_key, _up, flush='async')
|
| 380 |
-
|
| 381 |
-
def rename_choice_values(self, username, change):
|
| 382 |
-
"""Apply an arbitrary workspace rewrite (wave 20, item 15 / C-RENAME).
|
| 383 |
-
|
| 384 |
-
β NAMED FOR ITS ONE CALLER RATHER THAN EXPOSED AS A GENERIC `mutate`, deliberately. A
|
| 385 |
-
public "do anything to the workspace" method is an invitation to put write logic in
|
| 386 |
-
callers instead of here, and every OTHER method on this class exists precisely because
|
| 387 |
-
that logic belongs in one place. Renaming a choice is the one operation that must touch
|
| 388 |
-
three strata AT ONCE β the field's `choices`, the cells in `overlays`, and the views that
|
| 389 |
-
filter or colour by the old value β inside a SINGLE transaction, because a rename that
|
| 390 |
-
updated the cells and not the filters would leave a saved view matching nothing.
|
| 391 |
-
|
| 392 |
-
`change(ws)` receives the whole workspace with every stratum pre-created (see `_update`).
|
| 393 |
-
"""
|
| 394 |
-
return self._update(username, change)
|
| 395 |
-
|
| 396 |
-
def save_active_view(self, username, view_id):
|
| 397 |
-
"""Remember which view this user last opened (owner item 3, 2026-07-31).
|
| 398 |
-
|
| 399 |
-
Presentation state, not authorisation: the READ side re-validates the id against what
|
| 400 |
-
the caller may actually see, so a stale or foreign id degrades to the default view
|
| 401 |
-
rather than granting anything. Stored per user like every other stratum.
|
| 402 |
-
"""
|
| 403 |
-
vid = str(view_id or '').strip()[:120]
|
| 404 |
-
if not vid or username == SHARED_KEY:
|
| 405 |
-
return
|
| 406 |
-
|
| 407 |
-
def _set(ws):
|
| 408 |
-
ws['activeViewId'] = vid
|
| 409 |
-
self._update(username, _set)
|
| 410 |
-
|
| 411 |
-
def save_record_layout(self, username, order):
|
| 412 |
-
"""The per-user RECORD-DETAIL field order (wave 2026-08-02, C-LAYOUT).
|
| 413 |
-
|
| 414 |
-
Presentation state for ONE surface β the record modal. Deliberately not view config:
|
| 415 |
-
the owner's ask is per-user, not per-view, and it must never reorder grid columns.
|
| 416 |
-
The event handler validated keys against the live field set; the wire re-validates at
|
| 417 |
-
serve time (aios_grid.workspace_wire), so a deleted field cannot outlive itself here.
|
| 418 |
-
An empty order clears the stratum back to "default order".
|
| 419 |
-
"""
|
| 420 |
-
if username == SHARED_KEY:
|
| 421 |
-
return
|
| 422 |
-
clean, seen = [], set()
|
| 423 |
-
for key in (order or [])[:200]:
|
| 424 |
-
key = str(key or '').strip()[:80]
|
| 425 |
-
if key and key not in seen:
|
| 426 |
-
seen.add(key)
|
| 427 |
-
clean.append(key)
|
| 428 |
-
|
| 429 |
-
def _set(ws):
|
| 430 |
-
if clean:
|
| 431 |
-
ws['recordLayout'] = {'order': clean}
|
| 432 |
-
else:
|
| 433 |
-
ws.pop('recordLayout', None)
|
| 434 |
-
|
| 435 |
-
self._update(username, _set)
|
| 436 |
-
|
| 437 |
-
def save_folders(self, username, folders, item_folders):
|
| 438 |
-
"""Replace the folder stratum wholesale (wave-8 I11).
|
| 439 |
-
|
| 440 |
-
Wholesale rather than per-folder because the caller has ALREADY validated the complete
|
| 441 |
-
picture through aios_grid.clean_folders / clean_item_folders, and those two are
|
| 442 |
-
interdependent: a placement is only legal while its folder exists, so committing them
|
| 443 |
-
separately would leave a window where a reader sees an item filed into a folder that is
|
| 444 |
-
not there yet. One write, one consistent state.
|
| 445 |
-
"""
|
| 446 |
-
def _set(ws):
|
| 447 |
-
ws['folders'] = dict(folders or {})
|
| 448 |
-
ws['itemFolders'] = dict(item_folders or {})
|
| 449 |
-
self._update(username, _set)
|
| 450 |
-
|
| 451 |
-
def save_view_order(self, username, order):
|
| 452 |
-
"""β WAVE-27 item 5 (contract C7) β this user's own ORDER for the views rail.
|
| 453 |
-
|
| 454 |
-
Wholesale, like `save_folders` above and for the same reason: the client sends the full
|
| 455 |
-
list it is looking at, not a delta, because a partial order cannot say where an UNNAMED
|
| 456 |
-
view went.
|
| 457 |
-
|
| 458 |
-
β PER USER, and it belongs in this stratum rather than on the view records themselves.
|
| 459 |
-
`aios_grid`'s own folder note argues it out for placements and every word applies: an
|
| 460 |
-
arrangement is a per-user ORGANISING act, not part of what a view IS β so keeping it out
|
| 461 |
-
of the view config means duplicating, sharing or exporting a view does not drag one
|
| 462 |
-
person's rail position along with it. It also means a SHARED view can sit in a different
|
| 463 |
-
place for each person who can see it, which is the only coherent answer once two people
|
| 464 |
-
share one view.
|
| 465 |
-
|
| 466 |
-
An empty list CLEARS the arrangement (back to server order) rather than storing `[]`.
|
| 467 |
-
"""
|
| 468 |
-
def _set(ws):
|
| 469 |
-
clean = []
|
| 470 |
-
seen = set()
|
| 471 |
-
for vid in (order or []):
|
| 472 |
-
vid = str(vid).strip()[:120]
|
| 473 |
-
if vid and vid not in seen:
|
| 474 |
-
seen.add(vid)
|
| 475 |
-
clean.append(vid)
|
| 476 |
-
if clean:
|
| 477 |
-
ws['viewOrder'] = clean
|
| 478 |
-
else:
|
| 479 |
-
ws.pop('viewOrder', None)
|
| 480 |
-
self._update(username, _set)
|
| 481 |
-
|
| 482 |
-
def shared_views(self, viewer, is_admin=False):
|
| 483 |
-
"""Every SHARED view this viewer may see, by id (wave-9 I17).
|
| 484 |
-
|
| 485 |
-
Read-only and independent of the viewer's own workspace: the caller merges. Returns
|
| 486 |
-
only what `_may_see` allows, so a caller cannot accidentally render somebody else's
|
| 487 |
-
personal view by forgetting to filter.
|
| 488 |
-
"""
|
| 489 |
-
try:
|
| 490 |
-
bucket = ((self._st.get(self.table_key) or {}).get(SHARED_KEY) or {}).get('views') or {}
|
| 491 |
-
except Exception:
|
| 492 |
-
return {}
|
| 493 |
-
return {vid: dict(v) for vid, v in bucket.items()
|
| 494 |
-
if _may_see(v, viewer, is_admin)}
|
| 495 |
-
|
| 496 |
-
def shared_view(self, view_id):
|
| 497 |
-
"""One shared view RAW β no visibility filter. For authorisation decisions only: a
|
| 498 |
-
caller must know a view exists and who owns it before it can decide whether the actor
|
| 499 |
-
may touch it. Never hand the result to a renderer without checking `_may_see`."""
|
| 500 |
-
try:
|
| 501 |
-
return ((self._st.get(self.table_key) or {}).get(SHARED_KEY) or {}
|
| 502 |
-
).get('views', {}).get(str(view_id))
|
| 503 |
-
except Exception:
|
| 504 |
-
return None
|
| 505 |
-
|
| 506 |
-
def save_view(self, username, view, shared=None, reserved_names=(), is_admin=False):
|
| 507 |
-
"""Upsert a SavedView into its ONE home β personal workspace or the shared bucket.
|
| 508 |
-
|
| 509 |
-
`shared` defaults to reading the view's own permissions (`is_shared`). Whichever home
|
| 510 |
-
it lands in, the view is REMOVED from the other, so a view can never exist as two
|
| 511 |
-
copies that diverge on the next edit.
|
| 512 |
-
|
| 513 |
-
β AUTHORISATION IS THE CALLER'S JOB and must happen BEFORE this is called β this layer
|
| 514 |
-
moves data and does not know who is asking. `_cl_handle_one` is the wall.
|
| 515 |
-
"""
|
| 516 |
-
view_id = str((view or {}).get('id') or '').strip()
|
| 517 |
-
if not view_id:
|
| 518 |
-
raise ValueError('view id is required')
|
| 519 |
-
if username == SHARED_KEY:
|
| 520 |
-
raise ValueError('reserved username')
|
| 521 |
-
to_shared = is_shared(view) if shared is None else bool(shared)
|
| 522 |
-
requested = dict(view)
|
| 523 |
-
accepted = {}
|
| 524 |
-
|
| 525 |
-
def _up(data):
|
| 526 |
-
# View names are tenant-global: every personal workspace plus the shared bucket.
|
| 527 |
-
# This deliberately includes views the actor cannot see. The only disclosed fact
|
| 528 |
-
# is that a display name is already taken, while the categorical "no duplicate
|
| 529 |
-
# view names" contract remains true when a personal view is later shared.
|
| 530 |
-
names = list(reserved_names or ())
|
| 531 |
-
for workspace in data.values():
|
| 532 |
-
if not isinstance(workspace, dict):
|
| 533 |
-
continue
|
| 534 |
-
names.extend(
|
| 535 |
-
value.get('name')
|
| 536 |
-
for candidate_id, value in (workspace.get('views') or {}).items()
|
| 537 |
-
if candidate_id != view_id and isinstance(value, dict)
|
| 538 |
-
)
|
| 539 |
-
payload = dict(requested)
|
| 540 |
-
payload['name'] = _unique_name(payload.get('name'), names)
|
| 541 |
-
accepted.clear()
|
| 542 |
-
accepted.update(payload)
|
| 543 |
-
if to_shared:
|
| 544 |
-
bucket = data.setdefault(SHARED_KEY, {})
|
| 545 |
-
bucket.setdefault('views', {})[view_id] = payload
|
| 546 |
-
# it may have lived in the creator's workspace before being shared
|
| 547 |
-
owner = data.get(payload.get('createdBy') or username) or {}
|
| 548 |
-
(owner.get('views') or {}).pop(view_id, None)
|
| 549 |
-
else:
|
| 550 |
-
ws = data.setdefault(username, {})
|
| 551 |
-
ws.setdefault('views', {})[view_id] = payload
|
| 552 |
-
(data.get(SHARED_KEY, {}).get('views') or {}).pop(view_id, None)
|
| 553 |
-
return data
|
| 554 |
-
|
| 555 |
-
self._st.update(self.table_key, _up, flush='async')
|
| 556 |
-
return dict(accepted)
|
| 557 |
-
|
| 558 |
-
def delete_view(self, username, view_id):
|
| 559 |
-
"""Delete a custom/list view override. The system all-rows view is guarded by caller.
|
| 560 |
-
|
| 561 |
-
Removes from BOTH homes: the caller has already authorised the delete, and leaving a
|
| 562 |
-
stale copy in the other bucket would resurrect the view on the next read.
|
| 563 |
-
"""
|
| 564 |
-
vid = str(view_id)
|
| 565 |
-
|
| 566 |
-
def _up(data):
|
| 567 |
-
(data.get(username, {}).get('views') or {}).pop(vid, None)
|
| 568 |
-
(data.get(SHARED_KEY, {}).get('views') or {}).pop(vid, None)
|
| 569 |
-
return data
|
| 570 |
-
|
| 571 |
-
self._st.update(self.table_key, _up, flush='async')
|
| 572 |
-
|
| 573 |
-
def save_field(self, username, field, reserved_names=(), correction_id=None):
|
| 574 |
-
"""Persist a column note or a user-created (custom_/measure_) field definition.
|
| 575 |
-
|
| 576 |
-
ββ W36-T25 β THE COLUMN SUMMARY GOES TO THE TENANT-WIDE STRATUM, EVERYTHING ELSE STAYS
|
| 577 |
-
PER USER, in ONE transaction. See `SHARED_FIELD_KEYS` above for the owner's report and for
|
| 578 |
-
why the line falls where it does. The returned `accepted` still carries the summary: the
|
| 579 |
-
caller is echoing back the field it just stored, and dropping a key from that echo would
|
| 580 |
-
tell the client its write was refused ([[read-path-cannot-witness-write-path]]).
|
| 581 |
-
"""
|
| 582 |
-
key = str((field or {}).get('key') or '').strip()
|
| 583 |
-
if not key:
|
| 584 |
-
raise ValueError('field key is required')
|
| 585 |
-
requested = dict(field)
|
| 586 |
-
accepted = {}
|
| 587 |
-
|
| 588 |
-
def _save(ws):
|
| 589 |
-
names = list(reserved_names or ())
|
| 590 |
-
names.extend(
|
| 591 |
-
value.get('label')
|
| 592 |
-
for candidate_key, value in (ws.get('fields') or {}).items()
|
| 593 |
-
if candidate_key != key and isinstance(value, dict)
|
| 594 |
-
)
|
| 595 |
-
payload = dict(requested)
|
| 596 |
-
payload.pop('labelCorrectedFrom', None)
|
| 597 |
-
payload.pop('labelCorrectionId', None)
|
| 598 |
-
requested_label = ' '.join(
|
| 599 |
-
str(payload.get('label') or 'Untitled').split())[:120].rstrip()
|
| 600 |
-
payload['label'] = _unique_name(requested_label, names)
|
| 601 |
-
corrections = ws.setdefault('fieldCorrections', {})
|
| 602 |
-
corrections.pop(key, None)
|
| 603 |
-
if payload['label'] != requested_label and correction_id:
|
| 604 |
-
corrections[key] = {
|
| 605 |
-
'label': payload['label'],
|
| 606 |
-
'labelCorrectedFrom': requested_label,
|
| 607 |
-
'labelCorrectionId': str(correction_id)[:180],
|
| 608 |
-
}
|
| 609 |
-
if not corrections:
|
| 610 |
-
ws.pop('fieldCorrections', None)
|
| 611 |
-
accepted.clear()
|
| 612 |
-
accepted.update(payload)
|
| 613 |
-
# ββ W36-T25 β SPLIT AFTER the label allocation and the correction bookkeeping, so
|
| 614 |
-
# both still see the whole payload, and BEFORE the per-user write.
|
| 615 |
-
mine, shared_now = split_shared(payload)
|
| 616 |
-
shared_write.clear()
|
| 617 |
-
shared_write.update(shared_now)
|
| 618 |
-
# β EMPTINESS IS JUDGED ON THE PER-USER HALF. A source column whose ONLY state was a
|
| 619 |
-
# summary now has no per-user state at all, and leaving an `{}` override behind is the
|
| 620 |
-
# meaningless row `source_override_is_empty` exists to prevent.
|
| 621 |
-
if source_override_is_empty(mine):
|
| 622 |
-
ws['fields'].pop(key, None)
|
| 623 |
-
else:
|
| 624 |
-
ws['fields'][key] = mine
|
| 625 |
-
|
| 626 |
-
shared_write = {}
|
| 627 |
-
|
| 628 |
-
def _share(shared_fields):
|
| 629 |
-
# β A CLEARED SUMMARY MUST REMOVE THE ROW, not leave an empty one: `workspace` treats
|
| 630 |
-
# any entry it finds as a live tenant-wide summary, so an `{'agg': ''}` husk would be
|
| 631 |
-
# skipped today and become a resurrection hazard the moment the read grows a second
|
| 632 |
-
# shared key. Absence is the only honest spelling of "nobody set one".
|
| 633 |
-
if shared_write:
|
| 634 |
-
shared_fields[key] = {**(shared_fields.get(key) or {}), **shared_write}
|
| 635 |
-
else:
|
| 636 |
-
shared_fields.pop(key, None)
|
| 637 |
-
|
| 638 |
-
self._update(username, _save, shared=_share)
|
| 639 |
-
return dict(accepted)
|
| 640 |
-
|
| 641 |
-
def duplicate_field(self, username, source_key, new_key, field,
|
| 642 |
-
reserved_names=(), correction_id=None):
|
| 643 |
-
"""Clone a user-created field in ONE store transaction (wave-5 item 1): the new
|
| 644 |
-
definition plus β for `custom_` overlay sources only β every stored cell value under
|
| 645 |
-
the source key. One transaction, because a def without its values (or values without a
|
| 646 |
-
def) is exactly the orphan state delete_field exists to prevent, in reverse.
|
| 647 |
-
The caller validated both keys (same created stratum) and stamped the clone's
|
| 648 |
-
createdBy; this layer only moves data."""
|
| 649 |
-
source_key = str(source_key or '').strip()
|
| 650 |
-
new_key = str(new_key or '').strip()
|
| 651 |
-
if not source_key or not new_key or source_key == new_key:
|
| 652 |
-
raise ValueError('duplicate_field needs two distinct keys')
|
| 653 |
-
requested = dict(field)
|
| 654 |
-
accepted = {}
|
| 655 |
-
|
| 656 |
-
def _dup(ws):
|
| 657 |
-
names = list(reserved_names or ())
|
| 658 |
-
names.extend(
|
| 659 |
-
value.get('label')
|
| 660 |
-
for candidate_key, value in (ws.get('fields') or {}).items()
|
| 661 |
-
if candidate_key != new_key and isinstance(value, dict)
|
| 662 |
-
)
|
| 663 |
-
payload = dict(requested)
|
| 664 |
-
payload.pop('labelCorrectedFrom', None)
|
| 665 |
-
payload.pop('labelCorrectionId', None)
|
| 666 |
-
requested_label = ' '.join(
|
| 667 |
-
str(payload.get('label') or 'Untitled').split())[:120].rstrip()
|
| 668 |
-
payload['label'] = _unique_name(requested_label, names)
|
| 669 |
-
corrections = ws.setdefault('fieldCorrections', {})
|
| 670 |
-
corrections.pop(new_key, None)
|
| 671 |
-
if payload['label'] != requested_label and correction_id:
|
| 672 |
-
corrections[new_key] = {
|
| 673 |
-
'label': payload['label'],
|
| 674 |
-
'labelCorrectedFrom': requested_label,
|
| 675 |
-
'labelCorrectionId': str(correction_id)[:180],
|
| 676 |
-
}
|
| 677 |
-
if not corrections:
|
| 678 |
-
ws.pop('fieldCorrections', None)
|
| 679 |
-
accepted.clear()
|
| 680 |
-
accepted.update(payload)
|
| 681 |
-
# β W36-T25: a CLONE carries the original's summary, and a summary is the database's
|
| 682 |
-
# (see `SHARED_FIELD_KEYS`). Storing it per user here would give the clone a different
|
| 683 |
-
# residency from every other column β one door out of three disagreeing about where a
|
| 684 |
-
# thing lives is how `save_field` and this function drift.
|
| 685 |
-
mine, shared_now = split_shared(payload)
|
| 686 |
-
shared_write.clear()
|
| 687 |
-
shared_write.update(shared_now)
|
| 688 |
-
ws['fields'][new_key] = mine
|
| 689 |
-
if source_key.startswith('custom_'):
|
| 690 |
-
for row in ws['overlays'].values():
|
| 691 |
-
if isinstance(row, dict) and source_key in row:
|
| 692 |
-
row[new_key] = row[source_key]
|
| 693 |
-
|
| 694 |
-
shared_write = {}
|
| 695 |
-
|
| 696 |
-
def _share(shared_fields):
|
| 697 |
-
if shared_write:
|
| 698 |
-
shared_fields[new_key] = {**(shared_fields.get(new_key) or {}), **shared_write}
|
| 699 |
-
else:
|
| 700 |
-
shared_fields.pop(new_key, None)
|
| 701 |
-
|
| 702 |
-
self._update(username, _dup, shared=_share)
|
| 703 |
-
return dict(accepted)
|
| 704 |
-
|
| 705 |
-
def delete_field(self, username, key):
|
| 706 |
-
"""Delete a USER-CREATED field definition outright (owner gap closed 2026-07-27).
|
| 707 |
-
|
| 708 |
-
Only the created strata ever reach here (`custom_` overlay fields, `measure_` formula
|
| 709 |
-
columns β the caller enforces the prefix). The stored overlay VALUES for the key are
|
| 710 |
-
scrubbed with it: a deleted column's cells must not linger as orphan data that would
|
| 711 |
-
silently resurface if the key were ever reused. Views referencing the key self-heal on
|
| 712 |
-
their next autosave (an unknown colId is dropped) β the rule every stale key rides.
|
| 713 |
-
|
| 714 |
-
ββ W36-T25 β AND THE TENANT-WIDE SUMMARY GOES WITH IT, for exactly the reason the
|
| 715 |
-
paragraph above gives about cells: an orphan `agg` under a deleted key is state nobody can
|
| 716 |
-
see and nobody can clear, and it would attach itself to the next column that happens to
|
| 717 |
-
take the key back. β This is the ONE stratum a per-user delete may reach across accounts,
|
| 718 |
-
and it is safe because the summary was never this user's to begin with β deleting the
|
| 719 |
-
COLUMN is a tenant-wide act already.
|
| 720 |
-
"""
|
| 721 |
-
key = str(key or '').strip()
|
| 722 |
-
if not key:
|
| 723 |
-
return
|
| 724 |
-
|
| 725 |
-
def _drop(ws):
|
| 726 |
-
ws['fields'].pop(key, None)
|
| 727 |
-
for row in ws['overlays'].values():
|
| 728 |
-
if isinstance(row, dict):
|
| 729 |
-
row.pop(key, None)
|
| 730 |
-
|
| 731 |
-
self._update(username, _drop, shared=lambda shared_fields: shared_fields.pop(key, None))
|
| 732 |
-
|
| 733 |
-
def patch_overlay(self, username, pid, updates):
|
| 734 |
-
"""Patch only the external editable stratum; never writes to the source system."""
|
| 735 |
-
clean = dict(updates or {})
|
| 736 |
-
if not clean:
|
| 737 |
-
return
|
| 738 |
-
|
| 739 |
-
def _patch(ws):
|
| 740 |
-
ws['overlays'].setdefault(str(int(pid)), {}).update(clean)
|
| 741 |
-
|
| 742 |
-
self._update(username, _patch)
|
| 743 |
-
|
| 744 |
-
|
| 745 |
-
def make(table_key, st=None):
|
| 746 |
-
return TableStore(table_key, st=st)
|
|
|
|
| 1 |
+
"""The generic per-user TABLE WORKSPACE store β the persistence half of the table-page factory.
|
| 2 |
+
|
| 3 |
+
One durable store key holds one table OBJECT's per-user Airtable-style state:
|
| 4 |
+
|
| 5 |
+
{username: {'views': {view_id: SavedView},
|
| 6 |
+
'fields': {field_key: Field}, # notes + custom_ + measure_ strata
|
| 7 |
+
'overlays': {str(pid): {field_key: value}}}}
|
| 8 |
+
|
| 9 |
+
`make(table_key)` returns the six operations a table page's host loop needs, closed over that
|
| 10 |
+
key. The Customer table's ops (`modules/customer_data.py`, key 'customer_table_workspace') are
|
| 11 |
+
these exact functions β the logic MOVED here 2026-07-27 so that duplicating the Customer table
|
| 12 |
+
pattern to a new object is a registry row + a config, not a copy of the store plumbing
|
| 13 |
+
(owner directive: the table-page factory).
|
| 14 |
+
|
| 15 |
+
A LIST (membership/formula semantics) is deliberately a different store from a VIEW
|
| 16 |
+
(presentation/query state) β see modules/customer_data.py's customer_lists key.
|
| 17 |
+
"""
|
| 18 |
+
import core.store as store
|
| 19 |
+
|
| 20 |
+
#: Wave-9 I17 β the SHARED bucket. Views whose permissions make them visible to anyone but
|
| 21 |
+
#: their creator live here instead of in a personal workspace, under a key that cannot collide
|
| 22 |
+
#: with a username (usernames come from core/users.py and are never dunder-wrapped; `is_shared`
|
| 23 |
+
#: guards it anyway). ONE HOME PER VIEW, never both: a view moved to personal is REMOVED from
|
| 24 |
+
#: here, and a view shared is removed from its creator's workspace. Two homes would mean two
|
| 25 |
+
#: divergent copies the moment either was edited.
|
| 26 |
+
SHARED_KEY = '__shared__'
|
| 27 |
+
|
| 28 |
+
#: ββ W36-T25 β THE COLUMN SUMMARY IS THE DATABASE'S, NOT ONE ACCOUNT'S.
|
| 29 |
+
#:
|
| 30 |
+
#: Owner item 1, second half, verbatim: *"When i filter Avi on my computer sum works for the Sales
|
| 31 |
+
#: last 365 days column but not when my colleague filter it β The sum of the field is not showing
|
| 32 |
+
#: at the bottom even when i zoom out. It only works on my screen??"* Measured while scouting it:
|
| 33 |
+
#: `save_field` writes the WHOLE field payload into `data[username]`, so a column summary β which
|
| 34 |
+
#: is a statement about the COLUMN, identical for every reader by construction β was stored once
|
| 35 |
+
#: per account. Set it, and the totals row exists for you and for nobody else. `computeAggs` then
|
| 36 |
+
#: paints no totals row at all for the colleague (`showTotals` is false when no field carries an
|
| 37 |
+
#: `agg`), which is exactly "not showing at the bottom".
|
| 38 |
+
#:
|
| 39 |
+
#: β ONLY `agg` MOVES, AND THE LINE IS NOT ARBITRARY. Width, column order, the note, the display
|
| 40 |
+
#: format and every `custom_`/`measure_` definition stay per user, because each of those is a
|
| 41 |
+
#: statement about how ONE PERSON reads the column. "Sum this column" is a statement about what
|
| 42 |
+
#: the column MEANS, and two accounts disagreeing about it is the defect, not a preference.
|
| 43 |
+
#:
|
| 44 |
+
#: β IT LIVES IN THE `__shared__` MEMBER OF THIS SAME BUCKET rather than in
|
| 45 |
+
#: `core/shared_overlay.py`'s `<key>__shared` document, and the reason is transactional: a summary
|
| 46 |
+
#: is written by the same `save_field` call that writes the note beside it, and the two must land
|
| 47 |
+
#: or fail together. `__shared__` is already this store's tenant-wide member (shared VIEWS live
|
| 48 |
+
#: there) and is guarded against colliding with a username by `is_shared` and by `core/users.py`'s
|
| 49 |
+
#: never-dunder rule, so there is no second bucket, no second flush and no second failure mode.
|
| 50 |
+
SHARED_FIELD_KEYS = ('agg',)
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def _may_see(view, viewer, is_admin=False):
|
| 54 |
+
"""Visibility for ONE shared view, fail-closed.
|
| 55 |
+
|
| 56 |
+
'collaborative' = everyone who can already open the module (the caller has gated that).
|
| 57 |
+
'users' = the named users, plus the creator, plus admins β an admin who could not
|
| 58 |
+
see a view could not administer it either.
|
| 59 |
+
Anything unrecognised returns False rather than defaulting open: an unreadable permission
|
| 60 |
+
must never widen access ([[aios-permissioning]] β no fail-open defaults).
|
| 61 |
+
"""
|
| 62 |
+
if not isinstance(view, dict):
|
| 63 |
+
return False
|
| 64 |
+
if view.get('createdBy') == viewer or is_admin:
|
| 65 |
+
return True
|
| 66 |
+
perms = view.get('permissions') or {}
|
| 67 |
+
edit = perms.get('edit')
|
| 68 |
+
if edit == 'collaborative':
|
| 69 |
+
return True
|
| 70 |
+
if edit == 'users':
|
| 71 |
+
return viewer in set(perms.get('users') or ())
|
| 72 |
+
return False # 'personal', absent, or junk
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def _may_edit(view, viewer, is_admin=False):
|
| 76 |
+
"""Who may WRITE a shared view. Same set as visibility today β the owner's item asks 'who
|
| 77 |
+
can edit' and lists who 'can have access', i.e. seeing and editing are one grant. Kept as a
|
| 78 |
+
separate function so they can diverge (a future read-only share) without hunting callers."""
|
| 79 |
+
return _may_see(view, viewer, is_admin)
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def _may_administer(view, viewer, is_admin=False):
|
| 83 |
+
"""Who may change a view's PERMISSIONS, or delete it: the creator or an admin ONLY.
|
| 84 |
+
|
| 85 |
+
Deliberately narrower than _may_edit. If a collaborator could rewrite `permissions` they
|
| 86 |
+
could grant themselves sole ownership of somebody else's view, or quietly widen a
|
| 87 |
+
users-scoped view to everyone β the classic privilege-escalation-by-edit hole.
|
| 88 |
+
"""
|
| 89 |
+
if not isinstance(view, dict):
|
| 90 |
+
return False
|
| 91 |
+
return bool(is_admin) or view.get('createdBy') == viewer
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def is_shared(view):
|
| 95 |
+
"""A view belongs in the shared bucket when its permissions reach beyond its creator."""
|
| 96 |
+
return ((view or {}).get('permissions') or {}).get('edit') in ('collaborative', 'users')
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
def _shared_fields(data):
|
| 100 |
+
"""The tenant-wide field stratum of one workspace document β `{field_key: {'agg': β¦}}`.
|
| 101 |
+
|
| 102 |
+
β TOTAL AND FAIL-SOFT: a document written before W36-T25 has no `__shared__` member at all,
|
| 103 |
+
and one written by a future version may have something else in it. Either way this answers an
|
| 104 |
+
empty mapping rather than raising, because the caller is a READ that must still serve the
|
| 105 |
+
workspace β a column with no summary is the state every column was in yesterday.
|
| 106 |
+
"""
|
| 107 |
+
shared = (data or {}).get(SHARED_KEY)
|
| 108 |
+
fields = (shared or {}).get('fields') if isinstance(shared, dict) else None
|
| 109 |
+
return fields if isinstance(fields, dict) else {}
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def split_shared(payload):
|
| 113 |
+
"""`(per_user, shared)` β one field payload divided at the strata boundary.
|
| 114 |
+
|
| 115 |
+
β W36-T25. `shared` carries only `SHARED_FIELD_KEYS` that are actually SET; `per_user` is the
|
| 116 |
+
payload without them. Split HERE rather than at each write door so the three doors that store
|
| 117 |
+
a field (`save_field`, `duplicate_field`, and the delete that must clear it) cannot come apart
|
| 118 |
+
about where a summary lives β which is the whole reason this module exists rather than being
|
| 119 |
+
copied per topic.
|
| 120 |
+
"""
|
| 121 |
+
payload = dict(payload or {})
|
| 122 |
+
shared = {}
|
| 123 |
+
for key in SHARED_FIELD_KEYS:
|
| 124 |
+
value = str(payload.pop(key, '') or '').strip()
|
| 125 |
+
if value:
|
| 126 |
+
shared[key] = value
|
| 127 |
+
return payload, shared
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
def source_override_is_empty(payload):
|
| 131 |
+
"""Does this stored definition of a SOURCE (non-custom) column carry any user state?
|
| 132 |
+
|
| 133 |
+
A cleared note on an immutable source field returns to the canonical schema instead of
|
| 134 |
+
leaving a meaningless override row. Custom fields remain even with an empty note β and so
|
| 135 |
+
does a PRESET field carrying a measure-window override (wave-2 item 8), a DISPLAY-format
|
| 136 |
+
override (wave-5 item 10), and, since W29-T83, a COLUMN SUMMARY.
|
| 137 |
+
|
| 138 |
+
β EVERY CLAUSE IS A SETTING A USER MADE, and each one omitted is a setting that silently
|
| 139 |
+
stops surviving a session. `agg` was missing: choosing Average on Customer's `Overdue days`
|
| 140 |
+
built an override whose only content was that summary, so this rule threw the whole row away
|
| 141 |
+
on write while the menu went on reading "Summary: Average" from the client's own optimistic
|
| 142 |
+
copy until the next login β a discarded WRITE wearing the face of a failed read
|
| 143 |
+
([[lost-write-looks-like-failed-read]]). Measured on `bac40c2`; a `ut_*` table, which stores
|
| 144 |
+
its definitions through another door entirely, kept it.
|
| 145 |
+
|
| 146 |
+
β ONE RULE, TWO CALLERS β here and `grid_events`' store-less fallback. Two copies of a
|
| 147 |
+
discard rule is how one of them keeps a write the other bins ([[one-evaluator-per-question]]).
|
| 148 |
+
β A CLEARED summary still drops the row, which is the intent: with nothing else set, the
|
| 149 |
+
column goes back to whatever the contract declares for it.
|
| 150 |
+
"""
|
| 151 |
+
payload = payload or {}
|
| 152 |
+
if payload.get('custom'):
|
| 153 |
+
return False
|
| 154 |
+
return (not str(payload.get('note') or '').strip()
|
| 155 |
+
and not isinstance(payload.get('measure'), dict)
|
| 156 |
+
and not isinstance(payload.get('format'), dict)
|
| 157 |
+
and not str(payload.get('agg') or '').strip())
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def _unique_name(wanted, existing, *, fallback='Untitled', max_len=120):
|
| 161 |
+
"""Allocate one human-facing name inside a store.update transaction.
|
| 162 |
+
|
| 163 |
+
Keys/ids remain structural identity. Names compare case-insensitively after collapsing
|
| 164 |
+
whitespace, because those variants are indistinguishable in the UI. This helper belongs
|
| 165 |
+
in the store layer: allocating from a pre-write snapshot lets two concurrent requests both
|
| 166 |
+
choose the same free name before either write lands.
|
| 167 |
+
"""
|
| 168 |
+
limit = max(1, int(max_len))
|
| 169 |
+
|
| 170 |
+
def _clean(value):
|
| 171 |
+
return ' '.join(str(value or '').split())
|
| 172 |
+
|
| 173 |
+
base = (_clean(wanted) or _clean(fallback) or 'Untitled')[:limit].rstrip()
|
| 174 |
+
taken = {_clean(value).casefold() for value in existing if _clean(value)}
|
| 175 |
+
if base.casefold() not in taken:
|
| 176 |
+
return base
|
| 177 |
+
index = 2
|
| 178 |
+
while True:
|
| 179 |
+
suffix = f' {index}'
|
| 180 |
+
stem = base[:max(0, limit - len(suffix))].rstrip()
|
| 181 |
+
candidate = f'{stem}{suffix}' if stem else str(index)[-limit:]
|
| 182 |
+
if candidate.casefold() not in taken:
|
| 183 |
+
return candidate
|
| 184 |
+
index += 1
|
| 185 |
+
|
| 186 |
+
|
| 187 |
+
class TableStore:
|
| 188 |
+
"""The six store operations for one table object's workspace, closed over its store key.
|
| 189 |
+
|
| 190 |
+
`st` (wave 18, C3-UT) is the STORE HANDLE β anything exposing `get(name)` /
|
| 191 |
+
`update(name, fn, flush=)`. Default = `core.store` (tenant #0, every existing caller,
|
| 192 |
+
zero behaviour change). The API passes the session's `TenantRuntime`, whose accessors
|
| 193 |
+
apply the tenant prefix / repo binding β which is what makes a user table created by a
|
| 194 |
+
Nurilab admin land in Nurilab's store instead of Royal's.
|
| 195 |
+
"""
|
| 196 |
+
|
| 197 |
+
def __init__(self, table_key, st=None):
|
| 198 |
+
self.table_key = table_key
|
| 199 |
+
self._st = st if st is not None else store
|
| 200 |
+
|
| 201 |
+
@property
|
| 202 |
+
def st(self):
|
| 203 |
+
"""The bound store handle β for SIBLING registries (core/shares) that must read the
|
| 204 |
+
same tenant's buckets this workspace lives in (wave 21, C1)."""
|
| 205 |
+
return self._st
|
| 206 |
+
|
| 207 |
+
def find_view(self, view_id):
|
| 208 |
+
"""`(owner_username, view)` for a view living in ANY personal stratum, else None.
|
| 209 |
+
|
| 210 |
+
β Wave 21 (item 9, C1): the R10 grant registry names bare ids, so projecting a granted
|
| 211 |
+
view means locating the OWNER's record inside this topic's bucket. Personal strata
|
| 212 |
+
only β the `__shared__` bucket has its own read path (`shared_views`), and serving one
|
| 213 |
+
view from two finders is how two copies drift."""
|
| 214 |
+
vid = str(view_id or '').strip()
|
| 215 |
+
if not vid:
|
| 216 |
+
return None
|
| 217 |
+
try:
|
| 218 |
+
data = self._st.get(self.table_key) or {}
|
| 219 |
+
except Exception:
|
| 220 |
+
return None
|
| 221 |
+
for username, ws in data.items():
|
| 222 |
+
if username == SHARED_KEY or not isinstance(ws, dict):
|
| 223 |
+
continue
|
| 224 |
+
v = (ws.get('views') or {}).get(vid)
|
| 225 |
+
if isinstance(v, dict):
|
| 226 |
+
return str(username), dict(v)
|
| 227 |
+
return None
|
| 228 |
+
|
| 229 |
+
def find_folder(self, folder_id):
|
| 230 |
+
"""`(owner_username, folder_row, {view_id: view})` for a VIEWS folder living in any
|
| 231 |
+
personal stratum, else None. `find_view`'s sibling, and here for the same reason.
|
| 232 |
+
|
| 233 |
+
β D-37 (wave 20's R10 remainder, closed 2026-08-05): the grant registry accepts kind
|
| 234 |
+
`folder` and has since wave 20, but only the VIEW kind was ever projected β so "share
|
| 235 |
+
this folder with Karen" recorded a row, listed under Shared with me, and put nothing on
|
| 236 |
+
Karen's screen. Projecting a folder means two lookups the view path does not need: WHO
|
| 237 |
+
owns it, and WHICH views are filed in it. Folder membership lives in the owner's
|
| 238 |
+
`itemFolders` map (item id -> folder id), never on the view record, so the views are
|
| 239 |
+
found by asking that map rather than by reading a list off the folder.
|
| 240 |
+
|
| 241 |
+
Views only (`folders['views']`): the cohort surface has its own store and its own
|
| 242 |
+
sharing question, and answering both here would make one function mean two things.
|
| 243 |
+
"""
|
| 244 |
+
fid = str(folder_id or '').strip()
|
| 245 |
+
if not fid:
|
| 246 |
+
return None
|
| 247 |
+
try:
|
| 248 |
+
data = self._st.get(self.table_key) or {}
|
| 249 |
+
except Exception:
|
| 250 |
+
return None
|
| 251 |
+
for username, ws in data.items():
|
| 252 |
+
if username == SHARED_KEY or not isinstance(ws, dict):
|
| 253 |
+
continue
|
| 254 |
+
rows = (ws.get('folders') or {}).get('views') or []
|
| 255 |
+
hit = next((f for f in rows
|
| 256 |
+
if isinstance(f, dict) and str(f.get('id') or '') == fid), None)
|
| 257 |
+
if not hit:
|
| 258 |
+
continue
|
| 259 |
+
# β `itemFolders` IS KEYED BY SURFACE FIRST (`{'views': {itemId: folderId}, β¦}`) β
|
| 260 |
+
# reading item ids off the top level finds the surface names instead and matches
|
| 261 |
+
# nothing, so the projection silently returns an EMPTY folder and the feature looks
|
| 262 |
+
# exactly as broken as it was before the fix. Caught by this change's own gate,
|
| 263 |
+
# which is the entire argument for writing one.
|
| 264 |
+
placed = (ws.get('itemFolders') or {}).get('views') or {}
|
| 265 |
+
views = ws.get('views') or {}
|
| 266 |
+
inside = {str(vid): dict(v) for vid, v in views.items()
|
| 267 |
+
if isinstance(v, dict) and str(placed.get(str(vid)) or '') == fid}
|
| 268 |
+
return str(username), dict(hit), inside
|
| 269 |
+
return None
|
| 270 |
+
|
| 271 |
+
# ---------------------------------------------------------------- read
|
| 272 |
+
def workspace(self, username, consume_corrections=True):
|
| 273 |
+
"""One user's durable workspace: always the full three-strata shape."""
|
| 274 |
+
try:
|
| 275 |
+
data = self._st.get(self.table_key) or {}
|
| 276 |
+
ws = data.get(username, {}) or {}
|
| 277 |
+
# A collision acknowledgement is protocol state, not part of a field definition.
|
| 278 |
+
# Consume it with the first fresh workspace payload after the correcting write,
|
| 279 |
+
# then splice a bounded copy into that payload only. Keeping it out of `fields`
|
| 280 |
+
# prevents an old request id surviving forever and overriding a later rename.
|
| 281 |
+
corrections = {}
|
| 282 |
+
if consume_corrections and ws.get('fieldCorrections'):
|
| 283 |
+
def _take(current):
|
| 284 |
+
current_ws = current.get(username) or {}
|
| 285 |
+
pending = current_ws.get('fieldCorrections') or {}
|
| 286 |
+
corrections.update({
|
| 287 |
+
str(key)[:80]: dict(value)
|
| 288 |
+
for key, value in pending.items()
|
| 289 |
+
if isinstance(value, dict)
|
| 290 |
+
})
|
| 291 |
+
current_ws.pop('fieldCorrections', None)
|
| 292 |
+
return current
|
| 293 |
+
|
| 294 |
+
data = self._st.update(self.table_key, _take, flush='async')
|
| 295 |
+
ws = (data or {}).get(username, {}) or {}
|
| 296 |
+
except Exception:
|
| 297 |
+
data = {}
|
| 298 |
+
ws = {}
|
| 299 |
+
corrections = {}
|
| 300 |
+
fields = {
|
| 301 |
+
key: dict(value) if isinstance(value, dict) else value
|
| 302 |
+
for key, value in (ws.get('fields') or {}).items()
|
| 303 |
+
}
|
| 304 |
+
# ββ W36-T25 β THE TENANT-WIDE COLUMN SUMMARY, MERGED OVER THIS USER'S STRATUM.
|
| 305 |
+
# β IT MUST BE ABLE TO CREATE AN ENTRY, not only decorate one, and that is the whole
|
| 306 |
+
# reason this is a merge rather than a lookup: the colleague who never touched the column
|
| 307 |
+
# has NO per-user record for it, so a decorate-only pass would have left them with exactly
|
| 308 |
+
# the blank totals row the owner reported. `aios_grid.workspace_wire` reads `meta['agg']`
|
| 309 |
+
# off whatever is here, base column or custom one alike.
|
| 310 |
+
for key, shared in _shared_fields(data).items():
|
| 311 |
+
agg = str((shared or {}).get('agg') or '').strip()
|
| 312 |
+
if not agg:
|
| 313 |
+
continue
|
| 314 |
+
entry = fields.get(key)
|
| 315 |
+
fields[key] = {**entry, 'agg': agg} if isinstance(entry, dict) else {'agg': agg}
|
| 316 |
+
for key, ack in corrections.items():
|
| 317 |
+
field = fields.get(key)
|
| 318 |
+
accepted_label = str(ack.get('label') or '')[:120]
|
| 319 |
+
requested_label = str(ack.get('labelCorrectedFrom') or '')[:120]
|
| 320 |
+
correction_id = str(ack.get('labelCorrectionId') or '')[:180]
|
| 321 |
+
# A newer field write clears/replaces the pending ack in the SAME transaction.
|
| 322 |
+
# The label check is an extra belt against ever attaching a stale ack to a newer
|
| 323 |
+
# definition if a future store implementation weakens that ordering.
|
| 324 |
+
if (isinstance(field, dict) and accepted_label
|
| 325 |
+
and str(field.get('label') or '') == accepted_label
|
| 326 |
+
and requested_label and correction_id):
|
| 327 |
+
field['labelCorrectedFrom'] = requested_label
|
| 328 |
+
field['labelCorrectionId'] = correction_id
|
| 329 |
+
out = {
|
| 330 |
+
'views': dict(ws.get('views') or {}),
|
| 331 |
+
'fields': fields,
|
| 332 |
+
'overlays': dict(ws.get('overlays') or {}),
|
| 333 |
+
# wave-8 I11 (C4): folders over the saved views / cohorts sidebars. A FOURTH
|
| 334 |
+
# stratum rather than a key on each item β see aios_grid.clean_folders for why
|
| 335 |
+
# (a cohort lives in another store, and filing is an organising act, not part of
|
| 336 |
+
# what a view is). Absent for every workspace saved before this wave, which is
|
| 337 |
+
# exactly "no folders yet".
|
| 338 |
+
'folders': dict(ws.get('folders') or {}),
|
| 339 |
+
'itemFolders': dict(ws.get('itemFolders') or {}),
|
| 340 |
+
}
|
| 341 |
+
# 2026-07-31 (owner item 3): WHERE THE USER LEFT OFF survives a new browser. The
|
| 342 |
+
# client's localStorage copy wins when present; this is the server's answer for a
|
| 343 |
+
# fresh profile, which used to fall all the way to the system default view.
|
| 344 |
+
if ws.get('activeViewId'):
|
| 345 |
+
out['activeViewId'] = str(ws['activeViewId'])
|
| 346 |
+
# Wave 2026-08-02 (C-LAYOUT): the per-user record-detail field order. A fifth
|
| 347 |
+
# stratum, absent until the user first reorders β exactly "default order".
|
| 348 |
+
if isinstance(ws.get('recordLayout'), dict):
|
| 349 |
+
out['recordLayout'] = dict(ws['recordLayout'])
|
| 350 |
+
return out
|
| 351 |
+
|
| 352 |
+
# ---------------------------------------------------------------- write
|
| 353 |
+
def _update(self, username, change, shared=None):
|
| 354 |
+
"""Apply `change(ws)` to this user's stratum, and `shared(shared_fields)` to the
|
| 355 |
+
tenant-wide one, in ONE transaction.
|
| 356 |
+
|
| 357 |
+
β W36-T25 β `shared` IS A SECOND CALLBACK RATHER THAN A SECOND `update`, and that is the
|
| 358 |
+
whole reason it exists here instead of at the caller. `save_field` writes a note (per
|
| 359 |
+
user) and a column summary (tenant-wide) from ONE payload; two transactions would let the
|
| 360 |
+
summary land while the note did not, and the store's own commit is asynchronous, so the
|
| 361 |
+
window is real rather than theoretical. One `update`, one flush, one failure mode.
|
| 362 |
+
"""
|
| 363 |
+
def _up(data):
|
| 364 |
+
ws = data.setdefault(username, {})
|
| 365 |
+
ws.setdefault('views', {})
|
| 366 |
+
ws.setdefault('fields', {})
|
| 367 |
+
ws.setdefault('overlays', {})
|
| 368 |
+
ws.setdefault('folders', {})
|
| 369 |
+
ws.setdefault('itemFolders', {})
|
| 370 |
+
change(ws)
|
| 371 |
+
if shared is not None:
|
| 372 |
+
shared(data.setdefault(SHARED_KEY, {}).setdefault('fields', {}))
|
| 373 |
+
return data
|
| 374 |
+
# flush='async' (wave-7 W3): this is THE hot path β every autosaved filter tweak,
|
| 375 |
+
# column note and typed overlay cell lands here inside the component round-trip, and
|
| 376 |
+
# the historical synchronous hub commit cost seconds per edit. The mutation applies to
|
| 377 |
+
# the in-process cache (read-your-writes for every subsequent render); the hub write
|
| 378 |
+
# coalesces in the background. Registry/auth writes elsewhere stay flush='sync'.
|
| 379 |
+
return self._st.update(self.table_key, _up, flush='async')
|
| 380 |
+
|
| 381 |
+
def rename_choice_values(self, username, change):
|
| 382 |
+
"""Apply an arbitrary workspace rewrite (wave 20, item 15 / C-RENAME).
|
| 383 |
+
|
| 384 |
+
β NAMED FOR ITS ONE CALLER RATHER THAN EXPOSED AS A GENERIC `mutate`, deliberately. A
|
| 385 |
+
public "do anything to the workspace" method is an invitation to put write logic in
|
| 386 |
+
callers instead of here, and every OTHER method on this class exists precisely because
|
| 387 |
+
that logic belongs in one place. Renaming a choice is the one operation that must touch
|
| 388 |
+
three strata AT ONCE β the field's `choices`, the cells in `overlays`, and the views that
|
| 389 |
+
filter or colour by the old value β inside a SINGLE transaction, because a rename that
|
| 390 |
+
updated the cells and not the filters would leave a saved view matching nothing.
|
| 391 |
+
|
| 392 |
+
`change(ws)` receives the whole workspace with every stratum pre-created (see `_update`).
|
| 393 |
+
"""
|
| 394 |
+
return self._update(username, change)
|
| 395 |
+
|
| 396 |
+
def save_active_view(self, username, view_id):
|
| 397 |
+
"""Remember which view this user last opened (owner item 3, 2026-07-31).
|
| 398 |
+
|
| 399 |
+
Presentation state, not authorisation: the READ side re-validates the id against what
|
| 400 |
+
the caller may actually see, so a stale or foreign id degrades to the default view
|
| 401 |
+
rather than granting anything. Stored per user like every other stratum.
|
| 402 |
+
"""
|
| 403 |
+
vid = str(view_id or '').strip()[:120]
|
| 404 |
+
if not vid or username == SHARED_KEY:
|
| 405 |
+
return
|
| 406 |
+
|
| 407 |
+
def _set(ws):
|
| 408 |
+
ws['activeViewId'] = vid
|
| 409 |
+
self._update(username, _set)
|
| 410 |
+
|
| 411 |
+
def save_record_layout(self, username, order):
|
| 412 |
+
"""The per-user RECORD-DETAIL field order (wave 2026-08-02, C-LAYOUT).
|
| 413 |
+
|
| 414 |
+
Presentation state for ONE surface β the record modal. Deliberately not view config:
|
| 415 |
+
the owner's ask is per-user, not per-view, and it must never reorder grid columns.
|
| 416 |
+
The event handler validated keys against the live field set; the wire re-validates at
|
| 417 |
+
serve time (aios_grid.workspace_wire), so a deleted field cannot outlive itself here.
|
| 418 |
+
An empty order clears the stratum back to "default order".
|
| 419 |
+
"""
|
| 420 |
+
if username == SHARED_KEY:
|
| 421 |
+
return
|
| 422 |
+
clean, seen = [], set()
|
| 423 |
+
for key in (order or [])[:200]:
|
| 424 |
+
key = str(key or '').strip()[:80]
|
| 425 |
+
if key and key not in seen:
|
| 426 |
+
seen.add(key)
|
| 427 |
+
clean.append(key)
|
| 428 |
+
|
| 429 |
+
def _set(ws):
|
| 430 |
+
if clean:
|
| 431 |
+
ws['recordLayout'] = {'order': clean}
|
| 432 |
+
else:
|
| 433 |
+
ws.pop('recordLayout', None)
|
| 434 |
+
|
| 435 |
+
self._update(username, _set)
|
| 436 |
+
|
| 437 |
+
def save_folders(self, username, folders, item_folders):
|
| 438 |
+
"""Replace the folder stratum wholesale (wave-8 I11).
|
| 439 |
+
|
| 440 |
+
Wholesale rather than per-folder because the caller has ALREADY validated the complete
|
| 441 |
+
picture through aios_grid.clean_folders / clean_item_folders, and those two are
|
| 442 |
+
interdependent: a placement is only legal while its folder exists, so committing them
|
| 443 |
+
separately would leave a window where a reader sees an item filed into a folder that is
|
| 444 |
+
not there yet. One write, one consistent state.
|
| 445 |
+
"""
|
| 446 |
+
def _set(ws):
|
| 447 |
+
ws['folders'] = dict(folders or {})
|
| 448 |
+
ws['itemFolders'] = dict(item_folders or {})
|
| 449 |
+
self._update(username, _set)
|
| 450 |
+
|
| 451 |
+
def save_view_order(self, username, order):
|
| 452 |
+
"""β WAVE-27 item 5 (contract C7) β this user's own ORDER for the views rail.
|
| 453 |
+
|
| 454 |
+
Wholesale, like `save_folders` above and for the same reason: the client sends the full
|
| 455 |
+
list it is looking at, not a delta, because a partial order cannot say where an UNNAMED
|
| 456 |
+
view went.
|
| 457 |
+
|
| 458 |
+
β PER USER, and it belongs in this stratum rather than on the view records themselves.
|
| 459 |
+
`aios_grid`'s own folder note argues it out for placements and every word applies: an
|
| 460 |
+
arrangement is a per-user ORGANISING act, not part of what a view IS β so keeping it out
|
| 461 |
+
of the view config means duplicating, sharing or exporting a view does not drag one
|
| 462 |
+
person's rail position along with it. It also means a SHARED view can sit in a different
|
| 463 |
+
place for each person who can see it, which is the only coherent answer once two people
|
| 464 |
+
share one view.
|
| 465 |
+
|
| 466 |
+
An empty list CLEARS the arrangement (back to server order) rather than storing `[]`.
|
| 467 |
+
"""
|
| 468 |
+
def _set(ws):
|
| 469 |
+
clean = []
|
| 470 |
+
seen = set()
|
| 471 |
+
for vid in (order or []):
|
| 472 |
+
vid = str(vid).strip()[:120]
|
| 473 |
+
if vid and vid not in seen:
|
| 474 |
+
seen.add(vid)
|
| 475 |
+
clean.append(vid)
|
| 476 |
+
if clean:
|
| 477 |
+
ws['viewOrder'] = clean
|
| 478 |
+
else:
|
| 479 |
+
ws.pop('viewOrder', None)
|
| 480 |
+
self._update(username, _set)
|
| 481 |
+
|
| 482 |
+
def shared_views(self, viewer, is_admin=False):
|
| 483 |
+
"""Every SHARED view this viewer may see, by id (wave-9 I17).
|
| 484 |
+
|
| 485 |
+
Read-only and independent of the viewer's own workspace: the caller merges. Returns
|
| 486 |
+
only what `_may_see` allows, so a caller cannot accidentally render somebody else's
|
| 487 |
+
personal view by forgetting to filter.
|
| 488 |
+
"""
|
| 489 |
+
try:
|
| 490 |
+
bucket = ((self._st.get(self.table_key) or {}).get(SHARED_KEY) or {}).get('views') or {}
|
| 491 |
+
except Exception:
|
| 492 |
+
return {}
|
| 493 |
+
return {vid: dict(v) for vid, v in bucket.items()
|
| 494 |
+
if _may_see(v, viewer, is_admin)}
|
| 495 |
+
|
| 496 |
+
def shared_view(self, view_id):
|
| 497 |
+
"""One shared view RAW β no visibility filter. For authorisation decisions only: a
|
| 498 |
+
caller must know a view exists and who owns it before it can decide whether the actor
|
| 499 |
+
may touch it. Never hand the result to a renderer without checking `_may_see`."""
|
| 500 |
+
try:
|
| 501 |
+
return ((self._st.get(self.table_key) or {}).get(SHARED_KEY) or {}
|
| 502 |
+
).get('views', {}).get(str(view_id))
|
| 503 |
+
except Exception:
|
| 504 |
+
return None
|
| 505 |
+
|
| 506 |
+
def save_view(self, username, view, shared=None, reserved_names=(), is_admin=False):
|
| 507 |
+
"""Upsert a SavedView into its ONE home β personal workspace or the shared bucket.
|
| 508 |
+
|
| 509 |
+
`shared` defaults to reading the view's own permissions (`is_shared`). Whichever home
|
| 510 |
+
it lands in, the view is REMOVED from the other, so a view can never exist as two
|
| 511 |
+
copies that diverge on the next edit.
|
| 512 |
+
|
| 513 |
+
β AUTHORISATION IS THE CALLER'S JOB and must happen BEFORE this is called β this layer
|
| 514 |
+
moves data and does not know who is asking. `_cl_handle_one` is the wall.
|
| 515 |
+
"""
|
| 516 |
+
view_id = str((view or {}).get('id') or '').strip()
|
| 517 |
+
if not view_id:
|
| 518 |
+
raise ValueError('view id is required')
|
| 519 |
+
if username == SHARED_KEY:
|
| 520 |
+
raise ValueError('reserved username')
|
| 521 |
+
to_shared = is_shared(view) if shared is None else bool(shared)
|
| 522 |
+
requested = dict(view)
|
| 523 |
+
accepted = {}
|
| 524 |
+
|
| 525 |
+
def _up(data):
|
| 526 |
+
# View names are tenant-global: every personal workspace plus the shared bucket.
|
| 527 |
+
# This deliberately includes views the actor cannot see. The only disclosed fact
|
| 528 |
+
# is that a display name is already taken, while the categorical "no duplicate
|
| 529 |
+
# view names" contract remains true when a personal view is later shared.
|
| 530 |
+
names = list(reserved_names or ())
|
| 531 |
+
for workspace in data.values():
|
| 532 |
+
if not isinstance(workspace, dict):
|
| 533 |
+
continue
|
| 534 |
+
names.extend(
|
| 535 |
+
value.get('name')
|
| 536 |
+
for candidate_id, value in (workspace.get('views') or {}).items()
|
| 537 |
+
if candidate_id != view_id and isinstance(value, dict)
|
| 538 |
+
)
|
| 539 |
+
payload = dict(requested)
|
| 540 |
+
payload['name'] = _unique_name(payload.get('name'), names)
|
| 541 |
+
accepted.clear()
|
| 542 |
+
accepted.update(payload)
|
| 543 |
+
if to_shared:
|
| 544 |
+
bucket = data.setdefault(SHARED_KEY, {})
|
| 545 |
+
bucket.setdefault('views', {})[view_id] = payload
|
| 546 |
+
# it may have lived in the creator's workspace before being shared
|
| 547 |
+
owner = data.get(payload.get('createdBy') or username) or {}
|
| 548 |
+
(owner.get('views') or {}).pop(view_id, None)
|
| 549 |
+
else:
|
| 550 |
+
ws = data.setdefault(username, {})
|
| 551 |
+
ws.setdefault('views', {})[view_id] = payload
|
| 552 |
+
(data.get(SHARED_KEY, {}).get('views') or {}).pop(view_id, None)
|
| 553 |
+
return data
|
| 554 |
+
|
| 555 |
+
self._st.update(self.table_key, _up, flush='async')
|
| 556 |
+
return dict(accepted)
|
| 557 |
+
|
| 558 |
+
def delete_view(self, username, view_id):
|
| 559 |
+
"""Delete a custom/list view override. The system all-rows view is guarded by caller.
|
| 560 |
+
|
| 561 |
+
Removes from BOTH homes: the caller has already authorised the delete, and leaving a
|
| 562 |
+
stale copy in the other bucket would resurrect the view on the next read.
|
| 563 |
+
"""
|
| 564 |
+
vid = str(view_id)
|
| 565 |
+
|
| 566 |
+
def _up(data):
|
| 567 |
+
(data.get(username, {}).get('views') or {}).pop(vid, None)
|
| 568 |
+
(data.get(SHARED_KEY, {}).get('views') or {}).pop(vid, None)
|
| 569 |
+
return data
|
| 570 |
+
|
| 571 |
+
self._st.update(self.table_key, _up, flush='async')
|
| 572 |
+
|
| 573 |
+
def save_field(self, username, field, reserved_names=(), correction_id=None):
|
| 574 |
+
"""Persist a column note or a user-created (custom_/measure_) field definition.
|
| 575 |
+
|
| 576 |
+
ββ W36-T25 β THE COLUMN SUMMARY GOES TO THE TENANT-WIDE STRATUM, EVERYTHING ELSE STAYS
|
| 577 |
+
PER USER, in ONE transaction. See `SHARED_FIELD_KEYS` above for the owner's report and for
|
| 578 |
+
why the line falls where it does. The returned `accepted` still carries the summary: the
|
| 579 |
+
caller is echoing back the field it just stored, and dropping a key from that echo would
|
| 580 |
+
tell the client its write was refused ([[read-path-cannot-witness-write-path]]).
|
| 581 |
+
"""
|
| 582 |
+
key = str((field or {}).get('key') or '').strip()
|
| 583 |
+
if not key:
|
| 584 |
+
raise ValueError('field key is required')
|
| 585 |
+
requested = dict(field)
|
| 586 |
+
accepted = {}
|
| 587 |
+
|
| 588 |
+
def _save(ws):
|
| 589 |
+
names = list(reserved_names or ())
|
| 590 |
+
names.extend(
|
| 591 |
+
value.get('label')
|
| 592 |
+
for candidate_key, value in (ws.get('fields') or {}).items()
|
| 593 |
+
if candidate_key != key and isinstance(value, dict)
|
| 594 |
+
)
|
| 595 |
+
payload = dict(requested)
|
| 596 |
+
payload.pop('labelCorrectedFrom', None)
|
| 597 |
+
payload.pop('labelCorrectionId', None)
|
| 598 |
+
requested_label = ' '.join(
|
| 599 |
+
str(payload.get('label') or 'Untitled').split())[:120].rstrip()
|
| 600 |
+
payload['label'] = _unique_name(requested_label, names)
|
| 601 |
+
corrections = ws.setdefault('fieldCorrections', {})
|
| 602 |
+
corrections.pop(key, None)
|
| 603 |
+
if payload['label'] != requested_label and correction_id:
|
| 604 |
+
corrections[key] = {
|
| 605 |
+
'label': payload['label'],
|
| 606 |
+
'labelCorrectedFrom': requested_label,
|
| 607 |
+
'labelCorrectionId': str(correction_id)[:180],
|
| 608 |
+
}
|
| 609 |
+
if not corrections:
|
| 610 |
+
ws.pop('fieldCorrections', None)
|
| 611 |
+
accepted.clear()
|
| 612 |
+
accepted.update(payload)
|
| 613 |
+
# ββ W36-T25 β SPLIT AFTER the label allocation and the correction bookkeeping, so
|
| 614 |
+
# both still see the whole payload, and BEFORE the per-user write.
|
| 615 |
+
mine, shared_now = split_shared(payload)
|
| 616 |
+
shared_write.clear()
|
| 617 |
+
shared_write.update(shared_now)
|
| 618 |
+
# β EMPTINESS IS JUDGED ON THE PER-USER HALF. A source column whose ONLY state was a
|
| 619 |
+
# summary now has no per-user state at all, and leaving an `{}` override behind is the
|
| 620 |
+
# meaningless row `source_override_is_empty` exists to prevent.
|
| 621 |
+
if source_override_is_empty(mine):
|
| 622 |
+
ws['fields'].pop(key, None)
|
| 623 |
+
else:
|
| 624 |
+
ws['fields'][key] = mine
|
| 625 |
+
|
| 626 |
+
shared_write = {}
|
| 627 |
+
|
| 628 |
+
def _share(shared_fields):
|
| 629 |
+
# β A CLEARED SUMMARY MUST REMOVE THE ROW, not leave an empty one: `workspace` treats
|
| 630 |
+
# any entry it finds as a live tenant-wide summary, so an `{'agg': ''}` husk would be
|
| 631 |
+
# skipped today and become a resurrection hazard the moment the read grows a second
|
| 632 |
+
# shared key. Absence is the only honest spelling of "nobody set one".
|
| 633 |
+
if shared_write:
|
| 634 |
+
shared_fields[key] = {**(shared_fields.get(key) or {}), **shared_write}
|
| 635 |
+
else:
|
| 636 |
+
shared_fields.pop(key, None)
|
| 637 |
+
|
| 638 |
+
self._update(username, _save, shared=_share)
|
| 639 |
+
return dict(accepted)
|
| 640 |
+
|
| 641 |
+
def duplicate_field(self, username, source_key, new_key, field,
|
| 642 |
+
reserved_names=(), correction_id=None):
|
| 643 |
+
"""Clone a user-created field in ONE store transaction (wave-5 item 1): the new
|
| 644 |
+
definition plus β for `custom_` overlay sources only β every stored cell value under
|
| 645 |
+
the source key. One transaction, because a def without its values (or values without a
|
| 646 |
+
def) is exactly the orphan state delete_field exists to prevent, in reverse.
|
| 647 |
+
The caller validated both keys (same created stratum) and stamped the clone's
|
| 648 |
+
createdBy; this layer only moves data."""
|
| 649 |
+
source_key = str(source_key or '').strip()
|
| 650 |
+
new_key = str(new_key or '').strip()
|
| 651 |
+
if not source_key or not new_key or source_key == new_key:
|
| 652 |
+
raise ValueError('duplicate_field needs two distinct keys')
|
| 653 |
+
requested = dict(field)
|
| 654 |
+
accepted = {}
|
| 655 |
+
|
| 656 |
+
def _dup(ws):
|
| 657 |
+
names = list(reserved_names or ())
|
| 658 |
+
names.extend(
|
| 659 |
+
value.get('label')
|
| 660 |
+
for candidate_key, value in (ws.get('fields') or {}).items()
|
| 661 |
+
if candidate_key != new_key and isinstance(value, dict)
|
| 662 |
+
)
|
| 663 |
+
payload = dict(requested)
|
| 664 |
+
payload.pop('labelCorrectedFrom', None)
|
| 665 |
+
payload.pop('labelCorrectionId', None)
|
| 666 |
+
requested_label = ' '.join(
|
| 667 |
+
str(payload.get('label') or 'Untitled').split())[:120].rstrip()
|
| 668 |
+
payload['label'] = _unique_name(requested_label, names)
|
| 669 |
+
corrections = ws.setdefault('fieldCorrections', {})
|
| 670 |
+
corrections.pop(new_key, None)
|
| 671 |
+
if payload['label'] != requested_label and correction_id:
|
| 672 |
+
corrections[new_key] = {
|
| 673 |
+
'label': payload['label'],
|
| 674 |
+
'labelCorrectedFrom': requested_label,
|
| 675 |
+
'labelCorrectionId': str(correction_id)[:180],
|
| 676 |
+
}
|
| 677 |
+
if not corrections:
|
| 678 |
+
ws.pop('fieldCorrections', None)
|
| 679 |
+
accepted.clear()
|
| 680 |
+
accepted.update(payload)
|
| 681 |
+
# β W36-T25: a CLONE carries the original's summary, and a summary is the database's
|
| 682 |
+
# (see `SHARED_FIELD_KEYS`). Storing it per user here would give the clone a different
|
| 683 |
+
# residency from every other column β one door out of three disagreeing about where a
|
| 684 |
+
# thing lives is how `save_field` and this function drift.
|
| 685 |
+
mine, shared_now = split_shared(payload)
|
| 686 |
+
shared_write.clear()
|
| 687 |
+
shared_write.update(shared_now)
|
| 688 |
+
ws['fields'][new_key] = mine
|
| 689 |
+
if source_key.startswith('custom_'):
|
| 690 |
+
for row in ws['overlays'].values():
|
| 691 |
+
if isinstance(row, dict) and source_key in row:
|
| 692 |
+
row[new_key] = row[source_key]
|
| 693 |
+
|
| 694 |
+
shared_write = {}
|
| 695 |
+
|
| 696 |
+
def _share(shared_fields):
|
| 697 |
+
if shared_write:
|
| 698 |
+
shared_fields[new_key] = {**(shared_fields.get(new_key) or {}), **shared_write}
|
| 699 |
+
else:
|
| 700 |
+
shared_fields.pop(new_key, None)
|
| 701 |
+
|
| 702 |
+
self._update(username, _dup, shared=_share)
|
| 703 |
+
return dict(accepted)
|
| 704 |
+
|
| 705 |
+
def delete_field(self, username, key):
|
| 706 |
+
"""Delete a USER-CREATED field definition outright (owner gap closed 2026-07-27).
|
| 707 |
+
|
| 708 |
+
Only the created strata ever reach here (`custom_` overlay fields, `measure_` formula
|
| 709 |
+
columns β the caller enforces the prefix). The stored overlay VALUES for the key are
|
| 710 |
+
scrubbed with it: a deleted column's cells must not linger as orphan data that would
|
| 711 |
+
silently resurface if the key were ever reused. Views referencing the key self-heal on
|
| 712 |
+
their next autosave (an unknown colId is dropped) β the rule every stale key rides.
|
| 713 |
+
|
| 714 |
+
ββ W36-T25 β AND THE TENANT-WIDE SUMMARY GOES WITH IT, for exactly the reason the
|
| 715 |
+
paragraph above gives about cells: an orphan `agg` under a deleted key is state nobody can
|
| 716 |
+
see and nobody can clear, and it would attach itself to the next column that happens to
|
| 717 |
+
take the key back. β This is the ONE stratum a per-user delete may reach across accounts,
|
| 718 |
+
and it is safe because the summary was never this user's to begin with β deleting the
|
| 719 |
+
COLUMN is a tenant-wide act already.
|
| 720 |
+
"""
|
| 721 |
+
key = str(key or '').strip()
|
| 722 |
+
if not key:
|
| 723 |
+
return
|
| 724 |
+
|
| 725 |
+
def _drop(ws):
|
| 726 |
+
ws['fields'].pop(key, None)
|
| 727 |
+
for row in ws['overlays'].values():
|
| 728 |
+
if isinstance(row, dict):
|
| 729 |
+
row.pop(key, None)
|
| 730 |
+
|
| 731 |
+
self._update(username, _drop, shared=lambda shared_fields: shared_fields.pop(key, None))
|
| 732 |
+
|
| 733 |
+
def patch_overlay(self, username, pid, updates):
|
| 734 |
+
"""Patch only the external editable stratum; never writes to the source system."""
|
| 735 |
+
clean = dict(updates or {})
|
| 736 |
+
if not clean:
|
| 737 |
+
return
|
| 738 |
+
|
| 739 |
+
def _patch(ws):
|
| 740 |
+
ws['overlays'].setdefault(str(int(pid)), {}).update(clean)
|
| 741 |
+
|
| 742 |
+
self._update(username, _patch)
|
| 743 |
+
|
| 744 |
+
|
| 745 |
+
def make(table_key, st=None):
|
| 746 |
+
return TableStore(table_key, st=st)
|
platform/core/user_tables.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
platform/harness/meta_store.py
CHANGED
|
@@ -1,521 +1,521 @@
|
|
| 1 |
-
"""meta_store.py β the REST loader for a mirror whose only other loader is XML-RPC (W31-T48).
|
| 2 |
-
|
| 3 |
-
python platform/harness/meta_store.py --sync pull into tenant #0's mirror
|
| 4 |
-
python platform/harness/meta_store.py --sync --tenant gtmlab
|
| 5 |
-
python platform/harness/meta_store.py --status what is in the mirror now
|
| 6 |
-
|
| 7 |
-
β WHY THIS IS A SIBLING AND NOT A ROW IN `datastore.ENTITIES`. `datastore.sync_entity` is
|
| 8 |
-
hardwired to XML-RPC β it does `import core.odoo as O` and speaks `search_read` β so no amount of
|
| 9 |
-
spec data makes it fetch over HTTPS. What IS reusable is everything *below* the fetch: the
|
| 10 |
-
per-tenant file (`path_for`), the process-singleton connection (`connect`), and the
|
| 11 |
-
delete-then-insert upsert. This module borrows those and brings its own reader. **The mirror is one
|
| 12 |
-
store with two loaders, not two stores.**
|
| 13 |
-
|
| 14 |
-
β THE SHAPE OF THE WHOLE THING, because it is the owner's actual request: *"we just need to pull
|
| 15 |
-
the data into our template database for Meta"*, working *"just like Odoo"*. Odoo's path is
|
| 16 |
-
`XML-RPC -> DuckDB mirror -> odoo_relational -> ut_odoo_* locked grids`. Meta's is
|
| 17 |
-
`Graph -> DuckDB mirror -> meta_relational -> ut_meta_* locked grids`. Same middle, same end, one
|
| 18 |
-
different first hop.
|
| 19 |
-
|
| 20 |
-
β EVERY COLUMN NAME BELOW WAS MEASURED, NOT READ OFF A DOC (R8). `proto/meta-entity-fields.json`
|
| 21 |
-
carries the run: each level's list is what the API ACCEPTED when asked, on a real ad account.
|
| 22 |
-
Ad Account 45 Β· Campaign 35 Β· Ad Set 54 Β· Ad 36 Β· Creative 58 Β· Insights 57
|
| 23 |
-
β AND `ACCEPTED` IS NOT `RETURNED`. Graph omits null fields from a response, so a 20-field ask
|
| 24 |
-
came back with 14 keys. Building a schema from what came back would drop ~30% of the columns with
|
| 25 |
-
nothing going red β which is exactly the "do not drop any column" instruction, broken silently.
|
| 26 |
-
The lists below are therefore the ASKED-AND-ACCEPTED set, and a column with no value is NULL.
|
| 27 |
-
|
| 28 |
-
β `owner` IS ABSENT FROM THE AD ACCOUNT LIST AND THAT IS A REPORTED LIMIT, NOT AN OMISSION: it
|
| 29 |
-
answers `403 (#200) Requires business_management permission`, which this token does not carry. One
|
| 30 |
-
column of 285. R6's second sentence β a limit that cannot be removed gets stated with its cause.
|
| 31 |
-
|
| 32 |
-
β IDS ARE TEXT, ALWAYS. A Meta object id is a 17-digit decimal string; `120273975028650555` is
|
| 33 |
-
larger than 2^53, so any float or JS-number path silently corrupts it. Same ruling as `ig_id`
|
| 34 |
-
(W26/R3), for the same reason, and it is why every `id` column here is VARCHAR.
|
| 35 |
-
"""
|
| 36 |
-
import argparse
|
| 37 |
-
import json
|
| 38 |
-
import os
|
| 39 |
-
import sys
|
| 40 |
-
import time
|
| 41 |
-
import urllib.error
|
| 42 |
-
import urllib.parse
|
| 43 |
-
import urllib.request
|
| 44 |
-
from pathlib import Path
|
| 45 |
-
|
| 46 |
-
_HERE = Path(__file__).resolve().parent
|
| 47 |
-
if str(_HERE.parent) not in sys.path:
|
| 48 |
-
sys.path.insert(0, str(_HERE.parent))
|
| 49 |
-
|
| 50 |
-
from harness import datastore # noqa: E402
|
| 51 |
-
|
| 52 |
-
GRAPH_VERSION = os.environ.get("META_GRAPH_VERSION") or "v21.0"
|
| 53 |
-
GRAPH = f"https://graph.facebook.com/{GRAPH_VERSION}"
|
| 54 |
-
|
| 55 |
-
#: Numeric columns, by name. Everything else is VARCHAR β including ids (see the header) and
|
| 56 |
-
#: including anything Graph returns as a nested object, which is stored as compact JSON text.
|
| 57 |
-
_INT = {"impressions", "reach", "clicks", "unique_clicks", "inline_link_clicks",
|
| 58 |
-
"inline_post_engagement", "full_view_impressions", "estimated_ad_recallers",
|
| 59 |
-
"account_status", "age", "timezone_id", "timezone_offset_hours_utc", "io_number",
|
| 60 |
-
"min_daily_budget", "min_campaign_group_spend_cap"}
|
| 61 |
-
_DBL = {"spend", "social_spend", "ctr", "unique_ctr", "cpc", "cpm", "cpp", "frequency",
|
| 62 |
-
"inline_link_click_ctr", "outbound_clicks_ctr", "amount_spent", "balance", "spend_cap",
|
| 63 |
-
"daily_budget", "lifetime_budget", "budget_remaining", "bid_amount", "canvas_avg_view_time",
|
| 64 |
-
"daily_min_spend_target", "daily_spend_cap", "lifetime_min_spend_target",
|
| 65 |
-
"lifetime_spend_cap", "lifetime_imps"}
|
| 66 |
-
|
| 67 |
-
#: β THE MEASURED CATALOG. `edge` is the connection on the ad account; `None` means the account
|
| 68 |
-
#: itself. `parent` names the column that ties a row to its parent, used by nothing here and by
|
| 69 |
-
#: the grid links in `meta_relational`.
|
| 70 |
-
ACCOUNT_FIELDS = (
|
| 71 |
-
"account_id account_status age amount_spent balance business_city business_country_code "
|
| 72 |
-
"business_name business_state business_street business_street2 business_zip capabilities "
|
| 73 |
-
"created_time currency disable_reason end_advertiser end_advertiser_name funding_source "
|
| 74 |
-
"has_migrated_permissions id io_number is_attribution_spec_system_default "
|
| 75 |
-
"is_direct_deals_enabled is_notifications_enabled is_personal is_prepay_account "
|
| 76 |
-
"is_tax_id_required line_numbers media_agency min_campaign_group_spend_cap min_daily_budget "
|
| 77 |
-
"name offsite_pixels_tos_accepted partner spend_cap tax_id tax_id_status tax_id_type "
|
| 78 |
-
"timezone_id timezone_name timezone_offset_hours_utc tos_accepted user_tasks user_tos_accepted"
|
| 79 |
-
).split()
|
| 80 |
-
|
| 81 |
-
CAMPAIGN_FIELDS = (
|
| 82 |
-
"account_id bid_strategy boosted_object_id budget_rebalance_flag budget_remaining buying_type "
|
| 83 |
-
"campaign_group_active_time can_create_brand_lift_study can_use_spend_cap configured_status "
|
| 84 |
-
"created_time daily_budget effective_status id is_skadnetwork_attribution issues_info "
|
| 85 |
-
"last_budget_toggling_time lifetime_budget name objective pacing_type primary_attribution "
|
| 86 |
-
"promoted_object smart_promotion_type source_campaign source_campaign_id special_ad_categories "
|
| 87 |
-
"special_ad_category special_ad_category_country spend_cap start_time status stop_time "
|
| 88 |
-
"topline_id updated_time"
|
| 89 |
-
).split()
|
| 90 |
-
|
| 91 |
-
ADSET_FIELDS = (
|
| 92 |
-
"account_id adlabels adset_schedule asset_feed_id attribution_spec bid_adjustments bid_amount "
|
| 93 |
-
"bid_constraints bid_info bid_strategy billing_event budget_remaining campaign "
|
| 94 |
-
"campaign_active_time campaign_attribution campaign_id configured_status created_time "
|
| 95 |
-
"creative_sequence daily_budget daily_min_spend_target daily_spend_cap destination_type "
|
| 96 |
-
"effective_status end_time frequency_control_specs id instagram_actor_id is_dynamic_creative "
|
| 97 |
-
"issues_info learning_stage_info lifetime_budget lifetime_imps lifetime_min_spend_target "
|
| 98 |
-
"lifetime_spend_cap multi_optimization_goal_weight name optimization_goal "
|
| 99 |
-
"optimization_sub_event pacing_type promoted_object recurring_budget_semantics review_feedback "
|
| 100 |
-
"rf_prediction_id source_adset source_adset_id start_time status targeting "
|
| 101 |
-
"targeting_optimization_types time_based_ad_rotation_id_blocks "
|
| 102 |
-
"time_based_ad_rotation_intervals updated_time use_new_app_click"
|
| 103 |
-
).split()
|
| 104 |
-
|
| 105 |
-
AD_FIELDS = (
|
| 106 |
-
"account_id ad_active_time ad_review_feedback ad_schedule_end_time ad_schedule_start_time "
|
| 107 |
-
"adlabels adset adset_id bid_amount bid_info bid_type campaign campaign_id configured_status "
|
| 108 |
-
"conversion_domain created_time creative demolink_hash display_sequence effective_status "
|
| 109 |
-
"engagement_audience failed_delivery_checks id issues_info last_updated_by_app_id name "
|
| 110 |
-
"preview_shareable_link priority recommendations source_ad source_ad_id status targeting "
|
| 111 |
-
"tracking_and_conversion_with_defaults tracking_specs updated_time"
|
| 112 |
-
).split()
|
| 113 |
-
|
| 114 |
-
CREATIVE_FIELDS = (
|
| 115 |
-
"account_id actor_id adlabels applink_treatment asset_feed_spec authorization_category body "
|
| 116 |
-
"branded_content_sponsor_page_id bundle_folder_id call_to_action_type categorization_criteria "
|
| 117 |
-
"category_media_source collaborative_ads_lsb_image_bank_id degrees_of_freedom_spec "
|
| 118 |
-
"destination_set_id dynamic_ad_voice effective_authorization_category "
|
| 119 |
-
"effective_instagram_media_id effective_object_story_id enable_direct_install "
|
| 120 |
-
"enable_launch_instant_app id image_crops image_hash image_url instagram_actor_id "
|
| 121 |
-
"instagram_permalink_url instagram_story_id instagram_user_id interactive_components_spec "
|
| 122 |
-
"link_deep_link_url link_destination_display_url link_og_id link_url "
|
| 123 |
-
"messenger_sponsored_message name object_id object_store_url object_story_id object_story_spec "
|
| 124 |
-
"object_type object_url place_page_set_id platform_customizations playable_asset_id "
|
| 125 |
-
"portrait_customizations product_set_id recommender_settings source_instagram_media_id status "
|
| 126 |
-
"template_url template_url_spec thumbnail_id thumbnail_url title url_tags "
|
| 127 |
-
"use_page_actor_override video_id"
|
| 128 |
-
).split()
|
| 129 |
-
|
| 130 |
-
INSIGHT_FIELDS = (
|
| 131 |
-
"account_currency account_id account_name action_values actions ad_id ad_name adset_id "
|
| 132 |
-
"adset_name attribution_setting buying_type campaign_id campaign_name canvas_avg_view_time "
|
| 133 |
-
"clicks conversion_rate_ranking conversion_values conversions cost_per_action_type "
|
| 134 |
-
"cost_per_inline_link_click cost_per_thruplay cost_per_unique_click cpc cpm cpp ctr date_start "
|
| 135 |
-
"date_stop engagement_rate_ranking estimated_ad_recallers frequency full_view_impressions "
|
| 136 |
-
"impressions inline_link_click_ctr inline_link_clicks inline_post_engagement objective "
|
| 137 |
-
"optimization_goal outbound_clicks outbound_clicks_ctr purchase_roas quality_ranking reach "
|
| 138 |
-
"social_spend spend unique_clicks unique_ctr unique_outbound_clicks "
|
| 139 |
-
"video_avg_time_watched_actions video_p100_watched_actions video_p25_watched_actions "
|
| 140 |
-
"video_p50_watched_actions video_p75_watched_actions video_p95_watched_actions "
|
| 141 |
-
"video_play_actions video_thruplay_watched_actions website_purchase_roas"
|
| 142 |
-
).split()
|
| 143 |
-
|
| 144 |
-
#: table -> (graph edge on the account | None, measured fields, parent column)
|
| 145 |
-
SPECS = {
|
| 146 |
-
"meta_ad_accounts": (None, ACCOUNT_FIELDS, None),
|
| 147 |
-
"meta_campaigns": ("campaigns", CAMPAIGN_FIELDS, "account_id"),
|
| 148 |
-
"meta_adsets": ("adsets", ADSET_FIELDS, "campaign_id"),
|
| 149 |
-
"meta_ads": ("ads", AD_FIELDS, "adset_id"),
|
| 150 |
-
"meta_creatives": ("adcreatives", CREATIVE_FIELDS, "account_id"),
|
| 151 |
-
}
|
| 152 |
-
|
| 153 |
-
#: The daily Insights grain (R2). One row per (ad, day) β the id is synthesised because Insights
|
| 154 |
-
#: has no id of its own, and it must be STABLE so a re-run updates instead of appending.
|
| 155 |
-
INSIGHTS_TABLE = "meta_insights_daily"
|
| 156 |
-
INSIGHTS_LEVEL = os.environ.get("META_INSIGHTS_LEVEL") or "ad"
|
| 157 |
-
|
| 158 |
-
#: β INSIGHTS IS FETCHED IN TIME SLICES, AND THE REASON IS MEASURED, NOT DEFENSIVE. All 57 fields
|
| 159 |
-
#: at ad level over `last_90d` with a 100-row page answers **HTTP 500 "An unknown error occurred"**
|
| 160 |
-
#: β Graph's way of saying the synchronous query is too heavy (the async report-run API is the
|
| 161 |
-
#: other answer, and it costs a poll loop this does not need). The SAME 57 fields over 7 days at
|
| 162 |
-
#: page 25 answer 200. So the window is walked in slices with every column intact:
|
| 163 |
-
#: 57 fields Β· ad level Β· 7d Β· limit 25 -> 200, 25 rows
|
| 164 |
-
#: 57 fields Β· account level Β· 7d -> 200, 7 rows
|
| 165 |
-
#: β Narrowing the FIELD list would also have "fixed" it, and that is the wrong fix twice over β
|
| 166 |
-
#: it drops columns R2 requires, and it does so invisibly.
|
| 167 |
-
INSIGHTS_DAYS = int(os.environ.get("META_INSIGHTS_DAYS") or 90)
|
| 168 |
-
INSIGHTS_SLICE = int(os.environ.get("META_INSIGHTS_SLICE_DAYS") or 7)
|
| 169 |
-
INSIGHTS_PAGE = int(os.environ.get("META_INSIGHTS_PAGE") or 25)
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
def _slices(days, size):
|
| 173 |
-
"""[(since, until)] covering the last `days`, oldest first, in `size`-day windows."""
|
| 174 |
-
from datetime import date, timedelta
|
| 175 |
-
end = date.today()
|
| 176 |
-
start = end - timedelta(days=max(1, days) - 1)
|
| 177 |
-
out, cur = [], start
|
| 178 |
-
while cur <= end:
|
| 179 |
-
stop = min(cur + timedelta(days=max(1, size) - 1), end)
|
| 180 |
-
out.append((cur.isoformat(), stop.isoformat()))
|
| 181 |
-
cur = stop + timedelta(days=1)
|
| 182 |
-
return out
|
| 183 |
-
|
| 184 |
-
#: β A REAL CEILING, AND R6's SECOND SENTENCE APPLIES TO IT. Graph pages at 25-100 rows; this is
|
| 185 |
-
#: the number of PAGES a single edge may walk before the loader stops and SAYS it stopped. It is
|
| 186 |
-
#: not a row cap on a connected source (R6 forbids that) β it is a runaway guard, and reaching it
|
| 187 |
-
#: is reported as a problem, never absorbed.
|
| 188 |
-
MAX_PAGES = int(os.environ.get("META_MAX_PAGES") or 200)
|
| 189 |
-
PAGE = int(os.environ.get("META_PAGE_SIZE") or 100)
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
class MetaError(RuntimeError):
|
| 193 |
-
"""A Graph refusal carrying Meta's own words. Safe to print: no token ever reaches it."""
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
def token():
|
| 197 |
-
"""The Meta token from the environment or gitignored `platform/.env`; "" when absent.
|
| 198 |
-
|
| 199 |
-
β Same resolver `aios-web/api/connectors_meta.py` uses. Duplicated deliberately and minimally:
|
| 200 |
-
`platform/` must not import from `aios-web/api/`, which is the layering rule this repo keeps
|
| 201 |
-
(`core` never imports up). Fifteen lines is the price of that boundary.
|
| 202 |
-
"""
|
| 203 |
-
tok = os.environ.get("META_ADS_ACCESS_TOKEN") or ""
|
| 204 |
-
if tok:
|
| 205 |
-
return tok.strip()
|
| 206 |
-
env = _HERE.parent / ".env"
|
| 207 |
-
if env.exists():
|
| 208 |
-
for line in env.read_text(encoding="utf-8", errors="replace").splitlines():
|
| 209 |
-
if line.strip().startswith("META_ADS_ACCESS_TOKEN"):
|
| 210 |
-
_, _, v = line.partition("=")
|
| 211 |
-
return v.strip().strip('"').strip("'")
|
| 212 |
-
return ""
|
| 213 |
-
|
| 214 |
-
|
| 215 |
-
def _get(path, tok, **params):
|
| 216 |
-
params["access_token"] = tok
|
| 217 |
-
url = f"{GRAPH}/{path.lstrip('/')}?" + urllib.parse.urlencode(params)
|
| 218 |
-
req = urllib.request.Request(url, headers={"User-Agent": "aios-meta-store/1"})
|
| 219 |
-
try:
|
| 220 |
-
with urllib.request.urlopen(req, timeout=120) as r:
|
| 221 |
-
return json.loads(r.read().decode("utf-8", "replace"))
|
| 222 |
-
except urllib.error.HTTPError as e:
|
| 223 |
-
try:
|
| 224 |
-
msg = (json.loads(e.read().decode("utf-8", "replace")).get("error") or {}
|
| 225 |
-
).get("message") or ""
|
| 226 |
-
except Exception:
|
| 227 |
-
msg = ""
|
| 228 |
-
raise MetaError(f"HTTP {e.code} on /{path.lstrip('/')}: {msg[:200]}") from None
|
| 229 |
-
except Exception as e:
|
| 230 |
-
raise MetaError(f"{type(e).__name__} on /{path.lstrip('/')}") from None
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
#: Graph's own words when a page is too heavy. It arrives as an HTTP **500**, not a 4xx, which is
|
| 234 |
-
#: why it cannot be treated as "the server is broken, give up".
|
| 235 |
-
_TOO_MUCH = "reduce the amount of data"
|
| 236 |
-
|
| 237 |
-
#: β THE ADS-MANAGEMENT RATE LIMIT, WHICH IS PER AD ACCOUNT AND NOT PER TOKEN. Measured: after a
|
| 238 |
-
#: heavy backfill Graph answers **HTTP 400 "There have been too many calls to this ad-account.
|
| 239 |
-
#: Wait a bit and try again."** It is a 4xx, so nothing about the status code says "retry" β the
|
| 240 |
-
#: MESSAGE is the only signal, which is why it is matched here rather than inferred from a code.
|
| 241 |
-
#: β It persists for minutes, so the backoff is measured in minutes and BOUNDED: after
|
| 242 |
-
#: `_RATE_TRIES` waits the loader STOPS and reports what it got, rather than sitting in a retry
|
| 243 |
-
#: loop nobody can see. A partial mirror that says it is partial beats a hung sync.
|
| 244 |
-
_RATE_LIMITED = "too many calls"
|
| 245 |
-
_RATE_TRIES = int(os.environ.get("META_RATE_TRIES") or 3)
|
| 246 |
-
_RATE_WAIT = int(os.environ.get("META_RATE_WAIT_S") or 90)
|
| 247 |
-
|
| 248 |
-
|
| 249 |
-
def _walk(path, tok, log, **params):
|
| 250 |
-
"""Every page of an edge, paged by CURSOR under our own parameters.
|
| 251 |
-
|
| 252 |
-
β IT DOES NOT FOLLOW GRAPH'S `paging.next` URL, AND THAT IS THE WHOLE POINT. Measured: the
|
| 253 |
-
first call to `/adcreatives` at limit=100 answers **HTTP 500 "Please reduce the amount of data
|
| 254 |
-
you're asking for"**, the retry at limit=25 succeeds β and then `next` carries the ORIGINAL
|
| 255 |
-
limit=100 and fails again on page 2. A backoff that cannot reach every page is not a backoff.
|
| 256 |
-
Re-issuing each page ourselves with `after=<cursor>` keeps the reduced limit for the whole walk.
|
| 257 |
-
|
| 258 |
-
β It shrinks the PAGE, never the FIELD LIST. Dropping columns to make a request fit is the
|
| 259 |
-
silent omission R2 forbids, and nothing downstream could see it. Fewer rows per call, always
|
| 260 |
-
every column per row.
|
| 261 |
-
|
| 262 |
-
-> (rows, hit_page_cap)
|
| 263 |
-
"""
|
| 264 |
-
out, pages = [], 0
|
| 265 |
-
limit = int(params.pop("limit", None) or PAGE)
|
| 266 |
-
after, waited = None, 0
|
| 267 |
-
while True:
|
| 268 |
-
call = dict(params, limit=limit)
|
| 269 |
-
if after:
|
| 270 |
-
call["after"] = after
|
| 271 |
-
try:
|
| 272 |
-
body = _get(path, tok, **call)
|
| 273 |
-
except MetaError as e:
|
| 274 |
-
if _TOO_MUCH in str(e) and limit > 5:
|
| 275 |
-
limit = max(5, limit // 4)
|
| 276 |
-
log(f" page too heavy for /{path} - retrying at limit={limit} "
|
| 277 |
-
f"(a payload ceiling, not a row cap; every column is still asked for)")
|
| 278 |
-
continue
|
| 279 |
-
if _RATE_LIMITED in str(e).lower() and waited < _RATE_TRIES:
|
| 280 |
-
waited += 1
|
| 281 |
-
log(f" rate-limited on /{path} (per AD ACCOUNT, not per token) - waiting "
|
| 282 |
-
f"{_RATE_WAIT}s, attempt {waited}/{_RATE_TRIES}")
|
| 283 |
-
time.sleep(_RATE_WAIT)
|
| 284 |
-
continue
|
| 285 |
-
if _RATE_LIMITED in str(e).lower():
|
| 286 |
-
log(f" !! GIVING UP on /{path} after {waited} waits: still rate-limited. "
|
| 287 |
-
f"{len(out)} row(s) collected so far are kept. Cause: the ads-management "
|
| 288 |
-
f"limit is per ad account and persists for minutes. Fix: re-run "
|
| 289 |
-
f"`--sync --only <table>` later; the upsert is idempotent.")
|
| 290 |
-
return out, True
|
| 291 |
-
raise
|
| 292 |
-
out.extend(body.get("data") or [])
|
| 293 |
-
pages += 1
|
| 294 |
-
after = ((body.get("paging") or {}).get("cursors") or {}).get("after")
|
| 295 |
-
has_next = bool((body.get("paging") or {}).get("next")) and bool(after)
|
| 296 |
-
if not has_next:
|
| 297 |
-
return out, False
|
| 298 |
-
if pages >= MAX_PAGES:
|
| 299 |
-
log(f" !! STOPPED at MAX_PAGES={MAX_PAGES} on /{path} with more pages left. "
|
| 300 |
-
f"Cause: a runaway guard, not a row cap. Fix: raise META_MAX_PAGES, or narrow the "
|
| 301 |
-
f"window with META_INSIGHTS_DAYS.")
|
| 302 |
-
return out, True
|
| 303 |
-
|
| 304 |
-
|
| 305 |
-
def _cell(value):
|
| 306 |
-
"""One Graph value -> one DuckDB cell. Nested structures become compact JSON TEXT rather than
|
| 307 |
-
being dropped: R2 says every field the API returns, and `targeting` is a field."""
|
| 308 |
-
if value is None or isinstance(value, (str, int, float, bool)):
|
| 309 |
-
return value
|
| 310 |
-
return json.dumps(value, separators=(",", ":"), ensure_ascii=False)
|
| 311 |
-
|
| 312 |
-
|
| 313 |
-
def _coltype(name, rows=None):
|
| 314 |
-
"""The DuckDB type for one column β decided by the DATA when there is data, by the name only
|
| 315 |
-
as a fallback.
|
| 316 |
-
|
| 317 |
-
β THE NAME LIST WAS WRONG AND ONLY THE API COULD SAY SO. `cost_per_unique_click`,
|
| 318 |
-
`cost_per_action_type`, `purchase_roas` and friends READ like money and are **arrays of
|
| 319 |
-
action-type objects** on the Insights edge:
|
| 320 |
-
[{"action_type":"outbound_click","value":"1.459854"}]
|
| 321 |
-
Typed DOUBLE from `_DBL`, the insert died with `Conversion Error: Could not convert string
|
| 322 |
-
'[{...}]' to DOUBLE` β after five entity tables had already been written, so the sync looked
|
| 323 |
-
like it worked and then blew up on the last table.
|
| 324 |
-
β Same principle the field CATALOG is built on, applied one layer down: **ask the response,
|
| 325 |
-
do not assert from a list.** A value that ever arrives as a list or dict is JSON TEXT, because
|
| 326 |
-
that is what `_cell` stores; anything else falls back to the measured name hints.
|
| 327 |
-
"""
|
| 328 |
-
if rows:
|
| 329 |
-
seen, kinds = 0, set()
|
| 330 |
-
for r in rows:
|
| 331 |
-
v = r.get(name)
|
| 332 |
-
if v is None or v == "":
|
| 333 |
-
continue
|
| 334 |
-
kinds.add("json" if isinstance(v, (list, dict)) else
|
| 335 |
-
"bool" if isinstance(v, bool) else
|
| 336 |
-
"int" if isinstance(v, int) else
|
| 337 |
-
"float" if isinstance(v, float) else "str")
|
| 338 |
-
seen += 1
|
| 339 |
-
if seen >= 200:
|
| 340 |
-
break
|
| 341 |
-
if kinds:
|
| 342 |
-
if "json" in kinds or "str" in kinds:
|
| 343 |
-
return "VARCHAR" # a JSON blob, or a numeric STRING
|
| 344 |
-
if kinds <= {"int", "bool"}:
|
| 345 |
-
return "BIGINT" if name in _INT else ("VARCHAR" if "bool" in kinds else "BIGINT")
|
| 346 |
-
return "DOUBLE"
|
| 347 |
-
if name in _INT:
|
| 348 |
-
return "BIGINT"
|
| 349 |
-
if name in _DBL:
|
| 350 |
-
return "DOUBLE"
|
| 351 |
-
return "VARCHAR" # ids included β see the header
|
| 352 |
-
|
| 353 |
-
|
| 354 |
-
def _ensure(con, table, fields, rows=None):
|
| 355 |
-
"""Create or widen the table, with every column typed from `rows` when they are available.
|
| 356 |
-
|
| 357 |
-
β AN EXISTING COLUMN WHOSE TYPE IS NOW WRONG IS REBUILT, NOT PATCHED. DuckDB cannot retype a
|
| 358 |
-
column in place, and this table is DERIVED data that can be re-pulled in minutes β so a type
|
| 359 |
-
disagreement drops and recreates rather than limping on with a column that refuses every
|
| 360 |
-
insert. The alternative is a mirror that is permanently unwritable for one bad guess.
|
| 361 |
-
"""
|
| 362 |
-
want = {f: _coltype(f, rows) for f in fields}
|
| 363 |
-
have = {r[1]: str(r[2]).upper() for r in con.execute(f"PRAGMA table_info('{table}')").fetchall()}
|
| 364 |
-
if have:
|
| 365 |
-
clash = [f for f, ty in want.items() if f in have and have[f] != ty
|
| 366 |
-
and not (have[f].startswith("VARCHAR") and ty == "VARCHAR")]
|
| 367 |
-
if clash:
|
| 368 |
-
con.execute(f"DROP TABLE {table}")
|
| 369 |
-
have = {}
|
| 370 |
-
if not have:
|
| 371 |
-
cols = ", ".join(f"{f} {want[f]}" for f in fields)
|
| 372 |
-
con.execute(f"CREATE TABLE IF NOT EXISTS {table} (id VARCHAR PRIMARY KEY, {cols})"
|
| 373 |
-
if "id" not in fields else
|
| 374 |
-
f"CREATE TABLE IF NOT EXISTS {table} ({cols})")
|
| 375 |
-
return
|
| 376 |
-
for f in fields:
|
| 377 |
-
if f not in have:
|
| 378 |
-
con.execute(f"ALTER TABLE {table} ADD COLUMN {f} {want[f]}")
|
| 379 |
-
|
| 380 |
-
|
| 381 |
-
def _upsert(con, table, fields, rows):
|
| 382 |
-
"""Delete-then-insert by id β the same idempotence `datastore._upsert` gives the Odoo half, so
|
| 383 |
-
a re-sync updates in place and can never append a second copy of the same object."""
|
| 384 |
-
if not rows:
|
| 385 |
-
return 0
|
| 386 |
-
cols = list(fields)
|
| 387 |
-
ids = [str(r.get("id") or "") for r in rows]
|
| 388 |
-
q = ",".join("?" for _ in ids)
|
| 389 |
-
con.execute(f"DELETE FROM {table} WHERE id IN ({q})", ids)
|
| 390 |
-
con.executemany(
|
| 391 |
-
f"INSERT INTO {table} ({', '.join(cols)}) VALUES ({', '.join('?' for _ in cols)})",
|
| 392 |
-
[[_cell(r.get(c)) for c in cols] for r in rows])
|
| 393 |
-
return len(rows)
|
| 394 |
-
|
| 395 |
-
|
| 396 |
-
def sync(tenant_key="royal-imports", tok=None, log=print, insights=True, insights_days=None):
|
| 397 |
-
"""Pull every level into the tenant's mirror. -> a report dict; raises only on a bad token.
|
| 398 |
-
|
| 399 |
-
Idempotent: re-running updates rows in place. Safe to call at boot and on the resync loop,
|
| 400 |
-
exactly as `odoo_relational.refresh` is.
|
| 401 |
-
"""
|
| 402 |
-
tok = tok or token()
|
| 403 |
-
# β THE WINDOW IS A PARAMETER, NOT AN ENVIRONMENT READ AT CALL TIME β and the difference cost a
|
| 404 |
-
# live deploy. `INSIGHTS_DAYS` binds at IMPORT (module scope), so `main._pull_meta`'s
|
| 405 |
-
# `os.environ.setdefault("META_INSIGHTS_DAYS", "7")` executed AFTER this module was already
|
| 406 |
-
# imported and changed nothing: every boot pulled **90 days**, not 7, which is the slow path
|
| 407 |
-
# that trips the per-ad-account rate limit and never finishes. The comment beside that call
|
| 408 |
-
# claimed a short window the code could not deliver.
|
| 409 |
-
# β The shape: **a knob read at import cannot be turned by a caller at runtime.** Passing it
|
| 410 |
-
# makes the caller's intent effective instead of aspirational; the env var stays the DEFAULT.
|
| 411 |
-
days = int(insights_days or INSIGHTS_DAYS)
|
| 412 |
-
report = {"tenant": tenant_key, "tables": {}, "problems": [], "accounts": []}
|
| 413 |
-
if not tok:
|
| 414 |
-
report["problems"].append(
|
| 415 |
-
"META_ADS_ACCESS_TOKEN is not set in the environment or in platform/.env, so nothing "
|
| 416 |
-
"was pulled. This is a missing CREDENTIAL, not a missing capability.")
|
| 417 |
-
return report
|
| 418 |
-
|
| 419 |
-
path = datastore.path_for(tenant_key)
|
| 420 |
-
if Path(datastore.DB_PATH) != Path(path):
|
| 421 |
-
datastore.use_path(path)
|
| 422 |
-
con = datastore.connect()
|
| 423 |
-
log(f" mirror: {Path(path).name}")
|
| 424 |
-
|
| 425 |
-
accts, _ = _walk("me/adaccounts", tok, log, fields="id,name", limit=PAGE)
|
| 426 |
-
report["accounts"] = [a.get("id") for a in accts]
|
| 427 |
-
if not accts:
|
| 428 |
-
report["problems"].append("the token reaches no ad accounts")
|
| 429 |
-
return report
|
| 430 |
-
|
| 431 |
-
# ββ the five entity levels ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 432 |
-
for table, (edge, fields, _parent) in SPECS.items():
|
| 433 |
-
rows, capped = [], False
|
| 434 |
-
for acct in accts:
|
| 435 |
-
aid = acct["id"]
|
| 436 |
-
if edge is None:
|
| 437 |
-
rows.append(_get(aid, tok, fields=",".join(fields)))
|
| 438 |
-
else:
|
| 439 |
-
got, hit = _walk(f"{aid}/{edge}", tok, log, fields=",".join(fields), limit=PAGE)
|
| 440 |
-
rows.extend(got)
|
| 441 |
-
capped = capped or hit
|
| 442 |
-
_ensure(con, table, fields, rows)
|
| 443 |
-
n = _upsert(con, table, fields, rows)
|
| 444 |
-
total = con.execute(f"SELECT count(*) FROM {table}").fetchone()[0]
|
| 445 |
-
report["tables"][table] = {"pulled": n, "in_mirror": total, "capped": capped}
|
| 446 |
-
log(f" {table:<20} pulled {n:>6} mirror total {total:>6}"
|
| 447 |
-
+ (" !! PAGE CAP HIT" if capped else ""))
|
| 448 |
-
if capped:
|
| 449 |
-
report["problems"].append(f"{table}: stopped at MAX_PAGES={MAX_PAGES}")
|
| 450 |
-
|
| 451 |
-
# ββ the daily Insights grain ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 452 |
-
if insights:
|
| 453 |
-
rows, capped = [], False
|
| 454 |
-
for acct in accts:
|
| 455 |
-
for since, until in _slices(days, INSIGHTS_SLICE):
|
| 456 |
-
got, hit = _walk(f"{acct['id']}/insights", tok, log,
|
| 457 |
-
fields=",".join(INSIGHT_FIELDS), level=INSIGHTS_LEVEL,
|
| 458 |
-
time_increment="1", limit=INSIGHTS_PAGE,
|
| 459 |
-
time_range=json.dumps({"since": since, "until": until}))
|
| 460 |
-
rows.extend(got)
|
| 461 |
-
capped = capped or hit
|
| 462 |
-
# β Insights rows carry no id. The key must be STABLE across runs or every re-sync
|
| 463 |
-
# appends a second copy of the same day β so it is composed from the grain itself.
|
| 464 |
-
for r in rows:
|
| 465 |
-
r["id"] = ":".join(str(r.get(k) or "") for k in
|
| 466 |
-
(f"{INSIGHTS_LEVEL}_id", "date_start", "date_stop"))
|
| 467 |
-
fields = ["id"] + INSIGHT_FIELDS
|
| 468 |
-
_ensure(con, INSIGHTS_TABLE, fields, rows)
|
| 469 |
-
n = _upsert(con, INSIGHTS_TABLE, fields, rows)
|
| 470 |
-
total = con.execute(f"SELECT count(*) FROM {INSIGHTS_TABLE}").fetchone()[0]
|
| 471 |
-
report["tables"][INSIGHTS_TABLE] = {"pulled": n, "in_mirror": total, "capped": capped}
|
| 472 |
-
log(f" {INSIGHTS_TABLE:<20} pulled {n:>6} mirror total {total:>6}"
|
| 473 |
-
+ (" !! PAGE CAP HIT" if capped else ""))
|
| 474 |
-
|
| 475 |
-
con.execute("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
|
| 476 |
-
["meta", "done", 0, f"{days}d/{INSIGHTS_SLICE}d@{INSIGHTS_LEVEL}",
|
| 477 |
-
sum(t["in_mirror"] for t in report["tables"].values()),
|
| 478 |
-
time.strftime("%Y-%m-%d %H:%M:%S")])
|
| 479 |
-
return report
|
| 480 |
-
|
| 481 |
-
|
| 482 |
-
def status(tenant_key="royal-imports"):
|
| 483 |
-
"""What is in the mirror right now, per table. Never fetches."""
|
| 484 |
-
path = datastore.path_for(tenant_key)
|
| 485 |
-
if Path(datastore.DB_PATH) != Path(path):
|
| 486 |
-
datastore.use_path(path)
|
| 487 |
-
out = {}
|
| 488 |
-
con = datastore.connect()
|
| 489 |
-
for table in list(SPECS) + [INSIGHTS_TABLE]:
|
| 490 |
-
try:
|
| 491 |
-
out[table] = con.execute(f"SELECT count(*) FROM {table}").fetchone()[0]
|
| 492 |
-
except Exception:
|
| 493 |
-
out[table] = None # table absent = never synced
|
| 494 |
-
return out
|
| 495 |
-
|
| 496 |
-
|
| 497 |
-
def main(argv=None):
|
| 498 |
-
ap = argparse.ArgumentParser(description="Pull Meta Ads into the tenant's DuckDB mirror.")
|
| 499 |
-
ap.add_argument("--sync", action="store_true")
|
| 500 |
-
ap.add_argument("--status", action="store_true")
|
| 501 |
-
ap.add_argument("--tenant", default="royal-imports")
|
| 502 |
-
ap.add_argument("--no-insights", action="store_true")
|
| 503 |
-
ap.add_argument("--insights-days", type=int, default=None,
|
| 504 |
-
help="override the Insights window for THIS run (default INSIGHTS_DAYS); the env var is the default, this is the caller's say")
|
| 505 |
-
a = ap.parse_args(argv)
|
| 506 |
-
if a.status:
|
| 507 |
-
for k, v in status(a.tenant).items():
|
| 508 |
-
print(f" {k:<20} {'(never synced)' if v is None else v}")
|
| 509 |
-
return 0
|
| 510 |
-
if not a.sync:
|
| 511 |
-
ap.print_help()
|
| 512 |
-
return 2
|
| 513 |
-
rep = sync(a.tenant, insights=not a.no_insights, insights_days=a.insights_days)
|
| 514 |
-
for p in rep["problems"]:
|
| 515 |
-
print(" PROBLEM:", p)
|
| 516 |
-
print(f" accounts: {len(rep['accounts'])} tables: {len(rep['tables'])}")
|
| 517 |
-
return 1 if rep["problems"] else 0
|
| 518 |
-
|
| 519 |
-
|
| 520 |
-
if __name__ == "__main__":
|
| 521 |
-
sys.exit(main())
|
|
|
|
| 1 |
+
"""meta_store.py β the REST loader for a mirror whose only other loader is XML-RPC (W31-T48).
|
| 2 |
+
|
| 3 |
+
python platform/harness/meta_store.py --sync pull into tenant #0's mirror
|
| 4 |
+
python platform/harness/meta_store.py --sync --tenant gtmlab
|
| 5 |
+
python platform/harness/meta_store.py --status what is in the mirror now
|
| 6 |
+
|
| 7 |
+
β WHY THIS IS A SIBLING AND NOT A ROW IN `datastore.ENTITIES`. `datastore.sync_entity` is
|
| 8 |
+
hardwired to XML-RPC β it does `import core.odoo as O` and speaks `search_read` β so no amount of
|
| 9 |
+
spec data makes it fetch over HTTPS. What IS reusable is everything *below* the fetch: the
|
| 10 |
+
per-tenant file (`path_for`), the process-singleton connection (`connect`), and the
|
| 11 |
+
delete-then-insert upsert. This module borrows those and brings its own reader. **The mirror is one
|
| 12 |
+
store with two loaders, not two stores.**
|
| 13 |
+
|
| 14 |
+
β THE SHAPE OF THE WHOLE THING, because it is the owner's actual request: *"we just need to pull
|
| 15 |
+
the data into our template database for Meta"*, working *"just like Odoo"*. Odoo's path is
|
| 16 |
+
`XML-RPC -> DuckDB mirror -> odoo_relational -> ut_odoo_* locked grids`. Meta's is
|
| 17 |
+
`Graph -> DuckDB mirror -> meta_relational -> ut_meta_* locked grids`. Same middle, same end, one
|
| 18 |
+
different first hop.
|
| 19 |
+
|
| 20 |
+
β EVERY COLUMN NAME BELOW WAS MEASURED, NOT READ OFF A DOC (R8). `proto/meta-entity-fields.json`
|
| 21 |
+
carries the run: each level's list is what the API ACCEPTED when asked, on a real ad account.
|
| 22 |
+
Ad Account 45 Β· Campaign 35 Β· Ad Set 54 Β· Ad 36 Β· Creative 58 Β· Insights 57
|
| 23 |
+
β AND `ACCEPTED` IS NOT `RETURNED`. Graph omits null fields from a response, so a 20-field ask
|
| 24 |
+
came back with 14 keys. Building a schema from what came back would drop ~30% of the columns with
|
| 25 |
+
nothing going red β which is exactly the "do not drop any column" instruction, broken silently.
|
| 26 |
+
The lists below are therefore the ASKED-AND-ACCEPTED set, and a column with no value is NULL.
|
| 27 |
+
|
| 28 |
+
β `owner` IS ABSENT FROM THE AD ACCOUNT LIST AND THAT IS A REPORTED LIMIT, NOT AN OMISSION: it
|
| 29 |
+
answers `403 (#200) Requires business_management permission`, which this token does not carry. One
|
| 30 |
+
column of 285. R6's second sentence β a limit that cannot be removed gets stated with its cause.
|
| 31 |
+
|
| 32 |
+
β IDS ARE TEXT, ALWAYS. A Meta object id is a 17-digit decimal string; `120273975028650555` is
|
| 33 |
+
larger than 2^53, so any float or JS-number path silently corrupts it. Same ruling as `ig_id`
|
| 34 |
+
(W26/R3), for the same reason, and it is why every `id` column here is VARCHAR.
|
| 35 |
+
"""
|
| 36 |
+
import argparse
|
| 37 |
+
import json
|
| 38 |
+
import os
|
| 39 |
+
import sys
|
| 40 |
+
import time
|
| 41 |
+
import urllib.error
|
| 42 |
+
import urllib.parse
|
| 43 |
+
import urllib.request
|
| 44 |
+
from pathlib import Path
|
| 45 |
+
|
| 46 |
+
_HERE = Path(__file__).resolve().parent
|
| 47 |
+
if str(_HERE.parent) not in sys.path:
|
| 48 |
+
sys.path.insert(0, str(_HERE.parent))
|
| 49 |
+
|
| 50 |
+
from harness import datastore # noqa: E402
|
| 51 |
+
|
| 52 |
+
GRAPH_VERSION = os.environ.get("META_GRAPH_VERSION") or "v21.0"
|
| 53 |
+
GRAPH = f"https://graph.facebook.com/{GRAPH_VERSION}"
|
| 54 |
+
|
| 55 |
+
#: Numeric columns, by name. Everything else is VARCHAR β including ids (see the header) and
|
| 56 |
+
#: including anything Graph returns as a nested object, which is stored as compact JSON text.
|
| 57 |
+
_INT = {"impressions", "reach", "clicks", "unique_clicks", "inline_link_clicks",
|
| 58 |
+
"inline_post_engagement", "full_view_impressions", "estimated_ad_recallers",
|
| 59 |
+
"account_status", "age", "timezone_id", "timezone_offset_hours_utc", "io_number",
|
| 60 |
+
"min_daily_budget", "min_campaign_group_spend_cap"}
|
| 61 |
+
_DBL = {"spend", "social_spend", "ctr", "unique_ctr", "cpc", "cpm", "cpp", "frequency",
|
| 62 |
+
"inline_link_click_ctr", "outbound_clicks_ctr", "amount_spent", "balance", "spend_cap",
|
| 63 |
+
"daily_budget", "lifetime_budget", "budget_remaining", "bid_amount", "canvas_avg_view_time",
|
| 64 |
+
"daily_min_spend_target", "daily_spend_cap", "lifetime_min_spend_target",
|
| 65 |
+
"lifetime_spend_cap", "lifetime_imps"}
|
| 66 |
+
|
| 67 |
+
#: β THE MEASURED CATALOG. `edge` is the connection on the ad account; `None` means the account
|
| 68 |
+
#: itself. `parent` names the column that ties a row to its parent, used by nothing here and by
|
| 69 |
+
#: the grid links in `meta_relational`.
|
| 70 |
+
ACCOUNT_FIELDS = (
|
| 71 |
+
"account_id account_status age amount_spent balance business_city business_country_code "
|
| 72 |
+
"business_name business_state business_street business_street2 business_zip capabilities "
|
| 73 |
+
"created_time currency disable_reason end_advertiser end_advertiser_name funding_source "
|
| 74 |
+
"has_migrated_permissions id io_number is_attribution_spec_system_default "
|
| 75 |
+
"is_direct_deals_enabled is_notifications_enabled is_personal is_prepay_account "
|
| 76 |
+
"is_tax_id_required line_numbers media_agency min_campaign_group_spend_cap min_daily_budget "
|
| 77 |
+
"name offsite_pixels_tos_accepted partner spend_cap tax_id tax_id_status tax_id_type "
|
| 78 |
+
"timezone_id timezone_name timezone_offset_hours_utc tos_accepted user_tasks user_tos_accepted"
|
| 79 |
+
).split()
|
| 80 |
+
|
| 81 |
+
CAMPAIGN_FIELDS = (
|
| 82 |
+
"account_id bid_strategy boosted_object_id budget_rebalance_flag budget_remaining buying_type "
|
| 83 |
+
"campaign_group_active_time can_create_brand_lift_study can_use_spend_cap configured_status "
|
| 84 |
+
"created_time daily_budget effective_status id is_skadnetwork_attribution issues_info "
|
| 85 |
+
"last_budget_toggling_time lifetime_budget name objective pacing_type primary_attribution "
|
| 86 |
+
"promoted_object smart_promotion_type source_campaign source_campaign_id special_ad_categories "
|
| 87 |
+
"special_ad_category special_ad_category_country spend_cap start_time status stop_time "
|
| 88 |
+
"topline_id updated_time"
|
| 89 |
+
).split()
|
| 90 |
+
|
| 91 |
+
ADSET_FIELDS = (
|
| 92 |
+
"account_id adlabels adset_schedule asset_feed_id attribution_spec bid_adjustments bid_amount "
|
| 93 |
+
"bid_constraints bid_info bid_strategy billing_event budget_remaining campaign "
|
| 94 |
+
"campaign_active_time campaign_attribution campaign_id configured_status created_time "
|
| 95 |
+
"creative_sequence daily_budget daily_min_spend_target daily_spend_cap destination_type "
|
| 96 |
+
"effective_status end_time frequency_control_specs id instagram_actor_id is_dynamic_creative "
|
| 97 |
+
"issues_info learning_stage_info lifetime_budget lifetime_imps lifetime_min_spend_target "
|
| 98 |
+
"lifetime_spend_cap multi_optimization_goal_weight name optimization_goal "
|
| 99 |
+
"optimization_sub_event pacing_type promoted_object recurring_budget_semantics review_feedback "
|
| 100 |
+
"rf_prediction_id source_adset source_adset_id start_time status targeting "
|
| 101 |
+
"targeting_optimization_types time_based_ad_rotation_id_blocks "
|
| 102 |
+
"time_based_ad_rotation_intervals updated_time use_new_app_click"
|
| 103 |
+
).split()
|
| 104 |
+
|
| 105 |
+
AD_FIELDS = (
|
| 106 |
+
"account_id ad_active_time ad_review_feedback ad_schedule_end_time ad_schedule_start_time "
|
| 107 |
+
"adlabels adset adset_id bid_amount bid_info bid_type campaign campaign_id configured_status "
|
| 108 |
+
"conversion_domain created_time creative demolink_hash display_sequence effective_status "
|
| 109 |
+
"engagement_audience failed_delivery_checks id issues_info last_updated_by_app_id name "
|
| 110 |
+
"preview_shareable_link priority recommendations source_ad source_ad_id status targeting "
|
| 111 |
+
"tracking_and_conversion_with_defaults tracking_specs updated_time"
|
| 112 |
+
).split()
|
| 113 |
+
|
| 114 |
+
CREATIVE_FIELDS = (
|
| 115 |
+
"account_id actor_id adlabels applink_treatment asset_feed_spec authorization_category body "
|
| 116 |
+
"branded_content_sponsor_page_id bundle_folder_id call_to_action_type categorization_criteria "
|
| 117 |
+
"category_media_source collaborative_ads_lsb_image_bank_id degrees_of_freedom_spec "
|
| 118 |
+
"destination_set_id dynamic_ad_voice effective_authorization_category "
|
| 119 |
+
"effective_instagram_media_id effective_object_story_id enable_direct_install "
|
| 120 |
+
"enable_launch_instant_app id image_crops image_hash image_url instagram_actor_id "
|
| 121 |
+
"instagram_permalink_url instagram_story_id instagram_user_id interactive_components_spec "
|
| 122 |
+
"link_deep_link_url link_destination_display_url link_og_id link_url "
|
| 123 |
+
"messenger_sponsored_message name object_id object_store_url object_story_id object_story_spec "
|
| 124 |
+
"object_type object_url place_page_set_id platform_customizations playable_asset_id "
|
| 125 |
+
"portrait_customizations product_set_id recommender_settings source_instagram_media_id status "
|
| 126 |
+
"template_url template_url_spec thumbnail_id thumbnail_url title url_tags "
|
| 127 |
+
"use_page_actor_override video_id"
|
| 128 |
+
).split()
|
| 129 |
+
|
| 130 |
+
INSIGHT_FIELDS = (
|
| 131 |
+
"account_currency account_id account_name action_values actions ad_id ad_name adset_id "
|
| 132 |
+
"adset_name attribution_setting buying_type campaign_id campaign_name canvas_avg_view_time "
|
| 133 |
+
"clicks conversion_rate_ranking conversion_values conversions cost_per_action_type "
|
| 134 |
+
"cost_per_inline_link_click cost_per_thruplay cost_per_unique_click cpc cpm cpp ctr date_start "
|
| 135 |
+
"date_stop engagement_rate_ranking estimated_ad_recallers frequency full_view_impressions "
|
| 136 |
+
"impressions inline_link_click_ctr inline_link_clicks inline_post_engagement objective "
|
| 137 |
+
"optimization_goal outbound_clicks outbound_clicks_ctr purchase_roas quality_ranking reach "
|
| 138 |
+
"social_spend spend unique_clicks unique_ctr unique_outbound_clicks "
|
| 139 |
+
"video_avg_time_watched_actions video_p100_watched_actions video_p25_watched_actions "
|
| 140 |
+
"video_p50_watched_actions video_p75_watched_actions video_p95_watched_actions "
|
| 141 |
+
"video_play_actions video_thruplay_watched_actions website_purchase_roas"
|
| 142 |
+
).split()
|
| 143 |
+
|
| 144 |
+
#: table -> (graph edge on the account | None, measured fields, parent column)
|
| 145 |
+
SPECS = {
|
| 146 |
+
"meta_ad_accounts": (None, ACCOUNT_FIELDS, None),
|
| 147 |
+
"meta_campaigns": ("campaigns", CAMPAIGN_FIELDS, "account_id"),
|
| 148 |
+
"meta_adsets": ("adsets", ADSET_FIELDS, "campaign_id"),
|
| 149 |
+
"meta_ads": ("ads", AD_FIELDS, "adset_id"),
|
| 150 |
+
"meta_creatives": ("adcreatives", CREATIVE_FIELDS, "account_id"),
|
| 151 |
+
}
|
| 152 |
+
|
| 153 |
+
#: The daily Insights grain (R2). One row per (ad, day) β the id is synthesised because Insights
|
| 154 |
+
#: has no id of its own, and it must be STABLE so a re-run updates instead of appending.
|
| 155 |
+
INSIGHTS_TABLE = "meta_insights_daily"
|
| 156 |
+
INSIGHTS_LEVEL = os.environ.get("META_INSIGHTS_LEVEL") or "ad"
|
| 157 |
+
|
| 158 |
+
#: β INSIGHTS IS FETCHED IN TIME SLICES, AND THE REASON IS MEASURED, NOT DEFENSIVE. All 57 fields
|
| 159 |
+
#: at ad level over `last_90d` with a 100-row page answers **HTTP 500 "An unknown error occurred"**
|
| 160 |
+
#: β Graph's way of saying the synchronous query is too heavy (the async report-run API is the
|
| 161 |
+
#: other answer, and it costs a poll loop this does not need). The SAME 57 fields over 7 days at
|
| 162 |
+
#: page 25 answer 200. So the window is walked in slices with every column intact:
|
| 163 |
+
#: 57 fields Β· ad level Β· 7d Β· limit 25 -> 200, 25 rows
|
| 164 |
+
#: 57 fields Β· account level Β· 7d -> 200, 7 rows
|
| 165 |
+
#: β Narrowing the FIELD list would also have "fixed" it, and that is the wrong fix twice over β
|
| 166 |
+
#: it drops columns R2 requires, and it does so invisibly.
|
| 167 |
+
INSIGHTS_DAYS = int(os.environ.get("META_INSIGHTS_DAYS") or 90)
|
| 168 |
+
INSIGHTS_SLICE = int(os.environ.get("META_INSIGHTS_SLICE_DAYS") or 7)
|
| 169 |
+
INSIGHTS_PAGE = int(os.environ.get("META_INSIGHTS_PAGE") or 25)
|
| 170 |
+
|
| 171 |
+
|
| 172 |
+
def _slices(days, size):
|
| 173 |
+
"""[(since, until)] covering the last `days`, oldest first, in `size`-day windows."""
|
| 174 |
+
from datetime import date, timedelta
|
| 175 |
+
end = date.today()
|
| 176 |
+
start = end - timedelta(days=max(1, days) - 1)
|
| 177 |
+
out, cur = [], start
|
| 178 |
+
while cur <= end:
|
| 179 |
+
stop = min(cur + timedelta(days=max(1, size) - 1), end)
|
| 180 |
+
out.append((cur.isoformat(), stop.isoformat()))
|
| 181 |
+
cur = stop + timedelta(days=1)
|
| 182 |
+
return out
|
| 183 |
+
|
| 184 |
+
#: β A REAL CEILING, AND R6's SECOND SENTENCE APPLIES TO IT. Graph pages at 25-100 rows; this is
|
| 185 |
+
#: the number of PAGES a single edge may walk before the loader stops and SAYS it stopped. It is
|
| 186 |
+
#: not a row cap on a connected source (R6 forbids that) β it is a runaway guard, and reaching it
|
| 187 |
+
#: is reported as a problem, never absorbed.
|
| 188 |
+
MAX_PAGES = int(os.environ.get("META_MAX_PAGES") or 200)
|
| 189 |
+
PAGE = int(os.environ.get("META_PAGE_SIZE") or 100)
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
class MetaError(RuntimeError):
|
| 193 |
+
"""A Graph refusal carrying Meta's own words. Safe to print: no token ever reaches it."""
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
def token():
|
| 197 |
+
"""The Meta token from the environment or gitignored `platform/.env`; "" when absent.
|
| 198 |
+
|
| 199 |
+
β Same resolver `aios-web/api/connectors_meta.py` uses. Duplicated deliberately and minimally:
|
| 200 |
+
`platform/` must not import from `aios-web/api/`, which is the layering rule this repo keeps
|
| 201 |
+
(`core` never imports up). Fifteen lines is the price of that boundary.
|
| 202 |
+
"""
|
| 203 |
+
tok = os.environ.get("META_ADS_ACCESS_TOKEN") or ""
|
| 204 |
+
if tok:
|
| 205 |
+
return tok.strip()
|
| 206 |
+
env = _HERE.parent / ".env"
|
| 207 |
+
if env.exists():
|
| 208 |
+
for line in env.read_text(encoding="utf-8", errors="replace").splitlines():
|
| 209 |
+
if line.strip().startswith("META_ADS_ACCESS_TOKEN"):
|
| 210 |
+
_, _, v = line.partition("=")
|
| 211 |
+
return v.strip().strip('"').strip("'")
|
| 212 |
+
return ""
|
| 213 |
+
|
| 214 |
+
|
| 215 |
+
def _get(path, tok, **params):
|
| 216 |
+
params["access_token"] = tok
|
| 217 |
+
url = f"{GRAPH}/{path.lstrip('/')}?" + urllib.parse.urlencode(params)
|
| 218 |
+
req = urllib.request.Request(url, headers={"User-Agent": "aios-meta-store/1"})
|
| 219 |
+
try:
|
| 220 |
+
with urllib.request.urlopen(req, timeout=120) as r:
|
| 221 |
+
return json.loads(r.read().decode("utf-8", "replace"))
|
| 222 |
+
except urllib.error.HTTPError as e:
|
| 223 |
+
try:
|
| 224 |
+
msg = (json.loads(e.read().decode("utf-8", "replace")).get("error") or {}
|
| 225 |
+
).get("message") or ""
|
| 226 |
+
except Exception:
|
| 227 |
+
msg = ""
|
| 228 |
+
raise MetaError(f"HTTP {e.code} on /{path.lstrip('/')}: {msg[:200]}") from None
|
| 229 |
+
except Exception as e:
|
| 230 |
+
raise MetaError(f"{type(e).__name__} on /{path.lstrip('/')}") from None
|
| 231 |
+
|
| 232 |
+
|
| 233 |
+
#: Graph's own words when a page is too heavy. It arrives as an HTTP **500**, not a 4xx, which is
|
| 234 |
+
#: why it cannot be treated as "the server is broken, give up".
|
| 235 |
+
_TOO_MUCH = "reduce the amount of data"
|
| 236 |
+
|
| 237 |
+
#: β THE ADS-MANAGEMENT RATE LIMIT, WHICH IS PER AD ACCOUNT AND NOT PER TOKEN. Measured: after a
|
| 238 |
+
#: heavy backfill Graph answers **HTTP 400 "There have been too many calls to this ad-account.
|
| 239 |
+
#: Wait a bit and try again."** It is a 4xx, so nothing about the status code says "retry" β the
|
| 240 |
+
#: MESSAGE is the only signal, which is why it is matched here rather than inferred from a code.
|
| 241 |
+
#: β It persists for minutes, so the backoff is measured in minutes and BOUNDED: after
|
| 242 |
+
#: `_RATE_TRIES` waits the loader STOPS and reports what it got, rather than sitting in a retry
|
| 243 |
+
#: loop nobody can see. A partial mirror that says it is partial beats a hung sync.
|
| 244 |
+
_RATE_LIMITED = "too many calls"
|
| 245 |
+
_RATE_TRIES = int(os.environ.get("META_RATE_TRIES") or 3)
|
| 246 |
+
_RATE_WAIT = int(os.environ.get("META_RATE_WAIT_S") or 90)
|
| 247 |
+
|
| 248 |
+
|
| 249 |
+
def _walk(path, tok, log, **params):
|
| 250 |
+
"""Every page of an edge, paged by CURSOR under our own parameters.
|
| 251 |
+
|
| 252 |
+
β IT DOES NOT FOLLOW GRAPH'S `paging.next` URL, AND THAT IS THE WHOLE POINT. Measured: the
|
| 253 |
+
first call to `/adcreatives` at limit=100 answers **HTTP 500 "Please reduce the amount of data
|
| 254 |
+
you're asking for"**, the retry at limit=25 succeeds β and then `next` carries the ORIGINAL
|
| 255 |
+
limit=100 and fails again on page 2. A backoff that cannot reach every page is not a backoff.
|
| 256 |
+
Re-issuing each page ourselves with `after=<cursor>` keeps the reduced limit for the whole walk.
|
| 257 |
+
|
| 258 |
+
β It shrinks the PAGE, never the FIELD LIST. Dropping columns to make a request fit is the
|
| 259 |
+
silent omission R2 forbids, and nothing downstream could see it. Fewer rows per call, always
|
| 260 |
+
every column per row.
|
| 261 |
+
|
| 262 |
+
-> (rows, hit_page_cap)
|
| 263 |
+
"""
|
| 264 |
+
out, pages = [], 0
|
| 265 |
+
limit = int(params.pop("limit", None) or PAGE)
|
| 266 |
+
after, waited = None, 0
|
| 267 |
+
while True:
|
| 268 |
+
call = dict(params, limit=limit)
|
| 269 |
+
if after:
|
| 270 |
+
call["after"] = after
|
| 271 |
+
try:
|
| 272 |
+
body = _get(path, tok, **call)
|
| 273 |
+
except MetaError as e:
|
| 274 |
+
if _TOO_MUCH in str(e) and limit > 5:
|
| 275 |
+
limit = max(5, limit // 4)
|
| 276 |
+
log(f" page too heavy for /{path} - retrying at limit={limit} "
|
| 277 |
+
f"(a payload ceiling, not a row cap; every column is still asked for)")
|
| 278 |
+
continue
|
| 279 |
+
if _RATE_LIMITED in str(e).lower() and waited < _RATE_TRIES:
|
| 280 |
+
waited += 1
|
| 281 |
+
log(f" rate-limited on /{path} (per AD ACCOUNT, not per token) - waiting "
|
| 282 |
+
f"{_RATE_WAIT}s, attempt {waited}/{_RATE_TRIES}")
|
| 283 |
+
time.sleep(_RATE_WAIT)
|
| 284 |
+
continue
|
| 285 |
+
if _RATE_LIMITED in str(e).lower():
|
| 286 |
+
log(f" !! GIVING UP on /{path} after {waited} waits: still rate-limited. "
|
| 287 |
+
f"{len(out)} row(s) collected so far are kept. Cause: the ads-management "
|
| 288 |
+
f"limit is per ad account and persists for minutes. Fix: re-run "
|
| 289 |
+
f"`--sync --only <table>` later; the upsert is idempotent.")
|
| 290 |
+
return out, True
|
| 291 |
+
raise
|
| 292 |
+
out.extend(body.get("data") or [])
|
| 293 |
+
pages += 1
|
| 294 |
+
after = ((body.get("paging") or {}).get("cursors") or {}).get("after")
|
| 295 |
+
has_next = bool((body.get("paging") or {}).get("next")) and bool(after)
|
| 296 |
+
if not has_next:
|
| 297 |
+
return out, False
|
| 298 |
+
if pages >= MAX_PAGES:
|
| 299 |
+
log(f" !! STOPPED at MAX_PAGES={MAX_PAGES} on /{path} with more pages left. "
|
| 300 |
+
f"Cause: a runaway guard, not a row cap. Fix: raise META_MAX_PAGES, or narrow the "
|
| 301 |
+
f"window with META_INSIGHTS_DAYS.")
|
| 302 |
+
return out, True
|
| 303 |
+
|
| 304 |
+
|
| 305 |
+
def _cell(value):
|
| 306 |
+
"""One Graph value -> one DuckDB cell. Nested structures become compact JSON TEXT rather than
|
| 307 |
+
being dropped: R2 says every field the API returns, and `targeting` is a field."""
|
| 308 |
+
if value is None or isinstance(value, (str, int, float, bool)):
|
| 309 |
+
return value
|
| 310 |
+
return json.dumps(value, separators=(",", ":"), ensure_ascii=False)
|
| 311 |
+
|
| 312 |
+
|
| 313 |
+
def _coltype(name, rows=None):
|
| 314 |
+
"""The DuckDB type for one column β decided by the DATA when there is data, by the name only
|
| 315 |
+
as a fallback.
|
| 316 |
+
|
| 317 |
+
β THE NAME LIST WAS WRONG AND ONLY THE API COULD SAY SO. `cost_per_unique_click`,
|
| 318 |
+
`cost_per_action_type`, `purchase_roas` and friends READ like money and are **arrays of
|
| 319 |
+
action-type objects** on the Insights edge:
|
| 320 |
+
[{"action_type":"outbound_click","value":"1.459854"}]
|
| 321 |
+
Typed DOUBLE from `_DBL`, the insert died with `Conversion Error: Could not convert string
|
| 322 |
+
'[{...}]' to DOUBLE` β after five entity tables had already been written, so the sync looked
|
| 323 |
+
like it worked and then blew up on the last table.
|
| 324 |
+
β Same principle the field CATALOG is built on, applied one layer down: **ask the response,
|
| 325 |
+
do not assert from a list.** A value that ever arrives as a list or dict is JSON TEXT, because
|
| 326 |
+
that is what `_cell` stores; anything else falls back to the measured name hints.
|
| 327 |
+
"""
|
| 328 |
+
if rows:
|
| 329 |
+
seen, kinds = 0, set()
|
| 330 |
+
for r in rows:
|
| 331 |
+
v = r.get(name)
|
| 332 |
+
if v is None or v == "":
|
| 333 |
+
continue
|
| 334 |
+
kinds.add("json" if isinstance(v, (list, dict)) else
|
| 335 |
+
"bool" if isinstance(v, bool) else
|
| 336 |
+
"int" if isinstance(v, int) else
|
| 337 |
+
"float" if isinstance(v, float) else "str")
|
| 338 |
+
seen += 1
|
| 339 |
+
if seen >= 200:
|
| 340 |
+
break
|
| 341 |
+
if kinds:
|
| 342 |
+
if "json" in kinds or "str" in kinds:
|
| 343 |
+
return "VARCHAR" # a JSON blob, or a numeric STRING
|
| 344 |
+
if kinds <= {"int", "bool"}:
|
| 345 |
+
return "BIGINT" if name in _INT else ("VARCHAR" if "bool" in kinds else "BIGINT")
|
| 346 |
+
return "DOUBLE"
|
| 347 |
+
if name in _INT:
|
| 348 |
+
return "BIGINT"
|
| 349 |
+
if name in _DBL:
|
| 350 |
+
return "DOUBLE"
|
| 351 |
+
return "VARCHAR" # ids included β see the header
|
| 352 |
+
|
| 353 |
+
|
| 354 |
+
def _ensure(con, table, fields, rows=None):
|
| 355 |
+
"""Create or widen the table, with every column typed from `rows` when they are available.
|
| 356 |
+
|
| 357 |
+
β AN EXISTING COLUMN WHOSE TYPE IS NOW WRONG IS REBUILT, NOT PATCHED. DuckDB cannot retype a
|
| 358 |
+
column in place, and this table is DERIVED data that can be re-pulled in minutes β so a type
|
| 359 |
+
disagreement drops and recreates rather than limping on with a column that refuses every
|
| 360 |
+
insert. The alternative is a mirror that is permanently unwritable for one bad guess.
|
| 361 |
+
"""
|
| 362 |
+
want = {f: _coltype(f, rows) for f in fields}
|
| 363 |
+
have = {r[1]: str(r[2]).upper() for r in con.execute(f"PRAGMA table_info('{table}')").fetchall()}
|
| 364 |
+
if have:
|
| 365 |
+
clash = [f for f, ty in want.items() if f in have and have[f] != ty
|
| 366 |
+
and not (have[f].startswith("VARCHAR") and ty == "VARCHAR")]
|
| 367 |
+
if clash:
|
| 368 |
+
con.execute(f"DROP TABLE {table}")
|
| 369 |
+
have = {}
|
| 370 |
+
if not have:
|
| 371 |
+
cols = ", ".join(f"{f} {want[f]}" for f in fields)
|
| 372 |
+
con.execute(f"CREATE TABLE IF NOT EXISTS {table} (id VARCHAR PRIMARY KEY, {cols})"
|
| 373 |
+
if "id" not in fields else
|
| 374 |
+
f"CREATE TABLE IF NOT EXISTS {table} ({cols})")
|
| 375 |
+
return
|
| 376 |
+
for f in fields:
|
| 377 |
+
if f not in have:
|
| 378 |
+
con.execute(f"ALTER TABLE {table} ADD COLUMN {f} {want[f]}")
|
| 379 |
+
|
| 380 |
+
|
| 381 |
+
def _upsert(con, table, fields, rows):
|
| 382 |
+
"""Delete-then-insert by id β the same idempotence `datastore._upsert` gives the Odoo half, so
|
| 383 |
+
a re-sync updates in place and can never append a second copy of the same object."""
|
| 384 |
+
if not rows:
|
| 385 |
+
return 0
|
| 386 |
+
cols = list(fields)
|
| 387 |
+
ids = [str(r.get("id") or "") for r in rows]
|
| 388 |
+
q = ",".join("?" for _ in ids)
|
| 389 |
+
con.execute(f"DELETE FROM {table} WHERE id IN ({q})", ids)
|
| 390 |
+
con.executemany(
|
| 391 |
+
f"INSERT INTO {table} ({', '.join(cols)}) VALUES ({', '.join('?' for _ in cols)})",
|
| 392 |
+
[[_cell(r.get(c)) for c in cols] for r in rows])
|
| 393 |
+
return len(rows)
|
| 394 |
+
|
| 395 |
+
|
| 396 |
+
def sync(tenant_key="royal-imports", tok=None, log=print, insights=True, insights_days=None):
|
| 397 |
+
"""Pull every level into the tenant's mirror. -> a report dict; raises only on a bad token.
|
| 398 |
+
|
| 399 |
+
Idempotent: re-running updates rows in place. Safe to call at boot and on the resync loop,
|
| 400 |
+
exactly as `odoo_relational.refresh` is.
|
| 401 |
+
"""
|
| 402 |
+
tok = tok or token()
|
| 403 |
+
# β THE WINDOW IS A PARAMETER, NOT AN ENVIRONMENT READ AT CALL TIME β and the difference cost a
|
| 404 |
+
# live deploy. `INSIGHTS_DAYS` binds at IMPORT (module scope), so `main._pull_meta`'s
|
| 405 |
+
# `os.environ.setdefault("META_INSIGHTS_DAYS", "7")` executed AFTER this module was already
|
| 406 |
+
# imported and changed nothing: every boot pulled **90 days**, not 7, which is the slow path
|
| 407 |
+
# that trips the per-ad-account rate limit and never finishes. The comment beside that call
|
| 408 |
+
# claimed a short window the code could not deliver.
|
| 409 |
+
# β The shape: **a knob read at import cannot be turned by a caller at runtime.** Passing it
|
| 410 |
+
# makes the caller's intent effective instead of aspirational; the env var stays the DEFAULT.
|
| 411 |
+
days = int(insights_days or INSIGHTS_DAYS)
|
| 412 |
+
report = {"tenant": tenant_key, "tables": {}, "problems": [], "accounts": []}
|
| 413 |
+
if not tok:
|
| 414 |
+
report["problems"].append(
|
| 415 |
+
"META_ADS_ACCESS_TOKEN is not set in the environment or in platform/.env, so nothing "
|
| 416 |
+
"was pulled. This is a missing CREDENTIAL, not a missing capability.")
|
| 417 |
+
return report
|
| 418 |
+
|
| 419 |
+
path = datastore.path_for(tenant_key)
|
| 420 |
+
if Path(datastore.DB_PATH) != Path(path):
|
| 421 |
+
datastore.use_path(path)
|
| 422 |
+
con = datastore.connect()
|
| 423 |
+
log(f" mirror: {Path(path).name}")
|
| 424 |
+
|
| 425 |
+
accts, _ = _walk("me/adaccounts", tok, log, fields="id,name", limit=PAGE)
|
| 426 |
+
report["accounts"] = [a.get("id") for a in accts]
|
| 427 |
+
if not accts:
|
| 428 |
+
report["problems"].append("the token reaches no ad accounts")
|
| 429 |
+
return report
|
| 430 |
+
|
| 431 |
+
# ββ the five entity levels ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 432 |
+
for table, (edge, fields, _parent) in SPECS.items():
|
| 433 |
+
rows, capped = [], False
|
| 434 |
+
for acct in accts:
|
| 435 |
+
aid = acct["id"]
|
| 436 |
+
if edge is None:
|
| 437 |
+
rows.append(_get(aid, tok, fields=",".join(fields)))
|
| 438 |
+
else:
|
| 439 |
+
got, hit = _walk(f"{aid}/{edge}", tok, log, fields=",".join(fields), limit=PAGE)
|
| 440 |
+
rows.extend(got)
|
| 441 |
+
capped = capped or hit
|
| 442 |
+
_ensure(con, table, fields, rows)
|
| 443 |
+
n = _upsert(con, table, fields, rows)
|
| 444 |
+
total = con.execute(f"SELECT count(*) FROM {table}").fetchone()[0]
|
| 445 |
+
report["tables"][table] = {"pulled": n, "in_mirror": total, "capped": capped}
|
| 446 |
+
log(f" {table:<20} pulled {n:>6} mirror total {total:>6}"
|
| 447 |
+
+ (" !! PAGE CAP HIT" if capped else ""))
|
| 448 |
+
if capped:
|
| 449 |
+
report["problems"].append(f"{table}: stopped at MAX_PAGES={MAX_PAGES}")
|
| 450 |
+
|
| 451 |
+
# ββ the daily Insights grain ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 452 |
+
if insights:
|
| 453 |
+
rows, capped = [], False
|
| 454 |
+
for acct in accts:
|
| 455 |
+
for since, until in _slices(days, INSIGHTS_SLICE):
|
| 456 |
+
got, hit = _walk(f"{acct['id']}/insights", tok, log,
|
| 457 |
+
fields=",".join(INSIGHT_FIELDS), level=INSIGHTS_LEVEL,
|
| 458 |
+
time_increment="1", limit=INSIGHTS_PAGE,
|
| 459 |
+
time_range=json.dumps({"since": since, "until": until}))
|
| 460 |
+
rows.extend(got)
|
| 461 |
+
capped = capped or hit
|
| 462 |
+
# β Insights rows carry no id. The key must be STABLE across runs or every re-sync
|
| 463 |
+
# appends a second copy of the same day β so it is composed from the grain itself.
|
| 464 |
+
for r in rows:
|
| 465 |
+
r["id"] = ":".join(str(r.get(k) or "") for k in
|
| 466 |
+
(f"{INSIGHTS_LEVEL}_id", "date_start", "date_stop"))
|
| 467 |
+
fields = ["id"] + INSIGHT_FIELDS
|
| 468 |
+
_ensure(con, INSIGHTS_TABLE, fields, rows)
|
| 469 |
+
n = _upsert(con, INSIGHTS_TABLE, fields, rows)
|
| 470 |
+
total = con.execute(f"SELECT count(*) FROM {INSIGHTS_TABLE}").fetchone()[0]
|
| 471 |
+
report["tables"][INSIGHTS_TABLE] = {"pulled": n, "in_mirror": total, "capped": capped}
|
| 472 |
+
log(f" {INSIGHTS_TABLE:<20} pulled {n:>6} mirror total {total:>6}"
|
| 473 |
+
+ (" !! PAGE CAP HIT" if capped else ""))
|
| 474 |
+
|
| 475 |
+
con.execute("INSERT OR REPLACE INTO _sync_state VALUES (?,?,?,?,?,?)",
|
| 476 |
+
["meta", "done", 0, f"{days}d/{INSIGHTS_SLICE}d@{INSIGHTS_LEVEL}",
|
| 477 |
+
sum(t["in_mirror"] for t in report["tables"].values()),
|
| 478 |
+
time.strftime("%Y-%m-%d %H:%M:%S")])
|
| 479 |
+
return report
|
| 480 |
+
|
| 481 |
+
|
| 482 |
+
def status(tenant_key="royal-imports"):
|
| 483 |
+
"""What is in the mirror right now, per table. Never fetches."""
|
| 484 |
+
path = datastore.path_for(tenant_key)
|
| 485 |
+
if Path(datastore.DB_PATH) != Path(path):
|
| 486 |
+
datastore.use_path(path)
|
| 487 |
+
out = {}
|
| 488 |
+
con = datastore.connect()
|
| 489 |
+
for table in list(SPECS) + [INSIGHTS_TABLE]:
|
| 490 |
+
try:
|
| 491 |
+
out[table] = con.execute(f"SELECT count(*) FROM {table}").fetchone()[0]
|
| 492 |
+
except Exception:
|
| 493 |
+
out[table] = None # table absent = never synced
|
| 494 |
+
return out
|
| 495 |
+
|
| 496 |
+
|
| 497 |
+
def main(argv=None):
|
| 498 |
+
ap = argparse.ArgumentParser(description="Pull Meta Ads into the tenant's DuckDB mirror.")
|
| 499 |
+
ap.add_argument("--sync", action="store_true")
|
| 500 |
+
ap.add_argument("--status", action="store_true")
|
| 501 |
+
ap.add_argument("--tenant", default="royal-imports")
|
| 502 |
+
ap.add_argument("--no-insights", action="store_true")
|
| 503 |
+
ap.add_argument("--insights-days", type=int, default=None,
|
| 504 |
+
help="override the Insights window for THIS run (default INSIGHTS_DAYS); the env var is the default, this is the caller's say")
|
| 505 |
+
a = ap.parse_args(argv)
|
| 506 |
+
if a.status:
|
| 507 |
+
for k, v in status(a.tenant).items():
|
| 508 |
+
print(f" {k:<20} {'(never synced)' if v is None else v}")
|
| 509 |
+
return 0
|
| 510 |
+
if not a.sync:
|
| 511 |
+
ap.print_help()
|
| 512 |
+
return 2
|
| 513 |
+
rep = sync(a.tenant, insights=not a.no_insights, insights_days=a.insights_days)
|
| 514 |
+
for p in rep["problems"]:
|
| 515 |
+
print(" PROBLEM:", p)
|
| 516 |
+
print(f" accounts: {len(rep['accounts'])} tables: {len(rep['tables'])}")
|
| 517 |
+
return 1 if rep["problems"] else 0
|
| 518 |
+
|
| 519 |
+
|
| 520 |
+
if __name__ == "__main__":
|
| 521 |
+
sys.exit(main())
|
platform/model/topics/odoo_accounts.yml
CHANGED
|
@@ -1,60 +1,60 @@
|
|
| 1 |
-
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
-
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
-
#
|
| 4 |
-
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
-
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
-
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
-
# trained on a schema the product does not have.
|
| 8 |
-
key: odoo_accounts
|
| 9 |
-
label: "Odoo GL accounts"
|
| 10 |
-
entity: account.account
|
| 11 |
-
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
-
grid: ut_odoo_accounts
|
| 13 |
-
subject: "odoo:account.account"
|
| 14 |
-
grain: "one row per GL account in the chart of accounts"
|
| 15 |
-
scope:
|
| 16 |
-
population: "every account.account record (192 measured)"
|
| 17 |
-
store:
|
| 18 |
-
table: account_account
|
| 19 |
-
alias: a
|
| 20 |
-
# NO date_col β a registry is not a dated event stream. Stated rather than
|
| 21 |
-
# omitted, so its absence reads as a fact and not as an unfinished file.
|
| 22 |
-
dims:
|
| 23 |
-
account_type: {label: "Account type"}
|
| 24 |
-
|
| 25 |
-
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 26 |
-
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 27 |
-
# from another topic; a `link` points at another database.
|
| 28 |
-
fields:
|
| 29 |
-
- key: account_code
|
| 30 |
-
label: "Code"
|
| 31 |
-
type: text
|
| 32 |
-
kind: data
|
| 33 |
-
- key: account_name
|
| 34 |
-
label: "Account"
|
| 35 |
-
type: text
|
| 36 |
-
kind: data
|
| 37 |
-
- key: odoo_id
|
| 38 |
-
label: "Odoo ID"
|
| 39 |
-
type: int
|
| 40 |
-
kind: data
|
| 41 |
-
means: "The `account.account` id. Also this row's id."
|
| 42 |
-
- key: account_type
|
| 43 |
-
label: "Type"
|
| 44 |
-
type: select
|
| 45 |
-
kind: data
|
| 46 |
-
- key: is_expense
|
| 47 |
-
label: "Expense account"
|
| 48 |
-
type: checkbox
|
| 49 |
-
kind: data
|
| 50 |
-
means: "Ticked for the expense family - the same predicate the semantic layer's gl_lines topic uses, so this column and that topic cannot disagree."
|
| 51 |
-
- key: refreshed
|
| 52 |
-
label: "Refreshed"
|
| 53 |
-
type: date
|
| 54 |
-
kind: data
|
| 55 |
-
means: "When this row was last reconciled against Odoo."
|
| 56 |
-
|
| 57 |
-
ai_context: >
|
| 58 |
-
The chart of accounts β the registry gl_lines posts against.
|
| 59 |
-
Use it to resolve an account code to a name or a type.
|
| 60 |
-
β `is_expense` is a DERIVED boolean standing in for Odoo's five-value `internal_group` (asset/liability/equity/income/expense), which the mirror does not yet carry.
|
|
|
|
| 1 |
+
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
+
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
+
#
|
| 4 |
+
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
+
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
+
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
+
# trained on a schema the product does not have.
|
| 8 |
+
key: odoo_accounts
|
| 9 |
+
label: "Odoo GL accounts"
|
| 10 |
+
entity: account.account
|
| 11 |
+
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
+
grid: ut_odoo_accounts
|
| 13 |
+
subject: "odoo:account.account"
|
| 14 |
+
grain: "one row per GL account in the chart of accounts"
|
| 15 |
+
scope:
|
| 16 |
+
population: "every account.account record (192 measured)"
|
| 17 |
+
store:
|
| 18 |
+
table: account_account
|
| 19 |
+
alias: a
|
| 20 |
+
# NO date_col β a registry is not a dated event stream. Stated rather than
|
| 21 |
+
# omitted, so its absence reads as a fact and not as an unfinished file.
|
| 22 |
+
dims:
|
| 23 |
+
account_type: {label: "Account type"}
|
| 24 |
+
|
| 25 |
+
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 26 |
+
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 27 |
+
# from another topic; a `link` points at another database.
|
| 28 |
+
fields:
|
| 29 |
+
- key: account_code
|
| 30 |
+
label: "Code"
|
| 31 |
+
type: text
|
| 32 |
+
kind: data
|
| 33 |
+
- key: account_name
|
| 34 |
+
label: "Account"
|
| 35 |
+
type: text
|
| 36 |
+
kind: data
|
| 37 |
+
- key: odoo_id
|
| 38 |
+
label: "Odoo ID"
|
| 39 |
+
type: int
|
| 40 |
+
kind: data
|
| 41 |
+
means: "The `account.account` id. Also this row's id."
|
| 42 |
+
- key: account_type
|
| 43 |
+
label: "Type"
|
| 44 |
+
type: select
|
| 45 |
+
kind: data
|
| 46 |
+
- key: is_expense
|
| 47 |
+
label: "Expense account"
|
| 48 |
+
type: checkbox
|
| 49 |
+
kind: data
|
| 50 |
+
means: "Ticked for the expense family - the same predicate the semantic layer's gl_lines topic uses, so this column and that topic cannot disagree."
|
| 51 |
+
- key: refreshed
|
| 52 |
+
label: "Refreshed"
|
| 53 |
+
type: date
|
| 54 |
+
kind: data
|
| 55 |
+
means: "When this row was last reconciled against Odoo."
|
| 56 |
+
|
| 57 |
+
ai_context: >
|
| 58 |
+
The chart of accounts β the registry gl_lines posts against.
|
| 59 |
+
Use it to resolve an account code to a name or a type.
|
| 60 |
+
β `is_expense` is a DERIVED boolean standing in for Odoo's five-value `internal_group` (asset/liability/equity/income/expense), which the mirror does not yet carry.
|
platform/model/topics/odoo_agents.yml
CHANGED
|
@@ -1,59 +1,59 @@
|
|
| 1 |
-
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
-
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
-
#
|
| 4 |
-
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
-
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
-
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
-
# trained on a schema the product does not have.
|
| 8 |
-
key: odoo_agents
|
| 9 |
-
label: "Odoo agents"
|
| 10 |
-
entity: res.partner
|
| 11 |
-
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
-
grid: ut_odoo_agents
|
| 13 |
-
subject: "odoo:res.partner.agent"
|
| 14 |
-
grain: "one row per sales agent"
|
| 15 |
-
scope:
|
| 16 |
-
population: "a UNION of two disagreeing sources β partners carrying commission lines, and partners flagged res_partner.agent = TRUE"
|
| 17 |
-
why_union: "measured 2026-08-09: 16 carry commission lines, 17 carry the flag, and the union is 19 β either source alone silently drops real agents"
|
| 18 |
-
store:
|
| 19 |
-
table: res_partner
|
| 20 |
-
alias: p
|
| 21 |
-
# NO date_col β a registry is not a dated event stream. Stated rather than
|
| 22 |
-
# omitted, so its absence reads as a fact and not as an unfinished file.
|
| 23 |
-
|
| 24 |
-
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 25 |
-
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 26 |
-
# from another topic; a `link` points at another database.
|
| 27 |
-
fields:
|
| 28 |
-
- key: agent
|
| 29 |
-
label: "Agent"
|
| 30 |
-
type: text
|
| 31 |
-
kind: data
|
| 32 |
-
- key: odoo_id
|
| 33 |
-
label: "Odoo ID"
|
| 34 |
-
type: int
|
| 35 |
-
kind: data
|
| 36 |
-
means: "The `res.partner` id. Also this row's id."
|
| 37 |
-
- key: agent_id
|
| 38 |
-
label: "Odoo agent id"
|
| 39 |
-
type: int
|
| 40 |
-
kind: data
|
| 41 |
-
- key: flagged
|
| 42 |
-
label: "Flagged in Odoo"
|
| 43 |
-
type: checkbox
|
| 44 |
-
kind: data
|
| 45 |
-
means: "Ticked = `res.partner.agent` is set. Unticked agents were found by their commission lines instead - both are real, which is why this table is the union of the two."
|
| 46 |
-
- key: commissioned
|
| 47 |
-
label: "Has commission lines"
|
| 48 |
-
type: checkbox
|
| 49 |
-
kind: data
|
| 50 |
-
- key: refreshed
|
| 51 |
-
label: "Refreshed"
|
| 52 |
-
type: date
|
| 53 |
-
kind: data
|
| 54 |
-
means: "When this row was last reconciled against Odoo."
|
| 55 |
-
|
| 56 |
-
ai_context: >
|
| 57 |
-
The sales-agent registry.
|
| 58 |
-
`flagged` and `commissioned` are the two SOURCES, kept as separate columns rather than merged, because they disagree and the disagreement is information.
|
| 59 |
-
For an agent's BOOK use sales_lines/commission_lines with the agent dim.
|
|
|
|
| 1 |
+
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
+
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
+
#
|
| 4 |
+
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
+
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
+
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
+
# trained on a schema the product does not have.
|
| 8 |
+
key: odoo_agents
|
| 9 |
+
label: "Odoo agents"
|
| 10 |
+
entity: res.partner
|
| 11 |
+
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
+
grid: ut_odoo_agents
|
| 13 |
+
subject: "odoo:res.partner.agent"
|
| 14 |
+
grain: "one row per sales agent"
|
| 15 |
+
scope:
|
| 16 |
+
population: "a UNION of two disagreeing sources β partners carrying commission lines, and partners flagged res_partner.agent = TRUE"
|
| 17 |
+
why_union: "measured 2026-08-09: 16 carry commission lines, 17 carry the flag, and the union is 19 β either source alone silently drops real agents"
|
| 18 |
+
store:
|
| 19 |
+
table: res_partner
|
| 20 |
+
alias: p
|
| 21 |
+
# NO date_col β a registry is not a dated event stream. Stated rather than
|
| 22 |
+
# omitted, so its absence reads as a fact and not as an unfinished file.
|
| 23 |
+
|
| 24 |
+
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 25 |
+
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 26 |
+
# from another topic; a `link` points at another database.
|
| 27 |
+
fields:
|
| 28 |
+
- key: agent
|
| 29 |
+
label: "Agent"
|
| 30 |
+
type: text
|
| 31 |
+
kind: data
|
| 32 |
+
- key: odoo_id
|
| 33 |
+
label: "Odoo ID"
|
| 34 |
+
type: int
|
| 35 |
+
kind: data
|
| 36 |
+
means: "The `res.partner` id. Also this row's id."
|
| 37 |
+
- key: agent_id
|
| 38 |
+
label: "Odoo agent id"
|
| 39 |
+
type: int
|
| 40 |
+
kind: data
|
| 41 |
+
- key: flagged
|
| 42 |
+
label: "Flagged in Odoo"
|
| 43 |
+
type: checkbox
|
| 44 |
+
kind: data
|
| 45 |
+
means: "Ticked = `res.partner.agent` is set. Unticked agents were found by their commission lines instead - both are real, which is why this table is the union of the two."
|
| 46 |
+
- key: commissioned
|
| 47 |
+
label: "Has commission lines"
|
| 48 |
+
type: checkbox
|
| 49 |
+
kind: data
|
| 50 |
+
- key: refreshed
|
| 51 |
+
label: "Refreshed"
|
| 52 |
+
type: date
|
| 53 |
+
kind: data
|
| 54 |
+
means: "When this row was last reconciled against Odoo."
|
| 55 |
+
|
| 56 |
+
ai_context: >
|
| 57 |
+
The sales-agent registry.
|
| 58 |
+
`flagged` and `commissioned` are the two SOURCES, kept as separate columns rather than merged, because they disagree and the disagreement is information.
|
| 59 |
+
For an agent's BOOK use sales_lines/commission_lines with the agent dim.
|
platform/model/topics/odoo_bills.yml
CHANGED
|
@@ -1,86 +1,86 @@
|
|
| 1 |
-
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
-
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
-
#
|
| 4 |
-
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
-
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
-
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
-
# trained on a schema the product does not have.
|
| 8 |
-
key: odoo_bills
|
| 9 |
-
label: "Odoo vendor bills"
|
| 10 |
-
entity: account.move
|
| 11 |
-
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
-
grid: ut_odoo_bills
|
| 13 |
-
subject: "odoo:account.move.vendor"
|
| 14 |
-
grain: "one row per posted VENDOR bill or vendor credit note (document grain)"
|
| 15 |
-
scope:
|
| 16 |
-
population: "move_type in (in_invoice, in_refund) AND state = posted"
|
| 17 |
-
signs: "for payables the residual is negative on the Odoo side; take the absolute value for an AP figure"
|
| 18 |
-
store:
|
| 19 |
-
table: account_move
|
| 20 |
-
alias: m
|
| 21 |
-
date_col: "m.invoice_date"
|
| 22 |
-
dims:
|
| 23 |
-
vendor: {label: "Vendor"}
|
| 24 |
-
payment_state: {label: "Payment state"}
|
| 25 |
-
move_type: {label: "Document type"}
|
| 26 |
-
|
| 27 |
-
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 28 |
-
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 29 |
-
# from another topic; a `link` points at another database.
|
| 30 |
-
fields:
|
| 31 |
-
- key: bill_no
|
| 32 |
-
label: "Bill"
|
| 33 |
-
type: text
|
| 34 |
-
kind: data
|
| 35 |
-
- key: odoo_id
|
| 36 |
-
label: "Odoo ID"
|
| 37 |
-
type: int
|
| 38 |
-
kind: data
|
| 39 |
-
means: "The `account.move` id. Also this row's id."
|
| 40 |
-
- key: vendor
|
| 41 |
-
label: "Vendor"
|
| 42 |
-
type: text
|
| 43 |
-
kind: data
|
| 44 |
-
- key: vendor_id
|
| 45 |
-
label: "Odoo vendor id"
|
| 46 |
-
type: int
|
| 47 |
-
kind: data
|
| 48 |
-
- key: invoice_date
|
| 49 |
-
label: "Bill date"
|
| 50 |
-
type: date
|
| 51 |
-
kind: data
|
| 52 |
-
- key: due_date
|
| 53 |
-
label: "Due date"
|
| 54 |
-
type: date
|
| 55 |
-
kind: data
|
| 56 |
-
- key: amount_untaxed
|
| 57 |
-
label: "Billed $"
|
| 58 |
-
type: currency
|
| 59 |
-
kind: data
|
| 60 |
-
- key: residual
|
| 61 |
-
label: "Outstanding $"
|
| 62 |
-
type: currency
|
| 63 |
-
kind: data
|
| 64 |
-
- key: payment_state
|
| 65 |
-
label: "Payment state"
|
| 66 |
-
type: select
|
| 67 |
-
kind: data
|
| 68 |
-
- key: move_type
|
| 69 |
-
label: "Document"
|
| 70 |
-
type: select
|
| 71 |
-
kind: data
|
| 72 |
-
- key: vendor_link
|
| 73 |
-
label: "Vendor record"
|
| 74 |
-
type: link
|
| 75 |
-
kind: link
|
| 76 |
-
to_grid: ut_odoo_vendors
|
| 77 |
-
- key: refreshed
|
| 78 |
-
label: "Refreshed"
|
| 79 |
-
type: date
|
| 80 |
-
kind: data
|
| 81 |
-
means: "When this row was last reconciled against Odoo."
|
| 82 |
-
|
| 83 |
-
ai_context: >
|
| 84 |
-
Vendor bills at DOCUMENT grain β the payables side of account_move.
|
| 85 |
-
Use it for AP questions.
|
| 86 |
-
β The census found `ref` (the VENDOR's own invoice number), `journal_id` and `currency_id` populated in Odoo and absent from the mirror, so they cannot be answered from here yet.
|
|
|
|
| 1 |
+
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
+
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
+
#
|
| 4 |
+
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
+
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
+
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
+
# trained on a schema the product does not have.
|
| 8 |
+
key: odoo_bills
|
| 9 |
+
label: "Odoo vendor bills"
|
| 10 |
+
entity: account.move
|
| 11 |
+
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
+
grid: ut_odoo_bills
|
| 13 |
+
subject: "odoo:account.move.vendor"
|
| 14 |
+
grain: "one row per posted VENDOR bill or vendor credit note (document grain)"
|
| 15 |
+
scope:
|
| 16 |
+
population: "move_type in (in_invoice, in_refund) AND state = posted"
|
| 17 |
+
signs: "for payables the residual is negative on the Odoo side; take the absolute value for an AP figure"
|
| 18 |
+
store:
|
| 19 |
+
table: account_move
|
| 20 |
+
alias: m
|
| 21 |
+
date_col: "m.invoice_date"
|
| 22 |
+
dims:
|
| 23 |
+
vendor: {label: "Vendor"}
|
| 24 |
+
payment_state: {label: "Payment state"}
|
| 25 |
+
move_type: {label: "Document type"}
|
| 26 |
+
|
| 27 |
+
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 28 |
+
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 29 |
+
# from another topic; a `link` points at another database.
|
| 30 |
+
fields:
|
| 31 |
+
- key: bill_no
|
| 32 |
+
label: "Bill"
|
| 33 |
+
type: text
|
| 34 |
+
kind: data
|
| 35 |
+
- key: odoo_id
|
| 36 |
+
label: "Odoo ID"
|
| 37 |
+
type: int
|
| 38 |
+
kind: data
|
| 39 |
+
means: "The `account.move` id. Also this row's id."
|
| 40 |
+
- key: vendor
|
| 41 |
+
label: "Vendor"
|
| 42 |
+
type: text
|
| 43 |
+
kind: data
|
| 44 |
+
- key: vendor_id
|
| 45 |
+
label: "Odoo vendor id"
|
| 46 |
+
type: int
|
| 47 |
+
kind: data
|
| 48 |
+
- key: invoice_date
|
| 49 |
+
label: "Bill date"
|
| 50 |
+
type: date
|
| 51 |
+
kind: data
|
| 52 |
+
- key: due_date
|
| 53 |
+
label: "Due date"
|
| 54 |
+
type: date
|
| 55 |
+
kind: data
|
| 56 |
+
- key: amount_untaxed
|
| 57 |
+
label: "Billed $"
|
| 58 |
+
type: currency
|
| 59 |
+
kind: data
|
| 60 |
+
- key: residual
|
| 61 |
+
label: "Outstanding $"
|
| 62 |
+
type: currency
|
| 63 |
+
kind: data
|
| 64 |
+
- key: payment_state
|
| 65 |
+
label: "Payment state"
|
| 66 |
+
type: select
|
| 67 |
+
kind: data
|
| 68 |
+
- key: move_type
|
| 69 |
+
label: "Document"
|
| 70 |
+
type: select
|
| 71 |
+
kind: data
|
| 72 |
+
- key: vendor_link
|
| 73 |
+
label: "Vendor record"
|
| 74 |
+
type: link
|
| 75 |
+
kind: link
|
| 76 |
+
to_grid: ut_odoo_vendors
|
| 77 |
+
- key: refreshed
|
| 78 |
+
label: "Refreshed"
|
| 79 |
+
type: date
|
| 80 |
+
kind: data
|
| 81 |
+
means: "When this row was last reconciled against Odoo."
|
| 82 |
+
|
| 83 |
+
ai_context: >
|
| 84 |
+
Vendor bills at DOCUMENT grain β the payables side of account_move.
|
| 85 |
+
Use it for AP questions.
|
| 86 |
+
β The census found `ref` (the VENDOR's own invoice number), `journal_id` and `currency_id` populated in Odoo and absent from the mirror, so they cannot be answered from here yet.
|
platform/model/topics/odoo_customers.yml
CHANGED
|
@@ -1,213 +1,213 @@
|
|
| 1 |
-
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
-
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
-
#
|
| 4 |
-
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
-
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
-
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
-
# trained on a schema the product does not have.
|
| 8 |
-
key: odoo_customers
|
| 9 |
-
label: "Odoo customers"
|
| 10 |
-
entity: res.partner
|
| 11 |
-
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
-
grid: customer_data
|
| 13 |
-
subject: "odoo:res.partner"
|
| 14 |
-
grain: "one row per customer in tenant #0's scoped book β a REGISTRY, not a dated event stream. Identity is the Odoo `res.partner` id (the row's `pid` IS that id)."
|
| 15 |
-
scope:
|
| 16 |
-
population: "the customer book `modules/customer_data.pool()` builds; the grid's own contract states it as one row per customer who ordered in the last 24 months"
|
| 17 |
-
identity: "the Odoo res.partner id β exposed as the `partner_id` column (W33-T43) and used as the row pid, so there is ONE id per row and no second one"
|
| 18 |
-
merged: "W33-T44 retired `ut_odoo_customers`, which presented this same subject. This topic describes the SURVIVING database, `customer_data`, which keeps its store bucket so no saved view, grant, cohort or formula moved"
|
| 19 |
-
no_date: "a registry has no date dimension; ask sales_lines or customer_invoices for anything time-windowed about a customer"
|
| 20 |
-
store:
|
| 21 |
-
table: res_partner
|
| 22 |
-
alias: p
|
| 23 |
-
# NO date_col β a registry is not a dated event stream. Stated rather than
|
| 24 |
-
# omitted, so its absence reads as a fact and not as an unfinished file.
|
| 25 |
-
dims:
|
| 26 |
-
agent: {label: "Sales agent"}
|
| 27 |
-
state: {label: "State"}
|
| 28 |
-
country: {label: "Country"}
|
| 29 |
-
|
| 30 |
-
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 31 |
-
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 32 |
-
# from another topic; a `link` points at another database.
|
| 33 |
-
fields:
|
| 34 |
-
- key: customer
|
| 35 |
-
label: "Customer"
|
| 36 |
-
type: text
|
| 37 |
-
kind: data
|
| 38 |
-
means: "The customer's name in Odoo. One row per customer who ordered in the last 24 months."
|
| 39 |
-
- key: partner_id
|
| 40 |
-
label: "Odoo ID"
|
| 41 |
-
type: int
|
| 42 |
-
kind: data
|
| 43 |
-
means: "The Odoo res.partner id β this row's identity, and the key every Odoo document joins on. W33-T43 / owner item 12: 'One unique ID per database always.' It is DERIVED rather than read off the pool row because a customer row's pid IS the partner id, so the value is already on every row and a second copy in the pool would be a second source for one fact."
|
| 44 |
-
- key: odoo_status
|
| 45 |
-
label: "Odoo record"
|
| 46 |
-
type: status
|
| 47 |
-
kind: data
|
| 48 |
-
means: "Whether this customer still exists in Odoo. Archived means deleted there."
|
| 49 |
-
- key: agent
|
| 50 |
-
label: "Agent"
|
| 51 |
-
type: text
|
| 52 |
-
kind: data
|
| 53 |
-
means: "The sales agent who owns this account."
|
| 54 |
-
- key: dba
|
| 55 |
-
label: "DBA"
|
| 56 |
-
type: select
|
| 57 |
-
kind: data
|
| 58 |
-
means: "The brand this customer buys from - Fisch, Royal, or both. Amazon-channel orders are not a DBA."
|
| 59 |
-
- key: salesperson
|
| 60 |
-
label: "Salesperson"
|
| 61 |
-
type: text
|
| 62 |
-
kind: data
|
| 63 |
-
means: "Who keyed in most of this customer's orders β not the Agent, who owns the account."
|
| 64 |
-
- key: street
|
| 65 |
-
label: "Street"
|
| 66 |
-
type: text
|
| 67 |
-
kind: data
|
| 68 |
-
means: "First address line, from res.partner directly - not the geocoder, so a customer the map cannot place still shows its address."
|
| 69 |
-
- key: street2
|
| 70 |
-
label: "Street 2"
|
| 71 |
-
type: text
|
| 72 |
-
kind: data
|
| 73 |
-
means: "Second address line (suite, unit, floor) on the customer's Odoo address."
|
| 74 |
-
- key: city
|
| 75 |
-
label: "City"
|
| 76 |
-
type: text
|
| 77 |
-
kind: data
|
| 78 |
-
means: "City on the customer's Odoo address."
|
| 79 |
-
- key: state
|
| 80 |
-
label: "State"
|
| 81 |
-
type: text
|
| 82 |
-
kind: data
|
| 83 |
-
means: "State or province on the customer's Odoo address."
|
| 84 |
-
- key: country
|
| 85 |
-
label: "Country"
|
| 86 |
-
type: text
|
| 87 |
-
kind: data
|
| 88 |
-
means: "Country on the customer's Odoo address."
|
| 89 |
-
- key: zip
|
| 90 |
-
label: "ZIP"
|
| 91 |
-
type: text
|
| 92 |
-
kind: data
|
| 93 |
-
means: "Postal code on the customer's Odoo address."
|
| 94 |
-
- key: customer_since
|
| 95 |
-
label: "Customer since"
|
| 96 |
-
type: date
|
| 97 |
-
kind: data
|
| 98 |
-
means: "When this customer was first set up in Odoo."
|
| 99 |
-
- key: tags
|
| 100 |
-
label: "Tags"
|
| 101 |
-
type: text
|
| 102 |
-
kind: data
|
| 103 |
-
means: "Odoo labels on this customer, comma-separated."
|
| 104 |
-
- key: pricelist
|
| 105 |
-
label: "Price list"
|
| 106 |
-
type: text
|
| 107 |
-
kind: data
|
| 108 |
-
means: "The price list this customer buys on."
|
| 109 |
-
- key: payment_terms
|
| 110 |
-
label: "Payment terms"
|
| 111 |
-
type: text
|
| 112 |
-
kind: data
|
| 113 |
-
means: "Payment terms on this customer's account β Net 30, for example."
|
| 114 |
-
- key: last_order
|
| 115 |
-
label: "Last order"
|
| 116 |
-
type: date
|
| 117 |
-
kind: data
|
| 118 |
-
means: "Date of the most recent confirmed order."
|
| 119 |
-
- key: overdue_days
|
| 120 |
-
label: "Overdue days"
|
| 121 |
-
type: int
|
| 122 |
-
kind: data
|
| 123 |
-
means: "How many days late this customer is running against their own usual ordering rhythm."
|
| 124 |
-
- key: est_missed
|
| 125 |
-
label: "Est. missed $"
|
| 126 |
-
type: currency
|
| 127 |
-
kind: data
|
| 128 |
-
means: "Estimated sales missed while quiet: missed orders (capped at 3) times average order value. An estimate, not money owed."
|
| 129 |
-
- key: ar_open
|
| 130 |
-
label: "AR current $"
|
| 131 |
-
type: currency
|
| 132 |
-
kind: data
|
| 133 |
-
means: "Invoiced money owed but not yet due (a 5-day grace applies before it counts as overdue)."
|
| 134 |
-
- key: ar_overdue
|
| 135 |
-
label: "AR overdue $"
|
| 136 |
-
type: currency
|
| 137 |
-
kind: data
|
| 138 |
-
means: "Invoiced money past due β same basis as the Collections page."
|
| 139 |
-
- key: ar_outstanding
|
| 140 |
-
label: "AR outstanding $"
|
| 141 |
-
type: currency
|
| 142 |
-
kind: data
|
| 143 |
-
means: "Total invoiced money owed right now: AR current $ plus AR overdue $."
|
| 144 |
-
- key: ar_exposure
|
| 145 |
-
label: "Credit exposure $"
|
| 146 |
-
type: currency
|
| 147 |
-
kind: data
|
| 148 |
-
means: "The most you could be out if they stopped paying today: open, overdue, draft and not-yet-invoiced."
|
| 149 |
-
- key: ar_aged_1_30
|
| 150 |
-
label: "1-30 days $"
|
| 151 |
-
type: currency
|
| 152 |
-
kind: data
|
| 153 |
-
means: "Overdue between 1 and 30 days. The four aging buckets sum to AR overdue $."
|
| 154 |
-
- key: ar_aged_31_60
|
| 155 |
-
label: "31-60 days $"
|
| 156 |
-
type: currency
|
| 157 |
-
kind: data
|
| 158 |
-
means: "Overdue between 31 and 60 days. The four aging buckets sum to AR overdue $."
|
| 159 |
-
- key: ar_aged_61_90
|
| 160 |
-
label: "61-90 days $"
|
| 161 |
-
type: currency
|
| 162 |
-
kind: data
|
| 163 |
-
means: "Overdue between 61 and 90 days. The four aging buckets sum to AR overdue $."
|
| 164 |
-
- key: ar_aged_90_plus
|
| 165 |
-
label: "90+ days $"
|
| 166 |
-
type: currency
|
| 167 |
-
kind: data
|
| 168 |
-
means: "Overdue by more than 90 days. The four aging buckets sum to AR overdue $."
|
| 169 |
-
- key: days_to_pay
|
| 170 |
-
label: "Days to pay"
|
| 171 |
-
type: int
|
| 172 |
-
kind: data
|
| 173 |
-
means: "Average days to pay an invoice in full. Blank means no fully paid invoice yet."
|
| 174 |
-
- key: top_category
|
| 175 |
-
label: "Top category"
|
| 176 |
-
type: text
|
| 177 |
-
kind: data
|
| 178 |
-
means: "The category this customer spent the most on in the last 12 months."
|
| 179 |
-
- key: top_category_pct
|
| 180 |
-
label: "Top category %"
|
| 181 |
-
type: pct
|
| 182 |
-
kind: data
|
| 183 |
-
means: "Share of last-12-months spend that went to the top category."
|
| 184 |
-
- key: sku_count
|
| 185 |
-
label: "SKUs bought"
|
| 186 |
-
type: int
|
| 187 |
-
kind: data
|
| 188 |
-
means: "Distinct products bought in the last 12 months."
|
| 189 |
-
- key: top_sku
|
| 190 |
-
label: "Top SKU"
|
| 191 |
-
type: text
|
| 192 |
-
kind: data
|
| 193 |
-
means: "The product this customer spent the most on in the last 12 months."
|
| 194 |
-
- key: days_since
|
| 195 |
-
label: "Days since order"
|
| 196 |
-
type: int
|
| 197 |
-
kind: data
|
| 198 |
-
means: "Days since the last confirmed order."
|
| 199 |
-
- key: typical_gap_days
|
| 200 |
-
label: "Typical gap days"
|
| 201 |
-
type: int
|
| 202 |
-
kind: data
|
| 203 |
-
means: "Days this customer usually goes between orders, from their own history."
|
| 204 |
-
- key: notes
|
| 205 |
-
label: "Notes"
|
| 206 |
-
type: text
|
| 207 |
-
kind: data
|
| 208 |
-
means: "Your notes on this customer. Saved in this app only, visible only to you."
|
| 209 |
-
|
| 210 |
-
ai_context: >
|
| 211 |
-
The customer REGISTRY as a person sees it in the app β the database labelled 'Odoo customers'.
|
| 212 |
-
Use it to answer 'who is this customer', 'which customers exist', 'which agent owns them', 'where are they', and for the AR columns it carries (ar_open, ar_overdue, days_to_pay), which are reconciled by modules/ar.py.
|
| 213 |
-
β For anything WINDOWED or at line grain go to sales_lines / customer_invoices / receivables β those topics own the time dimension and this one has none.
|
|
|
|
| 1 |
+
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
+
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
+
#
|
| 4 |
+
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
+
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
+
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
+
# trained on a schema the product does not have.
|
| 8 |
+
key: odoo_customers
|
| 9 |
+
label: "Odoo customers"
|
| 10 |
+
entity: res.partner
|
| 11 |
+
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
+
grid: customer_data
|
| 13 |
+
subject: "odoo:res.partner"
|
| 14 |
+
grain: "one row per customer in tenant #0's scoped book β a REGISTRY, not a dated event stream. Identity is the Odoo `res.partner` id (the row's `pid` IS that id)."
|
| 15 |
+
scope:
|
| 16 |
+
population: "the customer book `modules/customer_data.pool()` builds; the grid's own contract states it as one row per customer who ordered in the last 24 months"
|
| 17 |
+
identity: "the Odoo res.partner id β exposed as the `partner_id` column (W33-T43) and used as the row pid, so there is ONE id per row and no second one"
|
| 18 |
+
merged: "W33-T44 retired `ut_odoo_customers`, which presented this same subject. This topic describes the SURVIVING database, `customer_data`, which keeps its store bucket so no saved view, grant, cohort or formula moved"
|
| 19 |
+
no_date: "a registry has no date dimension; ask sales_lines or customer_invoices for anything time-windowed about a customer"
|
| 20 |
+
store:
|
| 21 |
+
table: res_partner
|
| 22 |
+
alias: p
|
| 23 |
+
# NO date_col β a registry is not a dated event stream. Stated rather than
|
| 24 |
+
# omitted, so its absence reads as a fact and not as an unfinished file.
|
| 25 |
+
dims:
|
| 26 |
+
agent: {label: "Sales agent"}
|
| 27 |
+
state: {label: "State"}
|
| 28 |
+
country: {label: "Country"}
|
| 29 |
+
|
| 30 |
+
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 31 |
+
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 32 |
+
# from another topic; a `link` points at another database.
|
| 33 |
+
fields:
|
| 34 |
+
- key: customer
|
| 35 |
+
label: "Customer"
|
| 36 |
+
type: text
|
| 37 |
+
kind: data
|
| 38 |
+
means: "The customer's name in Odoo. One row per customer who ordered in the last 24 months."
|
| 39 |
+
- key: partner_id
|
| 40 |
+
label: "Odoo ID"
|
| 41 |
+
type: int
|
| 42 |
+
kind: data
|
| 43 |
+
means: "The Odoo res.partner id β this row's identity, and the key every Odoo document joins on. W33-T43 / owner item 12: 'One unique ID per database always.' It is DERIVED rather than read off the pool row because a customer row's pid IS the partner id, so the value is already on every row and a second copy in the pool would be a second source for one fact."
|
| 44 |
+
- key: odoo_status
|
| 45 |
+
label: "Odoo record"
|
| 46 |
+
type: status
|
| 47 |
+
kind: data
|
| 48 |
+
means: "Whether this customer still exists in Odoo. Archived means deleted there."
|
| 49 |
+
- key: agent
|
| 50 |
+
label: "Agent"
|
| 51 |
+
type: text
|
| 52 |
+
kind: data
|
| 53 |
+
means: "The sales agent who owns this account."
|
| 54 |
+
- key: dba
|
| 55 |
+
label: "DBA"
|
| 56 |
+
type: select
|
| 57 |
+
kind: data
|
| 58 |
+
means: "The brand this customer buys from - Fisch, Royal, or both. Amazon-channel orders are not a DBA."
|
| 59 |
+
- key: salesperson
|
| 60 |
+
label: "Salesperson"
|
| 61 |
+
type: text
|
| 62 |
+
kind: data
|
| 63 |
+
means: "Who keyed in most of this customer's orders β not the Agent, who owns the account."
|
| 64 |
+
- key: street
|
| 65 |
+
label: "Street"
|
| 66 |
+
type: text
|
| 67 |
+
kind: data
|
| 68 |
+
means: "First address line, from res.partner directly - not the geocoder, so a customer the map cannot place still shows its address."
|
| 69 |
+
- key: street2
|
| 70 |
+
label: "Street 2"
|
| 71 |
+
type: text
|
| 72 |
+
kind: data
|
| 73 |
+
means: "Second address line (suite, unit, floor) on the customer's Odoo address."
|
| 74 |
+
- key: city
|
| 75 |
+
label: "City"
|
| 76 |
+
type: text
|
| 77 |
+
kind: data
|
| 78 |
+
means: "City on the customer's Odoo address."
|
| 79 |
+
- key: state
|
| 80 |
+
label: "State"
|
| 81 |
+
type: text
|
| 82 |
+
kind: data
|
| 83 |
+
means: "State or province on the customer's Odoo address."
|
| 84 |
+
- key: country
|
| 85 |
+
label: "Country"
|
| 86 |
+
type: text
|
| 87 |
+
kind: data
|
| 88 |
+
means: "Country on the customer's Odoo address."
|
| 89 |
+
- key: zip
|
| 90 |
+
label: "ZIP"
|
| 91 |
+
type: text
|
| 92 |
+
kind: data
|
| 93 |
+
means: "Postal code on the customer's Odoo address."
|
| 94 |
+
- key: customer_since
|
| 95 |
+
label: "Customer since"
|
| 96 |
+
type: date
|
| 97 |
+
kind: data
|
| 98 |
+
means: "When this customer was first set up in Odoo."
|
| 99 |
+
- key: tags
|
| 100 |
+
label: "Tags"
|
| 101 |
+
type: text
|
| 102 |
+
kind: data
|
| 103 |
+
means: "Odoo labels on this customer, comma-separated."
|
| 104 |
+
- key: pricelist
|
| 105 |
+
label: "Price list"
|
| 106 |
+
type: text
|
| 107 |
+
kind: data
|
| 108 |
+
means: "The price list this customer buys on."
|
| 109 |
+
- key: payment_terms
|
| 110 |
+
label: "Payment terms"
|
| 111 |
+
type: text
|
| 112 |
+
kind: data
|
| 113 |
+
means: "Payment terms on this customer's account β Net 30, for example."
|
| 114 |
+
- key: last_order
|
| 115 |
+
label: "Last order"
|
| 116 |
+
type: date
|
| 117 |
+
kind: data
|
| 118 |
+
means: "Date of the most recent confirmed order."
|
| 119 |
+
- key: overdue_days
|
| 120 |
+
label: "Overdue days"
|
| 121 |
+
type: int
|
| 122 |
+
kind: data
|
| 123 |
+
means: "How many days late this customer is running against their own usual ordering rhythm."
|
| 124 |
+
- key: est_missed
|
| 125 |
+
label: "Est. missed $"
|
| 126 |
+
type: currency
|
| 127 |
+
kind: data
|
| 128 |
+
means: "Estimated sales missed while quiet: missed orders (capped at 3) times average order value. An estimate, not money owed."
|
| 129 |
+
- key: ar_open
|
| 130 |
+
label: "AR current $"
|
| 131 |
+
type: currency
|
| 132 |
+
kind: data
|
| 133 |
+
means: "Invoiced money owed but not yet due (a 5-day grace applies before it counts as overdue)."
|
| 134 |
+
- key: ar_overdue
|
| 135 |
+
label: "AR overdue $"
|
| 136 |
+
type: currency
|
| 137 |
+
kind: data
|
| 138 |
+
means: "Invoiced money past due β same basis as the Collections page."
|
| 139 |
+
- key: ar_outstanding
|
| 140 |
+
label: "AR outstanding $"
|
| 141 |
+
type: currency
|
| 142 |
+
kind: data
|
| 143 |
+
means: "Total invoiced money owed right now: AR current $ plus AR overdue $."
|
| 144 |
+
- key: ar_exposure
|
| 145 |
+
label: "Credit exposure $"
|
| 146 |
+
type: currency
|
| 147 |
+
kind: data
|
| 148 |
+
means: "The most you could be out if they stopped paying today: open, overdue, draft and not-yet-invoiced."
|
| 149 |
+
- key: ar_aged_1_30
|
| 150 |
+
label: "1-30 days $"
|
| 151 |
+
type: currency
|
| 152 |
+
kind: data
|
| 153 |
+
means: "Overdue between 1 and 30 days. The four aging buckets sum to AR overdue $."
|
| 154 |
+
- key: ar_aged_31_60
|
| 155 |
+
label: "31-60 days $"
|
| 156 |
+
type: currency
|
| 157 |
+
kind: data
|
| 158 |
+
means: "Overdue between 31 and 60 days. The four aging buckets sum to AR overdue $."
|
| 159 |
+
- key: ar_aged_61_90
|
| 160 |
+
label: "61-90 days $"
|
| 161 |
+
type: currency
|
| 162 |
+
kind: data
|
| 163 |
+
means: "Overdue between 61 and 90 days. The four aging buckets sum to AR overdue $."
|
| 164 |
+
- key: ar_aged_90_plus
|
| 165 |
+
label: "90+ days $"
|
| 166 |
+
type: currency
|
| 167 |
+
kind: data
|
| 168 |
+
means: "Overdue by more than 90 days. The four aging buckets sum to AR overdue $."
|
| 169 |
+
- key: days_to_pay
|
| 170 |
+
label: "Days to pay"
|
| 171 |
+
type: int
|
| 172 |
+
kind: data
|
| 173 |
+
means: "Average days to pay an invoice in full. Blank means no fully paid invoice yet."
|
| 174 |
+
- key: top_category
|
| 175 |
+
label: "Top category"
|
| 176 |
+
type: text
|
| 177 |
+
kind: data
|
| 178 |
+
means: "The category this customer spent the most on in the last 12 months."
|
| 179 |
+
- key: top_category_pct
|
| 180 |
+
label: "Top category %"
|
| 181 |
+
type: pct
|
| 182 |
+
kind: data
|
| 183 |
+
means: "Share of last-12-months spend that went to the top category."
|
| 184 |
+
- key: sku_count
|
| 185 |
+
label: "SKUs bought"
|
| 186 |
+
type: int
|
| 187 |
+
kind: data
|
| 188 |
+
means: "Distinct products bought in the last 12 months."
|
| 189 |
+
- key: top_sku
|
| 190 |
+
label: "Top SKU"
|
| 191 |
+
type: text
|
| 192 |
+
kind: data
|
| 193 |
+
means: "The product this customer spent the most on in the last 12 months."
|
| 194 |
+
- key: days_since
|
| 195 |
+
label: "Days since order"
|
| 196 |
+
type: int
|
| 197 |
+
kind: data
|
| 198 |
+
means: "Days since the last confirmed order."
|
| 199 |
+
- key: typical_gap_days
|
| 200 |
+
label: "Typical gap days"
|
| 201 |
+
type: int
|
| 202 |
+
kind: data
|
| 203 |
+
means: "Days this customer usually goes between orders, from their own history."
|
| 204 |
+
- key: notes
|
| 205 |
+
label: "Notes"
|
| 206 |
+
type: text
|
| 207 |
+
kind: data
|
| 208 |
+
means: "Your notes on this customer. Saved in this app only, visible only to you."
|
| 209 |
+
|
| 210 |
+
ai_context: >
|
| 211 |
+
The customer REGISTRY as a person sees it in the app β the database labelled 'Odoo customers'.
|
| 212 |
+
Use it to answer 'who is this customer', 'which customers exist', 'which agent owns them', 'where are they', and for the AR columns it carries (ar_open, ar_overdue, days_to_pay), which are reconciled by modules/ar.py.
|
| 213 |
+
β For anything WINDOWED or at line grain go to sales_lines / customer_invoices / receivables β those topics own the time dimension and this one has none.
|
platform/model/topics/odoo_invoices.yml
CHANGED
|
@@ -1,98 +1,98 @@
|
|
| 1 |
-
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
-
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
-
#
|
| 4 |
-
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
-
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
-
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
-
# trained on a schema the product does not have.
|
| 8 |
-
key: odoo_invoices
|
| 9 |
-
label: "Odoo invoices"
|
| 10 |
-
entity: account.move
|
| 11 |
-
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
-
grid: ut_odoo_invoices
|
| 13 |
-
subject: "odoo:account.move.customer"
|
| 14 |
-
grain: "one row per posted CUSTOMER invoice or credit note (document grain)"
|
| 15 |
-
scope:
|
| 16 |
-
population: "move_type in (out_invoice, out_refund) AND state = posted"
|
| 17 |
-
signs: "amount_residual_signed is POSITIVE for an invoice and NEGATIVE for a credit note, so summing it nets the credit notes correctly"
|
| 18 |
-
store:
|
| 19 |
-
table: account_move
|
| 20 |
-
alias: m
|
| 21 |
-
date_col: "m.invoice_date"
|
| 22 |
-
dims:
|
| 23 |
-
customer: {label: "Customer"}
|
| 24 |
-
payment_state: {label: "Payment state"}
|
| 25 |
-
move_type: {label: "Document type"}
|
| 26 |
-
|
| 27 |
-
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 28 |
-
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 29 |
-
# from another topic; a `link` points at another database.
|
| 30 |
-
fields:
|
| 31 |
-
- key: invoice_no
|
| 32 |
-
label: "Invoice"
|
| 33 |
-
type: text
|
| 34 |
-
kind: data
|
| 35 |
-
- key: odoo_id
|
| 36 |
-
label: "Odoo ID"
|
| 37 |
-
type: int
|
| 38 |
-
kind: data
|
| 39 |
-
means: "The `account.move` id. Also this row's id."
|
| 40 |
-
- key: customer
|
| 41 |
-
label: "Customer"
|
| 42 |
-
type: text
|
| 43 |
-
kind: data
|
| 44 |
-
- key: partner_id
|
| 45 |
-
label: "Odoo partner id"
|
| 46 |
-
type: int
|
| 47 |
-
kind: data
|
| 48 |
-
- key: invoice_date
|
| 49 |
-
label: "Invoice date"
|
| 50 |
-
type: date
|
| 51 |
-
kind: data
|
| 52 |
-
- key: due_date
|
| 53 |
-
label: "Due date"
|
| 54 |
-
type: date
|
| 55 |
-
kind: data
|
| 56 |
-
- key: residual
|
| 57 |
-
label: "Outstanding $"
|
| 58 |
-
type: currency
|
| 59 |
-
kind: data
|
| 60 |
-
means: "Odoo's signed residual. Exactly 0 on every settled document, which is why AR rollups need no filter."
|
| 61 |
-
- key: amount_untaxed
|
| 62 |
-
label: "Invoiced $"
|
| 63 |
-
type: currency
|
| 64 |
-
kind: data
|
| 65 |
-
- key: payment_state
|
| 66 |
-
label: "Payment state"
|
| 67 |
-
type: select
|
| 68 |
-
kind: data
|
| 69 |
-
- key: move_type
|
| 70 |
-
label: "Document"
|
| 71 |
-
type: select
|
| 72 |
-
kind: data
|
| 73 |
-
- key: wholesale_scope
|
| 74 |
-
label: "In wholesale scope"
|
| 75 |
-
type: checkbox
|
| 76 |
-
kind: data
|
| 77 |
-
means: "Unticked = the GIFTWARE DEALS / Amazon channel, which every wholesale metric in this product excludes. The row is kept so no Odoo id is missing; filter on this column to reconcile against the AR page."
|
| 78 |
-
- key: origin_order
|
| 79 |
-
label: "Source order"
|
| 80 |
-
type: text
|
| 81 |
-
kind: data
|
| 82 |
-
means: "Odoo's `invoice_origin` - usually the order name, sometimes blank."
|
| 83 |
-
- key: order_link
|
| 84 |
-
label: "Order record"
|
| 85 |
-
type: link
|
| 86 |
-
kind: link
|
| 87 |
-
to_grid: ut_odoo_orders
|
| 88 |
-
- key: refreshed
|
| 89 |
-
label: "Refreshed"
|
| 90 |
-
type: date
|
| 91 |
-
kind: data
|
| 92 |
-
means: "When this row was last reconciled against Odoo."
|
| 93 |
-
|
| 94 |
-
ai_context: >
|
| 95 |
-
Customer invoices at DOCUMENT grain β one row per invoice, not per line.
|
| 96 |
-
Use it for AR questions: what is open, what is overdue, what was invoiced.
|
| 97 |
-
For anything at product or line grain use invoice_lines; for the reconciled receivables view use receivables.
|
| 98 |
-
`origin_order` is the single stored key that joins an invoice back to the order it was raised from.
|
|
|
|
| 1 |
+
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
+
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
+
#
|
| 4 |
+
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
+
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
+
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
+
# trained on a schema the product does not have.
|
| 8 |
+
key: odoo_invoices
|
| 9 |
+
label: "Odoo invoices"
|
| 10 |
+
entity: account.move
|
| 11 |
+
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
+
grid: ut_odoo_invoices
|
| 13 |
+
subject: "odoo:account.move.customer"
|
| 14 |
+
grain: "one row per posted CUSTOMER invoice or credit note (document grain)"
|
| 15 |
+
scope:
|
| 16 |
+
population: "move_type in (out_invoice, out_refund) AND state = posted"
|
| 17 |
+
signs: "amount_residual_signed is POSITIVE for an invoice and NEGATIVE for a credit note, so summing it nets the credit notes correctly"
|
| 18 |
+
store:
|
| 19 |
+
table: account_move
|
| 20 |
+
alias: m
|
| 21 |
+
date_col: "m.invoice_date"
|
| 22 |
+
dims:
|
| 23 |
+
customer: {label: "Customer"}
|
| 24 |
+
payment_state: {label: "Payment state"}
|
| 25 |
+
move_type: {label: "Document type"}
|
| 26 |
+
|
| 27 |
+
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 28 |
+
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 29 |
+
# from another topic; a `link` points at another database.
|
| 30 |
+
fields:
|
| 31 |
+
- key: invoice_no
|
| 32 |
+
label: "Invoice"
|
| 33 |
+
type: text
|
| 34 |
+
kind: data
|
| 35 |
+
- key: odoo_id
|
| 36 |
+
label: "Odoo ID"
|
| 37 |
+
type: int
|
| 38 |
+
kind: data
|
| 39 |
+
means: "The `account.move` id. Also this row's id."
|
| 40 |
+
- key: customer
|
| 41 |
+
label: "Customer"
|
| 42 |
+
type: text
|
| 43 |
+
kind: data
|
| 44 |
+
- key: partner_id
|
| 45 |
+
label: "Odoo partner id"
|
| 46 |
+
type: int
|
| 47 |
+
kind: data
|
| 48 |
+
- key: invoice_date
|
| 49 |
+
label: "Invoice date"
|
| 50 |
+
type: date
|
| 51 |
+
kind: data
|
| 52 |
+
- key: due_date
|
| 53 |
+
label: "Due date"
|
| 54 |
+
type: date
|
| 55 |
+
kind: data
|
| 56 |
+
- key: residual
|
| 57 |
+
label: "Outstanding $"
|
| 58 |
+
type: currency
|
| 59 |
+
kind: data
|
| 60 |
+
means: "Odoo's signed residual. Exactly 0 on every settled document, which is why AR rollups need no filter."
|
| 61 |
+
- key: amount_untaxed
|
| 62 |
+
label: "Invoiced $"
|
| 63 |
+
type: currency
|
| 64 |
+
kind: data
|
| 65 |
+
- key: payment_state
|
| 66 |
+
label: "Payment state"
|
| 67 |
+
type: select
|
| 68 |
+
kind: data
|
| 69 |
+
- key: move_type
|
| 70 |
+
label: "Document"
|
| 71 |
+
type: select
|
| 72 |
+
kind: data
|
| 73 |
+
- key: wholesale_scope
|
| 74 |
+
label: "In wholesale scope"
|
| 75 |
+
type: checkbox
|
| 76 |
+
kind: data
|
| 77 |
+
means: "Unticked = the GIFTWARE DEALS / Amazon channel, which every wholesale metric in this product excludes. The row is kept so no Odoo id is missing; filter on this column to reconcile against the AR page."
|
| 78 |
+
- key: origin_order
|
| 79 |
+
label: "Source order"
|
| 80 |
+
type: text
|
| 81 |
+
kind: data
|
| 82 |
+
means: "Odoo's `invoice_origin` - usually the order name, sometimes blank."
|
| 83 |
+
- key: order_link
|
| 84 |
+
label: "Order record"
|
| 85 |
+
type: link
|
| 86 |
+
kind: link
|
| 87 |
+
to_grid: ut_odoo_orders
|
| 88 |
+
- key: refreshed
|
| 89 |
+
label: "Refreshed"
|
| 90 |
+
type: date
|
| 91 |
+
kind: data
|
| 92 |
+
means: "When this row was last reconciled against Odoo."
|
| 93 |
+
|
| 94 |
+
ai_context: >
|
| 95 |
+
Customer invoices at DOCUMENT grain β one row per invoice, not per line.
|
| 96 |
+
Use it for AR questions: what is open, what is overdue, what was invoiced.
|
| 97 |
+
For anything at product or line grain use invoice_lines; for the reconciled receivables view use receivables.
|
| 98 |
+
`origin_order` is the single stored key that joins an invoice back to the order it was raised from.
|
platform/model/topics/odoo_orders.yml
CHANGED
|
@@ -1,87 +1,87 @@
|
|
| 1 |
-
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
-
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
-
#
|
| 4 |
-
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
-
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
-
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
-
# trained on a schema the product does not have.
|
| 8 |
-
key: odoo_orders
|
| 9 |
-
label: "Odoo orders"
|
| 10 |
-
entity: sale.order
|
| 11 |
-
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
-
grid: ut_odoo_orders
|
| 13 |
-
subject: "odoo:sale.order"
|
| 14 |
-
grain: "one row per confirmed sales order (document grain)"
|
| 15 |
-
scope:
|
| 16 |
-
population: "state in (sale, done) β quotations and cancellations excluded"
|
| 17 |
-
vs_sales_orders: "β DISTINCT from the `sales_orders` topic: that one is the wholesale-scoped analytical view (teams 5/6, house accounts excluded); THIS one is the registry a person opens as a grid, and carries the row set the app shows"
|
| 18 |
-
store:
|
| 19 |
-
table: sale_order
|
| 20 |
-
alias: o
|
| 21 |
-
date_col: "o.date_order"
|
| 22 |
-
dims:
|
| 23 |
-
customer: {label: "Customer"}
|
| 24 |
-
team: {label: "Business unit"}
|
| 25 |
-
state: {label: "State"}
|
| 26 |
-
invoice_status: {label: "Invoice status"}
|
| 27 |
-
|
| 28 |
-
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 29 |
-
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 30 |
-
# from another topic; a `link` points at another database.
|
| 31 |
-
fields:
|
| 32 |
-
- key: order_no
|
| 33 |
-
label: "Order"
|
| 34 |
-
type: text
|
| 35 |
-
kind: data
|
| 36 |
-
- key: odoo_id
|
| 37 |
-
label: "Odoo ID"
|
| 38 |
-
type: int
|
| 39 |
-
kind: data
|
| 40 |
-
means: "The `sale.order` id. Also this row's id."
|
| 41 |
-
- key: customer
|
| 42 |
-
label: "Customer"
|
| 43 |
-
type: text
|
| 44 |
-
kind: data
|
| 45 |
-
- key: partner_id
|
| 46 |
-
label: "Odoo partner id"
|
| 47 |
-
type: int
|
| 48 |
-
kind: data
|
| 49 |
-
- key: order_date
|
| 50 |
-
label: "Order date"
|
| 51 |
-
type: date
|
| 52 |
-
kind: data
|
| 53 |
-
- key: amount_untaxed
|
| 54 |
-
label: "Order $"
|
| 55 |
-
type: currency
|
| 56 |
-
kind: data
|
| 57 |
-
- key: team
|
| 58 |
-
label: "Business unit"
|
| 59 |
-
type: select
|
| 60 |
-
kind: data
|
| 61 |
-
- key: state
|
| 62 |
-
label: "State"
|
| 63 |
-
type: select
|
| 64 |
-
kind: data
|
| 65 |
-
- key: invoice_status
|
| 66 |
-
label: "Invoice status"
|
| 67 |
-
type: select
|
| 68 |
-
kind: data
|
| 69 |
-
- key: wholesale_scope
|
| 70 |
-
label: "In wholesale scope"
|
| 71 |
-
type: checkbox
|
| 72 |
-
kind: data
|
| 73 |
-
means: "Unticked = the GIFTWARE DEALS / Amazon channel, which every wholesale metric in this product excludes. The row is kept so no Odoo id is missing; filter on this column to reconcile against the AR page."
|
| 74 |
-
- key: invoices
|
| 75 |
-
label: "Invoices"
|
| 76 |
-
type: link
|
| 77 |
-
kind: link
|
| 78 |
-
to_grid: ut_odoo_invoices
|
| 79 |
-
- key: refreshed
|
| 80 |
-
label: "Refreshed"
|
| 81 |
-
type: date
|
| 82 |
-
kind: data
|
| 83 |
-
means: "When this row was last reconciled against Odoo."
|
| 84 |
-
|
| 85 |
-
ai_context: >
|
| 86 |
-
Sales orders at DOCUMENT grain, as the app's grid shows them.
|
| 87 |
-
β If you are asked for revenue, prefer `sales_lines` (line grain, wholesale scope) or `sales_orders` β this topic exists to describe the ORDER RECORD a person can open, and its scope is the grid's, not the analytical model's.
|
|
|
|
| 1 |
+
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
+
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
+
#
|
| 4 |
+
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
+
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
+
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
+
# trained on a schema the product does not have.
|
| 8 |
+
key: odoo_orders
|
| 9 |
+
label: "Odoo orders"
|
| 10 |
+
entity: sale.order
|
| 11 |
+
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
+
grid: ut_odoo_orders
|
| 13 |
+
subject: "odoo:sale.order"
|
| 14 |
+
grain: "one row per confirmed sales order (document grain)"
|
| 15 |
+
scope:
|
| 16 |
+
population: "state in (sale, done) β quotations and cancellations excluded"
|
| 17 |
+
vs_sales_orders: "β DISTINCT from the `sales_orders` topic: that one is the wholesale-scoped analytical view (teams 5/6, house accounts excluded); THIS one is the registry a person opens as a grid, and carries the row set the app shows"
|
| 18 |
+
store:
|
| 19 |
+
table: sale_order
|
| 20 |
+
alias: o
|
| 21 |
+
date_col: "o.date_order"
|
| 22 |
+
dims:
|
| 23 |
+
customer: {label: "Customer"}
|
| 24 |
+
team: {label: "Business unit"}
|
| 25 |
+
state: {label: "State"}
|
| 26 |
+
invoice_status: {label: "Invoice status"}
|
| 27 |
+
|
| 28 |
+
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 29 |
+
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 30 |
+
# from another topic; a `link` points at another database.
|
| 31 |
+
fields:
|
| 32 |
+
- key: order_no
|
| 33 |
+
label: "Order"
|
| 34 |
+
type: text
|
| 35 |
+
kind: data
|
| 36 |
+
- key: odoo_id
|
| 37 |
+
label: "Odoo ID"
|
| 38 |
+
type: int
|
| 39 |
+
kind: data
|
| 40 |
+
means: "The `sale.order` id. Also this row's id."
|
| 41 |
+
- key: customer
|
| 42 |
+
label: "Customer"
|
| 43 |
+
type: text
|
| 44 |
+
kind: data
|
| 45 |
+
- key: partner_id
|
| 46 |
+
label: "Odoo partner id"
|
| 47 |
+
type: int
|
| 48 |
+
kind: data
|
| 49 |
+
- key: order_date
|
| 50 |
+
label: "Order date"
|
| 51 |
+
type: date
|
| 52 |
+
kind: data
|
| 53 |
+
- key: amount_untaxed
|
| 54 |
+
label: "Order $"
|
| 55 |
+
type: currency
|
| 56 |
+
kind: data
|
| 57 |
+
- key: team
|
| 58 |
+
label: "Business unit"
|
| 59 |
+
type: select
|
| 60 |
+
kind: data
|
| 61 |
+
- key: state
|
| 62 |
+
label: "State"
|
| 63 |
+
type: select
|
| 64 |
+
kind: data
|
| 65 |
+
- key: invoice_status
|
| 66 |
+
label: "Invoice status"
|
| 67 |
+
type: select
|
| 68 |
+
kind: data
|
| 69 |
+
- key: wholesale_scope
|
| 70 |
+
label: "In wholesale scope"
|
| 71 |
+
type: checkbox
|
| 72 |
+
kind: data
|
| 73 |
+
means: "Unticked = the GIFTWARE DEALS / Amazon channel, which every wholesale metric in this product excludes. The row is kept so no Odoo id is missing; filter on this column to reconcile against the AR page."
|
| 74 |
+
- key: invoices
|
| 75 |
+
label: "Invoices"
|
| 76 |
+
type: link
|
| 77 |
+
kind: link
|
| 78 |
+
to_grid: ut_odoo_invoices
|
| 79 |
+
- key: refreshed
|
| 80 |
+
label: "Refreshed"
|
| 81 |
+
type: date
|
| 82 |
+
kind: data
|
| 83 |
+
means: "When this row was last reconciled against Odoo."
|
| 84 |
+
|
| 85 |
+
ai_context: >
|
| 86 |
+
Sales orders at DOCUMENT grain, as the app's grid shows them.
|
| 87 |
+
β If you are asked for revenue, prefer `sales_lines` (line grain, wholesale scope) or `sales_orders` β this topic exists to describe the ORDER RECORD a person can open, and its scope is the grid's, not the analytical model's.
|
platform/model/topics/odoo_products.yml
CHANGED
|
@@ -1,147 +1,147 @@
|
|
| 1 |
-
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
-
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
-
#
|
| 4 |
-
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
-
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
-
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
-
# trained on a schema the product does not have.
|
| 8 |
-
key: odoo_products
|
| 9 |
-
label: "Odoo products"
|
| 10 |
-
entity: product.product
|
| 11 |
-
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
-
grid: product_data
|
| 13 |
-
subject: "odoo:product.product"
|
| 14 |
-
grain: "one row per SKU in the active catalogue β a CATALOGUE, not a dated event stream"
|
| 15 |
-
scope:
|
| 16 |
-
population: "every ACTIVE product, sold or not (measured 5,862). Deliberately not the sales-window universe: that lands at 2,717 and hides ~2,550 SKUs that have never sold in wholesale scope"
|
| 17 |
-
identity: "the SKU code (`default_code`). β 33 active products carry NO code and are keyed `pid:<odoo product id>` instead, so every active product has exactly one row and none is dropped"
|
| 18 |
-
merged: "W33-T44 retired `ut_odoo_products`, which presented this same subject on Odoo's product_id. This topic describes the SURVIVING database, `product_data`"
|
| 19 |
-
no_date: "a catalogue has no date dimension; ask sales_lines for movement"
|
| 20 |
-
store:
|
| 21 |
-
table: product_product
|
| 22 |
-
alias: pp
|
| 23 |
-
# NO date_col β a registry is not a dated event stream. Stated rather than
|
| 24 |
-
# omitted, so its absence reads as a fact and not as an unfinished file.
|
| 25 |
-
dims:
|
| 26 |
-
category: {label: "Category"}
|
| 27 |
-
supplier: {label: "Supplier"}
|
| 28 |
-
|
| 29 |
-
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 30 |
-
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 31 |
-
# from another topic; a `link` points at another database.
|
| 32 |
-
fields:
|
| 33 |
-
- key: code
|
| 34 |
-
label: "SKU"
|
| 35 |
-
type: text
|
| 36 |
-
kind: data
|
| 37 |
-
means: "The SKU code β the product's real business key. `pid` is a stable CRC32 of it because the grid keys on an integer."
|
| 38 |
-
- key: product
|
| 39 |
-
label: "Product"
|
| 40 |
-
type: text
|
| 41 |
-
kind: data
|
| 42 |
-
means: "Product name as it appears in Odoo."
|
| 43 |
-
- key: category
|
| 44 |
-
label: "Category"
|
| 45 |
-
type: select
|
| 46 |
-
kind: data
|
| 47 |
-
means: "Product category; '(uncategorized)' when Odoo carries none."
|
| 48 |
-
- key: supplier
|
| 49 |
-
label: "Supplier"
|
| 50 |
-
type: text
|
| 51 |
-
kind: data
|
| 52 |
-
means: "Who makes it. Editable here and shared with everyone in the workspace; seeded from the inventory mastersheet."
|
| 53 |
-
- key: origin_country
|
| 54 |
-
label: "Country"
|
| 55 |
-
type: text
|
| 56 |
-
kind: data
|
| 57 |
-
means: "Country of origin. Editable here and shared with everyone; seeded from the inventory mastersheet."
|
| 58 |
-
- key: lead_days
|
| 59 |
-
label: "Lead time (days)"
|
| 60 |
-
type: int
|
| 61 |
-
kind: data
|
| 62 |
-
means: "Order-to-arrival days for this supplier. Drives the buy signal. Editable and shared with everyone."
|
| 63 |
-
- key: first_cost
|
| 64 |
-
label: "First cost"
|
| 65 |
-
type: currency
|
| 66 |
-
kind: data
|
| 67 |
-
means: "Quoted unit cost at origin, before freight and duty. Editable and shared with everyone."
|
| 68 |
-
- key: price_fisch
|
| 69 |
-
label: "Fisch price"
|
| 70 |
-
type: currency
|
| 71 |
-
kind: data
|
| 72 |
-
means: "Fisch pricelist price for this SKU. Blank when that list prices it nowhere."
|
| 73 |
-
- key: price_royal_1
|
| 74 |
-
label: "Royal 1 price"
|
| 75 |
-
type: currency
|
| 76 |
-
kind: data
|
| 77 |
-
means: "Royal 1 pricelist price for this SKU. Blank when that list prices it nowhere."
|
| 78 |
-
- key: price_royal_2
|
| 79 |
-
label: "Royal 2 price"
|
| 80 |
-
type: currency
|
| 81 |
-
kind: data
|
| 82 |
-
means: "Royal 2 pricelist price for this SKU. Blank when that list prices it nowhere."
|
| 83 |
-
- key: rev_ytd
|
| 84 |
-
label: "Revenue YTD"
|
| 85 |
-
type: currency
|
| 86 |
-
kind: data
|
| 87 |
-
means: "Year-to-date revenue for this SKU, BU-scoped when the caller is."
|
| 88 |
-
- key: rev_ly
|
| 89 |
-
label: "Revenue LY"
|
| 90 |
-
type: currency
|
| 91 |
-
kind: data
|
| 92 |
-
means: "Same period last year β seasonal wholesale compares like for like."
|
| 93 |
-
- key: yoy_pct
|
| 94 |
-
label: "YoY %"
|
| 95 |
-
type: pct
|
| 96 |
-
kind: data
|
| 97 |
-
means: "Year-over-year change; null when last year was zero (a ratio to zero is not a number)."
|
| 98 |
-
- key: qty_ytd
|
| 99 |
-
label: "Units YTD"
|
| 100 |
-
type: int
|
| 101 |
-
kind: data
|
| 102 |
-
means: "Units sold year to date."
|
| 103 |
-
- key: orders_ytd
|
| 104 |
-
label: "Orders YTD"
|
| 105 |
-
type: int
|
| 106 |
-
kind: data
|
| 107 |
-
means: "Distinct orders containing this SKU, year to date."
|
| 108 |
-
- key: on_hand
|
| 109 |
-
label: "On hand"
|
| 110 |
-
type: int
|
| 111 |
-
kind: data
|
| 112 |
-
means: "Units in stock. CONSOLIDATED β one physical warehouse, not brand-tagged, so this column is ABSENT for a BU-scoped caller rather than silently company-wide."
|
| 113 |
-
- key: unit_cost
|
| 114 |
-
label: "Unit cost"
|
| 115 |
-
type: currency
|
| 116 |
-
kind: data
|
| 117 |
-
means: "Inventory unit cost. Consolidated; absent for a BU-scoped caller."
|
| 118 |
-
- key: inv_value
|
| 119 |
-
label: "Stock value"
|
| 120 |
-
type: currency
|
| 121 |
-
kind: data
|
| 122 |
-
means: "On-hand value at cost. Consolidated; absent for a BU-scoped caller."
|
| 123 |
-
- key: qty_ltm
|
| 124 |
-
label: "Units LTM"
|
| 125 |
-
type: int
|
| 126 |
-
kind: data
|
| 127 |
-
means: "Units sold in the last twelve months. Consolidated; absent for a BU-scoped caller."
|
| 128 |
-
- key: dos
|
| 129 |
-
label: "Days of supply"
|
| 130 |
-
type: int
|
| 131 |
-
kind: data
|
| 132 |
-
means: "Days of supply at the LTM rate; null means it never sells through. Consolidated; absent for a BU-scoped caller."
|
| 133 |
-
- key: cover_gap_d
|
| 134 |
-
label: "Cover gap (days)"
|
| 135 |
-
type: int
|
| 136 |
-
kind: data
|
| 137 |
-
means: "Days of supply minus lead time. Negative means it runs out before a reorder lands."
|
| 138 |
-
- key: stock_bucket
|
| 139 |
-
label: "Stock status"
|
| 140 |
-
type: select
|
| 141 |
-
kind: data
|
| 142 |
-
means: "Dead / excess / healthy bucket from the inventory module. Consolidated; absent for a BU-scoped caller."
|
| 143 |
-
|
| 144 |
-
ai_context: >
|
| 145 |
-
The product CATALOGUE β the database labelled 'Odoo products'.
|
| 146 |
-
Use it for 'what do we sell', 'what does it cost' (first_cost/unit_cost), 'who supplies it', 'what is on hand' and the stock-coverage columns.
|
| 147 |
-
β Revenue and unit columns here are WINDOWED SNAPSHOTS (rev_ytd, qty_ltm); for any other window, or for line grain, go to sales_lines.
|
|
|
|
| 1 |
+
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
+
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
+
#
|
| 4 |
+
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
+
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
+
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
+
# trained on a schema the product does not have.
|
| 8 |
+
key: odoo_products
|
| 9 |
+
label: "Odoo products"
|
| 10 |
+
entity: product.product
|
| 11 |
+
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
+
grid: product_data
|
| 13 |
+
subject: "odoo:product.product"
|
| 14 |
+
grain: "one row per SKU in the active catalogue β a CATALOGUE, not a dated event stream"
|
| 15 |
+
scope:
|
| 16 |
+
population: "every ACTIVE product, sold or not (measured 5,862). Deliberately not the sales-window universe: that lands at 2,717 and hides ~2,550 SKUs that have never sold in wholesale scope"
|
| 17 |
+
identity: "the SKU code (`default_code`). β 33 active products carry NO code and are keyed `pid:<odoo product id>` instead, so every active product has exactly one row and none is dropped"
|
| 18 |
+
merged: "W33-T44 retired `ut_odoo_products`, which presented this same subject on Odoo's product_id. This topic describes the SURVIVING database, `product_data`"
|
| 19 |
+
no_date: "a catalogue has no date dimension; ask sales_lines for movement"
|
| 20 |
+
store:
|
| 21 |
+
table: product_product
|
| 22 |
+
alias: pp
|
| 23 |
+
# NO date_col β a registry is not a dated event stream. Stated rather than
|
| 24 |
+
# omitted, so its absence reads as a fact and not as an unfinished file.
|
| 25 |
+
dims:
|
| 26 |
+
category: {label: "Category"}
|
| 27 |
+
supplier: {label: "Supplier"}
|
| 28 |
+
|
| 29 |
+
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 30 |
+
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 31 |
+
# from another topic; a `link` points at another database.
|
| 32 |
+
fields:
|
| 33 |
+
- key: code
|
| 34 |
+
label: "SKU"
|
| 35 |
+
type: text
|
| 36 |
+
kind: data
|
| 37 |
+
means: "The SKU code β the product's real business key. `pid` is a stable CRC32 of it because the grid keys on an integer."
|
| 38 |
+
- key: product
|
| 39 |
+
label: "Product"
|
| 40 |
+
type: text
|
| 41 |
+
kind: data
|
| 42 |
+
means: "Product name as it appears in Odoo."
|
| 43 |
+
- key: category
|
| 44 |
+
label: "Category"
|
| 45 |
+
type: select
|
| 46 |
+
kind: data
|
| 47 |
+
means: "Product category; '(uncategorized)' when Odoo carries none."
|
| 48 |
+
- key: supplier
|
| 49 |
+
label: "Supplier"
|
| 50 |
+
type: text
|
| 51 |
+
kind: data
|
| 52 |
+
means: "Who makes it. Editable here and shared with everyone in the workspace; seeded from the inventory mastersheet."
|
| 53 |
+
- key: origin_country
|
| 54 |
+
label: "Country"
|
| 55 |
+
type: text
|
| 56 |
+
kind: data
|
| 57 |
+
means: "Country of origin. Editable here and shared with everyone; seeded from the inventory mastersheet."
|
| 58 |
+
- key: lead_days
|
| 59 |
+
label: "Lead time (days)"
|
| 60 |
+
type: int
|
| 61 |
+
kind: data
|
| 62 |
+
means: "Order-to-arrival days for this supplier. Drives the buy signal. Editable and shared with everyone."
|
| 63 |
+
- key: first_cost
|
| 64 |
+
label: "First cost"
|
| 65 |
+
type: currency
|
| 66 |
+
kind: data
|
| 67 |
+
means: "Quoted unit cost at origin, before freight and duty. Editable and shared with everyone."
|
| 68 |
+
- key: price_fisch
|
| 69 |
+
label: "Fisch price"
|
| 70 |
+
type: currency
|
| 71 |
+
kind: data
|
| 72 |
+
means: "Fisch pricelist price for this SKU. Blank when that list prices it nowhere."
|
| 73 |
+
- key: price_royal_1
|
| 74 |
+
label: "Royal 1 price"
|
| 75 |
+
type: currency
|
| 76 |
+
kind: data
|
| 77 |
+
means: "Royal 1 pricelist price for this SKU. Blank when that list prices it nowhere."
|
| 78 |
+
- key: price_royal_2
|
| 79 |
+
label: "Royal 2 price"
|
| 80 |
+
type: currency
|
| 81 |
+
kind: data
|
| 82 |
+
means: "Royal 2 pricelist price for this SKU. Blank when that list prices it nowhere."
|
| 83 |
+
- key: rev_ytd
|
| 84 |
+
label: "Revenue YTD"
|
| 85 |
+
type: currency
|
| 86 |
+
kind: data
|
| 87 |
+
means: "Year-to-date revenue for this SKU, BU-scoped when the caller is."
|
| 88 |
+
- key: rev_ly
|
| 89 |
+
label: "Revenue LY"
|
| 90 |
+
type: currency
|
| 91 |
+
kind: data
|
| 92 |
+
means: "Same period last year β seasonal wholesale compares like for like."
|
| 93 |
+
- key: yoy_pct
|
| 94 |
+
label: "YoY %"
|
| 95 |
+
type: pct
|
| 96 |
+
kind: data
|
| 97 |
+
means: "Year-over-year change; null when last year was zero (a ratio to zero is not a number)."
|
| 98 |
+
- key: qty_ytd
|
| 99 |
+
label: "Units YTD"
|
| 100 |
+
type: int
|
| 101 |
+
kind: data
|
| 102 |
+
means: "Units sold year to date."
|
| 103 |
+
- key: orders_ytd
|
| 104 |
+
label: "Orders YTD"
|
| 105 |
+
type: int
|
| 106 |
+
kind: data
|
| 107 |
+
means: "Distinct orders containing this SKU, year to date."
|
| 108 |
+
- key: on_hand
|
| 109 |
+
label: "On hand"
|
| 110 |
+
type: int
|
| 111 |
+
kind: data
|
| 112 |
+
means: "Units in stock. CONSOLIDATED β one physical warehouse, not brand-tagged, so this column is ABSENT for a BU-scoped caller rather than silently company-wide."
|
| 113 |
+
- key: unit_cost
|
| 114 |
+
label: "Unit cost"
|
| 115 |
+
type: currency
|
| 116 |
+
kind: data
|
| 117 |
+
means: "Inventory unit cost. Consolidated; absent for a BU-scoped caller."
|
| 118 |
+
- key: inv_value
|
| 119 |
+
label: "Stock value"
|
| 120 |
+
type: currency
|
| 121 |
+
kind: data
|
| 122 |
+
means: "On-hand value at cost. Consolidated; absent for a BU-scoped caller."
|
| 123 |
+
- key: qty_ltm
|
| 124 |
+
label: "Units LTM"
|
| 125 |
+
type: int
|
| 126 |
+
kind: data
|
| 127 |
+
means: "Units sold in the last twelve months. Consolidated; absent for a BU-scoped caller."
|
| 128 |
+
- key: dos
|
| 129 |
+
label: "Days of supply"
|
| 130 |
+
type: int
|
| 131 |
+
kind: data
|
| 132 |
+
means: "Days of supply at the LTM rate; null means it never sells through. Consolidated; absent for a BU-scoped caller."
|
| 133 |
+
- key: cover_gap_d
|
| 134 |
+
label: "Cover gap (days)"
|
| 135 |
+
type: int
|
| 136 |
+
kind: data
|
| 137 |
+
means: "Days of supply minus lead time. Negative means it runs out before a reorder lands."
|
| 138 |
+
- key: stock_bucket
|
| 139 |
+
label: "Stock status"
|
| 140 |
+
type: select
|
| 141 |
+
kind: data
|
| 142 |
+
means: "Dead / excess / healthy bucket from the inventory module. Consolidated; absent for a BU-scoped caller."
|
| 143 |
+
|
| 144 |
+
ai_context: >
|
| 145 |
+
The product CATALOGUE β the database labelled 'Odoo products'.
|
| 146 |
+
Use it for 'what do we sell', 'what does it cost' (first_cost/unit_cost), 'who supplies it', 'what is on hand' and the stock-coverage columns.
|
| 147 |
+
β Revenue and unit columns here are WINDOWED SNAPSHOTS (rev_ytd, qty_ltm); for any other window, or for line grain, go to sales_lines.
|
platform/model/topics/odoo_vendors.yml
CHANGED
|
@@ -1,103 +1,103 @@
|
|
| 1 |
-
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
-
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
-
#
|
| 4 |
-
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
-
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
-
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
-
# trained on a schema the product does not have.
|
| 8 |
-
key: odoo_vendors
|
| 9 |
-
label: "Odoo vendors"
|
| 10 |
-
entity: res.partner
|
| 11 |
-
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
-
grid: ut_odoo_vendors
|
| 13 |
-
subject: "odoo:res.partner.vendor"
|
| 14 |
-
grain: "one row per partner we have posted a vendor bill to"
|
| 15 |
-
scope:
|
| 16 |
-
population: "DERIVED FROM THE BILLS β a partner with no posted vendor bill has no payable history to show, so the bill is the population"
|
| 17 |
-
asymmetry: "deliberately unlike odoo_customers, which is NOT derived from its documents; there is no second document universe for vendors"
|
| 18 |
-
overlap: "measured: only 9 of the vendors also appear in the customer population, so a vendor is not a customer row"
|
| 19 |
-
store:
|
| 20 |
-
table: res_partner
|
| 21 |
-
alias: p
|
| 22 |
-
# NO date_col β a registry is not a dated event stream. Stated rather than
|
| 23 |
-
# omitted, so its absence reads as a fact and not as an unfinished file.
|
| 24 |
-
dims:
|
| 25 |
-
country: {label: "Country"}
|
| 26 |
-
|
| 27 |
-
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 28 |
-
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 29 |
-
# from another topic; a `link` points at another database.
|
| 30 |
-
fields:
|
| 31 |
-
- key: vendor
|
| 32 |
-
label: "Vendor"
|
| 33 |
-
type: text
|
| 34 |
-
kind: data
|
| 35 |
-
- key: odoo_id
|
| 36 |
-
label: "Odoo ID"
|
| 37 |
-
type: int
|
| 38 |
-
kind: data
|
| 39 |
-
means: "The `res.partner` id. Also this row's id."
|
| 40 |
-
- key: vendor_id
|
| 41 |
-
label: "Odoo vendor id"
|
| 42 |
-
type: int
|
| 43 |
-
kind: data
|
| 44 |
-
- key: country
|
| 45 |
-
label: "Country"
|
| 46 |
-
type: text
|
| 47 |
-
kind: data
|
| 48 |
-
- key: email
|
| 49 |
-
label: "Email"
|
| 50 |
-
type: text
|
| 51 |
-
kind: data
|
| 52 |
-
- key: phone
|
| 53 |
-
label: "Phone"
|
| 54 |
-
type: text
|
| 55 |
-
kind: data
|
| 56 |
-
- key: mobile
|
| 57 |
-
label: "Mobile"
|
| 58 |
-
type: text
|
| 59 |
-
kind: data
|
| 60 |
-
- key: vat
|
| 61 |
-
label: "Tax ID"
|
| 62 |
-
type: text
|
| 63 |
-
kind: data
|
| 64 |
-
means: "Odoo `vat` β the vendor's tax/VAT registration number."
|
| 65 |
-
- key: vendor_ref
|
| 66 |
-
label: "Vendor reference"
|
| 67 |
-
type: text
|
| 68 |
-
kind: data
|
| 69 |
-
means: "Odoo `res.partner.ref` β our internal reference for this vendor."
|
| 70 |
-
- key: website
|
| 71 |
-
label: "Website"
|
| 72 |
-
type: url
|
| 73 |
-
kind: data
|
| 74 |
-
- key: street
|
| 75 |
-
label: "Street"
|
| 76 |
-
type: text
|
| 77 |
-
kind: data
|
| 78 |
-
- key: street2
|
| 79 |
-
label: "Street 2"
|
| 80 |
-
type: text
|
| 81 |
-
kind: data
|
| 82 |
-
- key: city
|
| 83 |
-
label: "City"
|
| 84 |
-
type: text
|
| 85 |
-
kind: data
|
| 86 |
-
- key: zip
|
| 87 |
-
label: "ZIP"
|
| 88 |
-
type: text
|
| 89 |
-
kind: data
|
| 90 |
-
- key: bills
|
| 91 |
-
label: "Bills"
|
| 92 |
-
type: link
|
| 93 |
-
kind: link
|
| 94 |
-
to_grid: ut_odoo_bills
|
| 95 |
-
- key: refreshed
|
| 96 |
-
label: "Refreshed"
|
| 97 |
-
type: date
|
| 98 |
-
kind: data
|
| 99 |
-
means: "When this row was last reconciled against Odoo."
|
| 100 |
-
|
| 101 |
-
ai_context: >
|
| 102 |
-
The vendor/supplier registry, with contact and tax identity (W33-T48 widened it from 5 columns to 15 after a census found 76 populated fields on the underlying partner)
|
| 103 |
-
β The contact columns are SPARSE by nature β measured 65/393 with an email, 13/393 with a tax id β so 'blank' means Odoo has no value, never that the sync failed.
|
|
|
|
| 1 |
+
# ββ W33-T49 (owner item 13) β an ENTITY topic: the semantic layer can finally see
|
| 2 |
+
# the databases a person actually opens, not only the line and document grains.
|
| 3 |
+
#
|
| 4 |
+
# β THE `fields:` BLOCK IS GENERATED FROM THE GRID'S OWN FIELD CONTRACT
|
| 5 |
+
# and is held to it by `verify_query.py::section_entity_topics`. Do not hand-edit a
|
| 6 |
+
# field row: add the column to the grid contract and re-emit, or the agent is being
|
| 7 |
+
# trained on a schema the product does not have.
|
| 8 |
+
key: odoo_vendors
|
| 9 |
+
label: "Odoo vendors"
|
| 10 |
+
entity: res.partner
|
| 11 |
+
# The database this topic DESCRIBES β the same store key the nav opens (W33-T46).
|
| 12 |
+
grid: ut_odoo_vendors
|
| 13 |
+
subject: "odoo:res.partner.vendor"
|
| 14 |
+
grain: "one row per partner we have posted a vendor bill to"
|
| 15 |
+
scope:
|
| 16 |
+
population: "DERIVED FROM THE BILLS β a partner with no posted vendor bill has no payable history to show, so the bill is the population"
|
| 17 |
+
asymmetry: "deliberately unlike odoo_customers, which is NOT derived from its documents; there is no second document universe for vendors"
|
| 18 |
+
overlap: "measured: only 9 of the vendors also appear in the customer population, so a vendor is not a customer row"
|
| 19 |
+
store:
|
| 20 |
+
table: res_partner
|
| 21 |
+
alias: p
|
| 22 |
+
# NO date_col β a registry is not a dated event stream. Stated rather than
|
| 23 |
+
# omitted, so its absence reads as a fact and not as an unfinished file.
|
| 24 |
+
dims:
|
| 25 |
+
country: {label: "Country"}
|
| 26 |
+
|
| 27 |
+
# key / label / type / kind, derived from the grid contract. `kind` says where the
|
| 28 |
+
# value COMES FROM: a `data` column is stored on the row; a `rollup` is computed
|
| 29 |
+
# from another topic; a `link` points at another database.
|
| 30 |
+
fields:
|
| 31 |
+
- key: vendor
|
| 32 |
+
label: "Vendor"
|
| 33 |
+
type: text
|
| 34 |
+
kind: data
|
| 35 |
+
- key: odoo_id
|
| 36 |
+
label: "Odoo ID"
|
| 37 |
+
type: int
|
| 38 |
+
kind: data
|
| 39 |
+
means: "The `res.partner` id. Also this row's id."
|
| 40 |
+
- key: vendor_id
|
| 41 |
+
label: "Odoo vendor id"
|
| 42 |
+
type: int
|
| 43 |
+
kind: data
|
| 44 |
+
- key: country
|
| 45 |
+
label: "Country"
|
| 46 |
+
type: text
|
| 47 |
+
kind: data
|
| 48 |
+
- key: email
|
| 49 |
+
label: "Email"
|
| 50 |
+
type: text
|
| 51 |
+
kind: data
|
| 52 |
+
- key: phone
|
| 53 |
+
label: "Phone"
|
| 54 |
+
type: text
|
| 55 |
+
kind: data
|
| 56 |
+
- key: mobile
|
| 57 |
+
label: "Mobile"
|
| 58 |
+
type: text
|
| 59 |
+
kind: data
|
| 60 |
+
- key: vat
|
| 61 |
+
label: "Tax ID"
|
| 62 |
+
type: text
|
| 63 |
+
kind: data
|
| 64 |
+
means: "Odoo `vat` β the vendor's tax/VAT registration number."
|
| 65 |
+
- key: vendor_ref
|
| 66 |
+
label: "Vendor reference"
|
| 67 |
+
type: text
|
| 68 |
+
kind: data
|
| 69 |
+
means: "Odoo `res.partner.ref` β our internal reference for this vendor."
|
| 70 |
+
- key: website
|
| 71 |
+
label: "Website"
|
| 72 |
+
type: url
|
| 73 |
+
kind: data
|
| 74 |
+
- key: street
|
| 75 |
+
label: "Street"
|
| 76 |
+
type: text
|
| 77 |
+
kind: data
|
| 78 |
+
- key: street2
|
| 79 |
+
label: "Street 2"
|
| 80 |
+
type: text
|
| 81 |
+
kind: data
|
| 82 |
+
- key: city
|
| 83 |
+
label: "City"
|
| 84 |
+
type: text
|
| 85 |
+
kind: data
|
| 86 |
+
- key: zip
|
| 87 |
+
label: "ZIP"
|
| 88 |
+
type: text
|
| 89 |
+
kind: data
|
| 90 |
+
- key: bills
|
| 91 |
+
label: "Bills"
|
| 92 |
+
type: link
|
| 93 |
+
kind: link
|
| 94 |
+
to_grid: ut_odoo_bills
|
| 95 |
+
- key: refreshed
|
| 96 |
+
label: "Refreshed"
|
| 97 |
+
type: date
|
| 98 |
+
kind: data
|
| 99 |
+
means: "When this row was last reconciled against Odoo."
|
| 100 |
+
|
| 101 |
+
ai_context: >
|
| 102 |
+
The vendor/supplier registry, with contact and tax identity (W33-T48 widened it from 5 columns to 15 after a census found 76 populated fields on the underlying partner)
|
| 103 |
+
β The contact columns are SPARSE by nature β measured 65/393 with an email, 13/393 with a tax id β so 'blank' means Odoo has no value, never that the sync failed.
|
platform/modules/collections_send.py
CHANGED
|
@@ -1,267 +1,267 @@
|
|
| 1 |
-
"""Collections statements β the send layer behind the Collections page's Statements section.
|
| 2 |
-
|
| 3 |
-
Folded in from the standalone collections_app (2026-07-05). THE one sanctioned exception to the
|
| 4 |
-
app's read-only-on-Odoo rule, unchanged from the standalone tool: WRITE is whitelisted to exactly
|
| 5 |
-
one operation β
|
| 6 |
-
|
| 7 |
-
create on mail.mail (queueing an outbound statement email)
|
| 8 |
-
|
| 9 |
-
A mail.mail record with state='outgoing' is picked up by Odoo's "Mail: Email Queue Manager" cron
|
| 10 |
-
(every ~15 min) and delivered through the company's Office 365 relay. Because we set model/res_id,
|
| 11 |
-
each sent statement also appears in the customer's chatter in Odoo β the audit log lives where AR
|
| 12 |
-
already works. This module has its OWN narrow XML-RPC client; the app-wide odoo_client stays
|
| 13 |
-
hard-blocking on all writes. The UI gates the section to admin users; SAFE_MODE is enforced HERE
|
| 14 |
-
in the data layer so the UI cannot bypass it.
|
| 15 |
-
|
| 16 |
-
Env (Space secrets / .env): ODOO_URL, ODOO_DB, ODOO_USER, ODOO_API_KEY
|
| 17 |
-
Optional: SAFE_MODE (default ON), SAFE_RECIPIENTS, SENDER_NAME, SENDER_EMAIL, REPLY_TO,
|
| 18 |
-
COMPANY_NAME, ROYAL_MAIL_SERVER_ID, ROYAL_AUTHOR_ID
|
| 19 |
-
"""
|
| 20 |
-
import os
|
| 21 |
-
import datetime as dt
|
| 22 |
-
import xmlrpc.client
|
| 23 |
-
from pathlib import Path
|
| 24 |
-
|
| 25 |
-
try: # self-contained: load credentials from the app root .env if present (HF uses Secrets)
|
| 26 |
-
from dotenv import load_dotenv
|
| 27 |
-
load_dotenv(Path(__file__).resolve().parents[1] / '.env')
|
| 28 |
-
except Exception:
|
| 29 |
-
pass
|
| 30 |
-
|
| 31 |
-
WRITE_WHITELIST = {('mail.mail', 'create')}
|
| 32 |
-
|
| 33 |
-
EXCLUDE_NAMES = {'GIFTWARE DEALS'} # the Amazon channel β not part of Fisch or Royal collections
|
| 34 |
-
DOMAIN = [('followup_reminder_type', '=', 'automatic'), ('credit', '>', 1)]
|
| 35 |
-
|
| 36 |
-
COMPANY = os.environ.get('COMPANY_NAME', 'Royal Imports')
|
| 37 |
-
|
| 38 |
-
# --- Sender identity (statements go out AS Royal Imports) ---
|
| 39 |
-
# Odoo routes outbound mail to the matching ir.mail_server by from_filter, and the Office 365
|
| 40 |
-
# relay only accepts sends as its authenticated address. The "Office 365 - Royal" server (id 2)
|
| 41 |
-
# authenticates as contact@royalimports.com with from_filter='contact@royalimports.com' β so the
|
| 42 |
-
# From MUST be that address for delivery to succeed. Friendly display name; replies routed to AR.
|
| 43 |
-
SENDER_NAME = os.environ.get('SENDER_NAME', 'Royal Imports Accounts Receivable')
|
| 44 |
-
SENDER_EMAIL = os.environ.get('SENDER_EMAIL', 'contact@royalimports.com')
|
| 45 |
-
REPLY_TO = os.environ.get('REPLY_TO', 'accounting@royalimports.com')
|
| 46 |
-
ROYAL_MAIL_SERVER_ID = int(os.environ.get('ROYAL_MAIL_SERVER_ID', '2'))
|
| 47 |
-
ROYAL_AUTHOR_ID = int(os.environ.get('ROYAL_AUTHOR_ID', '8978')) # "Royal Imports" partner
|
| 48 |
-
|
| 49 |
-
SENDER_DISPLAY = f'"{SENDER_NAME}" <{SENDER_EMAIL}>'
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
class WriteBlocked(RuntimeError):
|
| 53 |
-
pass
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
class SafeModeBlocked(RuntimeError):
|
| 57 |
-
pass
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
# --- Testing guardrail -------------------------------------------------------
|
| 61 |
-
# While SAFE_MODE is on, NO email can be queued to any address outside the allow-list β enforced
|
| 62 |
-
# here in the data layer so the UI cannot bypass it. Default: ON. To go live for real customers,
|
| 63 |
-
# set the Space secret SAFE_MODE=0.
|
| 64 |
-
SAFE_MODE = os.environ.get('SAFE_MODE', '1').strip().lower() not in ('0', 'false', 'no', '')
|
| 65 |
-
SAFE_RECIPIENTS = {a.strip().lower() for a in
|
| 66 |
-
os.environ.get('SAFE_RECIPIENTS', 'farhan@teamroyalimports.com').split(',')
|
| 67 |
-
if a.strip()}
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
def safe_recipient_ok(addr):
|
| 71 |
-
return (not SAFE_MODE) or (str(addr or '').strip().lower() in SAFE_RECIPIENTS)
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
class Odoo:
|
| 75 |
-
"""Narrow client: read anything, write ONLY the whitelisted mail.mail create."""
|
| 76 |
-
|
| 77 |
-
def __init__(self):
|
| 78 |
-
self.url = os.environ.get('ODOO_URL', '').rstrip('/')
|
| 79 |
-
self.db = os.environ.get('ODOO_DB', '')
|
| 80 |
-
self.user = os.environ.get('ODOO_USER', '')
|
| 81 |
-
self.key = os.environ.get('ODOO_API_KEY', '')
|
| 82 |
-
missing = [k for k, v in [('ODOO_URL', self.url), ('ODOO_DB', self.db),
|
| 83 |
-
('ODOO_USER', self.user), ('ODOO_API_KEY', self.key)] if not v]
|
| 84 |
-
if missing:
|
| 85 |
-
raise RuntimeError(f"Missing secrets: {', '.join(missing)}")
|
| 86 |
-
common = xmlrpc.client.ServerProxy(f'{self.url}/xmlrpc/2/common')
|
| 87 |
-
self.uid = common.authenticate(self.db, self.user, self.key, {})
|
| 88 |
-
if not self.uid:
|
| 89 |
-
raise RuntimeError('Odoo authentication failed')
|
| 90 |
-
self.models = xmlrpc.client.ServerProxy(f'{self.url}/xmlrpc/2/object')
|
| 91 |
-
|
| 92 |
-
def _exec(self, model, method, args, kwargs=None):
|
| 93 |
-
mutating = method in ('write', 'create', 'unlink', 'copy') or \
|
| 94 |
-
any(method.startswith(p) for p in ('action_', 'button_', 'do_', 'send_',
|
| 95 |
-
'set_', 'update_', 'process_'))
|
| 96 |
-
if mutating and (model, method) not in WRITE_WHITELIST:
|
| 97 |
-
raise WriteBlocked(f'{method} on {model} is not allowed from this app')
|
| 98 |
-
return self.models.execute_kw(self.db, self.uid, self.key,
|
| 99 |
-
model, method, args, kwargs or {})
|
| 100 |
-
|
| 101 |
-
def search_read(self, model, domain=None, fields=None, limit=None, order=None):
|
| 102 |
-
kw = {'fields': fields or []}
|
| 103 |
-
if limit is not None:
|
| 104 |
-
kw['limit'] = limit
|
| 105 |
-
if order:
|
| 106 |
-
kw['order'] = order
|
| 107 |
-
return self._exec(model, 'search_read', [domain or []], kw)
|
| 108 |
-
|
| 109 |
-
def queue_mail(self, payload):
|
| 110 |
-
"""The single allowed write: queue an outbound email."""
|
| 111 |
-
return self._exec('mail.mail', 'create', [payload])
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
# --------------------------------------------------------------- data builders
|
| 115 |
-
def load_collection_list(odoo):
|
| 116 |
-
"""The saved follow-up filter (Reminders=Automatic, Receivable>$1), GIFTWARE excluded,
|
| 117 |
-
with priority tiers."""
|
| 118 |
-
partners = odoo.search_read('res.partner', DOMAIN,
|
| 119 |
-
['name', 'credit', 'total_overdue', 'followup_status',
|
| 120 |
-
'followup_next_action_date', 'followup_responsible_id',
|
| 121 |
-
'email', 'phone', 'mobile'])
|
| 122 |
-
partners = [p for p in partners if (p['name'] or '').strip().upper() not in EXCLUDE_NAMES]
|
| 123 |
-
pids = [p['id'] for p in partners]
|
| 124 |
-
|
| 125 |
-
docs = odoo.search_read('account.move',
|
| 126 |
-
[('move_type', 'in', ['out_invoice', 'out_refund']), ('state', '=', 'posted'),
|
| 127 |
-
('payment_state', 'in', ['not_paid', 'partial']), ('partner_id', 'in', pids)],
|
| 128 |
-
['name', 'partner_id', 'move_type', 'invoice_date', 'invoice_date_due',
|
| 129 |
-
'amount_total', 'amount_residual_signed'])
|
| 130 |
-
|
| 131 |
-
today = dt.date.today()
|
| 132 |
-
by_partner = {}
|
| 133 |
-
for d in docs:
|
| 134 |
-
pid = d['partner_id'][0]
|
| 135 |
-
due = d.get('invoice_date_due')
|
| 136 |
-
try:
|
| 137 |
-
days = (today - dt.date.fromisoformat(due)).days if due else 0
|
| 138 |
-
except Exception:
|
| 139 |
-
days = 0
|
| 140 |
-
d['days_overdue'] = max(days, 0)
|
| 141 |
-
d['open_signed'] = d['amount_residual_signed']
|
| 142 |
-
by_partner.setdefault(pid, []).append(d)
|
| 143 |
-
|
| 144 |
-
rows = []
|
| 145 |
-
for p in partners:
|
| 146 |
-
pid = p['id']
|
| 147 |
-
odoo_overdue = float(p.get('total_overdue') or 0)
|
| 148 |
-
docs_p = sorted(by_partner.get(pid, []), key=lambda x: x.get('invoice_date_due') or '')
|
| 149 |
-
inv_overdue = sum(d['open_signed'] for d in docs_p if d['days_overdue'] > 0)
|
| 150 |
-
oldest = max((d['days_overdue'] for d in docs_p), default=0)
|
| 151 |
-
gap = inv_overdue - odoo_overdue
|
| 152 |
-
reconcile = abs(gap) > 50
|
| 153 |
-
|
| 154 |
-
if odoo_overdue >= 5000 or (odoo_overdue > 0 and oldest > 90):
|
| 155 |
-
tier = 'A-Urgent'
|
| 156 |
-
elif odoo_overdue >= 1000 or (odoo_overdue > 0 and oldest > 30):
|
| 157 |
-
tier = 'B-Active'
|
| 158 |
-
elif odoo_overdue > 0:
|
| 159 |
-
tier = 'C-Light'
|
| 160 |
-
else:
|
| 161 |
-
tier = 'Monitor'
|
| 162 |
-
|
| 163 |
-
rows.append({
|
| 164 |
-
'partner_id': pid,
|
| 165 |
-
'Customer': p['name'],
|
| 166 |
-
'Tier': tier,
|
| 167 |
-
'Overdue': odoo_overdue,
|
| 168 |
-
'Receivable': float(p.get('credit') or 0),
|
| 169 |
-
'Oldest (days)': oldest,
|
| 170 |
-
'Open Docs': len(docs_p),
|
| 171 |
-
'Email': p.get('email') or '',
|
| 172 |
-
'Phone': p.get('phone') or p.get('mobile') or '',
|
| 173 |
-
'Status': (p.get('followup_status') or '').replace('_', ' '),
|
| 174 |
-
'Reconcile?': 'YES' if reconcile else '',
|
| 175 |
-
'_docs': docs_p,
|
| 176 |
-
})
|
| 177 |
-
tier_rank = {'A-Urgent': 0, 'B-Active': 1, 'C-Light': 2, 'Monitor': 3}
|
| 178 |
-
rows.sort(key=lambda r: (tier_rank[r['Tier']], -r['Overdue']))
|
| 179 |
-
return rows
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
# --------------------------------------------------------------- statement email
|
| 183 |
-
# β STANDING RULE 2, AND THIS IS THE ONE STRING IN THE PRODUCT THAT LEAVES THE BUILDING.
|
| 184 |
-
# Every other finding `web_prose` reports is copy on a screen somebody here opens; this is the
|
| 185 |
-
# SUBJECT LINE of mail queued to a real debtor, over the single sanctioned Odoo write. It read
|
| 186 |
-
# 'Statement of Account (em dash) {company} (em dash) {month}' until W36-T42.
|
| 187 |
-
DEFAULT_SUBJECT = 'Statement of Account from {company}, {month}'
|
| 188 |
-
DEFAULT_INTRO = (
|
| 189 |
-
'Dear {customer},<br><br>'
|
| 190 |
-
'Please find below your current statement of account with {company}. '
|
| 191 |
-
'According to our records, the following invoices remain open:'
|
| 192 |
-
)
|
| 193 |
-
DEFAULT_FOOTER = (
|
| 194 |
-
'If you have already sent payment, please disregard this notice, and thank you. '
|
| 195 |
-
'For any questions about an invoice, simply reply to this email.<br><br>'
|
| 196 |
-
'Thank you for your business,<br>{company}<br>Accounts Receivable'
|
| 197 |
-
)
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
def render_statement_html(row, intro_tpl=DEFAULT_INTRO, footer_tpl=DEFAULT_FOOTER):
|
| 201 |
-
month = dt.date.today().strftime('%B %Y')
|
| 202 |
-
intro = intro_tpl.format(customer=row['Customer'], company=COMPANY, month=month)
|
| 203 |
-
footer = footer_tpl.format(customer=row['Customer'], company=COMPANY, month=month)
|
| 204 |
-
|
| 205 |
-
lines = []
|
| 206 |
-
total_open = 0.0
|
| 207 |
-
for d in row['_docs']:
|
| 208 |
-
kind = 'Credit Note' if d['move_type'] == 'out_refund' else 'Invoice'
|
| 209 |
-
amt = d['open_signed']
|
| 210 |
-
total_open += amt
|
| 211 |
-
overdue_txt = f"{d['days_overdue']}d overdue" if d['days_overdue'] > 0 else 'current'
|
| 212 |
-
color = '#C0392B' if d['days_overdue'] > 0 else '#1F4E78'
|
| 213 |
-
lines.append(
|
| 214 |
-
f"<tr><td style='padding:6px 10px;border-bottom:1px solid #e3e8ef'>{d['name']} <span style='color:#888'>({kind})</span></td>"
|
| 215 |
-
f"<td style='padding:6px 10px;border-bottom:1px solid #e3e8ef'>{d.get('invoice_date') or ''}</td>"
|
| 216 |
-
f"<td style='padding:6px 10px;border-bottom:1px solid #e3e8ef'>{d.get('invoice_date_due') or ''}</td>"
|
| 217 |
-
f"<td style='padding:6px 10px;border-bottom:1px solid #e3e8ef;color:{color}'>{overdue_txt}</td>"
|
| 218 |
-
f"<td style='padding:6px 10px;border-bottom:1px solid #e3e8ef;text-align:right'>${amt:,.2f}</td></tr>")
|
| 219 |
-
|
| 220 |
-
table = (
|
| 221 |
-
"<table style='border-collapse:collapse;font-size:14px;margin:14px 0'>"
|
| 222 |
-
"<tr style='background:#1F4E78;color:#fff'>"
|
| 223 |
-
"<th style='padding:7px 10px;text-align:left'>Document</th>"
|
| 224 |
-
"<th style='padding:7px 10px;text-align:left'>Date</th>"
|
| 225 |
-
"<th style='padding:7px 10px;text-align:left'>Due</th>"
|
| 226 |
-
"<th style='padding:7px 10px;text-align:left'>Status</th>"
|
| 227 |
-
"<th style='padding:7px 10px;text-align:right'>Open Balance</th></tr>"
|
| 228 |
-
+ ''.join(lines) +
|
| 229 |
-
f"<tr><td colspan='4' style='padding:8px 10px;font-weight:bold;text-align:right'>Total open</td>"
|
| 230 |
-
f"<td style='padding:8px 10px;font-weight:bold;text-align:right'>${total_open:,.2f}</td></tr>"
|
| 231 |
-
f"<tr><td colspan='4' style='padding:2px 10px;font-weight:bold;text-align:right;color:#C0392B'>Of which overdue</td>"
|
| 232 |
-
f"<td style='padding:2px 10px;font-weight:bold;text-align:right;color:#C0392B'>${row['Overdue']:,.2f}</td></tr>"
|
| 233 |
-
"</table>")
|
| 234 |
-
|
| 235 |
-
return (f"<div style='font-family:Calibri,Arial,sans-serif;color:#1a1a1a;font-size:14px'>"
|
| 236 |
-
f"{intro}{table}{footer}</div>")
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
def queue_statement(odoo, row, subject_tpl=DEFAULT_SUBJECT,
|
| 240 |
-
intro_tpl=DEFAULT_INTRO, footer_tpl=DEFAULT_FOOTER,
|
| 241 |
-
override_to=None):
|
| 242 |
-
"""Queue one statement email in Odoo. Returns mail.mail id.
|
| 243 |
-
override_to: send to a different address (used by the test-send button)."""
|
| 244 |
-
to = override_to or row['Email']
|
| 245 |
-
if not to:
|
| 246 |
-
raise ValueError(f"{row['Customer']} has no email address")
|
| 247 |
-
# Hard guardrail β refuse any recipient outside the allow-list while SAFE_MODE is on.
|
| 248 |
-
if not safe_recipient_ok(to):
|
| 249 |
-
raise SafeModeBlocked(
|
| 250 |
-
f"Guardrail ON: refusing to email {to}. Only {', '.join(sorted(SAFE_RECIPIENTS))} "
|
| 251 |
-
f"is allowed right now. (Set SAFE_MODE=0 to send to real customers.)")
|
| 252 |
-
month = dt.date.today().strftime('%B %Y')
|
| 253 |
-
subject = subject_tpl.format(customer=row['Customer'], company=COMPANY, month=month)
|
| 254 |
-
payload = {
|
| 255 |
-
'subject': subject,
|
| 256 |
-
'body_html': render_statement_html(row, intro_tpl, footer_tpl),
|
| 257 |
-
'email_to': to,
|
| 258 |
-
'email_from': SENDER_DISPLAY, # From: Royal Imports
|
| 259 |
-
'reply_to': REPLY_TO, # replies -> AR
|
| 260 |
-
'mail_server_id': ROYAL_MAIL_SERVER_ID, # force Royal O365 relay
|
| 261 |
-
'author_id': ROYAL_AUTHOR_ID, # clean attribution in chatter
|
| 262 |
-
'state': 'outgoing',
|
| 263 |
-
'auto_delete': False,
|
| 264 |
-
'model': 'res.partner',
|
| 265 |
-
'res_id': row['partner_id'],
|
| 266 |
-
}
|
| 267 |
-
return odoo.queue_mail(payload)
|
|
|
|
| 1 |
+
"""Collections statements β the send layer behind the Collections page's Statements section.
|
| 2 |
+
|
| 3 |
+
Folded in from the standalone collections_app (2026-07-05). THE one sanctioned exception to the
|
| 4 |
+
app's read-only-on-Odoo rule, unchanged from the standalone tool: WRITE is whitelisted to exactly
|
| 5 |
+
one operation β
|
| 6 |
+
|
| 7 |
+
create on mail.mail (queueing an outbound statement email)
|
| 8 |
+
|
| 9 |
+
A mail.mail record with state='outgoing' is picked up by Odoo's "Mail: Email Queue Manager" cron
|
| 10 |
+
(every ~15 min) and delivered through the company's Office 365 relay. Because we set model/res_id,
|
| 11 |
+
each sent statement also appears in the customer's chatter in Odoo β the audit log lives where AR
|
| 12 |
+
already works. This module has its OWN narrow XML-RPC client; the app-wide odoo_client stays
|
| 13 |
+
hard-blocking on all writes. The UI gates the section to admin users; SAFE_MODE is enforced HERE
|
| 14 |
+
in the data layer so the UI cannot bypass it.
|
| 15 |
+
|
| 16 |
+
Env (Space secrets / .env): ODOO_URL, ODOO_DB, ODOO_USER, ODOO_API_KEY
|
| 17 |
+
Optional: SAFE_MODE (default ON), SAFE_RECIPIENTS, SENDER_NAME, SENDER_EMAIL, REPLY_TO,
|
| 18 |
+
COMPANY_NAME, ROYAL_MAIL_SERVER_ID, ROYAL_AUTHOR_ID
|
| 19 |
+
"""
|
| 20 |
+
import os
|
| 21 |
+
import datetime as dt
|
| 22 |
+
import xmlrpc.client
|
| 23 |
+
from pathlib import Path
|
| 24 |
+
|
| 25 |
+
try: # self-contained: load credentials from the app root .env if present (HF uses Secrets)
|
| 26 |
+
from dotenv import load_dotenv
|
| 27 |
+
load_dotenv(Path(__file__).resolve().parents[1] / '.env')
|
| 28 |
+
except Exception:
|
| 29 |
+
pass
|
| 30 |
+
|
| 31 |
+
WRITE_WHITELIST = {('mail.mail', 'create')}
|
| 32 |
+
|
| 33 |
+
EXCLUDE_NAMES = {'GIFTWARE DEALS'} # the Amazon channel β not part of Fisch or Royal collections
|
| 34 |
+
DOMAIN = [('followup_reminder_type', '=', 'automatic'), ('credit', '>', 1)]
|
| 35 |
+
|
| 36 |
+
COMPANY = os.environ.get('COMPANY_NAME', 'Royal Imports')
|
| 37 |
+
|
| 38 |
+
# --- Sender identity (statements go out AS Royal Imports) ---
|
| 39 |
+
# Odoo routes outbound mail to the matching ir.mail_server by from_filter, and the Office 365
|
| 40 |
+
# relay only accepts sends as its authenticated address. The "Office 365 - Royal" server (id 2)
|
| 41 |
+
# authenticates as contact@royalimports.com with from_filter='contact@royalimports.com' β so the
|
| 42 |
+
# From MUST be that address for delivery to succeed. Friendly display name; replies routed to AR.
|
| 43 |
+
SENDER_NAME = os.environ.get('SENDER_NAME', 'Royal Imports Accounts Receivable')
|
| 44 |
+
SENDER_EMAIL = os.environ.get('SENDER_EMAIL', 'contact@royalimports.com')
|
| 45 |
+
REPLY_TO = os.environ.get('REPLY_TO', 'accounting@royalimports.com')
|
| 46 |
+
ROYAL_MAIL_SERVER_ID = int(os.environ.get('ROYAL_MAIL_SERVER_ID', '2'))
|
| 47 |
+
ROYAL_AUTHOR_ID = int(os.environ.get('ROYAL_AUTHOR_ID', '8978')) # "Royal Imports" partner
|
| 48 |
+
|
| 49 |
+
SENDER_DISPLAY = f'"{SENDER_NAME}" <{SENDER_EMAIL}>'
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
class WriteBlocked(RuntimeError):
|
| 53 |
+
pass
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
class SafeModeBlocked(RuntimeError):
|
| 57 |
+
pass
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
# --- Testing guardrail -------------------------------------------------------
|
| 61 |
+
# While SAFE_MODE is on, NO email can be queued to any address outside the allow-list β enforced
|
| 62 |
+
# here in the data layer so the UI cannot bypass it. Default: ON. To go live for real customers,
|
| 63 |
+
# set the Space secret SAFE_MODE=0.
|
| 64 |
+
SAFE_MODE = os.environ.get('SAFE_MODE', '1').strip().lower() not in ('0', 'false', 'no', '')
|
| 65 |
+
SAFE_RECIPIENTS = {a.strip().lower() for a in
|
| 66 |
+
os.environ.get('SAFE_RECIPIENTS', 'farhan@teamroyalimports.com').split(',')
|
| 67 |
+
if a.strip()}
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def safe_recipient_ok(addr):
|
| 71 |
+
return (not SAFE_MODE) or (str(addr or '').strip().lower() in SAFE_RECIPIENTS)
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
class Odoo:
|
| 75 |
+
"""Narrow client: read anything, write ONLY the whitelisted mail.mail create."""
|
| 76 |
+
|
| 77 |
+
def __init__(self):
|
| 78 |
+
self.url = os.environ.get('ODOO_URL', '').rstrip('/')
|
| 79 |
+
self.db = os.environ.get('ODOO_DB', '')
|
| 80 |
+
self.user = os.environ.get('ODOO_USER', '')
|
| 81 |
+
self.key = os.environ.get('ODOO_API_KEY', '')
|
| 82 |
+
missing = [k for k, v in [('ODOO_URL', self.url), ('ODOO_DB', self.db),
|
| 83 |
+
('ODOO_USER', self.user), ('ODOO_API_KEY', self.key)] if not v]
|
| 84 |
+
if missing:
|
| 85 |
+
raise RuntimeError(f"Missing secrets: {', '.join(missing)}")
|
| 86 |
+
common = xmlrpc.client.ServerProxy(f'{self.url}/xmlrpc/2/common')
|
| 87 |
+
self.uid = common.authenticate(self.db, self.user, self.key, {})
|
| 88 |
+
if not self.uid:
|
| 89 |
+
raise RuntimeError('Odoo authentication failed')
|
| 90 |
+
self.models = xmlrpc.client.ServerProxy(f'{self.url}/xmlrpc/2/object')
|
| 91 |
+
|
| 92 |
+
def _exec(self, model, method, args, kwargs=None):
|
| 93 |
+
mutating = method in ('write', 'create', 'unlink', 'copy') or \
|
| 94 |
+
any(method.startswith(p) for p in ('action_', 'button_', 'do_', 'send_',
|
| 95 |
+
'set_', 'update_', 'process_'))
|
| 96 |
+
if mutating and (model, method) not in WRITE_WHITELIST:
|
| 97 |
+
raise WriteBlocked(f'{method} on {model} is not allowed from this app')
|
| 98 |
+
return self.models.execute_kw(self.db, self.uid, self.key,
|
| 99 |
+
model, method, args, kwargs or {})
|
| 100 |
+
|
| 101 |
+
def search_read(self, model, domain=None, fields=None, limit=None, order=None):
|
| 102 |
+
kw = {'fields': fields or []}
|
| 103 |
+
if limit is not None:
|
| 104 |
+
kw['limit'] = limit
|
| 105 |
+
if order:
|
| 106 |
+
kw['order'] = order
|
| 107 |
+
return self._exec(model, 'search_read', [domain or []], kw)
|
| 108 |
+
|
| 109 |
+
def queue_mail(self, payload):
|
| 110 |
+
"""The single allowed write: queue an outbound email."""
|
| 111 |
+
return self._exec('mail.mail', 'create', [payload])
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
# --------------------------------------------------------------- data builders
|
| 115 |
+
def load_collection_list(odoo):
|
| 116 |
+
"""The saved follow-up filter (Reminders=Automatic, Receivable>$1), GIFTWARE excluded,
|
| 117 |
+
with priority tiers."""
|
| 118 |
+
partners = odoo.search_read('res.partner', DOMAIN,
|
| 119 |
+
['name', 'credit', 'total_overdue', 'followup_status',
|
| 120 |
+
'followup_next_action_date', 'followup_responsible_id',
|
| 121 |
+
'email', 'phone', 'mobile'])
|
| 122 |
+
partners = [p for p in partners if (p['name'] or '').strip().upper() not in EXCLUDE_NAMES]
|
| 123 |
+
pids = [p['id'] for p in partners]
|
| 124 |
+
|
| 125 |
+
docs = odoo.search_read('account.move',
|
| 126 |
+
[('move_type', 'in', ['out_invoice', 'out_refund']), ('state', '=', 'posted'),
|
| 127 |
+
('payment_state', 'in', ['not_paid', 'partial']), ('partner_id', 'in', pids)],
|
| 128 |
+
['name', 'partner_id', 'move_type', 'invoice_date', 'invoice_date_due',
|
| 129 |
+
'amount_total', 'amount_residual_signed'])
|
| 130 |
+
|
| 131 |
+
today = dt.date.today()
|
| 132 |
+
by_partner = {}
|
| 133 |
+
for d in docs:
|
| 134 |
+
pid = d['partner_id'][0]
|
| 135 |
+
due = d.get('invoice_date_due')
|
| 136 |
+
try:
|
| 137 |
+
days = (today - dt.date.fromisoformat(due)).days if due else 0
|
| 138 |
+
except Exception:
|
| 139 |
+
days = 0
|
| 140 |
+
d['days_overdue'] = max(days, 0)
|
| 141 |
+
d['open_signed'] = d['amount_residual_signed']
|
| 142 |
+
by_partner.setdefault(pid, []).append(d)
|
| 143 |
+
|
| 144 |
+
rows = []
|
| 145 |
+
for p in partners:
|
| 146 |
+
pid = p['id']
|
| 147 |
+
odoo_overdue = float(p.get('total_overdue') or 0)
|
| 148 |
+
docs_p = sorted(by_partner.get(pid, []), key=lambda x: x.get('invoice_date_due') or '')
|
| 149 |
+
inv_overdue = sum(d['open_signed'] for d in docs_p if d['days_overdue'] > 0)
|
| 150 |
+
oldest = max((d['days_overdue'] for d in docs_p), default=0)
|
| 151 |
+
gap = inv_overdue - odoo_overdue
|
| 152 |
+
reconcile = abs(gap) > 50
|
| 153 |
+
|
| 154 |
+
if odoo_overdue >= 5000 or (odoo_overdue > 0 and oldest > 90):
|
| 155 |
+
tier = 'A-Urgent'
|
| 156 |
+
elif odoo_overdue >= 1000 or (odoo_overdue > 0 and oldest > 30):
|
| 157 |
+
tier = 'B-Active'
|
| 158 |
+
elif odoo_overdue > 0:
|
| 159 |
+
tier = 'C-Light'
|
| 160 |
+
else:
|
| 161 |
+
tier = 'Monitor'
|
| 162 |
+
|
| 163 |
+
rows.append({
|
| 164 |
+
'partner_id': pid,
|
| 165 |
+
'Customer': p['name'],
|
| 166 |
+
'Tier': tier,
|
| 167 |
+
'Overdue': odoo_overdue,
|
| 168 |
+
'Receivable': float(p.get('credit') or 0),
|
| 169 |
+
'Oldest (days)': oldest,
|
| 170 |
+
'Open Docs': len(docs_p),
|
| 171 |
+
'Email': p.get('email') or '',
|
| 172 |
+
'Phone': p.get('phone') or p.get('mobile') or '',
|
| 173 |
+
'Status': (p.get('followup_status') or '').replace('_', ' '),
|
| 174 |
+
'Reconcile?': 'YES' if reconcile else '',
|
| 175 |
+
'_docs': docs_p,
|
| 176 |
+
})
|
| 177 |
+
tier_rank = {'A-Urgent': 0, 'B-Active': 1, 'C-Light': 2, 'Monitor': 3}
|
| 178 |
+
rows.sort(key=lambda r: (tier_rank[r['Tier']], -r['Overdue']))
|
| 179 |
+
return rows
|
| 180 |
+
|
| 181 |
+
|
| 182 |
+
# --------------------------------------------------------------- statement email
|
| 183 |
+
# β STANDING RULE 2, AND THIS IS THE ONE STRING IN THE PRODUCT THAT LEAVES THE BUILDING.
|
| 184 |
+
# Every other finding `web_prose` reports is copy on a screen somebody here opens; this is the
|
| 185 |
+
# SUBJECT LINE of mail queued to a real debtor, over the single sanctioned Odoo write. It read
|
| 186 |
+
# 'Statement of Account (em dash) {company} (em dash) {month}' until W36-T42.
|
| 187 |
+
DEFAULT_SUBJECT = 'Statement of Account from {company}, {month}'
|
| 188 |
+
DEFAULT_INTRO = (
|
| 189 |
+
'Dear {customer},<br><br>'
|
| 190 |
+
'Please find below your current statement of account with {company}. '
|
| 191 |
+
'According to our records, the following invoices remain open:'
|
| 192 |
+
)
|
| 193 |
+
DEFAULT_FOOTER = (
|
| 194 |
+
'If you have already sent payment, please disregard this notice, and thank you. '
|
| 195 |
+
'For any questions about an invoice, simply reply to this email.<br><br>'
|
| 196 |
+
'Thank you for your business,<br>{company}<br>Accounts Receivable'
|
| 197 |
+
)
|
| 198 |
+
|
| 199 |
+
|
| 200 |
+
def render_statement_html(row, intro_tpl=DEFAULT_INTRO, footer_tpl=DEFAULT_FOOTER):
|
| 201 |
+
month = dt.date.today().strftime('%B %Y')
|
| 202 |
+
intro = intro_tpl.format(customer=row['Customer'], company=COMPANY, month=month)
|
| 203 |
+
footer = footer_tpl.format(customer=row['Customer'], company=COMPANY, month=month)
|
| 204 |
+
|
| 205 |
+
lines = []
|
| 206 |
+
total_open = 0.0
|
| 207 |
+
for d in row['_docs']:
|
| 208 |
+
kind = 'Credit Note' if d['move_type'] == 'out_refund' else 'Invoice'
|
| 209 |
+
amt = d['open_signed']
|
| 210 |
+
total_open += amt
|
| 211 |
+
overdue_txt = f"{d['days_overdue']}d overdue" if d['days_overdue'] > 0 else 'current'
|
| 212 |
+
color = '#C0392B' if d['days_overdue'] > 0 else '#1F4E78'
|
| 213 |
+
lines.append(
|
| 214 |
+
f"<tr><td style='padding:6px 10px;border-bottom:1px solid #e3e8ef'>{d['name']} <span style='color:#888'>({kind})</span></td>"
|
| 215 |
+
f"<td style='padding:6px 10px;border-bottom:1px solid #e3e8ef'>{d.get('invoice_date') or ''}</td>"
|
| 216 |
+
f"<td style='padding:6px 10px;border-bottom:1px solid #e3e8ef'>{d.get('invoice_date_due') or ''}</td>"
|
| 217 |
+
f"<td style='padding:6px 10px;border-bottom:1px solid #e3e8ef;color:{color}'>{overdue_txt}</td>"
|
| 218 |
+
f"<td style='padding:6px 10px;border-bottom:1px solid #e3e8ef;text-align:right'>${amt:,.2f}</td></tr>")
|
| 219 |
+
|
| 220 |
+
table = (
|
| 221 |
+
"<table style='border-collapse:collapse;font-size:14px;margin:14px 0'>"
|
| 222 |
+
"<tr style='background:#1F4E78;color:#fff'>"
|
| 223 |
+
"<th style='padding:7px 10px;text-align:left'>Document</th>"
|
| 224 |
+
"<th style='padding:7px 10px;text-align:left'>Date</th>"
|
| 225 |
+
"<th style='padding:7px 10px;text-align:left'>Due</th>"
|
| 226 |
+
"<th style='padding:7px 10px;text-align:left'>Status</th>"
|
| 227 |
+
"<th style='padding:7px 10px;text-align:right'>Open Balance</th></tr>"
|
| 228 |
+
+ ''.join(lines) +
|
| 229 |
+
f"<tr><td colspan='4' style='padding:8px 10px;font-weight:bold;text-align:right'>Total open</td>"
|
| 230 |
+
f"<td style='padding:8px 10px;font-weight:bold;text-align:right'>${total_open:,.2f}</td></tr>"
|
| 231 |
+
f"<tr><td colspan='4' style='padding:2px 10px;font-weight:bold;text-align:right;color:#C0392B'>Of which overdue</td>"
|
| 232 |
+
f"<td style='padding:2px 10px;font-weight:bold;text-align:right;color:#C0392B'>${row['Overdue']:,.2f}</td></tr>"
|
| 233 |
+
"</table>")
|
| 234 |
+
|
| 235 |
+
return (f"<div style='font-family:Calibri,Arial,sans-serif;color:#1a1a1a;font-size:14px'>"
|
| 236 |
+
f"{intro}{table}{footer}</div>")
|
| 237 |
+
|
| 238 |
+
|
| 239 |
+
def queue_statement(odoo, row, subject_tpl=DEFAULT_SUBJECT,
|
| 240 |
+
intro_tpl=DEFAULT_INTRO, footer_tpl=DEFAULT_FOOTER,
|
| 241 |
+
override_to=None):
|
| 242 |
+
"""Queue one statement email in Odoo. Returns mail.mail id.
|
| 243 |
+
override_to: send to a different address (used by the test-send button)."""
|
| 244 |
+
to = override_to or row['Email']
|
| 245 |
+
if not to:
|
| 246 |
+
raise ValueError(f"{row['Customer']} has no email address")
|
| 247 |
+
# Hard guardrail β refuse any recipient outside the allow-list while SAFE_MODE is on.
|
| 248 |
+
if not safe_recipient_ok(to):
|
| 249 |
+
raise SafeModeBlocked(
|
| 250 |
+
f"Guardrail ON: refusing to email {to}. Only {', '.join(sorted(SAFE_RECIPIENTS))} "
|
| 251 |
+
f"is allowed right now. (Set SAFE_MODE=0 to send to real customers.)")
|
| 252 |
+
month = dt.date.today().strftime('%B %Y')
|
| 253 |
+
subject = subject_tpl.format(customer=row['Customer'], company=COMPANY, month=month)
|
| 254 |
+
payload = {
|
| 255 |
+
'subject': subject,
|
| 256 |
+
'body_html': render_statement_html(row, intro_tpl, footer_tpl),
|
| 257 |
+
'email_to': to,
|
| 258 |
+
'email_from': SENDER_DISPLAY, # From: Royal Imports
|
| 259 |
+
'reply_to': REPLY_TO, # replies -> AR
|
| 260 |
+
'mail_server_id': ROYAL_MAIL_SERVER_ID, # force Royal O365 relay
|
| 261 |
+
'author_id': ROYAL_AUTHOR_ID, # clean attribution in chatter
|
| 262 |
+
'state': 'outgoing',
|
| 263 |
+
'auto_delete': False,
|
| 264 |
+
'model': 'res.partner',
|
| 265 |
+
'res_id': row['partner_id'],
|
| 266 |
+
}
|
| 267 |
+
return odoo.queue_mail(payload)
|
platform/modules/product_data.py
CHANGED
|
@@ -1,804 +1,804 @@
|
|
| 1 |
-
"""modules/product_data.py β the PRODUCT table's pool (wave 15 item 9/10, contract C-TOPIC).
|
| 2 |
-
|
| 3 |
-
The second object on the table-page factory: same grid, same engine, same permission wall β the
|
| 4 |
-
only difference is the field schema and the identity. `modules/customer_data.pool()` is the
|
| 5 |
-
template and this deliberately mirrors its signature and its row shape.
|
| 6 |
-
|
| 7 |
-
β THREE DECISIONS THIS FILE HAD TO MAKE, EACH RECORDED BECAUSE A LATER READER WILL WONDER.
|
| 8 |
-
|
| 9 |
-
1. **THE IDENTITY IS A SKU CODE, WHICH IS A STRING, AND THE GRID WANTS AN INTEGER `pid`.**
|
| 10 |
-
Cohort membership, `allowed_pids`, the measure channel and `rows_from_pool` all key on an
|
| 11 |
-
integer. So each row carries BOTH: `pid` (a stable CRC32 of the code, so the same SKU gets the
|
| 12 |
-
same id on every pull and across processes β never an enumeration index, which would reshuffle
|
| 13 |
-
whenever the catalogue changes) and `code`, the real business key, as a visible column.
|
| 14 |
-
`_assert_no_pid_collision` fails the BUILD rather than the read: two SKUs sharing a pid would
|
| 15 |
-
silently merge in every downstream set operation, and a loud build failure is the only version
|
| 16 |
-
of that anyone would notice.
|
| 17 |
-
|
| 18 |
-
2. **THE SCOPE RULE β inventory columns are CONSOLIDATED and are therefore OMITTED for a
|
| 19 |
-
BU-scoped caller.** `products.directory(t, team_id)` is brand-shaped; `inventory.sku_inventory`
|
| 20 |
-
is explicitly NOT (its own docstring: "on-hand stock is one physical warehouse, not
|
| 21 |
-
brand-tagged"). Joining them for a Fisch-only user would put BU-shaped revenue beside
|
| 22 |
-
company-wide stock IN THE SAME ROW β the mixed-scope value defect wave 15's amendment 3 exists
|
| 23 |
-
for, arriving through a different door. The honest options were "omit" or "label the columns
|
| 24 |
-
company-wide"; omit is the fail-closed one, and a column that is absent asks a question,
|
| 25 |
-
whereas a column that is silently company-wide answers one wrongly.
|
| 26 |
-
|
| 27 |
-
3. **WHAT IS NOT HERE, NAMED RATHER THAN QUIETLY MISSING.** R4 lists ~20 fields. Vendor and
|
| 28 |
-
COUNTRY are not in Odoo at all β they live in the inventory WORKBOOK
|
| 29 |
-
([[odoo-vendor-country-origin]]) and need a loader this module deliberately does not invent.
|
| 30 |
-
Margin % comes from `modules/pricing.table`, which is a heavier build (channel-rate cost
|
| 31 |
-
allocation) and is left for the wave that needs it. `validate()` reconciles what SHIPS; it
|
| 32 |
-
does not pretend to cover columns that are absent.
|
| 33 |
-
|
| 34 |
-
4. **THE POOL IS CATALOGUE-FIRST, WITH REVENUE LEFT-JOINED** (wave 29, owner item 22 / R12,
|
| 35 |
-
2026-08-11). It was `for r in products.directory(...)` β and `directory()`'s row set IS the
|
| 36 |
-
union of two revenue read-groups, so **a SKU that never sold could not exist** and the grid
|
| 37 |
-
showed **2,717** of **5,875** active products. Three things that will be re-derived otherwise:
|
| 38 |
-
|
| 39 |
-
Β· β THE CAUSE IS A JOIN, NOT A LIMIT. No row cap exists on this path. Removing the date
|
| 40 |
-
window from `directory()` would ALSO be wrong twice: it breaks four SKU-health metrics
|
| 41 |
-
that legitimately want a sales window, and it lands at 3,327 (all-time-sold), because
|
| 42 |
-
~2,550 active SKUs have never sold in wholesale scope at all.
|
| 43 |
-
Β· β REVENUE ON A NEVER-SOLD ROW IS **BLANK, NOT $0** (R12). A row that appears in the
|
| 44 |
-
revenue universe carries measured numbers INCLUDING a real 0.0 β it sold last year and
|
| 45 |
-
not this one, and that zero is a measurement. A row that appears in NO revenue read
|
| 46 |
-
carries `None`, which the wire keeps (`aios_grid._round` passes None through) and the
|
| 47 |
-
client renders as an empty cell. Blank is an admission; zero is a measurement.
|
| 48 |
-
Β· β A BU-SCOPED CALLER NOW SEES THE WHOLE CATALOGUE, with ITS OWN revenue and blanks where
|
| 49 |
-
that BU never sold. That is not decision 2's mixed-scope defect: the catalogue is the ROW
|
| 50 |
-
UNIVERSE, not a company-wide VALUE sitting beside a BU-shaped one. `product.product`
|
| 51 |
-
carries no team, so a catalogue cannot be BU-shaped at all β which is exactly why the two
|
| 52 |
-
sources are JOINED rather than merged.
|
| 53 |
-
"""
|
| 54 |
-
import json
|
| 55 |
-
import zlib
|
| 56 |
-
from pathlib import Path
|
| 57 |
-
|
| 58 |
-
import core.odoo as O
|
| 59 |
-
import core.periods as P
|
| 60 |
-
import core.shared_overlay as shared_overlay
|
| 61 |
-
import core.table_store as table_store
|
| 62 |
-
import modules.products as products
|
| 63 |
-
|
| 64 |
-
#: POOL-ROW KEYS that exist ONLY on a consolidated pull. See decision 2.
|
| 65 |
-
#: `cover_gap_d` / `buy_now` (wave 17) join it because both are computed FROM `dos`, and a buy
|
| 66 |
-
#: signal built on a stock number the caller cannot see would be a recommendation nobody could
|
| 67 |
-
#: check ([[no-unverifiable-aggregates]]).
|
| 68 |
-
#:
|
| 69 |
-
#: β ROW KEYS, NOT FIELDS, and the distinction started mattering on 2026-08-03. Two readers:
|
| 70 |
-
#: `verify_perm_scope` asserts the shape of a POOL ROW against this, and `routes_products`
|
| 71 |
-
#: narrows the FIELD list with it. Since owner item 5 `buy_now` is a row key with no Field β
|
| 72 |
-
#: computed as `validate()`'s oracle, projected onto no wire β so it belongs here for the first
|
| 73 |
-
#: reader and is inert for the second. Removing it would stop the scope gate proving that a
|
| 74 |
-
#: BU-scoped pull withholds it.
|
| 75 |
-
#:
|
| 76 |
-
#: ββ OWNER RULING 2026-08-11 β THIS SET IS NOW EMPTY, AND THAT IS THE POINT. Verbatim:
|
| 77 |
-
#: *"Just scope any inventory with Sales from Fisch, leave the rest."* Withholding the whole
|
| 78 |
-
#: inventory block from a BU-scoped reader meant a Fisch salesperson could not see whether
|
| 79 |
-
#: anything was IN STOCK, which is the first question they ask. The ruling splits the block by
|
| 80 |
-
#: what a business unit can actually shape:
|
| 81 |
-
#:
|
| 82 |
-
#: * `on_hand` / `unit_cost` / `inv_value` β ONE physical warehouse, no Fisch shelf and no
|
| 83 |
-
#: Royal shelf. Served UNSCOPED to everybody, identical in both pulls. ("leave the rest")
|
| 84 |
-
#: * `qty_ltm` / `dos` / `stock_bucket` / `cover_gap_d` / `buy_now` β all SALES-derived, so a
|
| 85 |
-
#: BU-scoped pull recomputes them from THAT unit's LTM units. ("scope any inventory with
|
| 86 |
-
#: Sales from Fisch")
|
| 87 |
-
#:
|
| 88 |
-
#: This deliberately relaxes decision 2's "never a company-wide value beside a BU-shaped one":
|
| 89 |
-
#: stock is not a value a BU can own, and a column that is absent for a Fisch reader asks a
|
| 90 |
-
#: question they cannot answer anywhere else in the product. `verify_perm_scope` no longer
|
| 91 |
-
#: asserts the columns are WITHHELD β it asserts the split above, which is a stronger claim and
|
| 92 |
-
#: cannot pass on an empty scoped payload (the shape the old rule and an outage share).
|
| 93 |
-
CONSOLIDATED_ONLY = ()
|
| 94 |
-
|
| 95 |
-
#: β POOL-ROW KEYS THAT DELIBERATELY HAVE NO FIELD β they must never reach a browser.
|
| 96 |
-
#:
|
| 97 |
-
#: `rows_from_pool` projects strictly through the field contract, so "no Field" already means
|
| 98 |
-
#: "no cell on the wire". This tuple makes that a CHECKED fact rather than a consequence nobody
|
| 99 |
-
#: is watching: `verify_perm_scope` asserts every member is absent from BOTH product contracts,
|
| 100 |
-
#: and that every OTHER `CONSOLIDATED_ONLY` key is present in the consolidated one.
|
| 101 |
-
#:
|
| 102 |
-
#: Written because the two tuples silently diverged the moment owner item 5 retired `buy_now`'s
|
| 103 |
-
#: Field while keeping its computation, and the gate β which was asserting a FIELD property from
|
| 104 |
-
#: a ROW-key list β went red with no way to tell a deliberate divergence from a dropped column.
|
| 105 |
-
UNSHIPPED_ROW_KEYS = ("buy_now",)
|
| 106 |
-
|
| 107 |
-
#: Wave 17 (owner item 13, ruling R3) β the SUPPLIER MASTER, from the curated mastersheet map
|
| 108 |
-
#: `procurement_suppliers.json` (3,963 SKUs; supplier on 3,624, lead time on 3,563). NOT Odoo:
|
| 109 |
-
#: the owner confirmed this data came from the Inventory System Mastersheet, which is why
|
| 110 |
-
#: `modules/product_data`'s header said Vendor/Country were "NOT HERE, named rather than
|
| 111 |
-
#: quietly missing" and needed "a loader this module deliberately does not invent". This is
|
| 112 |
-
#: that loader.
|
| 113 |
-
#:
|
| 114 |
-
#: β THESE ARE CONTRACT COLUMNS, NOT USER-CREATED FIELDS, AND THE REASON IS A PERMISSION FACT.
|
| 115 |
-
#: A user-created field and its values live in the PER-USER strata (`table_store.workspace`
|
| 116 |
-
#: reads `store.get(key)[username]`); only VIEWS are shared. So a shared "Buy list" view that
|
| 117 |
-
#: filtered on a user-created column would, for every OTHER account, name a column that does not
|
| 118 |
-
#: exist β and an unknown column is an INACTIVE condition in the tri-state engine, which IGNORES
|
| 119 |
-
#: it and therefore WIDENS. The buy list would silently show the whole catalogue to everyone but
|
| 120 |
-
#: its author. Contract columns are identical for every reader, so the view means one thing.
|
| 121 |
-
#:
|
| 122 |
-
#: ββ 2026-08-12 (W30-T36) β THE OWNER'S ASK IS NOW DELIVERED, AND THE PARAGRAPH ABOVE IS WHY IT
|
| 123 |
-
#: TOOK THREE WAVES. Owner: *"turn this Excel sheet into a User created Field that we can edit."*
|
| 124 |
-
#: It was parked because a per-user column silently WIDENS a shared view β not because editing was
|
| 125 |
-
#: hard. Wave 29's `core/shared_overlay.py` (whose header quotes this very comment) is the stratum
|
| 126 |
-
#: that removes the objection: **one value per (row, column) for the whole tenant**, so the column
|
| 127 |
-
#: still means ONE thing to every reader and a shared view still filters honestly.
|
| 128 |
-
#: β The four columns below are now `source: "overlay"` + `shared: true` in the canonical
|
| 129 |
-
#: contract, their values live in `<TABLE_KEY>__shared`, and the master map is what SEEDS an
|
| 130 |
-
#: unedited cell rather than what freezes it. See `SHARED_KEYS` and `_ProductTableStore`.
|
| 131 |
-
_SUPPLIER_MAP_PATH = Path(__file__).resolve().parent.parent / "procurement_suppliers.json"
|
| 132 |
-
_SUPPLIER_CACHE = {}
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
def supplier_master():
|
| 136 |
-
"""`{code: {supplier, lead_days, origin_country, first_cost}}`, read once per process.
|
| 137 |
-
|
| 138 |
-
Degrades to `{}` when the file is unreadable, matching `_inventory_by_code`: a product table
|
| 139 |
-
that will not render because a master map is missing is a worse failure than one with blank
|
| 140 |
-
supplier columns.
|
| 141 |
-
"""
|
| 142 |
-
if _SUPPLIER_CACHE:
|
| 143 |
-
return _SUPPLIER_CACHE
|
| 144 |
-
try:
|
| 145 |
-
raw = json.loads(_SUPPLIER_MAP_PATH.read_text(encoding="utf-8"))
|
| 146 |
-
except Exception:
|
| 147 |
-
return {}
|
| 148 |
-
for code, meta in (raw or {}).items():
|
| 149 |
-
if not isinstance(meta, dict):
|
| 150 |
-
continue
|
| 151 |
-
lead = meta.get("lead")
|
| 152 |
-
_SUPPLIER_CACHE[str(code)] = {
|
| 153 |
-
"supplier": (meta.get("vendor") or "") or None,
|
| 154 |
-
"lead_days": int(lead) if isinstance(lead, (int, float)) else None,
|
| 155 |
-
"origin_country": (meta.get("country") or "") or None,
|
| 156 |
-
"first_cost": meta.get("first_cost"),
|
| 157 |
-
}
|
| 158 |
-
return _SUPPLIER_CACHE
|
| 159 |
-
|
| 160 |
-
#: Wave 16 C-TOPIC β the PRODUCT table's OWN workspace bucket. β Never the customer one:
|
| 161 |
-
#: product pids are CRC32 hashes of SKU codes and customer pids are Odoo partner ids, so in a
|
| 162 |
-
#: SHARED overlay bucket a hash collision would silently write a product note onto somebody's
|
| 163 |
-
#: customer (or the reverse). Separate store keys make that structurally impossible, which is
|
| 164 |
-
#: the whole reason the table-page factory exists ("a new table object gets its own
|
| 165 |
-
#: table_store.make('<its>_table_workspace')").
|
| 166 |
-
TABLE_KEY = 'product_table_workspace'
|
| 167 |
-
|
| 168 |
-
_GRID_FIELDS_PATH = Path(__file__).resolve().parent.parent / 'aios_grid_fields.json'
|
| 169 |
-
_SHARED_KEYS = None
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
def SHARED_KEYS():
|
| 173 |
-
"""The product columns whose values are TENANT-WIDE β derived from the canonical contract's
|
| 174 |
-
own `shared: true`, never typed out a second time.
|
| 175 |
-
|
| 176 |
-
β IT DELIBERATELY DOES NOT SWALLOW A READ FAILURE. Degrading to `()` would send a shared
|
| 177 |
-
write into the per-user stratum with nothing going wrong anywhere β the widening defect
|
| 178 |
-
reappearing silently, which is the one outcome this whole mechanism exists to prevent. If the
|
| 179 |
-
canonical contract is unreadable the product grid cannot render at all (`pd_fields` parses the
|
| 180 |
-
same file with no guard), so a raise here costs nothing that was still working.
|
| 181 |
-
"""
|
| 182 |
-
global _SHARED_KEYS
|
| 183 |
-
if _SHARED_KEYS is None:
|
| 184 |
-
doc = json.loads(_GRID_FIELDS_PATH.read_text(encoding='utf-8'))
|
| 185 |
-
_SHARED_KEYS = tuple(f['key'] for f in (doc.get('product_data') or {}).get('fields') or []
|
| 186 |
-
if f.get('shared'))
|
| 187 |
-
return _SHARED_KEYS
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
class _ProductTableStore(table_store.TableStore):
|
| 191 |
-
"""The product workspace, with the SHARED columns routed to the tenant-wide stratum.
|
| 192 |
-
|
| 193 |
-
ββ THIS SUBCLASS IS THE WHOLE OF W30-T36's WRITE PATH, AND THE REASON IT LIVES HERE RATHER
|
| 194 |
-
THAN AT A ROUTE IS THAT **THE BROWSER NEVER CALLS `PATCH /products/{pid}`** β measured, zero
|
| 195 |
-
call sites in `aios-web/web/src`. A cell edit travels `POST /grid/events` β `grid_events.
|
| 196 |
-
handle_one` β `_tops(ctx).patch_overlay(...)`, and `_tops` returns `ctx.table`, which
|
| 197 |
-
`routes_grid._ctx` sets to `TABLE_OPS` for the product scope. So this object IS the seam both
|
| 198 |
-
doors pass through; intercepting at either route would have left the other one writing a
|
| 199 |
-
per-user value that only its author could see.
|
| 200 |
-
|
| 201 |
-
β `st=self.st`, NEVER the module default. The shared stratum must resolve to the SAME store
|
| 202 |
-
handle as the per-user one it sits beside β `_tops`' own comment explains that a split, where
|
| 203 |
-
one side is tenant-scoped and the other is not, is worse than a stated residency error because
|
| 204 |
-
a user's value would vanish the moment they saved it. Reading `self.st` means both strata move
|
| 205 |
-
together the day that singleton gains a tenant handle.
|
| 206 |
-
"""
|
| 207 |
-
|
| 208 |
-
def patch_overlay(self, username, pid, updates):
|
| 209 |
-
clean = dict(updates or {})
|
| 210 |
-
if not clean:
|
| 211 |
-
return
|
| 212 |
-
keys = set(SHARED_KEYS())
|
| 213 |
-
shared = {k: v for k, v in clean.items() if k in keys}
|
| 214 |
-
personal = {k: v for k, v in clean.items() if k not in keys}
|
| 215 |
-
if shared:
|
| 216 |
-
shared_overlay.put_cells(TABLE_KEY, pid, shared, st=self.st)
|
| 217 |
-
if personal:
|
| 218 |
-
super().patch_overlay(username, pid, personal)
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
TABLE_OPS = _ProductTableStore(TABLE_KEY)
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
def shared_cells(pids, st=None):
|
| 225 |
-
"""`{"<pid>": {key: value}}` for the SHARED columns of the rows named by `pids`.
|
| 226 |
-
|
| 227 |
-
β `pids` is required and positional all the way down β `shared_overlay.cells` refuses to serve
|
| 228 |
-
"everything" by design, and the caller here always holds an already-scoped pool.
|
| 229 |
-
"""
|
| 230 |
-
return shared_overlay.cells(TABLE_KEY, pids, st=st if st is not None else TABLE_OPS.st)
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
def sku_pid(code):
|
| 234 |
-
"""A stable integer id for a SKU code. CRC32, masked to 31 bits so it is always positive and
|
| 235 |
-
always JSON-safe. Stable across processes and pulls, which an enumeration index is not."""
|
| 236 |
-
return zlib.crc32(str(code).encode("utf-8")) & 0x7FFFFFFF
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
def _assert_no_pid_collision(rows):
|
| 240 |
-
"""Two SKUs sharing a pid would MERGE in every set operation downstream β cohort membership,
|
| 241 |
-
allowed_pids, the measure channel β and nothing would report it. Fail the build instead."""
|
| 242 |
-
seen = {}
|
| 243 |
-
for r in rows:
|
| 244 |
-
prior = seen.get(r["pid"])
|
| 245 |
-
if prior is not None and prior != r["code"]:
|
| 246 |
-
raise ValueError(
|
| 247 |
-
f"product_data: pid collision β {prior!r} and {r['code']!r} both hash to "
|
| 248 |
-
f"{r['pid']}. Downstream set operations would merge them silently; widen the id "
|
| 249 |
-
f"before shipping this catalogue.")
|
| 250 |
-
seen[r["pid"]] = r["code"]
|
| 251 |
-
|
| 252 |
-
|
| 253 |
-
def _inventory_by_code(t):
|
| 254 |
-
"""`{code: {...}}` from the inventory module, or `{}` if it cannot be read.
|
| 255 |
-
|
| 256 |
-
Degrades to empty rather than raising, matching `customer_data._pool_build`'s treatment of
|
| 257 |
-
its own slow families: a product table that will not render because inventory is momentarily
|
| 258 |
-
unreachable is a worse failure than one with blank stock columns.
|
| 259 |
-
"""
|
| 260 |
-
try:
|
| 261 |
-
import modules.inventory as inventory
|
| 262 |
-
return inventory.sku_inventory(t=t) or {}
|
| 263 |
-
except Exception:
|
| 264 |
-
return {}
|
| 265 |
-
|
| 266 |
-
|
| 267 |
-
def _bu_ltm_share(t, team_id):
|
| 268 |
-
"""`{code: 0.0..1.0}` β this unit's SHARE of the SKU's last-twelve-months units.
|
| 269 |
-
|
| 270 |
-
β OWNER 2026-08-11: *"scope any inventory with Sales from Fisch"*. The velocity half of the
|
| 271 |
-
inventory block has to be re-shaped by a BU fact, and this is that fact.
|
| 272 |
-
|
| 273 |
-
β A SHARE, NOT THE UNIT COUNT ITSELF β AND THE REASON IS A MEASURED IMPOSSIBILITY. My first
|
| 274 |
-
version returned `products._sku_rev(...)['qty']` and used it directly as the scoped `qty_ltm`,
|
| 275 |
-
leaving the consolidated column on `inventory.sku_inventory`'s own figure. Two readers of one
|
| 276 |
-
question ([[one-question-two-normalizers]]): they disagree, and on 5 SKUs of 5,871 the live
|
| 277 |
-
check found **Fisch's LTM units EXCEEDING the company's** β a subset larger than its superset,
|
| 278 |
-
which no reader could explain and no reconciliation could survive.
|
| 279 |
-
|
| 280 |
-
Both halves of the ratio come from the SAME read here, so the share is in [0, 1] by
|
| 281 |
-
construction and the scoped figure can never exceed the consolidated one. It also leaves the
|
| 282 |
-
consolidated column exactly as it was β the Inventory page and the Product grid still agree
|
| 283 |
-
about company-wide units, which is what "leave the rest" asked for.
|
| 284 |
-
|
| 285 |
-
`all_qty == 0` implies `bu_qty == 0` (same reader), so a share of 0 is the honest answer for
|
| 286 |
-
"this unit never sold it": `_bucket` turns that into 'No recent sales', not zero cover.
|
| 287 |
-
|
| 288 |
-
Degrades to `{}` on any failure, matching `_inventory_by_code` β and a missing code then takes
|
| 289 |
-
the `_DEFAULT_SHARE` below rather than a silent 0.
|
| 290 |
-
"""
|
| 291 |
-
try:
|
| 292 |
-
f, to = P.ltm(t)
|
| 293 |
-
allq = {code: (r.get('qty') or 0.0)
|
| 294 |
-
for code, r in (products._sku_rev(f, to, None) or {}).items()}
|
| 295 |
-
buq = {code: (r.get('qty') or 0.0)
|
| 296 |
-
for code, r in (products._sku_rev(f, to, team_id) or {}).items()}
|
| 297 |
-
except Exception:
|
| 298 |
-
return {}
|
| 299 |
-
out = {}
|
| 300 |
-
for code, total in allq.items():
|
| 301 |
-
out[code] = min(1.0, max(0.0, (buq.get(code, 0.0) / total))) if total > 0 else 0.0
|
| 302 |
-
return out
|
| 303 |
-
|
| 304 |
-
|
| 305 |
-
#: What a SKU absent from the LTM sales read is worth to a business unit. ZERO β it did not sell
|
| 306 |
-
#: in anybody's book over the window, so no unit can claim its velocity. Named rather than
|
| 307 |
-
#: inlined so the choice is visible: the alternative (1.0, "assume it is all ours") would print
|
| 308 |
-
#: company-wide cover on a BU grid, which is the mixed-scope defect this whole rule avoids.
|
| 309 |
-
_DEFAULT_SHARE = 0.0
|
| 310 |
-
|
| 311 |
-
|
| 312 |
-
def _rescope_inventory(e, share):
|
| 313 |
-
"""`(qty_ltm, dos, bucket)` recomputed for ONE business unit's sales rate.
|
| 314 |
-
|
| 315 |
-
The formulas are `inventory.sku_inventory`'s, applied to a BU-shaped numerator β NOT a second
|
| 316 |
-
idea of what days-of-supply means. `_bucket` is imported from there for the same reason: two
|
| 317 |
-
copies of a threshold table is how the Product grid and the Inventory page start disagreeing
|
| 318 |
-
about which SKUs are dead.
|
| 319 |
-
|
| 320 |
-
β `on_hand` is whatever the warehouse holds, unscoped β so a Fisch reader's `dos` answers
|
| 321 |
-
"how long does ALL our stock last at Fisch's rate", which is the question a Fisch salesperson
|
| 322 |
-
actually has. It is deliberately NOT a pro-rated share of the shelf: there is no such shelf,
|
| 323 |
-
and inventing one would put a number on screen that no Odoo query could reproduce.
|
| 324 |
-
"""
|
| 325 |
-
on_hand = e.get("on_hand")
|
| 326 |
-
if on_hand is None:
|
| 327 |
-
return None, None, None # no inventory row for this SKU: blank, never zero
|
| 328 |
-
qty = float(e.get("qty_ltm") or 0.0) * float(share or 0.0)
|
| 329 |
-
daily = qty / 365.0
|
| 330 |
-
if daily > 0:
|
| 331 |
-
dos_raw = on_hand / daily
|
| 332 |
-
else:
|
| 333 |
-
dos_raw = float('inf') if on_hand > 0 else 0.0
|
| 334 |
-
try:
|
| 335 |
-
import modules.inventory as inventory
|
| 336 |
-
bucket = inventory._bucket(dos_raw, on_hand, qty)
|
| 337 |
-
except Exception:
|
| 338 |
-
bucket = None
|
| 339 |
-
return qty, (None if dos_raw == float('inf') else round(float(dos_raw), 0)), bucket
|
| 340 |
-
|
| 341 |
-
|
| 342 |
-
def _catalogue_by_code():
|
| 343 |
-
"""`{code: {'product', 'category'}}` β the CATALOGUE universe this pool is built from.
|
| 344 |
-
|
| 345 |
-
β UNLIKE `_inventory_by_code`, THIS ONE RAISES, and the asymmetry is the whole point.
|
| 346 |
-
Inventory degrades to `{}` because a product table with blank stock columns beats one that
|
| 347 |
-
will not render. The CATALOGUE is not a column β it is the ROW SET. A catalogue read that
|
| 348 |
-
failed quietly would drop the grid straight back to the sold-only 2,717 with every gate
|
| 349 |
-
green and nothing on screen saying so, which is the defect this seam exists to end
|
| 350 |
-
([[gate-can-report-green-on-nothing]]). `routes_products._pool_for` already turns the raise
|
| 351 |
-
into a 503 that names the cause, so the loud failure has somewhere honest to land.
|
| 352 |
-
|
| 353 |
-
It exists as a `pd`-level function rather than an inline `products.catalogue()` call for the
|
| 354 |
-
same reason `_inventory_by_code` does: it is the seam `verify_perm_scope`'s section H stubs
|
| 355 |
-
to build a pool without Odoo.
|
| 356 |
-
"""
|
| 357 |
-
return products.catalogue()
|
| 358 |
-
|
| 359 |
-
|
| 360 |
-
def _pricelist_by_code():
|
| 361 |
-
"""`({code: {price_*: price}}, report)` from `products.pricelist_by_code`, or `({}, β¦)`.
|
| 362 |
-
|
| 363 |
-
A SEAM for the same two reasons `_inventory_by_code` is one: it degrades rather than raises
|
| 364 |
-
(these are columns, not the row set), and `verify_perm_scope`'s section H stubs it to build a
|
| 365 |
-
pool without Odoo. β Unstubbed there, section H would reach live Odoo through the back door
|
| 366 |
-
and the whole file would stop being runnable offline.
|
| 367 |
-
"""
|
| 368 |
-
try:
|
| 369 |
-
return products.pricelist_by_code()
|
| 370 |
-
except Exception as e:
|
| 371 |
-
return {}, {"error": f"{type(e).__name__}: {str(e)[:200]}"}
|
| 372 |
-
|
| 373 |
-
|
| 374 |
-
def pool(team_id=None, t=None):
|
| 375 |
-
"""One row per ACTIVE product β the PRODUCT analogue of `customer_data.pool`.
|
| 376 |
-
|
| 377 |
-
CATALOGUE-FIRST, REVENUE LEFT-JOINED (R12 β see decision 4 in the module header). The row set
|
| 378 |
-
is `products.catalogue()`; `products.directory()` supplies the revenue columns for the SKUs
|
| 379 |
-
that sold in its window and contributes NO rows of its own.
|
| 380 |
-
|
| 381 |
-
`team_id` shapes the revenue columns exactly as it does for customers (`products.directory`
|
| 382 |
-
passes it into `_sku_rev`), which is why `core.perm_scope.derive_pool_scope` must keep
|
| 383 |
-
driving it rather than a post-filter deciding the BU. It does NOT shape the row set: a
|
| 384 |
-
catalogue has no team.
|
| 385 |
-
"""
|
| 386 |
-
t = t or P.today()
|
| 387 |
-
consolidated = team_id is None
|
| 388 |
-
# β OWNER 2026-08-11: read inventory on EVERY pull, not just a consolidated one. The stock
|
| 389 |
-
# itself is company-wide; only the sales-derived half is re-scoped, by `_rescope_inventory`.
|
| 390 |
-
inv = _inventory_by_code(t)
|
| 391 |
-
bu_share = {} if consolidated else _bu_ltm_share(t, team_id)
|
| 392 |
-
sup = supplier_master()
|
| 393 |
-
prices, _price_report = _pricelist_by_code()
|
| 394 |
-
cat = _catalogue_by_code()
|
| 395 |
-
# The LEFT side of the join, indexed by the same code key. β A code here that the catalogue
|
| 396 |
-
# does not carry belongs to a product no ACTIVE record claims β archived, and R12 keeps those
|
| 397 |
-
# out. `validate()` asserts that of Odoo rather than assuming it, and reports the revenue
|
| 398 |
-
# that therefore sits outside the grid (MEASURED 2026-08-11: 3 codes, $0.00 YTD / $294.50 LY).
|
| 399 |
-
rev = {r["code"]: r for r in products.directory(t=t, team_id=team_id)}
|
| 400 |
-
|
| 401 |
-
rows = []
|
| 402 |
-
for code, meta in cat.items():
|
| 403 |
-
r = rev.get(code)
|
| 404 |
-
row = {
|
| 405 |
-
"pid": sku_pid(code),
|
| 406 |
-
"code": code,
|
| 407 |
-
# β ONE source for the name and the category, the CATALOGUE β not the sale line's
|
| 408 |
-
# m2o. For a re-SKUed code the line's name can be the ARCHIVED record's; the active
|
| 409 |
-
# record's `display_name` is the current truth, and it is the same string for every
|
| 410 |
-
# SKU that is not re-SKUed. `directory()` derives the category identically.
|
| 411 |
-
"product": meta.get("product") or code,
|
| 412 |
-
"category": meta.get("category") or "(uncategorized)",
|
| 413 |
-
# ββ W33-T43 (R2 / amendment A2) β ODOO'S OWN PRODUCT ID, beside the hashed `pid`.
|
| 414 |
-
# R2 retires `ut_odoo_products` onto this key and keeps every data column it had; this
|
| 415 |
-
# is that column. β NOT DERIVABLE DOWNSTREAM: `pid` is `crc32(default_code)` here,
|
| 416 |
-
# unlike a customer row whose `pid` IS the partner id, so `aios_grid.py` cannot recover
|
| 417 |
-
# it and it has to be carried from `products.catalogue()`.
|
| 418 |
-
# β `None`, never 0, when the catalogue somehow has no id β 0 is a real Odoo id.
|
| 419 |
-
"product_id": meta.get("id"),
|
| 420 |
-
# β R12 β BLANK, NEVER $0, on a SKU no revenue read ever saw. A row that IS in `rev`
|
| 421 |
-
# keeps its measured numbers including a real 0.0 (it sold last year, not this one).
|
| 422 |
-
"rev_ytd": r.get("rev_ytd", 0.0) if r else None,
|
| 423 |
-
"rev_ly": r.get("rev_ly", 0.0) if r else None,
|
| 424 |
-
"yoy_pct": r.get("yoy_pct") if r else None,
|
| 425 |
-
"qty_ytd": r.get("qty_ytd", 0.0) if r else None,
|
| 426 |
-
"orders_ytd": r.get("orders_ytd", 0) if r else None,
|
| 427 |
-
}
|
| 428 |
-
# β WAVE 30 W30-T34 β the PRICELIST stratum, one column per declared list. Catalogue
|
| 429 |
-
# data like the supplier block, so it rides every pull, scoped or not: a price book is
|
| 430 |
-
# not a thing a business unit owns a slice of, and a Fisch reader who cannot see the
|
| 431 |
-
# Fisch price is exactly who this column is for.
|
| 432 |
-
#
|
| 433 |
-
# β `None`, NEVER 0 AND NEVER A FALLBACK, for the 1,883 SKUs no list prices. That is
|
| 434 |
-
# W29-T52's own negative control: the single `3_global` rule computes over `list_price`,
|
| 435 |
-
# which is 1.00 on 5,817 of 5,871 products, so the fallback is not a cheaper answer β
|
| 436 |
-
# it is a wrong one wearing a currency sign.
|
| 437 |
-
p = prices.get(code) or {}
|
| 438 |
-
for _col, _name in products.PRICELIST_COLUMNS:
|
| 439 |
-
row[_col] = p.get(_col)
|
| 440 |
-
# Wave 17 R3 β the supplier master, on every pull (it is catalogue data, not stock).
|
| 441 |
-
s = sup.get(code) or {}
|
| 442 |
-
row.update({
|
| 443 |
-
"supplier": s.get("supplier") or "(none)",
|
| 444 |
-
"lead_days": s.get("lead_days"),
|
| 445 |
-
"origin_country": s.get("origin_country") or "(none)",
|
| 446 |
-
"first_cost": s.get("first_cost"),
|
| 447 |
-
})
|
| 448 |
-
e = inv.get(code) or {}
|
| 449 |
-
if consolidated:
|
| 450 |
-
qty_ltm, dos, bucket = e.get("qty_ltm"), e.get("dos"), e.get("bucket")
|
| 451 |
-
else:
|
| 452 |
-
qty_ltm, dos, bucket = _rescope_inventory(e, bu_share.get(code, _DEFAULT_SHARE))
|
| 453 |
-
row.update({
|
| 454 |
-
# UNSCOPED on purpose (owner 2026-08-11): one warehouse, no per-brand shelf.
|
| 455 |
-
"on_hand": e.get("on_hand"),
|
| 456 |
-
"unit_cost": e.get("unit_cost"),
|
| 457 |
-
"inv_value": e.get("inv_value"),
|
| 458 |
-
# SCOPED: these three are functions of how fast THIS unit sells the SKU.
|
| 459 |
-
"qty_ltm": qty_ltm,
|
| 460 |
-
"dos": dos,
|
| 461 |
-
"stock_bucket": bucket,
|
| 462 |
-
})
|
| 463 |
-
# THE BUY TRIGGER (R3), ported from `modules/procurement`'s lead-time-cover rule:
|
| 464 |
-
# buy when the shelf runs out before a reorder could land.
|
| 465 |
-
#
|
| 466 |
-
# β BLANK, NOT "OK", WHEN EITHER INPUT IS MISSING. 339 SKUs have no supplier and 400
|
| 467 |
-
# no lead time; `dos` is null for anything that never sells through. "We don't know"
|
| 468 |
-
# and "you're fine" are different sentences, and only one of them is safe to print
|
| 469 |
-
# beside a purchasing decision.
|
| 470 |
-
#
|
| 471 |
-
# β `buy_now` SHIPS NOWHERE ANY MORE, and is computed anyway. OWNER 2026-08-03: it
|
| 472 |
-
# is not a preset field β it is a formula over the two columns beside it, and the
|
| 473 |
-
# platform has a formula field type for exactly that. Its Field is gone from
|
| 474 |
-
# `aios_grid_fields.json`, and `rows_from_pool` projects rows STRICTLY through that
|
| 475 |
-
# contract, so no Field means no cell on the wire. Nothing renders this key.
|
| 476 |
-
#
|
| 477 |
-
# It stays computed because deleting it would delete the PROOF. `validate()` below
|
| 478 |
-
# reconciles the formula's predicate against this one, row for row, which is the only
|
| 479 |
-
# thing that makes "the same exact figures" a claim rather than an assertion β and
|
| 480 |
-
# this repo has the scar already (wave 17: archiving `ar` "would have DELETED the
|
| 481 |
-
# proof"). Two comparisons per SKU is what that costs.
|
| 482 |
-
#
|
| 483 |
-
# β READS THE SCOPED `dos`, NOT `e['dos']`. On a Fisch pull the cover gap must answer
|
| 484 |
-
# "does the shelf outlast a reorder AT FISCH'S RATE" β reading the consolidated figure
|
| 485 |
-
# here would print a buy signal computed from both units' velocity beside a days-of-supply
|
| 486 |
-
# computed from one, and the two columns would disagree on the same row.
|
| 487 |
-
lead = s.get("lead_days")
|
| 488 |
-
if isinstance(dos, (int, float)) and isinstance(lead, (int, float)) and lead > 0:
|
| 489 |
-
row["cover_gap_d"] = int(round(dos - lead))
|
| 490 |
-
row["buy_now"] = "Buy now" if dos < lead else "OK"
|
| 491 |
-
else:
|
| 492 |
-
row["cover_gap_d"] = None
|
| 493 |
-
row["buy_now"] = None
|
| 494 |
-
rows.append(row)
|
| 495 |
-
|
| 496 |
-
# `or 0.0` reads a BLANK as zero FOR SORTING ONLY β never for the cell. A never-sold SKU
|
| 497 |
-
# settles among the zero-revenue ones at the bottom, which is where it belongs; the stored
|
| 498 |
-
# value stays None so nothing downstream can mistake "we never saw a sale" for "we measured
|
| 499 |
-
# nothing sold".
|
| 500 |
-
rows.sort(key=lambda r: -(r["rev_ytd"] or 0.0))
|
| 501 |
-
_assert_no_pid_collision(rows)
|
| 502 |
-
return rows
|
| 503 |
-
|
| 504 |
-
|
| 505 |
-
#: β THE BUY SIGNAL, AS THE FORMULA FIELD EVALUATES IT (owner, 2026-08-03).
|
| 506 |
-
#:
|
| 507 |
-
#: The owner's ruling was that "Buy signal" is not a preset field β it is a formula over two
|
| 508 |
-
#: columns the product table already carries, and the platform has a formula field type for it.
|
| 509 |
-
#: The formula, verbatim, is what `_seed_wave17.BUY_SIGNAL_FORMULA` creates and what the JSON
|
| 510 |
-
#: contract's `_product_removed_buy_now` note records:
|
| 511 |
-
#:
|
| 512 |
-
#: IF({lead_days} > 0, IF({dos} < {lead_days}, "Buy now", "OK"), "")
|
| 513 |
-
#:
|
| 514 |
-
#: This function is a PORT of how `customer-grid/formulaEngine.ts` evaluates that tree, not a
|
| 515 |
-
#: restatement of the business rule β that is the whole point, because the two could drift and
|
| 516 |
-
#: `validate()` is where the drift must show. Three engine behaviours it reproduces exactly:
|
| 517 |
-
#:
|
| 518 |
-
#: Β· a `ref` to a missing/non-numeric cell is None (`case "ref"` returns null for anything
|
| 519 |
-
#: that is not a finite number or a string);
|
| 520 |
-
#: Β· `cmp` returns BLANK unless BOTH sides read as numbers β it never coerces a blank to 0,
|
| 521 |
-
#: which is the difference between this and a filter engine's `toNum(null) === 0`;
|
| 522 |
-
#: Β· `IF` with a non-boolean condition returns BLANK ("no truthiness"), so a blank comparison
|
| 523 |
-
#: propagates out as a blank cell rather than taking the false branch.
|
| 524 |
-
BUY_SIGNAL_FORMULA = 'IF({lead_days} > 0, IF({dos} < {lead_days}, "Buy now", "OK"), "")'
|
| 525 |
-
|
| 526 |
-
|
| 527 |
-
def _buy_signal_formula(row):
|
| 528 |
-
"""Evaluate `BUY_SIGNAL_FORMULA` over one pool row -> 'Buy now' | 'OK' | '' (blank)."""
|
| 529 |
-
def num(v):
|
| 530 |
-
# The engine's `ref` + `asNumber`: booleans are not values here, and a non-finite
|
| 531 |
-
# number is null. `isinstance(True, int)` is True in Python, so bool is excluded first.
|
| 532 |
-
if isinstance(v, bool) or not isinstance(v, (int, float)):
|
| 533 |
-
return None
|
| 534 |
-
return v if v == v and v not in (float('inf'), float('-inf')) else None
|
| 535 |
-
|
| 536 |
-
lead, dos = num(row.get("lead_days")), num(row.get("dos"))
|
| 537 |
-
if lead is None: # `{lead_days} > 0` is blank -> IF(blank, β¦) is blank
|
| 538 |
-
return ""
|
| 539 |
-
if not lead > 0:
|
| 540 |
-
return "" # the formula's own else-branch
|
| 541 |
-
if dos is None: # `{dos} < {lead_days}` is blank -> IF(blank, β¦) is blank
|
| 542 |
-
return ""
|
| 543 |
-
return "Buy now" if dos < lead else "OK"
|
| 544 |
-
|
| 545 |
-
|
| 546 |
-
def validate(team_id=None, t=None):
|
| 547 |
-
"""Reconcile what SHIPS to an independent aggregate β the platform's own rule that a number
|
| 548 |
-
which does not tie to Odoo does not ship.
|
| 549 |
-
|
| 550 |
-
ββ THE POPULATION LEG IS NEW, AND IT IS HERE BECAUSE THIS FUNCTION USED TO BE SELF-SEALING
|
| 551 |
-
(wave 29, contract C8). It reconciled Ξ£ per-SKU YTD revenue against `products._sku_rev` β the
|
| 552 |
-
same function `pool()` built its rows from. BOTH SIDES CAME FROM THE JOIN, so the oracle could
|
| 553 |
-
not see a missing row, and a grid holding 2,717 of 5,875 active products passed for a wave.
|
| 554 |
-
No gate anywhere asserted a pool row COUNT against an independent Odoo `search_count` either.
|
| 555 |
-
That is the leg below, and it is the control that would have caught it
|
| 556 |
-
([[no-unverifiable-aggregates]]).
|
| 557 |
-
|
| 558 |
-
β The revenue leg was NOT deleted with it β it answers a different question ("did the join
|
| 559 |
-
lose money?") and is still the right shape for that one. What changed is that it now has to
|
| 560 |
-
account for revenue belonging to codes NO ACTIVE PRODUCT CARRIES, because R12 keeps archived
|
| 561 |
-
products out of the grid; the total is decomposed rather than compared loosely, so a growing
|
| 562 |
-
"outside the grid" figure shows up as a number rather than as slack in a tolerance.
|
| 563 |
-
|
| 564 |
-
Still deliberately NOT covered, and said rather than implied: the inventory columns are a
|
| 565 |
-
different module's reconciliation, and they are absent on a scoped pull anyway.
|
| 566 |
-
"""
|
| 567 |
-
t = t or P.today()
|
| 568 |
-
rows = pool(team_id=team_id, t=t)
|
| 569 |
-
|
| 570 |
-
# ββ THE POPULATION, AGAINST AN ORACLE THAT CANNOT SEE OUR JOIN. A bare Odoo count of active
|
| 571 |
-
# products, asked fresh β not a `len()` over anything this module or `products.catalogue()`
|
| 572 |
-
# built. NC: drop one row from `pool()` and this goes red; that is the whole point of it.
|
| 573 |
-
n_active = products.catalogue_count()
|
| 574 |
-
checks = [{
|
| 575 |
-
"check": "the pool holds one row per ACTIVE product (R12) β row count == an INDEPENDENT "
|
| 576 |
-
"Odoo search_count('product.product', active=True)",
|
| 577 |
-
"ours": len(rows), "theirs": n_active, "ok": len(rows) == n_active,
|
| 578 |
-
}]
|
| 579 |
-
|
| 580 |
-
yf, yt = P.ytd(t)
|
| 581 |
-
sku_rev = products._sku_rev(yf, yt, team_id) or {}
|
| 582 |
-
in_pool = {r["code"] for r in rows}
|
| 583 |
-
outside = {c: v for c, v in sku_rev.items() if c not in in_pool}
|
| 584 |
-
ours = round(sum(r["rev_ytd"] or 0.0 for r in rows), 2)
|
| 585 |
-
theirs = round(sum(v.get("rev", 0.0) for v in sku_rev.values()), 2)
|
| 586 |
-
dropped = round(sum(v.get("rev", 0.0) for v in outside.values()), 2)
|
| 587 |
-
checks.append({
|
| 588 |
-
"check": "Ξ£ per-SKU YTD revenue in the grid + Ξ£ YTD revenue of codes no ACTIVE product "
|
| 589 |
-
"carries == modules.products._sku_rev over the same window",
|
| 590 |
-
"ours": round(ours + dropped, 2), "theirs": theirs,
|
| 591 |
-
"ok": abs((ours + dropped) - theirs) < 0.01,
|
| 592 |
-
"detail": {"in_the_grid": ours, "outside_the_grid": dropped,
|
| 593 |
-
"codes_outside": len(outside)},
|
| 594 |
-
})
|
| 595 |
-
# β AND THE EXCLUSION IS ASSERTED, NOT ASSUMED. "Everything I dropped was archived" is
|
| 596 |
-
# trivially true when checked against the dict that did the dropping β that is the
|
| 597 |
-
# self-sealing shape all over again. So it is asked of ODOO. A code here that a live ACTIVE
|
| 598 |
-
# product carries means `products.catalogue()` missed a row that has revenue, which is the
|
| 599 |
-
# 2,717 defect returning in a smaller costume.
|
| 600 |
-
# β `pid:N` keys are not `default_code`s and are skipped: an UNCODED active product is in the
|
| 601 |
-
# catalogue under its own `pid:N` key and therefore cannot be in `outside` at all.
|
| 602 |
-
coded_outside = sorted(c for c in outside if not str(c).startswith("pid:"))
|
| 603 |
-
still_active = (O.get_odoo().search_count(
|
| 604 |
-
'product.product', [('active', '=', True), ('default_code', 'in', coded_outside)])
|
| 605 |
-
if coded_outside else 0)
|
| 606 |
-
checks.append({
|
| 607 |
-
"check": "every SKU with revenue but NO grid row is genuinely ARCHIVED (R12: archived "
|
| 608 |
-
"stay out) β asked of Odoo, never of the catalogue that did the dropping",
|
| 609 |
-
"ours": still_active, "theirs": 0, "ok": still_active == 0,
|
| 610 |
-
"detail": {"codes": coded_outside[:10], "n_codes": len(outside)},
|
| 611 |
-
})
|
| 612 |
-
# ββ ββ WAVE 30 W30-T34 β THE PRICELIST COLUMNS, AGAINST ORACLES THAT CANNOT SEE OUR JOIN ββ
|
| 613 |
-
#
|
| 614 |
-
# Three legs, because the column can fail in three different ways and only one of them is a
|
| 615 |
-
# count. `products.pricelist_by_code` builds a `{code: {col: price}}` map by expanding each
|
| 616 |
-
# rule over the products it names, IN PYTHON β that expansion is the fragile part, so every
|
| 617 |
-
# leg below re-asks ODOO instead of re-reading the map.
|
| 618 |
-
price_report = _pricelist_by_code()[1]
|
| 619 |
-
_pl_rows = O.search_read('product.pricelist', [], ['id', 'name'])
|
| 620 |
-
_pl_id = {}
|
| 621 |
-
for _p in _pl_rows:
|
| 622 |
-
_pl_id.setdefault(str(_p.get('name') or '').strip(), _p['id'])
|
| 623 |
-
_today = P.today().isoformat()
|
| 624 |
-
|
| 625 |
-
_targets_memo = {}
|
| 626 |
-
|
| 627 |
-
def _rule_targets(list_name):
|
| 628 |
-
"""(variant_ids, template_ids) a pricelist prices today β read FRESH from Odoo.
|
| 629 |
-
|
| 630 |
-
Memoised for the LIFE OF THIS CALL only: legs 1 and 2 both need all three lists, and
|
| 631 |
-
without this `validate()` makes six identical round trips instead of three (measured:
|
| 632 |
-
~48s of the run). Deliberately NOT an `lru_cache` β an oracle that survives the process
|
| 633 |
-
is an oracle reading yesterday's Odoo.
|
| 634 |
-
"""
|
| 635 |
-
if list_name in _targets_memo:
|
| 636 |
-
return _targets_memo[list_name]
|
| 637 |
-
var, tmpl = set(), set()
|
| 638 |
-
plid = _pl_id.get(list_name)
|
| 639 |
-
if plid is None:
|
| 640 |
-
return var, tmpl
|
| 641 |
-
for r in O.search_read(
|
| 642 |
-
'product.pricelist.item',
|
| 643 |
-
[('pricelist_id', '=', plid), ('compute_price', '=', 'fixed'),
|
| 644 |
-
('applied_on', 'in', ['0_product_variant', '1_product'])],
|
| 645 |
-
['product_id', 'product_tmpl_id', 'applied_on', 'fixed_price',
|
| 646 |
-
'date_start', 'date_end']):
|
| 647 |
-
ds, de = str(r.get('date_start') or '')[:10], str(r.get('date_end') or '')[:10]
|
| 648 |
-
if (ds and ds > _today) or (de and de < _today):
|
| 649 |
-
continue
|
| 650 |
-
fp = r.get('fixed_price')
|
| 651 |
-
if not isinstance(fp, (int, float)) or fp <= 0:
|
| 652 |
-
continue
|
| 653 |
-
if r.get('applied_on') == '0_product_variant' and r.get('product_id'):
|
| 654 |
-
var.add(O.m2o_id(r['product_id']))
|
| 655 |
-
elif r.get('product_tmpl_id'):
|
| 656 |
-
tmpl.add(O.m2o_id(r['product_tmpl_id']))
|
| 657 |
-
_targets_memo[list_name] = (var, tmpl)
|
| 658 |
-
return var, tmpl
|
| 659 |
-
|
| 660 |
-
# LEG 1 β COVERAGE PER LIST. Ours: cells we filled. Theirs: a bare Odoo `search_count` of
|
| 661 |
-
# ACTIVE products a fresh read of that list's rules reaches. The rule set is shared (it IS
|
| 662 |
-
# the data) but the EXPANSION is not, and the expansion is what breaks.
|
| 663 |
-
for _col, _name in products.PRICELIST_COLUMNS:
|
| 664 |
-
_var, _tmpl = _rule_targets(_name)
|
| 665 |
-
_dom = [('active', '=', True), '|', ('id', 'in', sorted(_var)),
|
| 666 |
-
('product_tmpl_id', 'in', sorted(_tmpl))]
|
| 667 |
-
theirs = O.get_odoo().search_count('product.product', _dom) if (_var or _tmpl) else 0
|
| 668 |
-
ours = sum(1 for r in rows if isinstance(r.get(_col), (int, float)))
|
| 669 |
-
checks.append({
|
| 670 |
-
"check": f"{_name} pricelist: SKUs priced in the grid == an INDEPENDENT Odoo count "
|
| 671 |
-
f"of ACTIVE products its date-valid fixed rules reach",
|
| 672 |
-
"ours": ours, "theirs": theirs, "ok": ours == theirs,
|
| 673 |
-
"detail": {"column": _col, "variant_rules": len(_var), "template_rules": len(_tmpl)},
|
| 674 |
-
})
|
| 675 |
-
|
| 676 |
-
# LEG 2 β β THE TICKET'S OWN NEGATIVE CONTROL, AS A LEG. A SKU with no specific item must
|
| 677 |
-
# render BLANK. Asked of ODOO, never of the map that produced the blank β "everything I left
|
| 678 |
-
# empty was genuinely unpriced" is trivially true when checked against the dict that emptied
|
| 679 |
-
# it, which is the self-sealing shape the population leg above exists to end.
|
| 680 |
-
_unpriced = [r for r in rows
|
| 681 |
-
if not any(isinstance(r.get(c), (int, float))
|
| 682 |
-
for c, _n in products.PRICELIST_COLUMNS)]
|
| 683 |
-
_codes = {r["code"] for r in _unpriced}
|
| 684 |
-
_stray = 0
|
| 685 |
-
if _codes:
|
| 686 |
-
# Resolve those codes back to Odoo ids and ask whether ANY declared list prices them.
|
| 687 |
-
_ids, _tmpls = set(), set()
|
| 688 |
-
for _p in O.search_read('product.product',
|
| 689 |
-
[('active', '=', True),
|
| 690 |
-
('default_code', 'in', sorted(c for c in _codes
|
| 691 |
-
if not c.startswith("pid:")))],
|
| 692 |
-
['id', 'product_tmpl_id']):
|
| 693 |
-
_ids.add(_p['id'])
|
| 694 |
-
_tmpls.add(O.m2o_id(_p.get('product_tmpl_id')))
|
| 695 |
-
for _col, _name in products.PRICELIST_COLUMNS:
|
| 696 |
-
_v, _t = _rule_targets(_name)
|
| 697 |
-
_stray += len((_v & _ids) | ({t for t in _tmpls if t in _t}))
|
| 698 |
-
checks.append({
|
| 699 |
-
"check": "every SKU rendering a BLANK price is genuinely unpriced on all three declared "
|
| 700 |
-
"lists (W29-T52's NC: no fallback dressed as a price) β asked of Odoo",
|
| 701 |
-
"ours": _stray, "theirs": 0, "ok": _stray == 0,
|
| 702 |
-
"detail": {"blank_skus": len(_unpriced), "priced_skus": len(rows) - len(_unpriced)},
|
| 703 |
-
})
|
| 704 |
-
|
| 705 |
-
# LEG 3 β β R6's SECOND SENTENCE, AS A NUMBER. *"If there is lag or it can't be done, you
|
| 706 |
-
# need to explicitly tell me why and recommend a fix."* Everything the reader cannot see is
|
| 707 |
-
# counted here rather than dropped. The ASSERTION is the one thing that must never be true β
|
| 708 |
-
# a price cell that is present and not a positive number, i.e. a "free" SKU β while the
|
| 709 |
-
# rest rides `detail` so it is reported without reddening an honest day's data.
|
| 710 |
-
_bad = [r["code"] for r in rows
|
| 711 |
-
for c, _n in products.PRICELIST_COLUMNS
|
| 712 |
-
if r.get(c) is not None and not (isinstance(r.get(c), (int, float)) and r[c] > 0)]
|
| 713 |
-
# β MEASURED 2026-08-12 AND NOT FIXABLE FROM THIS FENCE: `aios_grid.rows_from_pool` sends
|
| 714 |
-
# every non-text field through `_round(v)` = `round(v)` with no ndigits, i.e. to a whole
|
| 715 |
-
# dollar. 52.1% of fixed prices carry cents and the median relative error is 5.26%, so this
|
| 716 |
-
# column ties to the cent HERE and ships rounded. Reported, never silently enforced.
|
| 717 |
-
_cents = sum(1 for r in rows for c, _n in products.PRICELIST_COLUMNS
|
| 718 |
-
if isinstance(r.get(c), (int, float)) and abs(r[c] - round(r[c])) > 1e-9)
|
| 719 |
-
_to_zero = sum(1 for r in rows for c, _n in products.PRICELIST_COLUMNS
|
| 720 |
-
if isinstance(r.get(c), (int, float)) and round(r[c]) == 0)
|
| 721 |
-
checks.append({
|
| 722 |
-
"check": "no price cell is present-but-not-a-positive-number (a 0 would read as FREE; "
|
| 723 |
-
"an unset rule must be blank), and what the reader cannot see is COUNTED",
|
| 724 |
-
"ours": len(_bad), "theirs": 0, "ok": not _bad,
|
| 725 |
-
"detail": {"bad_cells": _bad[:10], "reader_report": price_report,
|
| 726 |
-
"wire_rounding_loses_cents_on": _cents,
|
| 727 |
-
"wire_rounding_to_zero_dollars": _to_zero},
|
| 728 |
-
})
|
| 729 |
-
|
| 730 |
-
# WAVE 17 R3 β the BUY SIGNAL must be a total, exact partition of the catalogue, and every
|
| 731 |
-
# member of it must be re-derivable from the two columns beside it. A signal somebody buys
|
| 732 |
-
# stock on cannot be "mostly right": the failure that matters is a row that says OK because
|
| 733 |
-
# an input was missing, so the blank leg is checked as hard as the other two.
|
| 734 |
-
if team_id is None: # consolidated only β the inputs exist only there
|
| 735 |
-
buy = [r for r in rows if r.get("buy_now") == "Buy now"]
|
| 736 |
-
ok_rows = [r for r in rows if r.get("buy_now") == "OK"]
|
| 737 |
-
blank = [r for r in rows if r.get("buy_now") is None]
|
| 738 |
-
mis = sum(1 for r in buy if not (r["dos"] < r["lead_days"]))
|
| 739 |
-
mis += sum(1 for r in ok_rows if not (r["dos"] >= r["lead_days"]))
|
| 740 |
-
# A blank must be UNKNOWN β never a row we could have answered and quietly did not.
|
| 741 |
-
mis += sum(1 for r in blank
|
| 742 |
-
if isinstance(r.get("dos"), (int, float))
|
| 743 |
-
and isinstance(r.get("lead_days"), (int, float)) and r["lead_days"] > 0)
|
| 744 |
-
checks.append({
|
| 745 |
-
"check": "Buy signal partitions the catalogue (buy + ok + unknown == rows, none "
|
| 746 |
-
"misclassified)",
|
| 747 |
-
"ours": len(buy) + len(ok_rows) + len(blank) - mis, "theirs": len(rows),
|
| 748 |
-
"ok": mis == 0 and len(buy) + len(ok_rows) + len(blank) == len(rows),
|
| 749 |
-
"detail": {"buy_now": len(buy), "ok": len(ok_rows), "unknown": len(blank),
|
| 750 |
-
"misclassified": mis},
|
| 751 |
-
})
|
| 752 |
-
# β OWNER 2026-08-03 β "use the formula fields to come up to the same EXACT figures".
|
| 753 |
-
#
|
| 754 |
-
# This is that sentence, as a check. The retired preset column and the formula that
|
| 755 |
-
# replaces it must agree on every SKU, in all three states, or the replacement is not a
|
| 756 |
-
# replacement. `_buy_signal_formula` is a line-by-line port of the client formula
|
| 757 |
-
# engine's evaluation of
|
| 758 |
-
#
|
| 759 |
-
# IF({lead_days} > 0, IF({dos} < {lead_days}, "Buy now", "OK"), "")
|
| 760 |
-
#
|
| 761 |
-
# including the part that is easy to get wrong: a comparison against a BLANK is blank,
|
| 762 |
-
# never a coerced 0. (The client engine's `cmp` returns null unless both sides read as
|
| 763 |
-
# numbers, and `IF` refuses a non-boolean condition β so a missing `dos` yields "" and
|
| 764 |
-
# not "Buy now". A filter engine would have said `0 < 30` and swept in every SKU that
|
| 765 |
-
# never sells through; the formula engine does not, and this check is what holds it.)
|
| 766 |
-
#
|
| 767 |
-
# β It compares SETS OF SKUs, not counts. Two different partitions can share a shape.
|
| 768 |
-
disagree = sorted(r["code"] for r in rows
|
| 769 |
-
if (r.get("buy_now") or "") != _buy_signal_formula(r))
|
| 770 |
-
checks.append({
|
| 771 |
-
"check": 'Buy signal as a FORMULA field == the retired preset column, per SKU '
|
| 772 |
-
'(IF({lead_days} > 0, IF({dos} < {lead_days}, "Buy now", "OK"), ""))',
|
| 773 |
-
"ours": len(rows) - len(disagree), "theirs": len(rows),
|
| 774 |
-
"ok": not disagree,
|
| 775 |
-
"detail": {"disagreeing_skus": disagree[:10], "n_disagree": len(disagree)},
|
| 776 |
-
})
|
| 777 |
-
# The other half of "the same figures": the SHARED Buy list view can no longer filter on
|
| 778 |
-
# the retired column, and its replacement conditions must select the same SKUs. They are
|
| 779 |
-
# `dos isNotEmpty AND lead_days isNotEmpty AND lead_days > 0 AND dos < lead_days`
|
| 780 |
-
# (_seed_wave17.views), so this reproduces exactly that conjunction.
|
| 781 |
-
#
|
| 782 |
-
# β THE NEGATIVE CONTROL IS WHY THIS IS NOT `cover_gap_d < 0`, which reads like the
|
| 783 |
-
# obvious filter and is WRONG: `cover_gap_d` is `int(round(dos - lead))`, so a genuine
|
| 784 |
-
# gap of -0.4 days rounds to 0 and that SKU drops off a buy list it belongs on.
|
| 785 |
-
view_rows = {r["code"] for r in rows
|
| 786 |
-
if isinstance(r.get("dos"), (int, float))
|
| 787 |
-
and isinstance(r.get("lead_days"), (int, float))
|
| 788 |
-
and r["lead_days"] > 0 and r["dos"] < r["lead_days"]}
|
| 789 |
-
signal_rows = {r["code"] for r in buy}
|
| 790 |
-
rounding_would_miss = sorted(
|
| 791 |
-
c for c in signal_rows
|
| 792 |
-
if next((r for r in rows if r["code"] == c), {}).get("cover_gap_d") == 0)
|
| 793 |
-
checks.append({
|
| 794 |
-
"check": "Buy list view conditions (dos/lead_days, no retired column) select "
|
| 795 |
-
"exactly the Buy-now SKUs",
|
| 796 |
-
"ours": len(view_rows), "theirs": len(signal_rows),
|
| 797 |
-
"ok": view_rows == signal_rows,
|
| 798 |
-
"detail": {"only_in_view": sorted(view_rows - signal_rows)[:10],
|
| 799 |
-
"only_in_signal": sorted(signal_rows - view_rows)[:10],
|
| 800 |
-
# Reported, not asserted: how many SKUs a `cover_gap_d < 0` filter would
|
| 801 |
-
# have silently dropped. 0 today does not make that filter correct.
|
| 802 |
-
"cover_gap_rounds_to_zero": len(rounding_would_miss)},
|
| 803 |
-
})
|
| 804 |
-
return checks
|
|
|
|
| 1 |
+
"""modules/product_data.py β the PRODUCT table's pool (wave 15 item 9/10, contract C-TOPIC).
|
| 2 |
+
|
| 3 |
+
The second object on the table-page factory: same grid, same engine, same permission wall β the
|
| 4 |
+
only difference is the field schema and the identity. `modules/customer_data.pool()` is the
|
| 5 |
+
template and this deliberately mirrors its signature and its row shape.
|
| 6 |
+
|
| 7 |
+
β THREE DECISIONS THIS FILE HAD TO MAKE, EACH RECORDED BECAUSE A LATER READER WILL WONDER.
|
| 8 |
+
|
| 9 |
+
1. **THE IDENTITY IS A SKU CODE, WHICH IS A STRING, AND THE GRID WANTS AN INTEGER `pid`.**
|
| 10 |
+
Cohort membership, `allowed_pids`, the measure channel and `rows_from_pool` all key on an
|
| 11 |
+
integer. So each row carries BOTH: `pid` (a stable CRC32 of the code, so the same SKU gets the
|
| 12 |
+
same id on every pull and across processes β never an enumeration index, which would reshuffle
|
| 13 |
+
whenever the catalogue changes) and `code`, the real business key, as a visible column.
|
| 14 |
+
`_assert_no_pid_collision` fails the BUILD rather than the read: two SKUs sharing a pid would
|
| 15 |
+
silently merge in every downstream set operation, and a loud build failure is the only version
|
| 16 |
+
of that anyone would notice.
|
| 17 |
+
|
| 18 |
+
2. **THE SCOPE RULE β inventory columns are CONSOLIDATED and are therefore OMITTED for a
|
| 19 |
+
BU-scoped caller.** `products.directory(t, team_id)` is brand-shaped; `inventory.sku_inventory`
|
| 20 |
+
is explicitly NOT (its own docstring: "on-hand stock is one physical warehouse, not
|
| 21 |
+
brand-tagged"). Joining them for a Fisch-only user would put BU-shaped revenue beside
|
| 22 |
+
company-wide stock IN THE SAME ROW β the mixed-scope value defect wave 15's amendment 3 exists
|
| 23 |
+
for, arriving through a different door. The honest options were "omit" or "label the columns
|
| 24 |
+
company-wide"; omit is the fail-closed one, and a column that is absent asks a question,
|
| 25 |
+
whereas a column that is silently company-wide answers one wrongly.
|
| 26 |
+
|
| 27 |
+
3. **WHAT IS NOT HERE, NAMED RATHER THAN QUIETLY MISSING.** R4 lists ~20 fields. Vendor and
|
| 28 |
+
COUNTRY are not in Odoo at all β they live in the inventory WORKBOOK
|
| 29 |
+
([[odoo-vendor-country-origin]]) and need a loader this module deliberately does not invent.
|
| 30 |
+
Margin % comes from `modules/pricing.table`, which is a heavier build (channel-rate cost
|
| 31 |
+
allocation) and is left for the wave that needs it. `validate()` reconciles what SHIPS; it
|
| 32 |
+
does not pretend to cover columns that are absent.
|
| 33 |
+
|
| 34 |
+
4. **THE POOL IS CATALOGUE-FIRST, WITH REVENUE LEFT-JOINED** (wave 29, owner item 22 / R12,
|
| 35 |
+
2026-08-11). It was `for r in products.directory(...)` β and `directory()`'s row set IS the
|
| 36 |
+
union of two revenue read-groups, so **a SKU that never sold could not exist** and the grid
|
| 37 |
+
showed **2,717** of **5,875** active products. Three things that will be re-derived otherwise:
|
| 38 |
+
|
| 39 |
+
Β· β THE CAUSE IS A JOIN, NOT A LIMIT. No row cap exists on this path. Removing the date
|
| 40 |
+
window from `directory()` would ALSO be wrong twice: it breaks four SKU-health metrics
|
| 41 |
+
that legitimately want a sales window, and it lands at 3,327 (all-time-sold), because
|
| 42 |
+
~2,550 active SKUs have never sold in wholesale scope at all.
|
| 43 |
+
Β· β REVENUE ON A NEVER-SOLD ROW IS **BLANK, NOT $0** (R12). A row that appears in the
|
| 44 |
+
revenue universe carries measured numbers INCLUDING a real 0.0 β it sold last year and
|
| 45 |
+
not this one, and that zero is a measurement. A row that appears in NO revenue read
|
| 46 |
+
carries `None`, which the wire keeps (`aios_grid._round` passes None through) and the
|
| 47 |
+
client renders as an empty cell. Blank is an admission; zero is a measurement.
|
| 48 |
+
Β· β A BU-SCOPED CALLER NOW SEES THE WHOLE CATALOGUE, with ITS OWN revenue and blanks where
|
| 49 |
+
that BU never sold. That is not decision 2's mixed-scope defect: the catalogue is the ROW
|
| 50 |
+
UNIVERSE, not a company-wide VALUE sitting beside a BU-shaped one. `product.product`
|
| 51 |
+
carries no team, so a catalogue cannot be BU-shaped at all β which is exactly why the two
|
| 52 |
+
sources are JOINED rather than merged.
|
| 53 |
+
"""
|
| 54 |
+
import json
|
| 55 |
+
import zlib
|
| 56 |
+
from pathlib import Path
|
| 57 |
+
|
| 58 |
+
import core.odoo as O
|
| 59 |
+
import core.periods as P
|
| 60 |
+
import core.shared_overlay as shared_overlay
|
| 61 |
+
import core.table_store as table_store
|
| 62 |
+
import modules.products as products
|
| 63 |
+
|
| 64 |
+
#: POOL-ROW KEYS that exist ONLY on a consolidated pull. See decision 2.
|
| 65 |
+
#: `cover_gap_d` / `buy_now` (wave 17) join it because both are computed FROM `dos`, and a buy
|
| 66 |
+
#: signal built on a stock number the caller cannot see would be a recommendation nobody could
|
| 67 |
+
#: check ([[no-unverifiable-aggregates]]).
|
| 68 |
+
#:
|
| 69 |
+
#: β ROW KEYS, NOT FIELDS, and the distinction started mattering on 2026-08-03. Two readers:
|
| 70 |
+
#: `verify_perm_scope` asserts the shape of a POOL ROW against this, and `routes_products`
|
| 71 |
+
#: narrows the FIELD list with it. Since owner item 5 `buy_now` is a row key with no Field β
|
| 72 |
+
#: computed as `validate()`'s oracle, projected onto no wire β so it belongs here for the first
|
| 73 |
+
#: reader and is inert for the second. Removing it would stop the scope gate proving that a
|
| 74 |
+
#: BU-scoped pull withholds it.
|
| 75 |
+
#:
|
| 76 |
+
#: ββ OWNER RULING 2026-08-11 β THIS SET IS NOW EMPTY, AND THAT IS THE POINT. Verbatim:
|
| 77 |
+
#: *"Just scope any inventory with Sales from Fisch, leave the rest."* Withholding the whole
|
| 78 |
+
#: inventory block from a BU-scoped reader meant a Fisch salesperson could not see whether
|
| 79 |
+
#: anything was IN STOCK, which is the first question they ask. The ruling splits the block by
|
| 80 |
+
#: what a business unit can actually shape:
|
| 81 |
+
#:
|
| 82 |
+
#: * `on_hand` / `unit_cost` / `inv_value` β ONE physical warehouse, no Fisch shelf and no
|
| 83 |
+
#: Royal shelf. Served UNSCOPED to everybody, identical in both pulls. ("leave the rest")
|
| 84 |
+
#: * `qty_ltm` / `dos` / `stock_bucket` / `cover_gap_d` / `buy_now` β all SALES-derived, so a
|
| 85 |
+
#: BU-scoped pull recomputes them from THAT unit's LTM units. ("scope any inventory with
|
| 86 |
+
#: Sales from Fisch")
|
| 87 |
+
#:
|
| 88 |
+
#: This deliberately relaxes decision 2's "never a company-wide value beside a BU-shaped one":
|
| 89 |
+
#: stock is not a value a BU can own, and a column that is absent for a Fisch reader asks a
|
| 90 |
+
#: question they cannot answer anywhere else in the product. `verify_perm_scope` no longer
|
| 91 |
+
#: asserts the columns are WITHHELD β it asserts the split above, which is a stronger claim and
|
| 92 |
+
#: cannot pass on an empty scoped payload (the shape the old rule and an outage share).
|
| 93 |
+
CONSOLIDATED_ONLY = ()
|
| 94 |
+
|
| 95 |
+
#: β POOL-ROW KEYS THAT DELIBERATELY HAVE NO FIELD β they must never reach a browser.
|
| 96 |
+
#:
|
| 97 |
+
#: `rows_from_pool` projects strictly through the field contract, so "no Field" already means
|
| 98 |
+
#: "no cell on the wire". This tuple makes that a CHECKED fact rather than a consequence nobody
|
| 99 |
+
#: is watching: `verify_perm_scope` asserts every member is absent from BOTH product contracts,
|
| 100 |
+
#: and that every OTHER `CONSOLIDATED_ONLY` key is present in the consolidated one.
|
| 101 |
+
#:
|
| 102 |
+
#: Written because the two tuples silently diverged the moment owner item 5 retired `buy_now`'s
|
| 103 |
+
#: Field while keeping its computation, and the gate β which was asserting a FIELD property from
|
| 104 |
+
#: a ROW-key list β went red with no way to tell a deliberate divergence from a dropped column.
|
| 105 |
+
UNSHIPPED_ROW_KEYS = ("buy_now",)
|
| 106 |
+
|
| 107 |
+
#: Wave 17 (owner item 13, ruling R3) β the SUPPLIER MASTER, from the curated mastersheet map
|
| 108 |
+
#: `procurement_suppliers.json` (3,963 SKUs; supplier on 3,624, lead time on 3,563). NOT Odoo:
|
| 109 |
+
#: the owner confirmed this data came from the Inventory System Mastersheet, which is why
|
| 110 |
+
#: `modules/product_data`'s header said Vendor/Country were "NOT HERE, named rather than
|
| 111 |
+
#: quietly missing" and needed "a loader this module deliberately does not invent". This is
|
| 112 |
+
#: that loader.
|
| 113 |
+
#:
|
| 114 |
+
#: β THESE ARE CONTRACT COLUMNS, NOT USER-CREATED FIELDS, AND THE REASON IS A PERMISSION FACT.
|
| 115 |
+
#: A user-created field and its values live in the PER-USER strata (`table_store.workspace`
|
| 116 |
+
#: reads `store.get(key)[username]`); only VIEWS are shared. So a shared "Buy list" view that
|
| 117 |
+
#: filtered on a user-created column would, for every OTHER account, name a column that does not
|
| 118 |
+
#: exist β and an unknown column is an INACTIVE condition in the tri-state engine, which IGNORES
|
| 119 |
+
#: it and therefore WIDENS. The buy list would silently show the whole catalogue to everyone but
|
| 120 |
+
#: its author. Contract columns are identical for every reader, so the view means one thing.
|
| 121 |
+
#:
|
| 122 |
+
#: ββ 2026-08-12 (W30-T36) β THE OWNER'S ASK IS NOW DELIVERED, AND THE PARAGRAPH ABOVE IS WHY IT
|
| 123 |
+
#: TOOK THREE WAVES. Owner: *"turn this Excel sheet into a User created Field that we can edit."*
|
| 124 |
+
#: It was parked because a per-user column silently WIDENS a shared view β not because editing was
|
| 125 |
+
#: hard. Wave 29's `core/shared_overlay.py` (whose header quotes this very comment) is the stratum
|
| 126 |
+
#: that removes the objection: **one value per (row, column) for the whole tenant**, so the column
|
| 127 |
+
#: still means ONE thing to every reader and a shared view still filters honestly.
|
| 128 |
+
#: β The four columns below are now `source: "overlay"` + `shared: true` in the canonical
|
| 129 |
+
#: contract, their values live in `<TABLE_KEY>__shared`, and the master map is what SEEDS an
|
| 130 |
+
#: unedited cell rather than what freezes it. See `SHARED_KEYS` and `_ProductTableStore`.
|
| 131 |
+
_SUPPLIER_MAP_PATH = Path(__file__).resolve().parent.parent / "procurement_suppliers.json"
|
| 132 |
+
_SUPPLIER_CACHE = {}
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
def supplier_master():
|
| 136 |
+
"""`{code: {supplier, lead_days, origin_country, first_cost}}`, read once per process.
|
| 137 |
+
|
| 138 |
+
Degrades to `{}` when the file is unreadable, matching `_inventory_by_code`: a product table
|
| 139 |
+
that will not render because a master map is missing is a worse failure than one with blank
|
| 140 |
+
supplier columns.
|
| 141 |
+
"""
|
| 142 |
+
if _SUPPLIER_CACHE:
|
| 143 |
+
return _SUPPLIER_CACHE
|
| 144 |
+
try:
|
| 145 |
+
raw = json.loads(_SUPPLIER_MAP_PATH.read_text(encoding="utf-8"))
|
| 146 |
+
except Exception:
|
| 147 |
+
return {}
|
| 148 |
+
for code, meta in (raw or {}).items():
|
| 149 |
+
if not isinstance(meta, dict):
|
| 150 |
+
continue
|
| 151 |
+
lead = meta.get("lead")
|
| 152 |
+
_SUPPLIER_CACHE[str(code)] = {
|
| 153 |
+
"supplier": (meta.get("vendor") or "") or None,
|
| 154 |
+
"lead_days": int(lead) if isinstance(lead, (int, float)) else None,
|
| 155 |
+
"origin_country": (meta.get("country") or "") or None,
|
| 156 |
+
"first_cost": meta.get("first_cost"),
|
| 157 |
+
}
|
| 158 |
+
return _SUPPLIER_CACHE
|
| 159 |
+
|
| 160 |
+
#: Wave 16 C-TOPIC β the PRODUCT table's OWN workspace bucket. β Never the customer one:
|
| 161 |
+
#: product pids are CRC32 hashes of SKU codes and customer pids are Odoo partner ids, so in a
|
| 162 |
+
#: SHARED overlay bucket a hash collision would silently write a product note onto somebody's
|
| 163 |
+
#: customer (or the reverse). Separate store keys make that structurally impossible, which is
|
| 164 |
+
#: the whole reason the table-page factory exists ("a new table object gets its own
|
| 165 |
+
#: table_store.make('<its>_table_workspace')").
|
| 166 |
+
TABLE_KEY = 'product_table_workspace'
|
| 167 |
+
|
| 168 |
+
_GRID_FIELDS_PATH = Path(__file__).resolve().parent.parent / 'aios_grid_fields.json'
|
| 169 |
+
_SHARED_KEYS = None
|
| 170 |
+
|
| 171 |
+
|
| 172 |
+
def SHARED_KEYS():
|
| 173 |
+
"""The product columns whose values are TENANT-WIDE β derived from the canonical contract's
|
| 174 |
+
own `shared: true`, never typed out a second time.
|
| 175 |
+
|
| 176 |
+
β IT DELIBERATELY DOES NOT SWALLOW A READ FAILURE. Degrading to `()` would send a shared
|
| 177 |
+
write into the per-user stratum with nothing going wrong anywhere β the widening defect
|
| 178 |
+
reappearing silently, which is the one outcome this whole mechanism exists to prevent. If the
|
| 179 |
+
canonical contract is unreadable the product grid cannot render at all (`pd_fields` parses the
|
| 180 |
+
same file with no guard), so a raise here costs nothing that was still working.
|
| 181 |
+
"""
|
| 182 |
+
global _SHARED_KEYS
|
| 183 |
+
if _SHARED_KEYS is None:
|
| 184 |
+
doc = json.loads(_GRID_FIELDS_PATH.read_text(encoding='utf-8'))
|
| 185 |
+
_SHARED_KEYS = tuple(f['key'] for f in (doc.get('product_data') or {}).get('fields') or []
|
| 186 |
+
if f.get('shared'))
|
| 187 |
+
return _SHARED_KEYS
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
class _ProductTableStore(table_store.TableStore):
|
| 191 |
+
"""The product workspace, with the SHARED columns routed to the tenant-wide stratum.
|
| 192 |
+
|
| 193 |
+
ββ THIS SUBCLASS IS THE WHOLE OF W30-T36's WRITE PATH, AND THE REASON IT LIVES HERE RATHER
|
| 194 |
+
THAN AT A ROUTE IS THAT **THE BROWSER NEVER CALLS `PATCH /products/{pid}`** β measured, zero
|
| 195 |
+
call sites in `aios-web/web/src`. A cell edit travels `POST /grid/events` β `grid_events.
|
| 196 |
+
handle_one` β `_tops(ctx).patch_overlay(...)`, and `_tops` returns `ctx.table`, which
|
| 197 |
+
`routes_grid._ctx` sets to `TABLE_OPS` for the product scope. So this object IS the seam both
|
| 198 |
+
doors pass through; intercepting at either route would have left the other one writing a
|
| 199 |
+
per-user value that only its author could see.
|
| 200 |
+
|
| 201 |
+
β `st=self.st`, NEVER the module default. The shared stratum must resolve to the SAME store
|
| 202 |
+
handle as the per-user one it sits beside β `_tops`' own comment explains that a split, where
|
| 203 |
+
one side is tenant-scoped and the other is not, is worse than a stated residency error because
|
| 204 |
+
a user's value would vanish the moment they saved it. Reading `self.st` means both strata move
|
| 205 |
+
together the day that singleton gains a tenant handle.
|
| 206 |
+
"""
|
| 207 |
+
|
| 208 |
+
def patch_overlay(self, username, pid, updates):
|
| 209 |
+
clean = dict(updates or {})
|
| 210 |
+
if not clean:
|
| 211 |
+
return
|
| 212 |
+
keys = set(SHARED_KEYS())
|
| 213 |
+
shared = {k: v for k, v in clean.items() if k in keys}
|
| 214 |
+
personal = {k: v for k, v in clean.items() if k not in keys}
|
| 215 |
+
if shared:
|
| 216 |
+
shared_overlay.put_cells(TABLE_KEY, pid, shared, st=self.st)
|
| 217 |
+
if personal:
|
| 218 |
+
super().patch_overlay(username, pid, personal)
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
TABLE_OPS = _ProductTableStore(TABLE_KEY)
|
| 222 |
+
|
| 223 |
+
|
| 224 |
+
def shared_cells(pids, st=None):
|
| 225 |
+
"""`{"<pid>": {key: value}}` for the SHARED columns of the rows named by `pids`.
|
| 226 |
+
|
| 227 |
+
β `pids` is required and positional all the way down β `shared_overlay.cells` refuses to serve
|
| 228 |
+
"everything" by design, and the caller here always holds an already-scoped pool.
|
| 229 |
+
"""
|
| 230 |
+
return shared_overlay.cells(TABLE_KEY, pids, st=st if st is not None else TABLE_OPS.st)
|
| 231 |
+
|
| 232 |
+
|
| 233 |
+
def sku_pid(code):
|
| 234 |
+
"""A stable integer id for a SKU code. CRC32, masked to 31 bits so it is always positive and
|
| 235 |
+
always JSON-safe. Stable across processes and pulls, which an enumeration index is not."""
|
| 236 |
+
return zlib.crc32(str(code).encode("utf-8")) & 0x7FFFFFFF
|
| 237 |
+
|
| 238 |
+
|
| 239 |
+
def _assert_no_pid_collision(rows):
|
| 240 |
+
"""Two SKUs sharing a pid would MERGE in every set operation downstream β cohort membership,
|
| 241 |
+
allowed_pids, the measure channel β and nothing would report it. Fail the build instead."""
|
| 242 |
+
seen = {}
|
| 243 |
+
for r in rows:
|
| 244 |
+
prior = seen.get(r["pid"])
|
| 245 |
+
if prior is not None and prior != r["code"]:
|
| 246 |
+
raise ValueError(
|
| 247 |
+
f"product_data: pid collision β {prior!r} and {r['code']!r} both hash to "
|
| 248 |
+
f"{r['pid']}. Downstream set operations would merge them silently; widen the id "
|
| 249 |
+
f"before shipping this catalogue.")
|
| 250 |
+
seen[r["pid"]] = r["code"]
|
| 251 |
+
|
| 252 |
+
|
| 253 |
+
def _inventory_by_code(t):
|
| 254 |
+
"""`{code: {...}}` from the inventory module, or `{}` if it cannot be read.
|
| 255 |
+
|
| 256 |
+
Degrades to empty rather than raising, matching `customer_data._pool_build`'s treatment of
|
| 257 |
+
its own slow families: a product table that will not render because inventory is momentarily
|
| 258 |
+
unreachable is a worse failure than one with blank stock columns.
|
| 259 |
+
"""
|
| 260 |
+
try:
|
| 261 |
+
import modules.inventory as inventory
|
| 262 |
+
return inventory.sku_inventory(t=t) or {}
|
| 263 |
+
except Exception:
|
| 264 |
+
return {}
|
| 265 |
+
|
| 266 |
+
|
| 267 |
+
def _bu_ltm_share(t, team_id):
|
| 268 |
+
"""`{code: 0.0..1.0}` β this unit's SHARE of the SKU's last-twelve-months units.
|
| 269 |
+
|
| 270 |
+
β OWNER 2026-08-11: *"scope any inventory with Sales from Fisch"*. The velocity half of the
|
| 271 |
+
inventory block has to be re-shaped by a BU fact, and this is that fact.
|
| 272 |
+
|
| 273 |
+
β A SHARE, NOT THE UNIT COUNT ITSELF β AND THE REASON IS A MEASURED IMPOSSIBILITY. My first
|
| 274 |
+
version returned `products._sku_rev(...)['qty']` and used it directly as the scoped `qty_ltm`,
|
| 275 |
+
leaving the consolidated column on `inventory.sku_inventory`'s own figure. Two readers of one
|
| 276 |
+
question ([[one-question-two-normalizers]]): they disagree, and on 5 SKUs of 5,871 the live
|
| 277 |
+
check found **Fisch's LTM units EXCEEDING the company's** β a subset larger than its superset,
|
| 278 |
+
which no reader could explain and no reconciliation could survive.
|
| 279 |
+
|
| 280 |
+
Both halves of the ratio come from the SAME read here, so the share is in [0, 1] by
|
| 281 |
+
construction and the scoped figure can never exceed the consolidated one. It also leaves the
|
| 282 |
+
consolidated column exactly as it was β the Inventory page and the Product grid still agree
|
| 283 |
+
about company-wide units, which is what "leave the rest" asked for.
|
| 284 |
+
|
| 285 |
+
`all_qty == 0` implies `bu_qty == 0` (same reader), so a share of 0 is the honest answer for
|
| 286 |
+
"this unit never sold it": `_bucket` turns that into 'No recent sales', not zero cover.
|
| 287 |
+
|
| 288 |
+
Degrades to `{}` on any failure, matching `_inventory_by_code` β and a missing code then takes
|
| 289 |
+
the `_DEFAULT_SHARE` below rather than a silent 0.
|
| 290 |
+
"""
|
| 291 |
+
try:
|
| 292 |
+
f, to = P.ltm(t)
|
| 293 |
+
allq = {code: (r.get('qty') or 0.0)
|
| 294 |
+
for code, r in (products._sku_rev(f, to, None) or {}).items()}
|
| 295 |
+
buq = {code: (r.get('qty') or 0.0)
|
| 296 |
+
for code, r in (products._sku_rev(f, to, team_id) or {}).items()}
|
| 297 |
+
except Exception:
|
| 298 |
+
return {}
|
| 299 |
+
out = {}
|
| 300 |
+
for code, total in allq.items():
|
| 301 |
+
out[code] = min(1.0, max(0.0, (buq.get(code, 0.0) / total))) if total > 0 else 0.0
|
| 302 |
+
return out
|
| 303 |
+
|
| 304 |
+
|
| 305 |
+
#: What a SKU absent from the LTM sales read is worth to a business unit. ZERO β it did not sell
|
| 306 |
+
#: in anybody's book over the window, so no unit can claim its velocity. Named rather than
|
| 307 |
+
#: inlined so the choice is visible: the alternative (1.0, "assume it is all ours") would print
|
| 308 |
+
#: company-wide cover on a BU grid, which is the mixed-scope defect this whole rule avoids.
|
| 309 |
+
_DEFAULT_SHARE = 0.0
|
| 310 |
+
|
| 311 |
+
|
| 312 |
+
def _rescope_inventory(e, share):
|
| 313 |
+
"""`(qty_ltm, dos, bucket)` recomputed for ONE business unit's sales rate.
|
| 314 |
+
|
| 315 |
+
The formulas are `inventory.sku_inventory`'s, applied to a BU-shaped numerator β NOT a second
|
| 316 |
+
idea of what days-of-supply means. `_bucket` is imported from there for the same reason: two
|
| 317 |
+
copies of a threshold table is how the Product grid and the Inventory page start disagreeing
|
| 318 |
+
about which SKUs are dead.
|
| 319 |
+
|
| 320 |
+
β `on_hand` is whatever the warehouse holds, unscoped β so a Fisch reader's `dos` answers
|
| 321 |
+
"how long does ALL our stock last at Fisch's rate", which is the question a Fisch salesperson
|
| 322 |
+
actually has. It is deliberately NOT a pro-rated share of the shelf: there is no such shelf,
|
| 323 |
+
and inventing one would put a number on screen that no Odoo query could reproduce.
|
| 324 |
+
"""
|
| 325 |
+
on_hand = e.get("on_hand")
|
| 326 |
+
if on_hand is None:
|
| 327 |
+
return None, None, None # no inventory row for this SKU: blank, never zero
|
| 328 |
+
qty = float(e.get("qty_ltm") or 0.0) * float(share or 0.0)
|
| 329 |
+
daily = qty / 365.0
|
| 330 |
+
if daily > 0:
|
| 331 |
+
dos_raw = on_hand / daily
|
| 332 |
+
else:
|
| 333 |
+
dos_raw = float('inf') if on_hand > 0 else 0.0
|
| 334 |
+
try:
|
| 335 |
+
import modules.inventory as inventory
|
| 336 |
+
bucket = inventory._bucket(dos_raw, on_hand, qty)
|
| 337 |
+
except Exception:
|
| 338 |
+
bucket = None
|
| 339 |
+
return qty, (None if dos_raw == float('inf') else round(float(dos_raw), 0)), bucket
|
| 340 |
+
|
| 341 |
+
|
| 342 |
+
def _catalogue_by_code():
|
| 343 |
+
"""`{code: {'product', 'category'}}` β the CATALOGUE universe this pool is built from.
|
| 344 |
+
|
| 345 |
+
β UNLIKE `_inventory_by_code`, THIS ONE RAISES, and the asymmetry is the whole point.
|
| 346 |
+
Inventory degrades to `{}` because a product table with blank stock columns beats one that
|
| 347 |
+
will not render. The CATALOGUE is not a column β it is the ROW SET. A catalogue read that
|
| 348 |
+
failed quietly would drop the grid straight back to the sold-only 2,717 with every gate
|
| 349 |
+
green and nothing on screen saying so, which is the defect this seam exists to end
|
| 350 |
+
([[gate-can-report-green-on-nothing]]). `routes_products._pool_for` already turns the raise
|
| 351 |
+
into a 503 that names the cause, so the loud failure has somewhere honest to land.
|
| 352 |
+
|
| 353 |
+
It exists as a `pd`-level function rather than an inline `products.catalogue()` call for the
|
| 354 |
+
same reason `_inventory_by_code` does: it is the seam `verify_perm_scope`'s section H stubs
|
| 355 |
+
to build a pool without Odoo.
|
| 356 |
+
"""
|
| 357 |
+
return products.catalogue()
|
| 358 |
+
|
| 359 |
+
|
| 360 |
+
def _pricelist_by_code():
|
| 361 |
+
"""`({code: {price_*: price}}, report)` from `products.pricelist_by_code`, or `({}, β¦)`.
|
| 362 |
+
|
| 363 |
+
A SEAM for the same two reasons `_inventory_by_code` is one: it degrades rather than raises
|
| 364 |
+
(these are columns, not the row set), and `verify_perm_scope`'s section H stubs it to build a
|
| 365 |
+
pool without Odoo. β Unstubbed there, section H would reach live Odoo through the back door
|
| 366 |
+
and the whole file would stop being runnable offline.
|
| 367 |
+
"""
|
| 368 |
+
try:
|
| 369 |
+
return products.pricelist_by_code()
|
| 370 |
+
except Exception as e:
|
| 371 |
+
return {}, {"error": f"{type(e).__name__}: {str(e)[:200]}"}
|
| 372 |
+
|
| 373 |
+
|
| 374 |
+
def pool(team_id=None, t=None):
|
| 375 |
+
"""One row per ACTIVE product β the PRODUCT analogue of `customer_data.pool`.
|
| 376 |
+
|
| 377 |
+
CATALOGUE-FIRST, REVENUE LEFT-JOINED (R12 β see decision 4 in the module header). The row set
|
| 378 |
+
is `products.catalogue()`; `products.directory()` supplies the revenue columns for the SKUs
|
| 379 |
+
that sold in its window and contributes NO rows of its own.
|
| 380 |
+
|
| 381 |
+
`team_id` shapes the revenue columns exactly as it does for customers (`products.directory`
|
| 382 |
+
passes it into `_sku_rev`), which is why `core.perm_scope.derive_pool_scope` must keep
|
| 383 |
+
driving it rather than a post-filter deciding the BU. It does NOT shape the row set: a
|
| 384 |
+
catalogue has no team.
|
| 385 |
+
"""
|
| 386 |
+
t = t or P.today()
|
| 387 |
+
consolidated = team_id is None
|
| 388 |
+
# β OWNER 2026-08-11: read inventory on EVERY pull, not just a consolidated one. The stock
|
| 389 |
+
# itself is company-wide; only the sales-derived half is re-scoped, by `_rescope_inventory`.
|
| 390 |
+
inv = _inventory_by_code(t)
|
| 391 |
+
bu_share = {} if consolidated else _bu_ltm_share(t, team_id)
|
| 392 |
+
sup = supplier_master()
|
| 393 |
+
prices, _price_report = _pricelist_by_code()
|
| 394 |
+
cat = _catalogue_by_code()
|
| 395 |
+
# The LEFT side of the join, indexed by the same code key. β A code here that the catalogue
|
| 396 |
+
# does not carry belongs to a product no ACTIVE record claims β archived, and R12 keeps those
|
| 397 |
+
# out. `validate()` asserts that of Odoo rather than assuming it, and reports the revenue
|
| 398 |
+
# that therefore sits outside the grid (MEASURED 2026-08-11: 3 codes, $0.00 YTD / $294.50 LY).
|
| 399 |
+
rev = {r["code"]: r for r in products.directory(t=t, team_id=team_id)}
|
| 400 |
+
|
| 401 |
+
rows = []
|
| 402 |
+
for code, meta in cat.items():
|
| 403 |
+
r = rev.get(code)
|
| 404 |
+
row = {
|
| 405 |
+
"pid": sku_pid(code),
|
| 406 |
+
"code": code,
|
| 407 |
+
# β ONE source for the name and the category, the CATALOGUE β not the sale line's
|
| 408 |
+
# m2o. For a re-SKUed code the line's name can be the ARCHIVED record's; the active
|
| 409 |
+
# record's `display_name` is the current truth, and it is the same string for every
|
| 410 |
+
# SKU that is not re-SKUed. `directory()` derives the category identically.
|
| 411 |
+
"product": meta.get("product") or code,
|
| 412 |
+
"category": meta.get("category") or "(uncategorized)",
|
| 413 |
+
# ββ W33-T43 (R2 / amendment A2) β ODOO'S OWN PRODUCT ID, beside the hashed `pid`.
|
| 414 |
+
# R2 retires `ut_odoo_products` onto this key and keeps every data column it had; this
|
| 415 |
+
# is that column. β NOT DERIVABLE DOWNSTREAM: `pid` is `crc32(default_code)` here,
|
| 416 |
+
# unlike a customer row whose `pid` IS the partner id, so `aios_grid.py` cannot recover
|
| 417 |
+
# it and it has to be carried from `products.catalogue()`.
|
| 418 |
+
# β `None`, never 0, when the catalogue somehow has no id β 0 is a real Odoo id.
|
| 419 |
+
"product_id": meta.get("id"),
|
| 420 |
+
# β R12 β BLANK, NEVER $0, on a SKU no revenue read ever saw. A row that IS in `rev`
|
| 421 |
+
# keeps its measured numbers including a real 0.0 (it sold last year, not this one).
|
| 422 |
+
"rev_ytd": r.get("rev_ytd", 0.0) if r else None,
|
| 423 |
+
"rev_ly": r.get("rev_ly", 0.0) if r else None,
|
| 424 |
+
"yoy_pct": r.get("yoy_pct") if r else None,
|
| 425 |
+
"qty_ytd": r.get("qty_ytd", 0.0) if r else None,
|
| 426 |
+
"orders_ytd": r.get("orders_ytd", 0) if r else None,
|
| 427 |
+
}
|
| 428 |
+
# β WAVE 30 W30-T34 β the PRICELIST stratum, one column per declared list. Catalogue
|
| 429 |
+
# data like the supplier block, so it rides every pull, scoped or not: a price book is
|
| 430 |
+
# not a thing a business unit owns a slice of, and a Fisch reader who cannot see the
|
| 431 |
+
# Fisch price is exactly who this column is for.
|
| 432 |
+
#
|
| 433 |
+
# β `None`, NEVER 0 AND NEVER A FALLBACK, for the 1,883 SKUs no list prices. That is
|
| 434 |
+
# W29-T52's own negative control: the single `3_global` rule computes over `list_price`,
|
| 435 |
+
# which is 1.00 on 5,817 of 5,871 products, so the fallback is not a cheaper answer β
|
| 436 |
+
# it is a wrong one wearing a currency sign.
|
| 437 |
+
p = prices.get(code) or {}
|
| 438 |
+
for _col, _name in products.PRICELIST_COLUMNS:
|
| 439 |
+
row[_col] = p.get(_col)
|
| 440 |
+
# Wave 17 R3 β the supplier master, on every pull (it is catalogue data, not stock).
|
| 441 |
+
s = sup.get(code) or {}
|
| 442 |
+
row.update({
|
| 443 |
+
"supplier": s.get("supplier") or "(none)",
|
| 444 |
+
"lead_days": s.get("lead_days"),
|
| 445 |
+
"origin_country": s.get("origin_country") or "(none)",
|
| 446 |
+
"first_cost": s.get("first_cost"),
|
| 447 |
+
})
|
| 448 |
+
e = inv.get(code) or {}
|
| 449 |
+
if consolidated:
|
| 450 |
+
qty_ltm, dos, bucket = e.get("qty_ltm"), e.get("dos"), e.get("bucket")
|
| 451 |
+
else:
|
| 452 |
+
qty_ltm, dos, bucket = _rescope_inventory(e, bu_share.get(code, _DEFAULT_SHARE))
|
| 453 |
+
row.update({
|
| 454 |
+
# UNSCOPED on purpose (owner 2026-08-11): one warehouse, no per-brand shelf.
|
| 455 |
+
"on_hand": e.get("on_hand"),
|
| 456 |
+
"unit_cost": e.get("unit_cost"),
|
| 457 |
+
"inv_value": e.get("inv_value"),
|
| 458 |
+
# SCOPED: these three are functions of how fast THIS unit sells the SKU.
|
| 459 |
+
"qty_ltm": qty_ltm,
|
| 460 |
+
"dos": dos,
|
| 461 |
+
"stock_bucket": bucket,
|
| 462 |
+
})
|
| 463 |
+
# THE BUY TRIGGER (R3), ported from `modules/procurement`'s lead-time-cover rule:
|
| 464 |
+
# buy when the shelf runs out before a reorder could land.
|
| 465 |
+
#
|
| 466 |
+
# β BLANK, NOT "OK", WHEN EITHER INPUT IS MISSING. 339 SKUs have no supplier and 400
|
| 467 |
+
# no lead time; `dos` is null for anything that never sells through. "We don't know"
|
| 468 |
+
# and "you're fine" are different sentences, and only one of them is safe to print
|
| 469 |
+
# beside a purchasing decision.
|
| 470 |
+
#
|
| 471 |
+
# β `buy_now` SHIPS NOWHERE ANY MORE, and is computed anyway. OWNER 2026-08-03: it
|
| 472 |
+
# is not a preset field β it is a formula over the two columns beside it, and the
|
| 473 |
+
# platform has a formula field type for exactly that. Its Field is gone from
|
| 474 |
+
# `aios_grid_fields.json`, and `rows_from_pool` projects rows STRICTLY through that
|
| 475 |
+
# contract, so no Field means no cell on the wire. Nothing renders this key.
|
| 476 |
+
#
|
| 477 |
+
# It stays computed because deleting it would delete the PROOF. `validate()` below
|
| 478 |
+
# reconciles the formula's predicate against this one, row for row, which is the only
|
| 479 |
+
# thing that makes "the same exact figures" a claim rather than an assertion β and
|
| 480 |
+
# this repo has the scar already (wave 17: archiving `ar` "would have DELETED the
|
| 481 |
+
# proof"). Two comparisons per SKU is what that costs.
|
| 482 |
+
#
|
| 483 |
+
# β READS THE SCOPED `dos`, NOT `e['dos']`. On a Fisch pull the cover gap must answer
|
| 484 |
+
# "does the shelf outlast a reorder AT FISCH'S RATE" β reading the consolidated figure
|
| 485 |
+
# here would print a buy signal computed from both units' velocity beside a days-of-supply
|
| 486 |
+
# computed from one, and the two columns would disagree on the same row.
|
| 487 |
+
lead = s.get("lead_days")
|
| 488 |
+
if isinstance(dos, (int, float)) and isinstance(lead, (int, float)) and lead > 0:
|
| 489 |
+
row["cover_gap_d"] = int(round(dos - lead))
|
| 490 |
+
row["buy_now"] = "Buy now" if dos < lead else "OK"
|
| 491 |
+
else:
|
| 492 |
+
row["cover_gap_d"] = None
|
| 493 |
+
row["buy_now"] = None
|
| 494 |
+
rows.append(row)
|
| 495 |
+
|
| 496 |
+
# `or 0.0` reads a BLANK as zero FOR SORTING ONLY β never for the cell. A never-sold SKU
|
| 497 |
+
# settles among the zero-revenue ones at the bottom, which is where it belongs; the stored
|
| 498 |
+
# value stays None so nothing downstream can mistake "we never saw a sale" for "we measured
|
| 499 |
+
# nothing sold".
|
| 500 |
+
rows.sort(key=lambda r: -(r["rev_ytd"] or 0.0))
|
| 501 |
+
_assert_no_pid_collision(rows)
|
| 502 |
+
return rows
|
| 503 |
+
|
| 504 |
+
|
| 505 |
+
#: β THE BUY SIGNAL, AS THE FORMULA FIELD EVALUATES IT (owner, 2026-08-03).
|
| 506 |
+
#:
|
| 507 |
+
#: The owner's ruling was that "Buy signal" is not a preset field β it is a formula over two
|
| 508 |
+
#: columns the product table already carries, and the platform has a formula field type for it.
|
| 509 |
+
#: The formula, verbatim, is what `_seed_wave17.BUY_SIGNAL_FORMULA` creates and what the JSON
|
| 510 |
+
#: contract's `_product_removed_buy_now` note records:
|
| 511 |
+
#:
|
| 512 |
+
#: IF({lead_days} > 0, IF({dos} < {lead_days}, "Buy now", "OK"), "")
|
| 513 |
+
#:
|
| 514 |
+
#: This function is a PORT of how `customer-grid/formulaEngine.ts` evaluates that tree, not a
|
| 515 |
+
#: restatement of the business rule β that is the whole point, because the two could drift and
|
| 516 |
+
#: `validate()` is where the drift must show. Three engine behaviours it reproduces exactly:
|
| 517 |
+
#:
|
| 518 |
+
#: Β· a `ref` to a missing/non-numeric cell is None (`case "ref"` returns null for anything
|
| 519 |
+
#: that is not a finite number or a string);
|
| 520 |
+
#: Β· `cmp` returns BLANK unless BOTH sides read as numbers β it never coerces a blank to 0,
|
| 521 |
+
#: which is the difference between this and a filter engine's `toNum(null) === 0`;
|
| 522 |
+
#: Β· `IF` with a non-boolean condition returns BLANK ("no truthiness"), so a blank comparison
|
| 523 |
+
#: propagates out as a blank cell rather than taking the false branch.
|
| 524 |
+
BUY_SIGNAL_FORMULA = 'IF({lead_days} > 0, IF({dos} < {lead_days}, "Buy now", "OK"), "")'
|
| 525 |
+
|
| 526 |
+
|
| 527 |
+
def _buy_signal_formula(row):
|
| 528 |
+
"""Evaluate `BUY_SIGNAL_FORMULA` over one pool row -> 'Buy now' | 'OK' | '' (blank)."""
|
| 529 |
+
def num(v):
|
| 530 |
+
# The engine's `ref` + `asNumber`: booleans are not values here, and a non-finite
|
| 531 |
+
# number is null. `isinstance(True, int)` is True in Python, so bool is excluded first.
|
| 532 |
+
if isinstance(v, bool) or not isinstance(v, (int, float)):
|
| 533 |
+
return None
|
| 534 |
+
return v if v == v and v not in (float('inf'), float('-inf')) else None
|
| 535 |
+
|
| 536 |
+
lead, dos = num(row.get("lead_days")), num(row.get("dos"))
|
| 537 |
+
if lead is None: # `{lead_days} > 0` is blank -> IF(blank, β¦) is blank
|
| 538 |
+
return ""
|
| 539 |
+
if not lead > 0:
|
| 540 |
+
return "" # the formula's own else-branch
|
| 541 |
+
if dos is None: # `{dos} < {lead_days}` is blank -> IF(blank, β¦) is blank
|
| 542 |
+
return ""
|
| 543 |
+
return "Buy now" if dos < lead else "OK"
|
| 544 |
+
|
| 545 |
+
|
| 546 |
+
def validate(team_id=None, t=None):
|
| 547 |
+
"""Reconcile what SHIPS to an independent aggregate β the platform's own rule that a number
|
| 548 |
+
which does not tie to Odoo does not ship.
|
| 549 |
+
|
| 550 |
+
ββ THE POPULATION LEG IS NEW, AND IT IS HERE BECAUSE THIS FUNCTION USED TO BE SELF-SEALING
|
| 551 |
+
(wave 29, contract C8). It reconciled Ξ£ per-SKU YTD revenue against `products._sku_rev` β the
|
| 552 |
+
same function `pool()` built its rows from. BOTH SIDES CAME FROM THE JOIN, so the oracle could
|
| 553 |
+
not see a missing row, and a grid holding 2,717 of 5,875 active products passed for a wave.
|
| 554 |
+
No gate anywhere asserted a pool row COUNT against an independent Odoo `search_count` either.
|
| 555 |
+
That is the leg below, and it is the control that would have caught it
|
| 556 |
+
([[no-unverifiable-aggregates]]).
|
| 557 |
+
|
| 558 |
+
β The revenue leg was NOT deleted with it β it answers a different question ("did the join
|
| 559 |
+
lose money?") and is still the right shape for that one. What changed is that it now has to
|
| 560 |
+
account for revenue belonging to codes NO ACTIVE PRODUCT CARRIES, because R12 keeps archived
|
| 561 |
+
products out of the grid; the total is decomposed rather than compared loosely, so a growing
|
| 562 |
+
"outside the grid" figure shows up as a number rather than as slack in a tolerance.
|
| 563 |
+
|
| 564 |
+
Still deliberately NOT covered, and said rather than implied: the inventory columns are a
|
| 565 |
+
different module's reconciliation, and they are absent on a scoped pull anyway.
|
| 566 |
+
"""
|
| 567 |
+
t = t or P.today()
|
| 568 |
+
rows = pool(team_id=team_id, t=t)
|
| 569 |
+
|
| 570 |
+
# ββ THE POPULATION, AGAINST AN ORACLE THAT CANNOT SEE OUR JOIN. A bare Odoo count of active
|
| 571 |
+
# products, asked fresh β not a `len()` over anything this module or `products.catalogue()`
|
| 572 |
+
# built. NC: drop one row from `pool()` and this goes red; that is the whole point of it.
|
| 573 |
+
n_active = products.catalogue_count()
|
| 574 |
+
checks = [{
|
| 575 |
+
"check": "the pool holds one row per ACTIVE product (R12) β row count == an INDEPENDENT "
|
| 576 |
+
"Odoo search_count('product.product', active=True)",
|
| 577 |
+
"ours": len(rows), "theirs": n_active, "ok": len(rows) == n_active,
|
| 578 |
+
}]
|
| 579 |
+
|
| 580 |
+
yf, yt = P.ytd(t)
|
| 581 |
+
sku_rev = products._sku_rev(yf, yt, team_id) or {}
|
| 582 |
+
in_pool = {r["code"] for r in rows}
|
| 583 |
+
outside = {c: v for c, v in sku_rev.items() if c not in in_pool}
|
| 584 |
+
ours = round(sum(r["rev_ytd"] or 0.0 for r in rows), 2)
|
| 585 |
+
theirs = round(sum(v.get("rev", 0.0) for v in sku_rev.values()), 2)
|
| 586 |
+
dropped = round(sum(v.get("rev", 0.0) for v in outside.values()), 2)
|
| 587 |
+
checks.append({
|
| 588 |
+
"check": "Ξ£ per-SKU YTD revenue in the grid + Ξ£ YTD revenue of codes no ACTIVE product "
|
| 589 |
+
"carries == modules.products._sku_rev over the same window",
|
| 590 |
+
"ours": round(ours + dropped, 2), "theirs": theirs,
|
| 591 |
+
"ok": abs((ours + dropped) - theirs) < 0.01,
|
| 592 |
+
"detail": {"in_the_grid": ours, "outside_the_grid": dropped,
|
| 593 |
+
"codes_outside": len(outside)},
|
| 594 |
+
})
|
| 595 |
+
# β AND THE EXCLUSION IS ASSERTED, NOT ASSUMED. "Everything I dropped was archived" is
|
| 596 |
+
# trivially true when checked against the dict that did the dropping β that is the
|
| 597 |
+
# self-sealing shape all over again. So it is asked of ODOO. A code here that a live ACTIVE
|
| 598 |
+
# product carries means `products.catalogue()` missed a row that has revenue, which is the
|
| 599 |
+
# 2,717 defect returning in a smaller costume.
|
| 600 |
+
# β `pid:N` keys are not `default_code`s and are skipped: an UNCODED active product is in the
|
| 601 |
+
# catalogue under its own `pid:N` key and therefore cannot be in `outside` at all.
|
| 602 |
+
coded_outside = sorted(c for c in outside if not str(c).startswith("pid:"))
|
| 603 |
+
still_active = (O.get_odoo().search_count(
|
| 604 |
+
'product.product', [('active', '=', True), ('default_code', 'in', coded_outside)])
|
| 605 |
+
if coded_outside else 0)
|
| 606 |
+
checks.append({
|
| 607 |
+
"check": "every SKU with revenue but NO grid row is genuinely ARCHIVED (R12: archived "
|
| 608 |
+
"stay out) β asked of Odoo, never of the catalogue that did the dropping",
|
| 609 |
+
"ours": still_active, "theirs": 0, "ok": still_active == 0,
|
| 610 |
+
"detail": {"codes": coded_outside[:10], "n_codes": len(outside)},
|
| 611 |
+
})
|
| 612 |
+
# ββ ββ WAVE 30 W30-T34 β THE PRICELIST COLUMNS, AGAINST ORACLES THAT CANNOT SEE OUR JOIN ββ
|
| 613 |
+
#
|
| 614 |
+
# Three legs, because the column can fail in three different ways and only one of them is a
|
| 615 |
+
# count. `products.pricelist_by_code` builds a `{code: {col: price}}` map by expanding each
|
| 616 |
+
# rule over the products it names, IN PYTHON β that expansion is the fragile part, so every
|
| 617 |
+
# leg below re-asks ODOO instead of re-reading the map.
|
| 618 |
+
price_report = _pricelist_by_code()[1]
|
| 619 |
+
_pl_rows = O.search_read('product.pricelist', [], ['id', 'name'])
|
| 620 |
+
_pl_id = {}
|
| 621 |
+
for _p in _pl_rows:
|
| 622 |
+
_pl_id.setdefault(str(_p.get('name') or '').strip(), _p['id'])
|
| 623 |
+
_today = P.today().isoformat()
|
| 624 |
+
|
| 625 |
+
_targets_memo = {}
|
| 626 |
+
|
| 627 |
+
def _rule_targets(list_name):
|
| 628 |
+
"""(variant_ids, template_ids) a pricelist prices today β read FRESH from Odoo.
|
| 629 |
+
|
| 630 |
+
Memoised for the LIFE OF THIS CALL only: legs 1 and 2 both need all three lists, and
|
| 631 |
+
without this `validate()` makes six identical round trips instead of three (measured:
|
| 632 |
+
~48s of the run). Deliberately NOT an `lru_cache` β an oracle that survives the process
|
| 633 |
+
is an oracle reading yesterday's Odoo.
|
| 634 |
+
"""
|
| 635 |
+
if list_name in _targets_memo:
|
| 636 |
+
return _targets_memo[list_name]
|
| 637 |
+
var, tmpl = set(), set()
|
| 638 |
+
plid = _pl_id.get(list_name)
|
| 639 |
+
if plid is None:
|
| 640 |
+
return var, tmpl
|
| 641 |
+
for r in O.search_read(
|
| 642 |
+
'product.pricelist.item',
|
| 643 |
+
[('pricelist_id', '=', plid), ('compute_price', '=', 'fixed'),
|
| 644 |
+
('applied_on', 'in', ['0_product_variant', '1_product'])],
|
| 645 |
+
['product_id', 'product_tmpl_id', 'applied_on', 'fixed_price',
|
| 646 |
+
'date_start', 'date_end']):
|
| 647 |
+
ds, de = str(r.get('date_start') or '')[:10], str(r.get('date_end') or '')[:10]
|
| 648 |
+
if (ds and ds > _today) or (de and de < _today):
|
| 649 |
+
continue
|
| 650 |
+
fp = r.get('fixed_price')
|
| 651 |
+
if not isinstance(fp, (int, float)) or fp <= 0:
|
| 652 |
+
continue
|
| 653 |
+
if r.get('applied_on') == '0_product_variant' and r.get('product_id'):
|
| 654 |
+
var.add(O.m2o_id(r['product_id']))
|
| 655 |
+
elif r.get('product_tmpl_id'):
|
| 656 |
+
tmpl.add(O.m2o_id(r['product_tmpl_id']))
|
| 657 |
+
_targets_memo[list_name] = (var, tmpl)
|
| 658 |
+
return var, tmpl
|
| 659 |
+
|
| 660 |
+
# LEG 1 β COVERAGE PER LIST. Ours: cells we filled. Theirs: a bare Odoo `search_count` of
|
| 661 |
+
# ACTIVE products a fresh read of that list's rules reaches. The rule set is shared (it IS
|
| 662 |
+
# the data) but the EXPANSION is not, and the expansion is what breaks.
|
| 663 |
+
for _col, _name in products.PRICELIST_COLUMNS:
|
| 664 |
+
_var, _tmpl = _rule_targets(_name)
|
| 665 |
+
_dom = [('active', '=', True), '|', ('id', 'in', sorted(_var)),
|
| 666 |
+
('product_tmpl_id', 'in', sorted(_tmpl))]
|
| 667 |
+
theirs = O.get_odoo().search_count('product.product', _dom) if (_var or _tmpl) else 0
|
| 668 |
+
ours = sum(1 for r in rows if isinstance(r.get(_col), (int, float)))
|
| 669 |
+
checks.append({
|
| 670 |
+
"check": f"{_name} pricelist: SKUs priced in the grid == an INDEPENDENT Odoo count "
|
| 671 |
+
f"of ACTIVE products its date-valid fixed rules reach",
|
| 672 |
+
"ours": ours, "theirs": theirs, "ok": ours == theirs,
|
| 673 |
+
"detail": {"column": _col, "variant_rules": len(_var), "template_rules": len(_tmpl)},
|
| 674 |
+
})
|
| 675 |
+
|
| 676 |
+
# LEG 2 β β THE TICKET'S OWN NEGATIVE CONTROL, AS A LEG. A SKU with no specific item must
|
| 677 |
+
# render BLANK. Asked of ODOO, never of the map that produced the blank β "everything I left
|
| 678 |
+
# empty was genuinely unpriced" is trivially true when checked against the dict that emptied
|
| 679 |
+
# it, which is the self-sealing shape the population leg above exists to end.
|
| 680 |
+
_unpriced = [r for r in rows
|
| 681 |
+
if not any(isinstance(r.get(c), (int, float))
|
| 682 |
+
for c, _n in products.PRICELIST_COLUMNS)]
|
| 683 |
+
_codes = {r["code"] for r in _unpriced}
|
| 684 |
+
_stray = 0
|
| 685 |
+
if _codes:
|
| 686 |
+
# Resolve those codes back to Odoo ids and ask whether ANY declared list prices them.
|
| 687 |
+
_ids, _tmpls = set(), set()
|
| 688 |
+
for _p in O.search_read('product.product',
|
| 689 |
+
[('active', '=', True),
|
| 690 |
+
('default_code', 'in', sorted(c for c in _codes
|
| 691 |
+
if not c.startswith("pid:")))],
|
| 692 |
+
['id', 'product_tmpl_id']):
|
| 693 |
+
_ids.add(_p['id'])
|
| 694 |
+
_tmpls.add(O.m2o_id(_p.get('product_tmpl_id')))
|
| 695 |
+
for _col, _name in products.PRICELIST_COLUMNS:
|
| 696 |
+
_v, _t = _rule_targets(_name)
|
| 697 |
+
_stray += len((_v & _ids) | ({t for t in _tmpls if t in _t}))
|
| 698 |
+
checks.append({
|
| 699 |
+
"check": "every SKU rendering a BLANK price is genuinely unpriced on all three declared "
|
| 700 |
+
"lists (W29-T52's NC: no fallback dressed as a price) β asked of Odoo",
|
| 701 |
+
"ours": _stray, "theirs": 0, "ok": _stray == 0,
|
| 702 |
+
"detail": {"blank_skus": len(_unpriced), "priced_skus": len(rows) - len(_unpriced)},
|
| 703 |
+
})
|
| 704 |
+
|
| 705 |
+
# LEG 3 β β R6's SECOND SENTENCE, AS A NUMBER. *"If there is lag or it can't be done, you
|
| 706 |
+
# need to explicitly tell me why and recommend a fix."* Everything the reader cannot see is
|
| 707 |
+
# counted here rather than dropped. The ASSERTION is the one thing that must never be true β
|
| 708 |
+
# a price cell that is present and not a positive number, i.e. a "free" SKU β while the
|
| 709 |
+
# rest rides `detail` so it is reported without reddening an honest day's data.
|
| 710 |
+
_bad = [r["code"] for r in rows
|
| 711 |
+
for c, _n in products.PRICELIST_COLUMNS
|
| 712 |
+
if r.get(c) is not None and not (isinstance(r.get(c), (int, float)) and r[c] > 0)]
|
| 713 |
+
# β MEASURED 2026-08-12 AND NOT FIXABLE FROM THIS FENCE: `aios_grid.rows_from_pool` sends
|
| 714 |
+
# every non-text field through `_round(v)` = `round(v)` with no ndigits, i.e. to a whole
|
| 715 |
+
# dollar. 52.1% of fixed prices carry cents and the median relative error is 5.26%, so this
|
| 716 |
+
# column ties to the cent HERE and ships rounded. Reported, never silently enforced.
|
| 717 |
+
_cents = sum(1 for r in rows for c, _n in products.PRICELIST_COLUMNS
|
| 718 |
+
if isinstance(r.get(c), (int, float)) and abs(r[c] - round(r[c])) > 1e-9)
|
| 719 |
+
_to_zero = sum(1 for r in rows for c, _n in products.PRICELIST_COLUMNS
|
| 720 |
+
if isinstance(r.get(c), (int, float)) and round(r[c]) == 0)
|
| 721 |
+
checks.append({
|
| 722 |
+
"check": "no price cell is present-but-not-a-positive-number (a 0 would read as FREE; "
|
| 723 |
+
"an unset rule must be blank), and what the reader cannot see is COUNTED",
|
| 724 |
+
"ours": len(_bad), "theirs": 0, "ok": not _bad,
|
| 725 |
+
"detail": {"bad_cells": _bad[:10], "reader_report": price_report,
|
| 726 |
+
"wire_rounding_loses_cents_on": _cents,
|
| 727 |
+
"wire_rounding_to_zero_dollars": _to_zero},
|
| 728 |
+
})
|
| 729 |
+
|
| 730 |
+
# WAVE 17 R3 β the BUY SIGNAL must be a total, exact partition of the catalogue, and every
|
| 731 |
+
# member of it must be re-derivable from the two columns beside it. A signal somebody buys
|
| 732 |
+
# stock on cannot be "mostly right": the failure that matters is a row that says OK because
|
| 733 |
+
# an input was missing, so the blank leg is checked as hard as the other two.
|
| 734 |
+
if team_id is None: # consolidated only β the inputs exist only there
|
| 735 |
+
buy = [r for r in rows if r.get("buy_now") == "Buy now"]
|
| 736 |
+
ok_rows = [r for r in rows if r.get("buy_now") == "OK"]
|
| 737 |
+
blank = [r for r in rows if r.get("buy_now") is None]
|
| 738 |
+
mis = sum(1 for r in buy if not (r["dos"] < r["lead_days"]))
|
| 739 |
+
mis += sum(1 for r in ok_rows if not (r["dos"] >= r["lead_days"]))
|
| 740 |
+
# A blank must be UNKNOWN β never a row we could have answered and quietly did not.
|
| 741 |
+
mis += sum(1 for r in blank
|
| 742 |
+
if isinstance(r.get("dos"), (int, float))
|
| 743 |
+
and isinstance(r.get("lead_days"), (int, float)) and r["lead_days"] > 0)
|
| 744 |
+
checks.append({
|
| 745 |
+
"check": "Buy signal partitions the catalogue (buy + ok + unknown == rows, none "
|
| 746 |
+
"misclassified)",
|
| 747 |
+
"ours": len(buy) + len(ok_rows) + len(blank) - mis, "theirs": len(rows),
|
| 748 |
+
"ok": mis == 0 and len(buy) + len(ok_rows) + len(blank) == len(rows),
|
| 749 |
+
"detail": {"buy_now": len(buy), "ok": len(ok_rows), "unknown": len(blank),
|
| 750 |
+
"misclassified": mis},
|
| 751 |
+
})
|
| 752 |
+
# β OWNER 2026-08-03 β "use the formula fields to come up to the same EXACT figures".
|
| 753 |
+
#
|
| 754 |
+
# This is that sentence, as a check. The retired preset column and the formula that
|
| 755 |
+
# replaces it must agree on every SKU, in all three states, or the replacement is not a
|
| 756 |
+
# replacement. `_buy_signal_formula` is a line-by-line port of the client formula
|
| 757 |
+
# engine's evaluation of
|
| 758 |
+
#
|
| 759 |
+
# IF({lead_days} > 0, IF({dos} < {lead_days}, "Buy now", "OK"), "")
|
| 760 |
+
#
|
| 761 |
+
# including the part that is easy to get wrong: a comparison against a BLANK is blank,
|
| 762 |
+
# never a coerced 0. (The client engine's `cmp` returns null unless both sides read as
|
| 763 |
+
# numbers, and `IF` refuses a non-boolean condition β so a missing `dos` yields "" and
|
| 764 |
+
# not "Buy now". A filter engine would have said `0 < 30` and swept in every SKU that
|
| 765 |
+
# never sells through; the formula engine does not, and this check is what holds it.)
|
| 766 |
+
#
|
| 767 |
+
# β It compares SETS OF SKUs, not counts. Two different partitions can share a shape.
|
| 768 |
+
disagree = sorted(r["code"] for r in rows
|
| 769 |
+
if (r.get("buy_now") or "") != _buy_signal_formula(r))
|
| 770 |
+
checks.append({
|
| 771 |
+
"check": 'Buy signal as a FORMULA field == the retired preset column, per SKU '
|
| 772 |
+
'(IF({lead_days} > 0, IF({dos} < {lead_days}, "Buy now", "OK"), ""))',
|
| 773 |
+
"ours": len(rows) - len(disagree), "theirs": len(rows),
|
| 774 |
+
"ok": not disagree,
|
| 775 |
+
"detail": {"disagreeing_skus": disagree[:10], "n_disagree": len(disagree)},
|
| 776 |
+
})
|
| 777 |
+
# The other half of "the same figures": the SHARED Buy list view can no longer filter on
|
| 778 |
+
# the retired column, and its replacement conditions must select the same SKUs. They are
|
| 779 |
+
# `dos isNotEmpty AND lead_days isNotEmpty AND lead_days > 0 AND dos < lead_days`
|
| 780 |
+
# (_seed_wave17.views), so this reproduces exactly that conjunction.
|
| 781 |
+
#
|
| 782 |
+
# β THE NEGATIVE CONTROL IS WHY THIS IS NOT `cover_gap_d < 0`, which reads like the
|
| 783 |
+
# obvious filter and is WRONG: `cover_gap_d` is `int(round(dos - lead))`, so a genuine
|
| 784 |
+
# gap of -0.4 days rounds to 0 and that SKU drops off a buy list it belongs on.
|
| 785 |
+
view_rows = {r["code"] for r in rows
|
| 786 |
+
if isinstance(r.get("dos"), (int, float))
|
| 787 |
+
and isinstance(r.get("lead_days"), (int, float))
|
| 788 |
+
and r["lead_days"] > 0 and r["dos"] < r["lead_days"]}
|
| 789 |
+
signal_rows = {r["code"] for r in buy}
|
| 790 |
+
rounding_would_miss = sorted(
|
| 791 |
+
c for c in signal_rows
|
| 792 |
+
if next((r for r in rows if r["code"] == c), {}).get("cover_gap_d") == 0)
|
| 793 |
+
checks.append({
|
| 794 |
+
"check": "Buy list view conditions (dos/lead_days, no retired column) select "
|
| 795 |
+
"exactly the Buy-now SKUs",
|
| 796 |
+
"ours": len(view_rows), "theirs": len(signal_rows),
|
| 797 |
+
"ok": view_rows == signal_rows,
|
| 798 |
+
"detail": {"only_in_view": sorted(view_rows - signal_rows)[:10],
|
| 799 |
+
"only_in_signal": sorted(signal_rows - view_rows)[:10],
|
| 800 |
+
# Reported, not asserted: how many SKUs a `cover_gap_d < 0` filter would
|
| 801 |
+
# have silently dropped. 0 today does not make that filter correct.
|
| 802 |
+
"cover_gap_rounds_to_zero": len(rounding_would_miss)},
|
| 803 |
+
})
|
| 804 |
+
return checks
|
platform/modules/products.py
CHANGED
|
@@ -1,737 +1,737 @@
|
|
| 1 |
-
"""Products / SKU module β SKU health: YoY movers (risers & decliners), zombie SKUs
|
| 2 |
-
(catalog rot β sellable, formerly selling, now dead), new winners, coverage collapse
|
| 3 |
-
(SKUs losing customer breadth β the FFS-recovery early-warning signal), and velocity leaders.
|
| 4 |
-
|
| 5 |
-
Per-SKU margin already lives in the Financial module (low_margin_skus); not duplicated here.
|
| 6 |
-
Basket / co-purchase (MBA) is intentionally deferred β the existing client app computes it
|
| 7 |
-
runtime-side, and a local co-occurrence pull over 74k LTM lines is the kind of heavy job the
|
| 8 |
-
project guardrails keep off the local PC. (See BACKLOG.)
|
| 9 |
-
|
| 10 |
-
All line-level (sale.order.line), RI+FFS scope, excluded accounts removed β reusing sale_line_domain.
|
| 11 |
-
Coverage (distinct customers per SKU) uses a 2-level read_group and is a touch slow (~15s/
|
| 12 |
-
window), so it's its own function the app calls lazily and caches.
|
| 13 |
-
"""
|
| 14 |
-
import sys
|
| 15 |
-
import datetime as dt
|
| 16 |
-
from pathlib import Path
|
| 17 |
-
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
| 18 |
-
from functools import lru_cache
|
| 19 |
-
import core.odoo as O
|
| 20 |
-
import core.periods as P
|
| 21 |
-
import modules.sales as sales_mod
|
| 22 |
-
|
| 23 |
-
# SKU-health views are about real products β exclude service/delivery pseudo-SKUs
|
| 24 |
-
# (Delivery Charges otherwise dominate movers/winners). Dot-path FILTER works (groupby
|
| 25 |
-
# on the dot-path does not). Validation reconciles on this same non-service universe.
|
| 26 |
-
_NO_SVC = [('product_id.type', '!=', 'service')]
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
@lru_cache(maxsize=1)
|
| 30 |
-
def _code_map():
|
| 31 |
-
"""product_id β SKU code. Multiple product records can share a default_code (re-SKUing
|
| 32 |
-
/duplicates); keying by code merges them so a re-coded item doesn't read as a fake
|
| 33 |
-
decliner + fake riser. Products without a code fall back to a per-id key. Archived
|
| 34 |
-
products are INCLUDED β re-SKUing typically archives the old record and creates a new
|
| 35 |
-
one under the same code, and the old record still carries last-year sales."""
|
| 36 |
-
prods = O.search_read('product.product', [('active', 'in', [True, False])], ['id', 'default_code'],
|
| 37 |
-
limit=50000)
|
| 38 |
-
return {p['id']: (str(p['default_code']).strip() if p.get('default_code') else f"pid:{p['id']}")
|
| 39 |
-
for p in prods}
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
def _sku_rev(date_from, date_to, team_id=None):
|
| 43 |
-
"""{sku_code: {'name','rev','qty','orders'}} over a window (services excluded,
|
| 44 |
-
duplicate product records merged by SKU code)."""
|
| 45 |
-
g = O.read_group('sale.order.line', O.sale_line_domain(date_from, date_to, team_id, extra=_NO_SVC),
|
| 46 |
-
['price_subtotal:sum', 'product_uom_qty:sum'], ['product_id'], lazy=False)
|
| 47 |
-
codes = _code_map()
|
| 48 |
-
out = {}
|
| 49 |
-
for r in g:
|
| 50 |
-
pid = O.m2o_id(r.get('product_id'))
|
| 51 |
-
if not pid:
|
| 52 |
-
continue
|
| 53 |
-
key = codes.get(pid, f"pid:{pid}")
|
| 54 |
-
e = out.setdefault(key, {'name': O.m2o_name(r.get('product_id')),
|
| 55 |
-
'rev': 0.0, 'qty': 0.0, 'orders': 0})
|
| 56 |
-
e['rev'] += r.get('price_subtotal') or 0.0
|
| 57 |
-
e['qty'] += r.get('product_uom_qty') or 0.0
|
| 58 |
-
e['orders'] += r.get('__count') or 0
|
| 59 |
-
return out
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
def yoy_movers(t=None, limit=20, team_id=None):
|
| 63 |
-
"""Top SKU risers and decliners by YTD-vs-same-period-LY revenue change."""
|
| 64 |
-
t = t or P.today()
|
| 65 |
-
yf, yt = P.ytd(t)
|
| 66 |
-
lf, lt = P.ytd_last_year(t)
|
| 67 |
-
this = _sku_rev(yf, yt, team_id)
|
| 68 |
-
last = _sku_rev(lf, lt, team_id)
|
| 69 |
-
rows = []
|
| 70 |
-
for pid in set(this) | set(last):
|
| 71 |
-
tr = this.get(pid, {'rev': 0.0, 'name': last.get(pid, {}).get('name', '')})
|
| 72 |
-
lr = last.get(pid, {'rev': 0.0})
|
| 73 |
-
name = this.get(pid, {}).get('name') or last.get(pid, {}).get('name') or ''
|
| 74 |
-
rows.append({'code': pid, 'product': name, 'rev_ytd': tr['rev'], 'rev_ly': lr['rev'],
|
| 75 |
-
'change': tr['rev'] - lr['rev']})
|
| 76 |
-
risers = sorted([r for r in rows if r['change'] > 0], key=lambda x: -x['change'])[:limit]
|
| 77 |
-
decliners = sorted([r for r in rows if r['change'] < 0], key=lambda x: x['change'])[:limit]
|
| 78 |
-
return {'risers': risers, 'decliners': decliners}
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
def zombie_skus(t=None, limit=30, min_prior=1500.0, team_id=None):
|
| 82 |
-
"""Catalog rot: SKUs that sold materially last year but are ~dead this year."""
|
| 83 |
-
t = t or P.today()
|
| 84 |
-
yf, yt = P.ytd(t)
|
| 85 |
-
lf, lt = P.ytd_last_year(t)
|
| 86 |
-
this = _sku_rev(yf, yt, team_id)
|
| 87 |
-
last = _sku_rev(lf, lt, team_id)
|
| 88 |
-
rows = []
|
| 89 |
-
for pid, lr in last.items():
|
| 90 |
-
if lr['rev'] < min_prior:
|
| 91 |
-
continue
|
| 92 |
-
tr = this.get(pid, {'rev': 0.0})
|
| 93 |
-
if tr['rev'] > 0.05 * lr['rev']: # still selling at >5% of prior β not a zombie
|
| 94 |
-
continue
|
| 95 |
-
rows.append({'code': pid, 'product': lr['name'], 'rev_ly': lr['rev'], 'rev_ytd': tr['rev'],
|
| 96 |
-
'lost': lr['rev'] - tr['rev']})
|
| 97 |
-
rows.sort(key=lambda x: -x['lost'])
|
| 98 |
-
return rows[:limit]
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
def new_winners(t=None, limit=20, min_this=1500.0, team_id=None):
|
| 102 |
-
"""SKUs that barely sold last year but are selling well this year."""
|
| 103 |
-
t = t or P.today()
|
| 104 |
-
yf, yt = P.ytd(t)
|
| 105 |
-
lf, lt = P.ytd_last_year(t)
|
| 106 |
-
this = _sku_rev(yf, yt, team_id)
|
| 107 |
-
last = _sku_rev(lf, lt, team_id)
|
| 108 |
-
rows = []
|
| 109 |
-
for pid, tr in this.items():
|
| 110 |
-
if tr['rev'] < min_this:
|
| 111 |
-
continue
|
| 112 |
-
lr = last.get(pid, {'rev': 0.0})
|
| 113 |
-
if lr['rev'] > 0.05 * tr['rev']:
|
| 114 |
-
continue
|
| 115 |
-
rows.append({'code': pid, 'product': tr['name'], 'rev_ytd': tr['rev'], 'rev_ly': lr['rev'],
|
| 116 |
-
'gained': tr['rev'] - lr['rev']})
|
| 117 |
-
rows.sort(key=lambda x: -x['gained'])
|
| 118 |
-
return rows[:limit]
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
def velocity_leaders(t=None, limit=25, team_id=None):
|
| 122 |
-
"""Top SKUs by LTM unit velocity (units/month) and revenue."""
|
| 123 |
-
t = t or P.today()
|
| 124 |
-
lf, lt = P.ltm(t)
|
| 125 |
-
sku = _sku_rev(lf, lt, team_id)
|
| 126 |
-
rows = [{'code': k, 'product': v['name'], 'units_ltm': v['qty'], 'units_per_mo': v['qty'] / 12.0,
|
| 127 |
-
'rev_ltm': v['rev'], 'orders_ltm': v['orders']} for k, v in sku.items()]
|
| 128 |
-
rows.sort(key=lambda x: -x['units_ltm'])
|
| 129 |
-
return rows[:limit]
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
def _code_category():
|
| 133 |
-
"""code -> category name. Built from _code_map (pid->code) + the sales category map
|
| 134 |
-
(pid->category); the first record carrying a code sets that code's category."""
|
| 135 |
-
codes = _code_map()
|
| 136 |
-
cats = sales_mod._product_cat()
|
| 137 |
-
out = {}
|
| 138 |
-
for pid, code in codes.items():
|
| 139 |
-
if code not in out and pid in cats:
|
| 140 |
-
out[code] = cats[pid]
|
| 141 |
-
return out
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
def directory(t=None, team_id=None):
|
| 145 |
-
"""Every SKU that sold this YTD or the same period last year, with the fields the SKU
|
| 146 |
-
drill-down filters/sorts on: category, YTD/LY revenue, YoY %, units and orders. Mirrors the
|
| 147 |
-
customer directory β the full list (filtering happens in the UI), sorted by YTD revenue."""
|
| 148 |
-
t = t or P.today()
|
| 149 |
-
yf, yt = P.ytd(t)
|
| 150 |
-
lf, lt = P.ytd_last_year(t)
|
| 151 |
-
this = _sku_rev(yf, yt, team_id)
|
| 152 |
-
last = _sku_rev(lf, lt, team_id)
|
| 153 |
-
code_cat = _code_category()
|
| 154 |
-
rows = []
|
| 155 |
-
for code in set(this) | set(last):
|
| 156 |
-
tr = this.get(code, {})
|
| 157 |
-
lr = last.get(code, {})
|
| 158 |
-
rev_ytd = tr.get('rev', 0.0)
|
| 159 |
-
rev_ly = lr.get('rev', 0.0)
|
| 160 |
-
rows.append({'code': code, 'product': tr.get('name') or lr.get('name') or code,
|
| 161 |
-
'category': code_cat.get(code, '(uncategorized)'),
|
| 162 |
-
'rev_ytd': rev_ytd, 'rev_ly': rev_ly, 'change': rev_ytd - rev_ly,
|
| 163 |
-
'yoy_pct': P.yoy_pct(rev_ytd, rev_ly),
|
| 164 |
-
'qty_ytd': tr.get('qty', 0.0), 'orders_ytd': tr.get('orders', 0)})
|
| 165 |
-
rows.sort(key=lambda r: -r['rev_ytd'])
|
| 166 |
-
return rows
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
def catalogue():
|
| 170 |
-
"""`{sku_code: {'product', 'category', 'id'}}` β EVERY ACTIVE product, sold or not.
|
| 171 |
-
|
| 172 |
-
β THIS IS THE CATALOGUE UNIVERSE, AND IT IS DELIBERATELY NOT `directory()`. `directory()`'s
|
| 173 |
-
row set IS the union of two revenue `read_group`s over `sale.order.line` (`:155`), so a SKU
|
| 174 |
-
that never sold cannot exist in it. That is CORRECT for its own callers β `yoy_movers`,
|
| 175 |
-
`zombie_skus`, `new_winners` and `categories` all legitimately want a sales-window universe β
|
| 176 |
-
and it is wrong for the PRODUCT GRID, which is a catalogue and was therefore showing 2,717 of
|
| 177 |
-
5,875 SKUs (wave 29, owner item 22 / ruling R12). The fix is this function plus a LEFT JOIN in
|
| 178 |
-
the consumer, never a window removed from `directory()`: removing the window alone lands at
|
| 179 |
-
3,327 (all-time-sold), because ~2,550 active SKUs have never sold in wholesale scope at all.
|
| 180 |
-
|
| 181 |
-
β NOT BU-SHAPED, and it cannot be: `product.product` carries no team. A catalogue is one
|
| 182 |
-
catalogue. `directory()` stays the BU-shaped half, which is why the two are joined rather than
|
| 183 |
-
merged β a scoped caller gets every SKU with ITS OWN revenue, blank where that BU never sold.
|
| 184 |
-
|
| 185 |
-
Keyed exactly like `_code_map()` β the `default_code`, or a `pid:N` fallback for the 33 active
|
| 186 |
-
records that carry none β so the join against `directory()` is code-for-code with no
|
| 187 |
-
normaliser. Names come from `display_name` (`[CODE] NAME`), which is what `O.m2o_name` yields
|
| 188 |
-
off a sale line, so a never-sold row wears the same format as a sold one.
|
| 189 |
-
|
| 190 |
-
β RAISES on a truncated read rather than returning a short catalogue. A silently short pull
|
| 191 |
-
would put the grid back at a plausible wrong number with every gate green β the exact failure
|
| 192 |
-
this function exists to end ([[no-unverifiable-aggregates]], and the same truncation guard
|
| 193 |
-
`modules/backorders.py:161` already uses).
|
| 194 |
-
"""
|
| 195 |
-
dom = [('active', '=', True)]
|
| 196 |
-
prods = O.search_read('product.product', dom, ['id', 'default_code', 'display_name', 'name'],
|
| 197 |
-
limit=50000)
|
| 198 |
-
n = O.get_odoo().search_count('product.product', dom)
|
| 199 |
-
if len(prods) != n:
|
| 200 |
-
raise ValueError(
|
| 201 |
-
f"products.catalogue: the product pull is TRUNCATED β read {len(prods)} rows against "
|
| 202 |
-
f"a search_count of {n}. A short catalogue renders as a plausible smaller grid with "
|
| 203 |
-
f"nothing reporting it; raise the limit before shipping this.")
|
| 204 |
-
code_cat = _code_category()
|
| 205 |
-
out = {}
|
| 206 |
-
for p in prods:
|
| 207 |
-
code = (str(p['default_code']).strip() if p.get('default_code') else f"pid:{p['id']}")
|
| 208 |
-
# First record wins, matching `_code_category`'s own convention. MEASURED 2026-08-11: zero
|
| 209 |
-
# active products share a code, so this branch is a guard, not a merge policy.
|
| 210 |
-
# ββ W33-T43 (R2 / amendment A2) β `id` RIDES THE ROW. Odoo's `product.product` id is
|
| 211 |
-
# already in hand (the `pid:{id}` fallback above uses it and then threw it away), and it is
|
| 212 |
-
# the ONE column `ut_odoo_products` had that `product_data` lacked. R2 merges onto the
|
| 213 |
-
# legacy key and keeps every data column, so retiring the twin without this would lose a
|
| 214 |
-
# real identifier rather than a navigational link.
|
| 215 |
-
# β A product's grid pid is `crc32(default_code)`, NOT the Odoo id β unlike a CUSTOMER row,
|
| 216 |
-
# whose pid IS the partner id. That asymmetry is exactly why this cannot be derived
|
| 217 |
-
# downstream in `aios_grid.py` and has to be carried from the source read.
|
| 218 |
-
out.setdefault(code, {'product': p.get('display_name') or p.get('name') or code,
|
| 219 |
-
'category': code_cat.get(code, '(uncategorized)'),
|
| 220 |
-
'id': p['id']})
|
| 221 |
-
return out
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
#: ββ WAVE 30 / W30-T34 (the carried W29-T52) β THE PRICELIST STRATUM, PER LIST.
|
| 225 |
-
#:
|
| 226 |
-
#: MEASURED LIVE 2026-08-12, and every one of these numbers shaped the design rather than
|
| 227 |
-
#: decorating it:
|
| 228 |
-
#:
|
| 229 |
-
#: Β· **Five pricelists, three of them material.** Date-valid fixed rules: Fisch 5,139,
|
| 230 |
-
#: Royal 1 2,702, Royal 2 2,605, Public Pricelist 17, Giftware Deals 1. At SKU level that
|
| 231 |
-
#: is Fisch 3,971 Β· Royal 1 1,920 Β· Royal 2 1,857 Β· Public 6 Β· Giftware 0.
|
| 232 |
-
#: Β· **3,988 of 5,871 active products carry a usable fixed rule; 1,883 carry NONE.**
|
| 233 |
-
#: Β· **1,853 SKUs are priced on all three lists AND THE LISTS DISAGREE** β median relative
|
| 234 |
-
#: spread 16%, max 82%. That is why this is three columns rather than one "the price" or a
|
| 235 |
-
#: low/high pair: neither can tell a Fisch rep what Fisch sells the SKU for, which is the
|
| 236 |
-
#: only question the column exists to answer.
|
| 237 |
-
#: Β· β **`list_price` is 1.00 on 5,817 of the 5,871** β so the single `3_global` rule
|
| 238 |
-
#: computed over it is meaningless, exactly as W29-T52 said. **This reader never falls
|
| 239 |
-
#: through to it**, and that is the ticket's own negative control: a SKU with no specific
|
| 240 |
-
#: item renders BLANK, never a fallback dressed as a price.
|
| 241 |
-
#:
|
| 242 |
-
#: β RESOLVED BY NAME, NOT BY ID. A pricelist renamed or deleted in Odoo makes its column go
|
| 243 |
-
#: blank and makes `report["lists_missing"]` name it β a hard-coded id would keep pointing at
|
| 244 |
-
#: whatever inherited it and mislabel every cell silently.
|
| 245 |
-
PRICELIST_COLUMNS = (
|
| 246 |
-
("price_fisch", "Fisch"),
|
| 247 |
-
("price_royal_1", "Royal 1"),
|
| 248 |
-
("price_royal_2", "Royal 2"),
|
| 249 |
-
)
|
| 250 |
-
|
| 251 |
-
|
| 252 |
-
def pricelist_by_code():
|
| 253 |
-
"""`({code: {column_key: price}}, report)` β the date-valid FIXED base-tier price for each
|
| 254 |
-
declared pricelist, keyed EXACTLY as `catalogue()` keys its rows.
|
| 255 |
-
|
| 256 |
-
β THE KEYING IS LOAD-BEARING, not a detail. `catalogue()` keys on `default_code` with a
|
| 257 |
-
`pid:{id}` fallback for the ~33 active records that carry none. Keying this map any other
|
| 258 |
-
way would leave those SKUs permanently blank while an oracle counting active products
|
| 259 |
-
counted them β a red gate on a working build, or worse, a silent hole nobody counts.
|
| 260 |
-
|
| 261 |
-
**The base tier, deliberately.** `pricecomp._tier_for` picks the highest `min_quantity` at
|
| 262 |
-
or below an order's quantity, because it is pricing a LINE that has one. A catalogue column
|
| 263 |
-
has no quantity in hand, so it takes the LOWEST `min_quantity` β the price at qty 1. Rules
|
| 264 |
-
above that break are a bulk price, and `report["qty_break_only"]` counts the SKUs whose only
|
| 265 |
-
rule sits on one (MEASURED: 10 rules of 10,464 carry a break at all).
|
| 266 |
-
|
| 267 |
-
**Variant rules beat template rules**, matching `pricecomp._tier_for`: a `0_product_variant`
|
| 268 |
-
rule is the more specific statement about this exact SKU.
|
| 269 |
-
|
| 270 |
-
β DEGRADES TO `({}, report)` on a read failure, matching `_inventory_by_code` rather than
|
| 271 |
-
`catalogue()`: these are COLUMNS, and a product grid that will not render because pricing is
|
| 272 |
-
momentarily unreachable is a worse failure than one with blank price columns. The blanks are
|
| 273 |
-
not silent β `product_data.validate()`'s coverage leg reconciles each column against a fresh
|
| 274 |
-
Odoo count and goes red at zero.
|
| 275 |
-
|
| 276 |
-
β **The report exists because R6's second sentence is law** (*"if it can't be done, you need
|
| 277 |
-
to explicitly tell me why and recommend a fix"*). Everything this reader CANNOT see is
|
| 278 |
-
counted rather than dropped: rules on pricelists the contract does not declare, non-fixed
|
| 279 |
-
(`formula`/`percent`) rules, the `3_global` fallback, and prices that only exist above a
|
| 280 |
-
quantity break.
|
| 281 |
-
"""
|
| 282 |
-
# β THE COUNTERS ARE DISJOINT AND ORDER-DEPENDENT, and saying so is the difference between a
|
| 283 |
-
# report and a misleading one. A rule is classified ONCE, by the first reason it is skipped:
|
| 284 |
-
# undeclared list β out of date β not fixed β global β zero price. So `rules_not_fixed: 0`
|
| 285 |
-
# means "no formula rule on a list we declare", NOT "this Odoo has no formula rules"
|
| 286 |
-
# (MEASURED 2026-08-12: it has exactly one, and it sits on Public Pricelist, which the
|
| 287 |
-
# contract does not declare β so it lands in `rules_undeclared_list`).
|
| 288 |
-
report = {"lists_missing": [], "rules_total": 0, "rules_undeclared_list": 0,
|
| 289 |
-
"rules_not_fixed": 0, "rules_global": 0, "rules_out_of_date": 0,
|
| 290 |
-
"rules_zero_price": 0, "qty_break_only": 0}
|
| 291 |
-
try:
|
| 292 |
-
pl_rows = O.search_read('product.pricelist', [], ['id', 'name'])
|
| 293 |
-
by_name = {}
|
| 294 |
-
for p in pl_rows:
|
| 295 |
-
by_name.setdefault(str(p.get('name') or '').strip(), p['id'])
|
| 296 |
-
wanted = {}
|
| 297 |
-
for col, name in PRICELIST_COLUMNS:
|
| 298 |
-
pid = by_name.get(name)
|
| 299 |
-
if pid is None:
|
| 300 |
-
report["lists_missing"].append(name)
|
| 301 |
-
else:
|
| 302 |
-
wanted[pid] = col
|
| 303 |
-
|
| 304 |
-
# ββ FILTERED SERVER-SIDE, AND THE REASON IS A MEASUREMENT, NOT A STYLE PREFERENCE.
|
| 305 |
-
# Reading all 10,464 rules with 9 fields and sorting them in Python costs **30.1s** on
|
| 306 |
-
# this connection; the same rules under a server-side domain with 6 fields cost
|
| 307 |
-
# **13.6s** (measured 2026-08-12, back to back). `pool()` runs this on a scope's
|
| 308 |
-
# first-ever build, so that 16.5s is 16.5s of somebody's page load.
|
| 309 |
-
#
|
| 310 |
-
# β The domain reproduces the Python predicate EXACTLY, and the null legs are the part
|
| 311 |
-
# that is easy to get wrong: an absent `date_start` is `False`, not a past date, so
|
| 312 |
-
# `('date_start','<=',today)` ALONE would drop every open-ended rule β which is almost
|
| 313 |
-
# all of them.
|
| 314 |
-
today = P.today().isoformat()
|
| 315 |
-
_declared = sorted(wanted)
|
| 316 |
-
_kinds = ['0_product_variant', '1_product']
|
| 317 |
-
_live = [('pricelist_id', 'in', _declared), ('compute_price', '=', 'fixed'),
|
| 318 |
-
('applied_on', 'in', _kinds),
|
| 319 |
-
'|', ('date_start', '=', False), ('date_start', '<=', today),
|
| 320 |
-
'|', ('date_end', '=', False), ('date_end', '>=', today),
|
| 321 |
-
('fixed_price', '>', 0)]
|
| 322 |
-
rules = O.search_read(
|
| 323 |
-
'product.pricelist.item', _live,
|
| 324 |
-
['pricelist_id', 'product_id', 'product_tmpl_id', 'applied_on', 'fixed_price',
|
| 325 |
-
'min_quantity']) if _declared else []
|
| 326 |
-
|
| 327 |
-
# β R6's SECOND SENTENCE, PAID FOR WITH `search_count` RATHER THAN A WIDER READ. Six
|
| 328 |
-
# counts cost ~1.2s together; the rules they count would cost 16s to read. What this
|
| 329 |
-
# reader cannot see is still REPORTED β it is just no longer transferred.
|
| 330 |
-
def _n(extra):
|
| 331 |
-
try:
|
| 332 |
-
return O.get_odoo().search_count('product.pricelist.item', extra)
|
| 333 |
-
except Exception:
|
| 334 |
-
return -1 # -1 reads as "not measured", never as zero
|
| 335 |
-
_dated = ['|', ('date_start', '=', False), ('date_start', '<=', today),
|
| 336 |
-
'|', ('date_end', '=', False), ('date_end', '>=', today)]
|
| 337 |
-
report["rules_total"] = _n([])
|
| 338 |
-
report["rules_undeclared_list"] = _n([('pricelist_id', 'not in', _declared)]) \
|
| 339 |
-
if _declared else report["rules_total"]
|
| 340 |
-
if _declared:
|
| 341 |
-
_on = [('pricelist_id', 'in', _declared)]
|
| 342 |
-
# `percent_price` and formula rules are read NOWHERE in this repo. A rule we cannot
|
| 343 |
-
# price is one the operator is TOLD about, never one that quietly becomes a blank.
|
| 344 |
-
report["rules_not_fixed"] = _n(_on + [('compute_price', '!=', 'fixed')])
|
| 345 |
-
report["rules_global"] = _n(_on + [('compute_price', '=', 'fixed'),
|
| 346 |
-
('applied_on', 'not in', _kinds)])
|
| 347 |
-
report["rules_out_of_date"] = (
|
| 348 |
-
_n(_on + [('compute_price', '=', 'fixed'), ('applied_on', 'in', _kinds)])
|
| 349 |
-
- _n(_on + [('compute_price', '=', 'fixed'),
|
| 350 |
-
('applied_on', 'in', _kinds)] + _dated))
|
| 351 |
-
# A zero price is not "free" β it is an unset rule. Blank says so; 0 does not.
|
| 352 |
-
report["rules_zero_price"] = _n(
|
| 353 |
-
_on + [('compute_price', '=', 'fixed'), ('applied_on', 'in', _kinds)]
|
| 354 |
-
+ _dated + [('fixed_price', '<=', 0)])
|
| 355 |
-
|
| 356 |
-
by_var, by_tmpl = {}, {}
|
| 357 |
-
for r in rules:
|
| 358 |
-
col = wanted[O.m2o_id(r.get('pricelist_id'))]
|
| 359 |
-
if r.get('applied_on') == '0_product_variant' and r.get('product_id'):
|
| 360 |
-
by_var.setdefault((col, O.m2o_id(r['product_id'])), []).append(r)
|
| 361 |
-
elif r.get('product_tmpl_id'):
|
| 362 |
-
by_tmpl.setdefault((col, O.m2o_id(r['product_tmpl_id'])), []).append(r)
|
| 363 |
-
|
| 364 |
-
dom = [('active', '=', True)]
|
| 365 |
-
prods = O.search_read('product.product', dom,
|
| 366 |
-
['id', 'default_code', 'product_tmpl_id'], limit=50000)
|
| 367 |
-
n = O.get_odoo().search_count('product.product', dom)
|
| 368 |
-
if len(prods) != n:
|
| 369 |
-
# Same guard, same reason as `catalogue()`: a short pull renders as a plausible
|
| 370 |
-
# smaller set of priced SKUs with nothing reporting it.
|
| 371 |
-
raise ValueError(
|
| 372 |
-
f"products.pricelist_by_code: the product pull is TRUNCATED β read {len(prods)} "
|
| 373 |
-
f"rows against a search_count of {n}.")
|
| 374 |
-
except Exception as e:
|
| 375 |
-
report["error"] = f"{type(e).__name__}: {str(e)[:200]}"
|
| 376 |
-
return {}, report
|
| 377 |
-
|
| 378 |
-
out = {}
|
| 379 |
-
for p in prods:
|
| 380 |
-
code = (str(p['default_code']).strip() if p.get('default_code') else f"pid:{p['id']}")
|
| 381 |
-
tmpl = O.m2o_id(p.get('product_tmpl_id'))
|
| 382 |
-
cells = {}
|
| 383 |
-
for col, _name in PRICELIST_COLUMNS:
|
| 384 |
-
cands = by_var.get((col, p['id'])) or by_tmpl.get((col, tmpl))
|
| 385 |
-
if not cands:
|
| 386 |
-
continue
|
| 387 |
-
base = min(cands, key=lambda r: r.get('min_quantity') or 0.0)
|
| 388 |
-
if (base.get('min_quantity') or 0.0) > 1.0:
|
| 389 |
-
report["qty_break_only"] += 1
|
| 390 |
-
cells[col] = base.get('fixed_price')
|
| 391 |
-
if cells:
|
| 392 |
-
out.setdefault(code, {}).update(cells)
|
| 393 |
-
return out, report
|
| 394 |
-
|
| 395 |
-
|
| 396 |
-
def catalogue_count():
|
| 397 |
-
"""The INDEPENDENT population oracle: Odoo's own count of active products.
|
| 398 |
-
|
| 399 |
-
Deliberately a bare `search_count` and not a `len()` over anything this module built β the
|
| 400 |
-
2,717 defect shipped silently for a wave because `product_data.validate()` derived BOTH sides
|
| 401 |
-
of its reconciliation from `_sku_rev`, so the oracle could never see a missing row.
|
| 402 |
-
"""
|
| 403 |
-
return O.get_odoo().search_count('product.product', [('active', '=', True)])
|
| 404 |
-
|
| 405 |
-
|
| 406 |
-
def categories(t=None, team_id=None):
|
| 407 |
-
"""Sorted distinct category names present in the SKU directory (for the drill-down filter)."""
|
| 408 |
-
return sorted({r['category'] for r in directory(t, team_id)})
|
| 409 |
-
|
| 410 |
-
|
| 411 |
-
def _coverage(date_from, date_to, team_id=None):
|
| 412 |
-
"""{sku_code: distinct_customer_count} via 2-level read_group (slow-ish), merged by code.
|
| 413 |
-
(A customer buying two records sharing a code can count twice; duplicates are rare, and
|
| 414 |
-
code-level avoids the bigger error of a re-SKUed product reading as full coverage loss.)"""
|
| 415 |
-
g = O.read_group('sale.order.line', O.sale_line_domain(date_from, date_to, team_id, extra=_NO_SVC),
|
| 416 |
-
['__count'], ['product_id', 'order_partner_id'], lazy=False)
|
| 417 |
-
codes = _code_map()
|
| 418 |
-
cov = {}
|
| 419 |
-
names = {}
|
| 420 |
-
for r in g:
|
| 421 |
-
pid = O.m2o_id(r.get('product_id'))
|
| 422 |
-
if not pid:
|
| 423 |
-
continue
|
| 424 |
-
key = codes.get(pid, f"pid:{pid}")
|
| 425 |
-
cov[key] = cov.get(key, 0) + 1
|
| 426 |
-
names.setdefault(key, O.m2o_name(r.get('product_id')))
|
| 427 |
-
return cov, names
|
| 428 |
-
|
| 429 |
-
|
| 430 |
-
def coverage_collapse(t=None, limit=25, min_prior_custs=8, team_id=None):
|
| 431 |
-
"""SKUs that lost the most customer breadth YoY β early warning a SKU is dying even if
|
| 432 |
-
revenue hasn't fully cratered yet."""
|
| 433 |
-
t = t or P.today()
|
| 434 |
-
yf, yt = P.ytd(t)
|
| 435 |
-
lf, lt = P.ytd_last_year(t)
|
| 436 |
-
cov_t, names_t = _coverage(yf, yt, team_id)
|
| 437 |
-
cov_l, names_l = _coverage(lf, lt, team_id)
|
| 438 |
-
rows = []
|
| 439 |
-
for pid, lc in cov_l.items():
|
| 440 |
-
if lc < min_prior_custs:
|
| 441 |
-
continue
|
| 442 |
-
tc = cov_t.get(pid, 0)
|
| 443 |
-
drop = lc - tc
|
| 444 |
-
if drop <= 0:
|
| 445 |
-
continue
|
| 446 |
-
rows.append({'code': pid, 'product': names_l.get(pid) or names_t.get(pid) or '',
|
| 447 |
-
'custs_ly': lc, 'custs_ytd': tc, 'lost_custs': drop,
|
| 448 |
-
'pct_drop': drop / lc * 100})
|
| 449 |
-
rows.sort(key=lambda x: (-x['lost_custs'], -x['pct_drop']))
|
| 450 |
-
return rows[:limit]
|
| 451 |
-
|
| 452 |
-
|
| 453 |
-
# ====================================================================== SKU DRAWER (mirror of customer drawer)
|
| 454 |
-
def _sku_product_ids(code):
|
| 455 |
-
"""All product.product ids sharing this SKU code (merged variants / archived records)."""
|
| 456 |
-
return [pid for pid, c in _code_map().items() if c == code]
|
| 457 |
-
|
| 458 |
-
|
| 459 |
-
def _sku_dom(pids, date_from, date_to, team_id=None):
|
| 460 |
-
return O.sale_line_domain(date_from, date_to, team_id, extra=[('product_id', 'in', pids)] + _NO_SVC)
|
| 461 |
-
|
| 462 |
-
|
| 463 |
-
def _sku_name_category(pids):
|
| 464 |
-
rows = O.search_read('product.product', [('id', 'in', pids)], ['name', 'categ_id'])
|
| 465 |
-
name = rows[0]['name'] if rows else '(unknown)'
|
| 466 |
-
catmap = sales_mod._product_cat()
|
| 467 |
-
main = next((catmap.get(p) for p in pids if catmap.get(p)), '(uncategorized)')
|
| 468 |
-
return name, main
|
| 469 |
-
|
| 470 |
-
|
| 471 |
-
def _sku_buyers(pids, date_from, date_to, team_id=None):
|
| 472 |
-
"""{partner_id: {'name','rev','qty'}} for buyers of this SKU over a window."""
|
| 473 |
-
g = O.read_group('sale.order.line', _sku_dom(pids, date_from, date_to, team_id),
|
| 474 |
-
['price_subtotal:sum', 'product_uom_qty:sum'], ['order_partner_id'], lazy=False)
|
| 475 |
-
out = {}
|
| 476 |
-
for r in g:
|
| 477 |
-
pid = O.m2o_id(r.get('order_partner_id'))
|
| 478 |
-
if pid:
|
| 479 |
-
out[pid] = {'name': O.m2o_name(r.get('order_partner_id')),
|
| 480 |
-
'rev': r.get('price_subtotal') or 0.0, 'qty': r.get('product_uom_qty') or 0.0}
|
| 481 |
-
return out
|
| 482 |
-
|
| 483 |
-
|
| 484 |
-
def sku_detail(code, t=None, team_id=None, n_months=13, allsku=None):
|
| 485 |
-
"""KPIs (rev/qty/buyers YoY, GM%), monthly trend, rank & % of BU for one SKU code.
|
| 486 |
-
Pass `allsku` (a cached _sku_rev YTD map) to skip the ~all-SKU rank read."""
|
| 487 |
-
t = t or P.today()
|
| 488 |
-
pids = _sku_product_ids(code)
|
| 489 |
-
if not pids:
|
| 490 |
-
return None
|
| 491 |
-
yf, yt = P.ytd(t)
|
| 492 |
-
lf, lt = P.ytd_last_year(t)
|
| 493 |
-
mf, mt = P.ltm(t)
|
| 494 |
-
name, category = _sku_name_category(pids)
|
| 495 |
-
|
| 496 |
-
def s(df, dtt, field='price_subtotal'):
|
| 497 |
-
return O.sum_field('sale.order.line', _sku_dom(pids, df, dtt, team_id), field)
|
| 498 |
-
rev_ytd, rev_ly = s(yf, yt), s(lf, lt)
|
| 499 |
-
qty_ytd, qty_ly = s(yf, yt, 'product_uom_qty'), s(lf, lt, 'product_uom_qty')
|
| 500 |
-
buyers_ytd = len(_sku_buyers(pids, yf, yt, team_id))
|
| 501 |
-
buyers_ly = len(_sku_buyers(pids, lf, lt, team_id))
|
| 502 |
-
g = O.read_group('sale.order.line', _sku_dom(pids, mf, mt, team_id),
|
| 503 |
-
['price_subtotal:sum', 'margin:sum'], [], lazy=False)
|
| 504 |
-
line_rev = (g[0].get('price_subtotal') if g else 0) or 0.0
|
| 505 |
-
margin = (g[0].get('margin') if g else 0) or 0.0
|
| 506 |
-
|
| 507 |
-
# Monthly trend: bin lines by their order's month in Python (dot-path month groupby is rejected
|
| 508 |
-
# on sale.order.line), in 2 reads instead of 26 point queries.
|
| 509 |
-
range_start = dt.date(t.year - 2, t.month, 1).isoformat()
|
| 510 |
-
lines = O.search_read('sale.order.line', _sku_dom(pids, range_start, t.isoformat(), team_id),
|
| 511 |
-
['price_subtotal', 'order_id'])
|
| 512 |
-
oids = list({O.m2o_id(line['order_id']) for line in lines if line.get('order_id')})
|
| 513 |
-
odate = {}
|
| 514 |
-
for i in range(0, len(oids), 1000):
|
| 515 |
-
for o in O.search_read('sale.order', [('id', 'in', oids[i:i + 1000])], ['date_order']):
|
| 516 |
-
if o.get('date_order'):
|
| 517 |
-
odate[o['id']] = str(o['date_order'])[:7]
|
| 518 |
-
mrev = {}
|
| 519 |
-
for line in lines:
|
| 520 |
-
ym = odate.get(O.m2o_id(line.get('order_id')))
|
| 521 |
-
if ym:
|
| 522 |
-
mrev[ym] = mrev.get(ym, 0.0) + (line.get('price_subtotal') or 0.0)
|
| 523 |
-
monthly = []
|
| 524 |
-
for ym, start, end in P.month_starts(n_months, t):
|
| 525 |
-
y, m = int(ym[:4]) - 1, int(ym[5:7])
|
| 526 |
-
monthly.append({'month': ym, 'revenue': mrev.get(ym, 0.0),
|
| 527 |
-
'revenue_ly': mrev.get(f'{y:04d}-{m:02d}', 0.0)})
|
| 528 |
-
|
| 529 |
-
allsku = allsku if allsku is not None else _sku_rev(yf, yt, team_id)
|
| 530 |
-
total = sum(v['rev'] for v in allsku.values()) or 1.0
|
| 531 |
-
rank = next((i + 1 for i, (c, _v) in enumerate(sorted(allsku.items(), key=lambda kv: -kv[1]['rev']))
|
| 532 |
-
if c == code), None)
|
| 533 |
-
return {
|
| 534 |
-
'code': code, 'name': name, 'category': category,
|
| 535 |
-
'rev_ytd': rev_ytd, 'rev_ly': rev_ly, 'rev_yoy_pct': P.yoy_pct(rev_ytd, rev_ly),
|
| 536 |
-
'qty_ytd': qty_ytd, 'qty_ly': qty_ly, 'qty_yoy_pct': P.yoy_pct(qty_ytd, qty_ly),
|
| 537 |
-
'buyers_ytd': buyers_ytd, 'buyers_ly': buyers_ly, 'buyers_delta': buyers_ytd - buyers_ly,
|
| 538 |
-
'gm_pct': (margin / line_rev * 100) if line_rev else 0.0, 'gm_dollars': margin,
|
| 539 |
-
'rank': rank, 'n_skus': len(allsku), 'pct_of_bu': rev_ytd / total * 100,
|
| 540 |
-
'monthly': monthly,
|
| 541 |
-
}
|
| 542 |
-
|
| 543 |
-
|
| 544 |
-
def sku_buyer_bridge(code, t=None, team_id=None, top=12):
|
| 545 |
-
"""Who drives the SKU's YoY: retained / new / churned buyers + a $-ranked churned call list."""
|
| 546 |
-
t = t or P.today()
|
| 547 |
-
pids = _sku_product_ids(code)
|
| 548 |
-
yf, yt = P.ytd(t)
|
| 549 |
-
lf, lt = P.ytd_last_year(t)
|
| 550 |
-
this, last = _sku_buyers(pids, yf, yt, team_id), _sku_buyers(pids, lf, lt, team_id)
|
| 551 |
-
tset, lset = set(this), set(last)
|
| 552 |
-
retained, new, churned = tset & lset, tset - lset, lset - tset
|
| 553 |
-
churned_list = sorted([{'pid': p, 'customer': last[p]['name'], 'ly_rev': last[p]['rev']}
|
| 554 |
-
for p in churned], key=lambda x: -x['ly_rev'])[:top]
|
| 555 |
-
return {'retained': {'n': len(retained), 'rev': sum(this[p]['rev'] for p in retained)},
|
| 556 |
-
'new': {'n': len(new), 'rev': sum(this[p]['rev'] for p in new)},
|
| 557 |
-
'churned': {'n': len(churned), 'rev': sum(last[p]['rev'] for p in churned)},
|
| 558 |
-
'buyer_retention_pct': (len(retained) / len(lset) * 100) if lset else 0.0,
|
| 559 |
-
'churned_buyers': churned_list, 'buyers_this': len(tset), 'buyers_last': len(lset),
|
| 560 |
-
'this_total': sum(v['rev'] for v in this.values())}
|
| 561 |
-
|
| 562 |
-
|
| 563 |
-
def sku_concentration(code, t=None, team_id=None):
|
| 564 |
-
"""Buyer-concentration risk: top-1/top-3 share, Herfindahl index, effective buyer count (LTM)."""
|
| 565 |
-
t = t or P.today()
|
| 566 |
-
mf, mt = P.ltm(t)
|
| 567 |
-
buyers = _sku_buyers(_sku_product_ids(code), mf, mt, team_id)
|
| 568 |
-
revs = sorted([v['rev'] for v in buyers.values()], reverse=True)
|
| 569 |
-
total = sum(revs) or 1.0
|
| 570 |
-
hhi = sum((r / total) ** 2 for r in revs)
|
| 571 |
-
return {'n_buyers': len(revs), 'top1_pct': (revs[0] / total * 100) if revs else 0.0,
|
| 572 |
-
'top3_pct': (sum(revs[:3]) / total * 100) if revs else 0.0,
|
| 573 |
-
'hhi': hhi, 'eff_buyers': (1 / hhi) if hhi else 0.0}
|
| 574 |
-
|
| 575 |
-
|
| 576 |
-
def sku_price_dispersion(code, t=None, team_id=None, cap=60):
|
| 577 |
-
"""Realized $/unit per buyer (LTM) vs the volume-weighted average; recoverable $ on below-VWAP
|
| 578 |
-
accounts. list_price is unreliable, so the dispersion among actual buyers is the margin lever."""
|
| 579 |
-
t = t or P.today()
|
| 580 |
-
mf, mt = P.ltm(t)
|
| 581 |
-
g = O.read_group('sale.order.line', _sku_dom(_sku_product_ids(code), mf, mt, team_id),
|
| 582 |
-
['price_subtotal:sum', 'product_uom_qty:sum'], ['order_partner_id'], lazy=False)
|
| 583 |
-
rows, tot_rev, tot_qty = [], 0.0, 0.0
|
| 584 |
-
for r in g:
|
| 585 |
-
pid = O.m2o_id(r.get('order_partner_id'))
|
| 586 |
-
rev = r.get('price_subtotal') or 0.0
|
| 587 |
-
qty = r.get('product_uom_qty') or 0.0
|
| 588 |
-
if not pid or qty <= 0:
|
| 589 |
-
continue
|
| 590 |
-
rows.append({'pid': pid, 'customer': O.m2o_name(r.get('order_partner_id')),
|
| 591 |
-
'price': rev / qty, 'qty': qty, 'rev': rev})
|
| 592 |
-
tot_rev += rev
|
| 593 |
-
tot_qty += qty
|
| 594 |
-
vwap = (tot_rev / tot_qty) if tot_qty else 0.0
|
| 595 |
-
for r in rows:
|
| 596 |
-
r['recoverable'] = max(0.0, vwap - r['price']) * r['qty']
|
| 597 |
-
rows.sort(key=lambda x: -x['recoverable'])
|
| 598 |
-
return {'vwap': vwap, 'n_buyers': len(rows),
|
| 599 |
-
'recoverable_total': sum(r['recoverable'] for r in rows), 'rows': rows[:cap]}
|
| 600 |
-
|
| 601 |
-
|
| 602 |
-
def sku_top_buyers(code, t=None, team_id=None, top=15):
|
| 603 |
-
"""Ranked buyers of this SKU (YTD) with YoY β clickable to open the customer drawer."""
|
| 604 |
-
t = t or P.today()
|
| 605 |
-
pids = _sku_product_ids(code)
|
| 606 |
-
yf, yt = P.ytd(t)
|
| 607 |
-
lf, lt = P.ytd_last_year(t)
|
| 608 |
-
this, last = _sku_buyers(pids, yf, yt, team_id), _sku_buyers(pids, lf, lt, team_id)
|
| 609 |
-
rows = [{'pid': p, 'customer': v['name'], 'rev': v['rev'], 'qty': v['qty'],
|
| 610 |
-
'yoy_pct': P.yoy_pct(v['rev'], last.get(p, {}).get('rev', 0.0))} for p, v in this.items()]
|
| 611 |
-
rows.sort(key=lambda x: -x['rev'])
|
| 612 |
-
return rows[:top]
|
| 613 |
-
|
| 614 |
-
|
| 615 |
-
_BUYER_TIERS = [('Whale (β₯$25k)', 25000.0), ('Large ($10β25k)', 10000.0),
|
| 616 |
-
('Mid ($2β10k)', 2000.0), ('Small (<$2k)', 0.0)]
|
| 617 |
-
|
| 618 |
-
|
| 619 |
-
def sku_customer_analysis(code, t=None, team_id=None, top=15):
|
| 620 |
-
"""WHO buys this SKU (LTM), as customers: the value-tier mix of its buyers (by each buyer's TOTAL
|
| 621 |
-
spend), and the accounts most DEPENDENT on it (this SKU as a share of their spend β who gets hurt
|
| 622 |
-
most if it stocks out / who to protect)."""
|
| 623 |
-
t = t or P.today()
|
| 624 |
-
lf, lt = P.ltm(t)
|
| 625 |
-
pids = _sku_product_ids(code)
|
| 626 |
-
empty = {'segments': [], 'dependency': [], 'n_buyers': 0, 'avg_dependency': 0.0}
|
| 627 |
-
if not pids:
|
| 628 |
-
return empty
|
| 629 |
-
buyers = _sku_buyers(pids, lf, lt, team_id) # {pid: {name, rev(this SKU), qty}}
|
| 630 |
-
if not buyers:
|
| 631 |
-
return empty
|
| 632 |
-
bpids = list(buyers)
|
| 633 |
-
g = O.read_group('sale.order', sales_mod.order_domain(lf, lt, team_id) + [('partner_id', 'in', bpids)],
|
| 634 |
-
['amount_untaxed:sum'], ['partner_id'], lazy=False)
|
| 635 |
-
total = {O.m2o_id(r['partner_id']): (r.get('amount_untaxed') or 0.0) for r in g if r.get('partner_id')}
|
| 636 |
-
|
| 637 |
-
def tier(rev):
|
| 638 |
-
for nm, lo in _BUYER_TIERS:
|
| 639 |
-
if rev >= lo:
|
| 640 |
-
return nm
|
| 641 |
-
return _BUYER_TIERS[-1][0]
|
| 642 |
-
seg = {nm: {'tier': nm, 'buyers': 0, 'sku_rev': 0.0} for nm, _ in _BUYER_TIERS}
|
| 643 |
-
dep = []
|
| 644 |
-
for p, v in buyers.items():
|
| 645 |
-
ct = total.get(p, v['rev']) or v['rev']
|
| 646 |
-
s = seg[tier(ct)]
|
| 647 |
-
s['buyers'] += 1
|
| 648 |
-
s['sku_rev'] += v['rev']
|
| 649 |
-
dep.append({'customer': v['name'], 'pid': p, 'sku_rev': v['rev'], 'cust_total': ct,
|
| 650 |
-
'dependency_pct': (v['rev'] / ct * 100) if ct else None})
|
| 651 |
-
tot = sum(s['sku_rev'] for s in seg.values()) or 1.0
|
| 652 |
-
segments = []
|
| 653 |
-
for nm, _ in _BUYER_TIERS:
|
| 654 |
-
s = seg[nm]
|
| 655 |
-
s['rev_share'] = s['sku_rev'] / tot * 100
|
| 656 |
-
s['avg_per_buyer'] = (s['sku_rev'] / s['buyers']) if s['buyers'] else 0.0
|
| 657 |
-
segments.append(s)
|
| 658 |
-
dep.sort(key=lambda x: -x['sku_rev'])
|
| 659 |
-
deps_known = [d['dependency_pct'] for d in dep if d['dependency_pct'] is not None]
|
| 660 |
-
return {'segments': segments, 'dependency': dep[:top], 'n_buyers': len(buyers),
|
| 661 |
-
'avg_dependency': (sum(deps_known) / len(deps_known)) if deps_known else 0.0}
|
| 662 |
-
|
| 663 |
-
|
| 664 |
-
def sku_whitespace(code, t=None, team_id=None, top=15):
|
| 665 |
-
"""Customers who buy this SKU's category but NOT this SKU β ranked prospect list."""
|
| 666 |
-
import modules.customers as cust_mod
|
| 667 |
-
t = t or P.today()
|
| 668 |
-
pids = _sku_product_ids(code)
|
| 669 |
-
_name, category = _sku_name_category(pids)
|
| 670 |
-
cat_buyers = cust_mod.category_buyers(category, team_id=team_id) or set()
|
| 671 |
-
mf, mt = P.ltm(t)
|
| 672 |
-
prospects = list(cat_buyers - set(_sku_buyers(pids, mf, mt, team_id)))
|
| 673 |
-
if not prospects:
|
| 674 |
-
return {'category': category, 'rows': []}
|
| 675 |
-
g = O.read_group('sale.order', sales_mod.order_domain(mf, mt, team_id) + [('partner_id', 'in', prospects)],
|
| 676 |
-
['amount_untaxed:sum'], ['partner_id'], lazy=False)
|
| 677 |
-
rows = [{'pid': O.m2o_id(r['partner_id']), 'customer': O.m2o_name(r['partner_id']),
|
| 678 |
-
'total_spend': r.get('amount_untaxed') or 0.0} for r in g if r.get('partner_id')]
|
| 679 |
-
rows.sort(key=lambda x: -x['total_spend'])
|
| 680 |
-
return {'category': category, 'rows': rows[:top]}
|
| 681 |
-
|
| 682 |
-
|
| 683 |
-
def sku_drawer_bundle(code, t=None, team_id=None, allsku=None):
|
| 684 |
-
"""The whole SKU drawer's first paint in one cached unit: sku_detail first (the not-found gate),
|
| 685 |
-
then the other six pulls CONCURRENTLY (O.parallel). Cuts a ~6-call cold open to ~max(call)."""
|
| 686 |
-
detail = sku_detail(code, t=t, team_id=team_id, allsku=allsku)
|
| 687 |
-
if detail is None:
|
| 688 |
-
return {'detail': None}
|
| 689 |
-
bridge, conc, price, buyers, white, ca = O.parallel([
|
| 690 |
-
lambda: sku_buyer_bridge(code, t, team_id),
|
| 691 |
-
lambda: sku_concentration(code, t, team_id),
|
| 692 |
-
lambda: sku_price_dispersion(code, t, team_id),
|
| 693 |
-
lambda: sku_top_buyers(code, t, team_id),
|
| 694 |
-
lambda: sku_whitespace(code, t, team_id),
|
| 695 |
-
lambda: sku_customer_analysis(code, t, team_id),
|
| 696 |
-
])
|
| 697 |
-
return {'detail': detail, 'bridge': bridge, 'conc': conc, 'price': price,
|
| 698 |
-
'buyers': buyers, 'white': white, 'ca': ca}
|
| 699 |
-
|
| 700 |
-
|
| 701 |
-
def validate(t=None, team_id=None):
|
| 702 |
-
"""Reconcile SKU metrics to independent Odoo aggregates. When team_id is set (a single BU
|
| 703 |
-
selected) every check runs SCOPED to that BU, so the validation panel never reconciles
|
| 704 |
-
against β or exposes β the other BU's numbers. All checks here are BU-scopeable (no cross-BU
|
| 705 |
-
mirror), so team_id threads straight through."""
|
| 706 |
-
t = t or P.today()
|
| 707 |
-
yf, yt = P.ytd(t)
|
| 708 |
-
checks = []
|
| 709 |
-
sku = _sku_rev(yf, yt, team_id)
|
| 710 |
-
sku_sum = sum(v['rev'] for v in sku.values())
|
| 711 |
-
line_total = O.sum_field('sale.order.line',
|
| 712 |
-
O.sale_line_domain(yf, yt, team_id, extra=_NO_SVC), 'price_subtotal')
|
| 713 |
-
checks.append({'check': 'SKU rev: Ξ£(per-SKU) == total line revenue, ex-services (YTD)',
|
| 714 |
-
'a': round(sku_sum, 2), 'b': round(line_total, 2),
|
| 715 |
-
'gap': round(sku_sum - line_total, 2),
|
| 716 |
-
'ok': abs(sku_sum - line_total) <= 1.0})
|
| 717 |
-
|
| 718 |
-
m = yoy_movers(t, limit=10**9, team_id=team_id)
|
| 719 |
-
movers_sum = sum(r['change'] for r in m['risers']) + sum(r['change'] for r in m['decliners'])
|
| 720 |
-
lf, lt = P.ytd_last_year(t)
|
| 721 |
-
last_total = O.sum_field('sale.order.line',
|
| 722 |
-
O.sale_line_domain(lf, lt, team_id, extra=_NO_SVC), 'price_subtotal')
|
| 723 |
-
checks.append({'check': 'SKU movers: Ξ£(Ξ) == (YTD β LY) total',
|
| 724 |
-
'a': round(movers_sum, 2), 'b': round(line_total - last_total, 2),
|
| 725 |
-
'gap': round(movers_sum - (line_total - last_total), 2),
|
| 726 |
-
'ok': abs(movers_sum - (line_total - last_total)) <= 1.0})
|
| 727 |
-
|
| 728 |
-
# SKU drawer: buyer-bridge reconciles β retained + new buyer revenue == this-period SKU revenue
|
| 729 |
-
top_code = max(sku.items(), key=lambda kv: kv[1]['rev'])[0] if sku else None
|
| 730 |
-
if top_code:
|
| 731 |
-
bb = sku_buyer_bridge(top_code, t, team_id=team_id)
|
| 732 |
-
recon = bb['retained']['rev'] + bb['new']['rev']
|
| 733 |
-
checks.append({'check': 'SKU drawer: retained+new buyer rev == SKU YTD revenue',
|
| 734 |
-
'a': round(recon, 2), 'b': round(bb['this_total'], 2),
|
| 735 |
-
'gap': round(recon - bb['this_total'], 2),
|
| 736 |
-
'ok': abs(recon - bb['this_total']) <= 1.0})
|
| 737 |
-
return checks
|
|
|
|
| 1 |
+
"""Products / SKU module β SKU health: YoY movers (risers & decliners), zombie SKUs
|
| 2 |
+
(catalog rot β sellable, formerly selling, now dead), new winners, coverage collapse
|
| 3 |
+
(SKUs losing customer breadth β the FFS-recovery early-warning signal), and velocity leaders.
|
| 4 |
+
|
| 5 |
+
Per-SKU margin already lives in the Financial module (low_margin_skus); not duplicated here.
|
| 6 |
+
Basket / co-purchase (MBA) is intentionally deferred β the existing client app computes it
|
| 7 |
+
runtime-side, and a local co-occurrence pull over 74k LTM lines is the kind of heavy job the
|
| 8 |
+
project guardrails keep off the local PC. (See BACKLOG.)
|
| 9 |
+
|
| 10 |
+
All line-level (sale.order.line), RI+FFS scope, excluded accounts removed β reusing sale_line_domain.
|
| 11 |
+
Coverage (distinct customers per SKU) uses a 2-level read_group and is a touch slow (~15s/
|
| 12 |
+
window), so it's its own function the app calls lazily and caches.
|
| 13 |
+
"""
|
| 14 |
+
import sys
|
| 15 |
+
import datetime as dt
|
| 16 |
+
from pathlib import Path
|
| 17 |
+
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
| 18 |
+
from functools import lru_cache
|
| 19 |
+
import core.odoo as O
|
| 20 |
+
import core.periods as P
|
| 21 |
+
import modules.sales as sales_mod
|
| 22 |
+
|
| 23 |
+
# SKU-health views are about real products β exclude service/delivery pseudo-SKUs
|
| 24 |
+
# (Delivery Charges otherwise dominate movers/winners). Dot-path FILTER works (groupby
|
| 25 |
+
# on the dot-path does not). Validation reconciles on this same non-service universe.
|
| 26 |
+
_NO_SVC = [('product_id.type', '!=', 'service')]
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
@lru_cache(maxsize=1)
|
| 30 |
+
def _code_map():
|
| 31 |
+
"""product_id β SKU code. Multiple product records can share a default_code (re-SKUing
|
| 32 |
+
/duplicates); keying by code merges them so a re-coded item doesn't read as a fake
|
| 33 |
+
decliner + fake riser. Products without a code fall back to a per-id key. Archived
|
| 34 |
+
products are INCLUDED β re-SKUing typically archives the old record and creates a new
|
| 35 |
+
one under the same code, and the old record still carries last-year sales."""
|
| 36 |
+
prods = O.search_read('product.product', [('active', 'in', [True, False])], ['id', 'default_code'],
|
| 37 |
+
limit=50000)
|
| 38 |
+
return {p['id']: (str(p['default_code']).strip() if p.get('default_code') else f"pid:{p['id']}")
|
| 39 |
+
for p in prods}
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def _sku_rev(date_from, date_to, team_id=None):
|
| 43 |
+
"""{sku_code: {'name','rev','qty','orders'}} over a window (services excluded,
|
| 44 |
+
duplicate product records merged by SKU code)."""
|
| 45 |
+
g = O.read_group('sale.order.line', O.sale_line_domain(date_from, date_to, team_id, extra=_NO_SVC),
|
| 46 |
+
['price_subtotal:sum', 'product_uom_qty:sum'], ['product_id'], lazy=False)
|
| 47 |
+
codes = _code_map()
|
| 48 |
+
out = {}
|
| 49 |
+
for r in g:
|
| 50 |
+
pid = O.m2o_id(r.get('product_id'))
|
| 51 |
+
if not pid:
|
| 52 |
+
continue
|
| 53 |
+
key = codes.get(pid, f"pid:{pid}")
|
| 54 |
+
e = out.setdefault(key, {'name': O.m2o_name(r.get('product_id')),
|
| 55 |
+
'rev': 0.0, 'qty': 0.0, 'orders': 0})
|
| 56 |
+
e['rev'] += r.get('price_subtotal') or 0.0
|
| 57 |
+
e['qty'] += r.get('product_uom_qty') or 0.0
|
| 58 |
+
e['orders'] += r.get('__count') or 0
|
| 59 |
+
return out
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def yoy_movers(t=None, limit=20, team_id=None):
|
| 63 |
+
"""Top SKU risers and decliners by YTD-vs-same-period-LY revenue change."""
|
| 64 |
+
t = t or P.today()
|
| 65 |
+
yf, yt = P.ytd(t)
|
| 66 |
+
lf, lt = P.ytd_last_year(t)
|
| 67 |
+
this = _sku_rev(yf, yt, team_id)
|
| 68 |
+
last = _sku_rev(lf, lt, team_id)
|
| 69 |
+
rows = []
|
| 70 |
+
for pid in set(this) | set(last):
|
| 71 |
+
tr = this.get(pid, {'rev': 0.0, 'name': last.get(pid, {}).get('name', '')})
|
| 72 |
+
lr = last.get(pid, {'rev': 0.0})
|
| 73 |
+
name = this.get(pid, {}).get('name') or last.get(pid, {}).get('name') or ''
|
| 74 |
+
rows.append({'code': pid, 'product': name, 'rev_ytd': tr['rev'], 'rev_ly': lr['rev'],
|
| 75 |
+
'change': tr['rev'] - lr['rev']})
|
| 76 |
+
risers = sorted([r for r in rows if r['change'] > 0], key=lambda x: -x['change'])[:limit]
|
| 77 |
+
decliners = sorted([r for r in rows if r['change'] < 0], key=lambda x: x['change'])[:limit]
|
| 78 |
+
return {'risers': risers, 'decliners': decliners}
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
def zombie_skus(t=None, limit=30, min_prior=1500.0, team_id=None):
|
| 82 |
+
"""Catalog rot: SKUs that sold materially last year but are ~dead this year."""
|
| 83 |
+
t = t or P.today()
|
| 84 |
+
yf, yt = P.ytd(t)
|
| 85 |
+
lf, lt = P.ytd_last_year(t)
|
| 86 |
+
this = _sku_rev(yf, yt, team_id)
|
| 87 |
+
last = _sku_rev(lf, lt, team_id)
|
| 88 |
+
rows = []
|
| 89 |
+
for pid, lr in last.items():
|
| 90 |
+
if lr['rev'] < min_prior:
|
| 91 |
+
continue
|
| 92 |
+
tr = this.get(pid, {'rev': 0.0})
|
| 93 |
+
if tr['rev'] > 0.05 * lr['rev']: # still selling at >5% of prior β not a zombie
|
| 94 |
+
continue
|
| 95 |
+
rows.append({'code': pid, 'product': lr['name'], 'rev_ly': lr['rev'], 'rev_ytd': tr['rev'],
|
| 96 |
+
'lost': lr['rev'] - tr['rev']})
|
| 97 |
+
rows.sort(key=lambda x: -x['lost'])
|
| 98 |
+
return rows[:limit]
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
def new_winners(t=None, limit=20, min_this=1500.0, team_id=None):
|
| 102 |
+
"""SKUs that barely sold last year but are selling well this year."""
|
| 103 |
+
t = t or P.today()
|
| 104 |
+
yf, yt = P.ytd(t)
|
| 105 |
+
lf, lt = P.ytd_last_year(t)
|
| 106 |
+
this = _sku_rev(yf, yt, team_id)
|
| 107 |
+
last = _sku_rev(lf, lt, team_id)
|
| 108 |
+
rows = []
|
| 109 |
+
for pid, tr in this.items():
|
| 110 |
+
if tr['rev'] < min_this:
|
| 111 |
+
continue
|
| 112 |
+
lr = last.get(pid, {'rev': 0.0})
|
| 113 |
+
if lr['rev'] > 0.05 * tr['rev']:
|
| 114 |
+
continue
|
| 115 |
+
rows.append({'code': pid, 'product': tr['name'], 'rev_ytd': tr['rev'], 'rev_ly': lr['rev'],
|
| 116 |
+
'gained': tr['rev'] - lr['rev']})
|
| 117 |
+
rows.sort(key=lambda x: -x['gained'])
|
| 118 |
+
return rows[:limit]
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def velocity_leaders(t=None, limit=25, team_id=None):
|
| 122 |
+
"""Top SKUs by LTM unit velocity (units/month) and revenue."""
|
| 123 |
+
t = t or P.today()
|
| 124 |
+
lf, lt = P.ltm(t)
|
| 125 |
+
sku = _sku_rev(lf, lt, team_id)
|
| 126 |
+
rows = [{'code': k, 'product': v['name'], 'units_ltm': v['qty'], 'units_per_mo': v['qty'] / 12.0,
|
| 127 |
+
'rev_ltm': v['rev'], 'orders_ltm': v['orders']} for k, v in sku.items()]
|
| 128 |
+
rows.sort(key=lambda x: -x['units_ltm'])
|
| 129 |
+
return rows[:limit]
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def _code_category():
|
| 133 |
+
"""code -> category name. Built from _code_map (pid->code) + the sales category map
|
| 134 |
+
(pid->category); the first record carrying a code sets that code's category."""
|
| 135 |
+
codes = _code_map()
|
| 136 |
+
cats = sales_mod._product_cat()
|
| 137 |
+
out = {}
|
| 138 |
+
for pid, code in codes.items():
|
| 139 |
+
if code not in out and pid in cats:
|
| 140 |
+
out[code] = cats[pid]
|
| 141 |
+
return out
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
def directory(t=None, team_id=None):
|
| 145 |
+
"""Every SKU that sold this YTD or the same period last year, with the fields the SKU
|
| 146 |
+
drill-down filters/sorts on: category, YTD/LY revenue, YoY %, units and orders. Mirrors the
|
| 147 |
+
customer directory β the full list (filtering happens in the UI), sorted by YTD revenue."""
|
| 148 |
+
t = t or P.today()
|
| 149 |
+
yf, yt = P.ytd(t)
|
| 150 |
+
lf, lt = P.ytd_last_year(t)
|
| 151 |
+
this = _sku_rev(yf, yt, team_id)
|
| 152 |
+
last = _sku_rev(lf, lt, team_id)
|
| 153 |
+
code_cat = _code_category()
|
| 154 |
+
rows = []
|
| 155 |
+
for code in set(this) | set(last):
|
| 156 |
+
tr = this.get(code, {})
|
| 157 |
+
lr = last.get(code, {})
|
| 158 |
+
rev_ytd = tr.get('rev', 0.0)
|
| 159 |
+
rev_ly = lr.get('rev', 0.0)
|
| 160 |
+
rows.append({'code': code, 'product': tr.get('name') or lr.get('name') or code,
|
| 161 |
+
'category': code_cat.get(code, '(uncategorized)'),
|
| 162 |
+
'rev_ytd': rev_ytd, 'rev_ly': rev_ly, 'change': rev_ytd - rev_ly,
|
| 163 |
+
'yoy_pct': P.yoy_pct(rev_ytd, rev_ly),
|
| 164 |
+
'qty_ytd': tr.get('qty', 0.0), 'orders_ytd': tr.get('orders', 0)})
|
| 165 |
+
rows.sort(key=lambda r: -r['rev_ytd'])
|
| 166 |
+
return rows
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
def catalogue():
|
| 170 |
+
"""`{sku_code: {'product', 'category', 'id'}}` β EVERY ACTIVE product, sold or not.
|
| 171 |
+
|
| 172 |
+
β THIS IS THE CATALOGUE UNIVERSE, AND IT IS DELIBERATELY NOT `directory()`. `directory()`'s
|
| 173 |
+
row set IS the union of two revenue `read_group`s over `sale.order.line` (`:155`), so a SKU
|
| 174 |
+
that never sold cannot exist in it. That is CORRECT for its own callers β `yoy_movers`,
|
| 175 |
+
`zombie_skus`, `new_winners` and `categories` all legitimately want a sales-window universe β
|
| 176 |
+
and it is wrong for the PRODUCT GRID, which is a catalogue and was therefore showing 2,717 of
|
| 177 |
+
5,875 SKUs (wave 29, owner item 22 / ruling R12). The fix is this function plus a LEFT JOIN in
|
| 178 |
+
the consumer, never a window removed from `directory()`: removing the window alone lands at
|
| 179 |
+
3,327 (all-time-sold), because ~2,550 active SKUs have never sold in wholesale scope at all.
|
| 180 |
+
|
| 181 |
+
β NOT BU-SHAPED, and it cannot be: `product.product` carries no team. A catalogue is one
|
| 182 |
+
catalogue. `directory()` stays the BU-shaped half, which is why the two are joined rather than
|
| 183 |
+
merged β a scoped caller gets every SKU with ITS OWN revenue, blank where that BU never sold.
|
| 184 |
+
|
| 185 |
+
Keyed exactly like `_code_map()` β the `default_code`, or a `pid:N` fallback for the 33 active
|
| 186 |
+
records that carry none β so the join against `directory()` is code-for-code with no
|
| 187 |
+
normaliser. Names come from `display_name` (`[CODE] NAME`), which is what `O.m2o_name` yields
|
| 188 |
+
off a sale line, so a never-sold row wears the same format as a sold one.
|
| 189 |
+
|
| 190 |
+
β RAISES on a truncated read rather than returning a short catalogue. A silently short pull
|
| 191 |
+
would put the grid back at a plausible wrong number with every gate green β the exact failure
|
| 192 |
+
this function exists to end ([[no-unverifiable-aggregates]], and the same truncation guard
|
| 193 |
+
`modules/backorders.py:161` already uses).
|
| 194 |
+
"""
|
| 195 |
+
dom = [('active', '=', True)]
|
| 196 |
+
prods = O.search_read('product.product', dom, ['id', 'default_code', 'display_name', 'name'],
|
| 197 |
+
limit=50000)
|
| 198 |
+
n = O.get_odoo().search_count('product.product', dom)
|
| 199 |
+
if len(prods) != n:
|
| 200 |
+
raise ValueError(
|
| 201 |
+
f"products.catalogue: the product pull is TRUNCATED β read {len(prods)} rows against "
|
| 202 |
+
f"a search_count of {n}. A short catalogue renders as a plausible smaller grid with "
|
| 203 |
+
f"nothing reporting it; raise the limit before shipping this.")
|
| 204 |
+
code_cat = _code_category()
|
| 205 |
+
out = {}
|
| 206 |
+
for p in prods:
|
| 207 |
+
code = (str(p['default_code']).strip() if p.get('default_code') else f"pid:{p['id']}")
|
| 208 |
+
# First record wins, matching `_code_category`'s own convention. MEASURED 2026-08-11: zero
|
| 209 |
+
# active products share a code, so this branch is a guard, not a merge policy.
|
| 210 |
+
# ββ W33-T43 (R2 / amendment A2) β `id` RIDES THE ROW. Odoo's `product.product` id is
|
| 211 |
+
# already in hand (the `pid:{id}` fallback above uses it and then threw it away), and it is
|
| 212 |
+
# the ONE column `ut_odoo_products` had that `product_data` lacked. R2 merges onto the
|
| 213 |
+
# legacy key and keeps every data column, so retiring the twin without this would lose a
|
| 214 |
+
# real identifier rather than a navigational link.
|
| 215 |
+
# β A product's grid pid is `crc32(default_code)`, NOT the Odoo id β unlike a CUSTOMER row,
|
| 216 |
+
# whose pid IS the partner id. That asymmetry is exactly why this cannot be derived
|
| 217 |
+
# downstream in `aios_grid.py` and has to be carried from the source read.
|
| 218 |
+
out.setdefault(code, {'product': p.get('display_name') or p.get('name') or code,
|
| 219 |
+
'category': code_cat.get(code, '(uncategorized)'),
|
| 220 |
+
'id': p['id']})
|
| 221 |
+
return out
|
| 222 |
+
|
| 223 |
+
|
| 224 |
+
#: ββ WAVE 30 / W30-T34 (the carried W29-T52) β THE PRICELIST STRATUM, PER LIST.
|
| 225 |
+
#:
|
| 226 |
+
#: MEASURED LIVE 2026-08-12, and every one of these numbers shaped the design rather than
|
| 227 |
+
#: decorating it:
|
| 228 |
+
#:
|
| 229 |
+
#: Β· **Five pricelists, three of them material.** Date-valid fixed rules: Fisch 5,139,
|
| 230 |
+
#: Royal 1 2,702, Royal 2 2,605, Public Pricelist 17, Giftware Deals 1. At SKU level that
|
| 231 |
+
#: is Fisch 3,971 Β· Royal 1 1,920 Β· Royal 2 1,857 Β· Public 6 Β· Giftware 0.
|
| 232 |
+
#: Β· **3,988 of 5,871 active products carry a usable fixed rule; 1,883 carry NONE.**
|
| 233 |
+
#: Β· **1,853 SKUs are priced on all three lists AND THE LISTS DISAGREE** β median relative
|
| 234 |
+
#: spread 16%, max 82%. That is why this is three columns rather than one "the price" or a
|
| 235 |
+
#: low/high pair: neither can tell a Fisch rep what Fisch sells the SKU for, which is the
|
| 236 |
+
#: only question the column exists to answer.
|
| 237 |
+
#: Β· β **`list_price` is 1.00 on 5,817 of the 5,871** β so the single `3_global` rule
|
| 238 |
+
#: computed over it is meaningless, exactly as W29-T52 said. **This reader never falls
|
| 239 |
+
#: through to it**, and that is the ticket's own negative control: a SKU with no specific
|
| 240 |
+
#: item renders BLANK, never a fallback dressed as a price.
|
| 241 |
+
#:
|
| 242 |
+
#: β RESOLVED BY NAME, NOT BY ID. A pricelist renamed or deleted in Odoo makes its column go
|
| 243 |
+
#: blank and makes `report["lists_missing"]` name it β a hard-coded id would keep pointing at
|
| 244 |
+
#: whatever inherited it and mislabel every cell silently.
|
| 245 |
+
PRICELIST_COLUMNS = (
|
| 246 |
+
("price_fisch", "Fisch"),
|
| 247 |
+
("price_royal_1", "Royal 1"),
|
| 248 |
+
("price_royal_2", "Royal 2"),
|
| 249 |
+
)
|
| 250 |
+
|
| 251 |
+
|
| 252 |
+
def pricelist_by_code():
|
| 253 |
+
"""`({code: {column_key: price}}, report)` β the date-valid FIXED base-tier price for each
|
| 254 |
+
declared pricelist, keyed EXACTLY as `catalogue()` keys its rows.
|
| 255 |
+
|
| 256 |
+
β THE KEYING IS LOAD-BEARING, not a detail. `catalogue()` keys on `default_code` with a
|
| 257 |
+
`pid:{id}` fallback for the ~33 active records that carry none. Keying this map any other
|
| 258 |
+
way would leave those SKUs permanently blank while an oracle counting active products
|
| 259 |
+
counted them β a red gate on a working build, or worse, a silent hole nobody counts.
|
| 260 |
+
|
| 261 |
+
**The base tier, deliberately.** `pricecomp._tier_for` picks the highest `min_quantity` at
|
| 262 |
+
or below an order's quantity, because it is pricing a LINE that has one. A catalogue column
|
| 263 |
+
has no quantity in hand, so it takes the LOWEST `min_quantity` β the price at qty 1. Rules
|
| 264 |
+
above that break are a bulk price, and `report["qty_break_only"]` counts the SKUs whose only
|
| 265 |
+
rule sits on one (MEASURED: 10 rules of 10,464 carry a break at all).
|
| 266 |
+
|
| 267 |
+
**Variant rules beat template rules**, matching `pricecomp._tier_for`: a `0_product_variant`
|
| 268 |
+
rule is the more specific statement about this exact SKU.
|
| 269 |
+
|
| 270 |
+
β DEGRADES TO `({}, report)` on a read failure, matching `_inventory_by_code` rather than
|
| 271 |
+
`catalogue()`: these are COLUMNS, and a product grid that will not render because pricing is
|
| 272 |
+
momentarily unreachable is a worse failure than one with blank price columns. The blanks are
|
| 273 |
+
not silent β `product_data.validate()`'s coverage leg reconciles each column against a fresh
|
| 274 |
+
Odoo count and goes red at zero.
|
| 275 |
+
|
| 276 |
+
β **The report exists because R6's second sentence is law** (*"if it can't be done, you need
|
| 277 |
+
to explicitly tell me why and recommend a fix"*). Everything this reader CANNOT see is
|
| 278 |
+
counted rather than dropped: rules on pricelists the contract does not declare, non-fixed
|
| 279 |
+
(`formula`/`percent`) rules, the `3_global` fallback, and prices that only exist above a
|
| 280 |
+
quantity break.
|
| 281 |
+
"""
|
| 282 |
+
# β THE COUNTERS ARE DISJOINT AND ORDER-DEPENDENT, and saying so is the difference between a
|
| 283 |
+
# report and a misleading one. A rule is classified ONCE, by the first reason it is skipped:
|
| 284 |
+
# undeclared list β out of date β not fixed β global β zero price. So `rules_not_fixed: 0`
|
| 285 |
+
# means "no formula rule on a list we declare", NOT "this Odoo has no formula rules"
|
| 286 |
+
# (MEASURED 2026-08-12: it has exactly one, and it sits on Public Pricelist, which the
|
| 287 |
+
# contract does not declare β so it lands in `rules_undeclared_list`).
|
| 288 |
+
report = {"lists_missing": [], "rules_total": 0, "rules_undeclared_list": 0,
|
| 289 |
+
"rules_not_fixed": 0, "rules_global": 0, "rules_out_of_date": 0,
|
| 290 |
+
"rules_zero_price": 0, "qty_break_only": 0}
|
| 291 |
+
try:
|
| 292 |
+
pl_rows = O.search_read('product.pricelist', [], ['id', 'name'])
|
| 293 |
+
by_name = {}
|
| 294 |
+
for p in pl_rows:
|
| 295 |
+
by_name.setdefault(str(p.get('name') or '').strip(), p['id'])
|
| 296 |
+
wanted = {}
|
| 297 |
+
for col, name in PRICELIST_COLUMNS:
|
| 298 |
+
pid = by_name.get(name)
|
| 299 |
+
if pid is None:
|
| 300 |
+
report["lists_missing"].append(name)
|
| 301 |
+
else:
|
| 302 |
+
wanted[pid] = col
|
| 303 |
+
|
| 304 |
+
# ββ FILTERED SERVER-SIDE, AND THE REASON IS A MEASUREMENT, NOT A STYLE PREFERENCE.
|
| 305 |
+
# Reading all 10,464 rules with 9 fields and sorting them in Python costs **30.1s** on
|
| 306 |
+
# this connection; the same rules under a server-side domain with 6 fields cost
|
| 307 |
+
# **13.6s** (measured 2026-08-12, back to back). `pool()` runs this on a scope's
|
| 308 |
+
# first-ever build, so that 16.5s is 16.5s of somebody's page load.
|
| 309 |
+
#
|
| 310 |
+
# β The domain reproduces the Python predicate EXACTLY, and the null legs are the part
|
| 311 |
+
# that is easy to get wrong: an absent `date_start` is `False`, not a past date, so
|
| 312 |
+
# `('date_start','<=',today)` ALONE would drop every open-ended rule β which is almost
|
| 313 |
+
# all of them.
|
| 314 |
+
today = P.today().isoformat()
|
| 315 |
+
_declared = sorted(wanted)
|
| 316 |
+
_kinds = ['0_product_variant', '1_product']
|
| 317 |
+
_live = [('pricelist_id', 'in', _declared), ('compute_price', '=', 'fixed'),
|
| 318 |
+
('applied_on', 'in', _kinds),
|
| 319 |
+
'|', ('date_start', '=', False), ('date_start', '<=', today),
|
| 320 |
+
'|', ('date_end', '=', False), ('date_end', '>=', today),
|
| 321 |
+
('fixed_price', '>', 0)]
|
| 322 |
+
rules = O.search_read(
|
| 323 |
+
'product.pricelist.item', _live,
|
| 324 |
+
['pricelist_id', 'product_id', 'product_tmpl_id', 'applied_on', 'fixed_price',
|
| 325 |
+
'min_quantity']) if _declared else []
|
| 326 |
+
|
| 327 |
+
# β R6's SECOND SENTENCE, PAID FOR WITH `search_count` RATHER THAN A WIDER READ. Six
|
| 328 |
+
# counts cost ~1.2s together; the rules they count would cost 16s to read. What this
|
| 329 |
+
# reader cannot see is still REPORTED β it is just no longer transferred.
|
| 330 |
+
def _n(extra):
|
| 331 |
+
try:
|
| 332 |
+
return O.get_odoo().search_count('product.pricelist.item', extra)
|
| 333 |
+
except Exception:
|
| 334 |
+
return -1 # -1 reads as "not measured", never as zero
|
| 335 |
+
_dated = ['|', ('date_start', '=', False), ('date_start', '<=', today),
|
| 336 |
+
'|', ('date_end', '=', False), ('date_end', '>=', today)]
|
| 337 |
+
report["rules_total"] = _n([])
|
| 338 |
+
report["rules_undeclared_list"] = _n([('pricelist_id', 'not in', _declared)]) \
|
| 339 |
+
if _declared else report["rules_total"]
|
| 340 |
+
if _declared:
|
| 341 |
+
_on = [('pricelist_id', 'in', _declared)]
|
| 342 |
+
# `percent_price` and formula rules are read NOWHERE in this repo. A rule we cannot
|
| 343 |
+
# price is one the operator is TOLD about, never one that quietly becomes a blank.
|
| 344 |
+
report["rules_not_fixed"] = _n(_on + [('compute_price', '!=', 'fixed')])
|
| 345 |
+
report["rules_global"] = _n(_on + [('compute_price', '=', 'fixed'),
|
| 346 |
+
('applied_on', 'not in', _kinds)])
|
| 347 |
+
report["rules_out_of_date"] = (
|
| 348 |
+
_n(_on + [('compute_price', '=', 'fixed'), ('applied_on', 'in', _kinds)])
|
| 349 |
+
- _n(_on + [('compute_price', '=', 'fixed'),
|
| 350 |
+
('applied_on', 'in', _kinds)] + _dated))
|
| 351 |
+
# A zero price is not "free" β it is an unset rule. Blank says so; 0 does not.
|
| 352 |
+
report["rules_zero_price"] = _n(
|
| 353 |
+
_on + [('compute_price', '=', 'fixed'), ('applied_on', 'in', _kinds)]
|
| 354 |
+
+ _dated + [('fixed_price', '<=', 0)])
|
| 355 |
+
|
| 356 |
+
by_var, by_tmpl = {}, {}
|
| 357 |
+
for r in rules:
|
| 358 |
+
col = wanted[O.m2o_id(r.get('pricelist_id'))]
|
| 359 |
+
if r.get('applied_on') == '0_product_variant' and r.get('product_id'):
|
| 360 |
+
by_var.setdefault((col, O.m2o_id(r['product_id'])), []).append(r)
|
| 361 |
+
elif r.get('product_tmpl_id'):
|
| 362 |
+
by_tmpl.setdefault((col, O.m2o_id(r['product_tmpl_id'])), []).append(r)
|
| 363 |
+
|
| 364 |
+
dom = [('active', '=', True)]
|
| 365 |
+
prods = O.search_read('product.product', dom,
|
| 366 |
+
['id', 'default_code', 'product_tmpl_id'], limit=50000)
|
| 367 |
+
n = O.get_odoo().search_count('product.product', dom)
|
| 368 |
+
if len(prods) != n:
|
| 369 |
+
# Same guard, same reason as `catalogue()`: a short pull renders as a plausible
|
| 370 |
+
# smaller set of priced SKUs with nothing reporting it.
|
| 371 |
+
raise ValueError(
|
| 372 |
+
f"products.pricelist_by_code: the product pull is TRUNCATED β read {len(prods)} "
|
| 373 |
+
f"rows against a search_count of {n}.")
|
| 374 |
+
except Exception as e:
|
| 375 |
+
report["error"] = f"{type(e).__name__}: {str(e)[:200]}"
|
| 376 |
+
return {}, report
|
| 377 |
+
|
| 378 |
+
out = {}
|
| 379 |
+
for p in prods:
|
| 380 |
+
code = (str(p['default_code']).strip() if p.get('default_code') else f"pid:{p['id']}")
|
| 381 |
+
tmpl = O.m2o_id(p.get('product_tmpl_id'))
|
| 382 |
+
cells = {}
|
| 383 |
+
for col, _name in PRICELIST_COLUMNS:
|
| 384 |
+
cands = by_var.get((col, p['id'])) or by_tmpl.get((col, tmpl))
|
| 385 |
+
if not cands:
|
| 386 |
+
continue
|
| 387 |
+
base = min(cands, key=lambda r: r.get('min_quantity') or 0.0)
|
| 388 |
+
if (base.get('min_quantity') or 0.0) > 1.0:
|
| 389 |
+
report["qty_break_only"] += 1
|
| 390 |
+
cells[col] = base.get('fixed_price')
|
| 391 |
+
if cells:
|
| 392 |
+
out.setdefault(code, {}).update(cells)
|
| 393 |
+
return out, report
|
| 394 |
+
|
| 395 |
+
|
| 396 |
+
def catalogue_count():
|
| 397 |
+
"""The INDEPENDENT population oracle: Odoo's own count of active products.
|
| 398 |
+
|
| 399 |
+
Deliberately a bare `search_count` and not a `len()` over anything this module built β the
|
| 400 |
+
2,717 defect shipped silently for a wave because `product_data.validate()` derived BOTH sides
|
| 401 |
+
of its reconciliation from `_sku_rev`, so the oracle could never see a missing row.
|
| 402 |
+
"""
|
| 403 |
+
return O.get_odoo().search_count('product.product', [('active', '=', True)])
|
| 404 |
+
|
| 405 |
+
|
| 406 |
+
def categories(t=None, team_id=None):
|
| 407 |
+
"""Sorted distinct category names present in the SKU directory (for the drill-down filter)."""
|
| 408 |
+
return sorted({r['category'] for r in directory(t, team_id)})
|
| 409 |
+
|
| 410 |
+
|
| 411 |
+
def _coverage(date_from, date_to, team_id=None):
|
| 412 |
+
"""{sku_code: distinct_customer_count} via 2-level read_group (slow-ish), merged by code.
|
| 413 |
+
(A customer buying two records sharing a code can count twice; duplicates are rare, and
|
| 414 |
+
code-level avoids the bigger error of a re-SKUed product reading as full coverage loss.)"""
|
| 415 |
+
g = O.read_group('sale.order.line', O.sale_line_domain(date_from, date_to, team_id, extra=_NO_SVC),
|
| 416 |
+
['__count'], ['product_id', 'order_partner_id'], lazy=False)
|
| 417 |
+
codes = _code_map()
|
| 418 |
+
cov = {}
|
| 419 |
+
names = {}
|
| 420 |
+
for r in g:
|
| 421 |
+
pid = O.m2o_id(r.get('product_id'))
|
| 422 |
+
if not pid:
|
| 423 |
+
continue
|
| 424 |
+
key = codes.get(pid, f"pid:{pid}")
|
| 425 |
+
cov[key] = cov.get(key, 0) + 1
|
| 426 |
+
names.setdefault(key, O.m2o_name(r.get('product_id')))
|
| 427 |
+
return cov, names
|
| 428 |
+
|
| 429 |
+
|
| 430 |
+
def coverage_collapse(t=None, limit=25, min_prior_custs=8, team_id=None):
|
| 431 |
+
"""SKUs that lost the most customer breadth YoY β early warning a SKU is dying even if
|
| 432 |
+
revenue hasn't fully cratered yet."""
|
| 433 |
+
t = t or P.today()
|
| 434 |
+
yf, yt = P.ytd(t)
|
| 435 |
+
lf, lt = P.ytd_last_year(t)
|
| 436 |
+
cov_t, names_t = _coverage(yf, yt, team_id)
|
| 437 |
+
cov_l, names_l = _coverage(lf, lt, team_id)
|
| 438 |
+
rows = []
|
| 439 |
+
for pid, lc in cov_l.items():
|
| 440 |
+
if lc < min_prior_custs:
|
| 441 |
+
continue
|
| 442 |
+
tc = cov_t.get(pid, 0)
|
| 443 |
+
drop = lc - tc
|
| 444 |
+
if drop <= 0:
|
| 445 |
+
continue
|
| 446 |
+
rows.append({'code': pid, 'product': names_l.get(pid) or names_t.get(pid) or '',
|
| 447 |
+
'custs_ly': lc, 'custs_ytd': tc, 'lost_custs': drop,
|
| 448 |
+
'pct_drop': drop / lc * 100})
|
| 449 |
+
rows.sort(key=lambda x: (-x['lost_custs'], -x['pct_drop']))
|
| 450 |
+
return rows[:limit]
|
| 451 |
+
|
| 452 |
+
|
| 453 |
+
# ====================================================================== SKU DRAWER (mirror of customer drawer)
|
| 454 |
+
def _sku_product_ids(code):
|
| 455 |
+
"""All product.product ids sharing this SKU code (merged variants / archived records)."""
|
| 456 |
+
return [pid for pid, c in _code_map().items() if c == code]
|
| 457 |
+
|
| 458 |
+
|
| 459 |
+
def _sku_dom(pids, date_from, date_to, team_id=None):
|
| 460 |
+
return O.sale_line_domain(date_from, date_to, team_id, extra=[('product_id', 'in', pids)] + _NO_SVC)
|
| 461 |
+
|
| 462 |
+
|
| 463 |
+
def _sku_name_category(pids):
|
| 464 |
+
rows = O.search_read('product.product', [('id', 'in', pids)], ['name', 'categ_id'])
|
| 465 |
+
name = rows[0]['name'] if rows else '(unknown)'
|
| 466 |
+
catmap = sales_mod._product_cat()
|
| 467 |
+
main = next((catmap.get(p) for p in pids if catmap.get(p)), '(uncategorized)')
|
| 468 |
+
return name, main
|
| 469 |
+
|
| 470 |
+
|
| 471 |
+
def _sku_buyers(pids, date_from, date_to, team_id=None):
|
| 472 |
+
"""{partner_id: {'name','rev','qty'}} for buyers of this SKU over a window."""
|
| 473 |
+
g = O.read_group('sale.order.line', _sku_dom(pids, date_from, date_to, team_id),
|
| 474 |
+
['price_subtotal:sum', 'product_uom_qty:sum'], ['order_partner_id'], lazy=False)
|
| 475 |
+
out = {}
|
| 476 |
+
for r in g:
|
| 477 |
+
pid = O.m2o_id(r.get('order_partner_id'))
|
| 478 |
+
if pid:
|
| 479 |
+
out[pid] = {'name': O.m2o_name(r.get('order_partner_id')),
|
| 480 |
+
'rev': r.get('price_subtotal') or 0.0, 'qty': r.get('product_uom_qty') or 0.0}
|
| 481 |
+
return out
|
| 482 |
+
|
| 483 |
+
|
| 484 |
+
def sku_detail(code, t=None, team_id=None, n_months=13, allsku=None):
|
| 485 |
+
"""KPIs (rev/qty/buyers YoY, GM%), monthly trend, rank & % of BU for one SKU code.
|
| 486 |
+
Pass `allsku` (a cached _sku_rev YTD map) to skip the ~all-SKU rank read."""
|
| 487 |
+
t = t or P.today()
|
| 488 |
+
pids = _sku_product_ids(code)
|
| 489 |
+
if not pids:
|
| 490 |
+
return None
|
| 491 |
+
yf, yt = P.ytd(t)
|
| 492 |
+
lf, lt = P.ytd_last_year(t)
|
| 493 |
+
mf, mt = P.ltm(t)
|
| 494 |
+
name, category = _sku_name_category(pids)
|
| 495 |
+
|
| 496 |
+
def s(df, dtt, field='price_subtotal'):
|
| 497 |
+
return O.sum_field('sale.order.line', _sku_dom(pids, df, dtt, team_id), field)
|
| 498 |
+
rev_ytd, rev_ly = s(yf, yt), s(lf, lt)
|
| 499 |
+
qty_ytd, qty_ly = s(yf, yt, 'product_uom_qty'), s(lf, lt, 'product_uom_qty')
|
| 500 |
+
buyers_ytd = len(_sku_buyers(pids, yf, yt, team_id))
|
| 501 |
+
buyers_ly = len(_sku_buyers(pids, lf, lt, team_id))
|
| 502 |
+
g = O.read_group('sale.order.line', _sku_dom(pids, mf, mt, team_id),
|
| 503 |
+
['price_subtotal:sum', 'margin:sum'], [], lazy=False)
|
| 504 |
+
line_rev = (g[0].get('price_subtotal') if g else 0) or 0.0
|
| 505 |
+
margin = (g[0].get('margin') if g else 0) or 0.0
|
| 506 |
+
|
| 507 |
+
# Monthly trend: bin lines by their order's month in Python (dot-path month groupby is rejected
|
| 508 |
+
# on sale.order.line), in 2 reads instead of 26 point queries.
|
| 509 |
+
range_start = dt.date(t.year - 2, t.month, 1).isoformat()
|
| 510 |
+
lines = O.search_read('sale.order.line', _sku_dom(pids, range_start, t.isoformat(), team_id),
|
| 511 |
+
['price_subtotal', 'order_id'])
|
| 512 |
+
oids = list({O.m2o_id(line['order_id']) for line in lines if line.get('order_id')})
|
| 513 |
+
odate = {}
|
| 514 |
+
for i in range(0, len(oids), 1000):
|
| 515 |
+
for o in O.search_read('sale.order', [('id', 'in', oids[i:i + 1000])], ['date_order']):
|
| 516 |
+
if o.get('date_order'):
|
| 517 |
+
odate[o['id']] = str(o['date_order'])[:7]
|
| 518 |
+
mrev = {}
|
| 519 |
+
for line in lines:
|
| 520 |
+
ym = odate.get(O.m2o_id(line.get('order_id')))
|
| 521 |
+
if ym:
|
| 522 |
+
mrev[ym] = mrev.get(ym, 0.0) + (line.get('price_subtotal') or 0.0)
|
| 523 |
+
monthly = []
|
| 524 |
+
for ym, start, end in P.month_starts(n_months, t):
|
| 525 |
+
y, m = int(ym[:4]) - 1, int(ym[5:7])
|
| 526 |
+
monthly.append({'month': ym, 'revenue': mrev.get(ym, 0.0),
|
| 527 |
+
'revenue_ly': mrev.get(f'{y:04d}-{m:02d}', 0.0)})
|
| 528 |
+
|
| 529 |
+
allsku = allsku if allsku is not None else _sku_rev(yf, yt, team_id)
|
| 530 |
+
total = sum(v['rev'] for v in allsku.values()) or 1.0
|
| 531 |
+
rank = next((i + 1 for i, (c, _v) in enumerate(sorted(allsku.items(), key=lambda kv: -kv[1]['rev']))
|
| 532 |
+
if c == code), None)
|
| 533 |
+
return {
|
| 534 |
+
'code': code, 'name': name, 'category': category,
|
| 535 |
+
'rev_ytd': rev_ytd, 'rev_ly': rev_ly, 'rev_yoy_pct': P.yoy_pct(rev_ytd, rev_ly),
|
| 536 |
+
'qty_ytd': qty_ytd, 'qty_ly': qty_ly, 'qty_yoy_pct': P.yoy_pct(qty_ytd, qty_ly),
|
| 537 |
+
'buyers_ytd': buyers_ytd, 'buyers_ly': buyers_ly, 'buyers_delta': buyers_ytd - buyers_ly,
|
| 538 |
+
'gm_pct': (margin / line_rev * 100) if line_rev else 0.0, 'gm_dollars': margin,
|
| 539 |
+
'rank': rank, 'n_skus': len(allsku), 'pct_of_bu': rev_ytd / total * 100,
|
| 540 |
+
'monthly': monthly,
|
| 541 |
+
}
|
| 542 |
+
|
| 543 |
+
|
| 544 |
+
def sku_buyer_bridge(code, t=None, team_id=None, top=12):
|
| 545 |
+
"""Who drives the SKU's YoY: retained / new / churned buyers + a $-ranked churned call list."""
|
| 546 |
+
t = t or P.today()
|
| 547 |
+
pids = _sku_product_ids(code)
|
| 548 |
+
yf, yt = P.ytd(t)
|
| 549 |
+
lf, lt = P.ytd_last_year(t)
|
| 550 |
+
this, last = _sku_buyers(pids, yf, yt, team_id), _sku_buyers(pids, lf, lt, team_id)
|
| 551 |
+
tset, lset = set(this), set(last)
|
| 552 |
+
retained, new, churned = tset & lset, tset - lset, lset - tset
|
| 553 |
+
churned_list = sorted([{'pid': p, 'customer': last[p]['name'], 'ly_rev': last[p]['rev']}
|
| 554 |
+
for p in churned], key=lambda x: -x['ly_rev'])[:top]
|
| 555 |
+
return {'retained': {'n': len(retained), 'rev': sum(this[p]['rev'] for p in retained)},
|
| 556 |
+
'new': {'n': len(new), 'rev': sum(this[p]['rev'] for p in new)},
|
| 557 |
+
'churned': {'n': len(churned), 'rev': sum(last[p]['rev'] for p in churned)},
|
| 558 |
+
'buyer_retention_pct': (len(retained) / len(lset) * 100) if lset else 0.0,
|
| 559 |
+
'churned_buyers': churned_list, 'buyers_this': len(tset), 'buyers_last': len(lset),
|
| 560 |
+
'this_total': sum(v['rev'] for v in this.values())}
|
| 561 |
+
|
| 562 |
+
|
| 563 |
+
def sku_concentration(code, t=None, team_id=None):
|
| 564 |
+
"""Buyer-concentration risk: top-1/top-3 share, Herfindahl index, effective buyer count (LTM)."""
|
| 565 |
+
t = t or P.today()
|
| 566 |
+
mf, mt = P.ltm(t)
|
| 567 |
+
buyers = _sku_buyers(_sku_product_ids(code), mf, mt, team_id)
|
| 568 |
+
revs = sorted([v['rev'] for v in buyers.values()], reverse=True)
|
| 569 |
+
total = sum(revs) or 1.0
|
| 570 |
+
hhi = sum((r / total) ** 2 for r in revs)
|
| 571 |
+
return {'n_buyers': len(revs), 'top1_pct': (revs[0] / total * 100) if revs else 0.0,
|
| 572 |
+
'top3_pct': (sum(revs[:3]) / total * 100) if revs else 0.0,
|
| 573 |
+
'hhi': hhi, 'eff_buyers': (1 / hhi) if hhi else 0.0}
|
| 574 |
+
|
| 575 |
+
|
| 576 |
+
def sku_price_dispersion(code, t=None, team_id=None, cap=60):
|
| 577 |
+
"""Realized $/unit per buyer (LTM) vs the volume-weighted average; recoverable $ on below-VWAP
|
| 578 |
+
accounts. list_price is unreliable, so the dispersion among actual buyers is the margin lever."""
|
| 579 |
+
t = t or P.today()
|
| 580 |
+
mf, mt = P.ltm(t)
|
| 581 |
+
g = O.read_group('sale.order.line', _sku_dom(_sku_product_ids(code), mf, mt, team_id),
|
| 582 |
+
['price_subtotal:sum', 'product_uom_qty:sum'], ['order_partner_id'], lazy=False)
|
| 583 |
+
rows, tot_rev, tot_qty = [], 0.0, 0.0
|
| 584 |
+
for r in g:
|
| 585 |
+
pid = O.m2o_id(r.get('order_partner_id'))
|
| 586 |
+
rev = r.get('price_subtotal') or 0.0
|
| 587 |
+
qty = r.get('product_uom_qty') or 0.0
|
| 588 |
+
if not pid or qty <= 0:
|
| 589 |
+
continue
|
| 590 |
+
rows.append({'pid': pid, 'customer': O.m2o_name(r.get('order_partner_id')),
|
| 591 |
+
'price': rev / qty, 'qty': qty, 'rev': rev})
|
| 592 |
+
tot_rev += rev
|
| 593 |
+
tot_qty += qty
|
| 594 |
+
vwap = (tot_rev / tot_qty) if tot_qty else 0.0
|
| 595 |
+
for r in rows:
|
| 596 |
+
r['recoverable'] = max(0.0, vwap - r['price']) * r['qty']
|
| 597 |
+
rows.sort(key=lambda x: -x['recoverable'])
|
| 598 |
+
return {'vwap': vwap, 'n_buyers': len(rows),
|
| 599 |
+
'recoverable_total': sum(r['recoverable'] for r in rows), 'rows': rows[:cap]}
|
| 600 |
+
|
| 601 |
+
|
| 602 |
+
def sku_top_buyers(code, t=None, team_id=None, top=15):
|
| 603 |
+
"""Ranked buyers of this SKU (YTD) with YoY β clickable to open the customer drawer."""
|
| 604 |
+
t = t or P.today()
|
| 605 |
+
pids = _sku_product_ids(code)
|
| 606 |
+
yf, yt = P.ytd(t)
|
| 607 |
+
lf, lt = P.ytd_last_year(t)
|
| 608 |
+
this, last = _sku_buyers(pids, yf, yt, team_id), _sku_buyers(pids, lf, lt, team_id)
|
| 609 |
+
rows = [{'pid': p, 'customer': v['name'], 'rev': v['rev'], 'qty': v['qty'],
|
| 610 |
+
'yoy_pct': P.yoy_pct(v['rev'], last.get(p, {}).get('rev', 0.0))} for p, v in this.items()]
|
| 611 |
+
rows.sort(key=lambda x: -x['rev'])
|
| 612 |
+
return rows[:top]
|
| 613 |
+
|
| 614 |
+
|
| 615 |
+
_BUYER_TIERS = [('Whale (β₯$25k)', 25000.0), ('Large ($10β25k)', 10000.0),
|
| 616 |
+
('Mid ($2β10k)', 2000.0), ('Small (<$2k)', 0.0)]
|
| 617 |
+
|
| 618 |
+
|
| 619 |
+
def sku_customer_analysis(code, t=None, team_id=None, top=15):
|
| 620 |
+
"""WHO buys this SKU (LTM), as customers: the value-tier mix of its buyers (by each buyer's TOTAL
|
| 621 |
+
spend), and the accounts most DEPENDENT on it (this SKU as a share of their spend β who gets hurt
|
| 622 |
+
most if it stocks out / who to protect)."""
|
| 623 |
+
t = t or P.today()
|
| 624 |
+
lf, lt = P.ltm(t)
|
| 625 |
+
pids = _sku_product_ids(code)
|
| 626 |
+
empty = {'segments': [], 'dependency': [], 'n_buyers': 0, 'avg_dependency': 0.0}
|
| 627 |
+
if not pids:
|
| 628 |
+
return empty
|
| 629 |
+
buyers = _sku_buyers(pids, lf, lt, team_id) # {pid: {name, rev(this SKU), qty}}
|
| 630 |
+
if not buyers:
|
| 631 |
+
return empty
|
| 632 |
+
bpids = list(buyers)
|
| 633 |
+
g = O.read_group('sale.order', sales_mod.order_domain(lf, lt, team_id) + [('partner_id', 'in', bpids)],
|
| 634 |
+
['amount_untaxed:sum'], ['partner_id'], lazy=False)
|
| 635 |
+
total = {O.m2o_id(r['partner_id']): (r.get('amount_untaxed') or 0.0) for r in g if r.get('partner_id')}
|
| 636 |
+
|
| 637 |
+
def tier(rev):
|
| 638 |
+
for nm, lo in _BUYER_TIERS:
|
| 639 |
+
if rev >= lo:
|
| 640 |
+
return nm
|
| 641 |
+
return _BUYER_TIERS[-1][0]
|
| 642 |
+
seg = {nm: {'tier': nm, 'buyers': 0, 'sku_rev': 0.0} for nm, _ in _BUYER_TIERS}
|
| 643 |
+
dep = []
|
| 644 |
+
for p, v in buyers.items():
|
| 645 |
+
ct = total.get(p, v['rev']) or v['rev']
|
| 646 |
+
s = seg[tier(ct)]
|
| 647 |
+
s['buyers'] += 1
|
| 648 |
+
s['sku_rev'] += v['rev']
|
| 649 |
+
dep.append({'customer': v['name'], 'pid': p, 'sku_rev': v['rev'], 'cust_total': ct,
|
| 650 |
+
'dependency_pct': (v['rev'] / ct * 100) if ct else None})
|
| 651 |
+
tot = sum(s['sku_rev'] for s in seg.values()) or 1.0
|
| 652 |
+
segments = []
|
| 653 |
+
for nm, _ in _BUYER_TIERS:
|
| 654 |
+
s = seg[nm]
|
| 655 |
+
s['rev_share'] = s['sku_rev'] / tot * 100
|
| 656 |
+
s['avg_per_buyer'] = (s['sku_rev'] / s['buyers']) if s['buyers'] else 0.0
|
| 657 |
+
segments.append(s)
|
| 658 |
+
dep.sort(key=lambda x: -x['sku_rev'])
|
| 659 |
+
deps_known = [d['dependency_pct'] for d in dep if d['dependency_pct'] is not None]
|
| 660 |
+
return {'segments': segments, 'dependency': dep[:top], 'n_buyers': len(buyers),
|
| 661 |
+
'avg_dependency': (sum(deps_known) / len(deps_known)) if deps_known else 0.0}
|
| 662 |
+
|
| 663 |
+
|
| 664 |
+
def sku_whitespace(code, t=None, team_id=None, top=15):
|
| 665 |
+
"""Customers who buy this SKU's category but NOT this SKU β ranked prospect list."""
|
| 666 |
+
import modules.customers as cust_mod
|
| 667 |
+
t = t or P.today()
|
| 668 |
+
pids = _sku_product_ids(code)
|
| 669 |
+
_name, category = _sku_name_category(pids)
|
| 670 |
+
cat_buyers = cust_mod.category_buyers(category, team_id=team_id) or set()
|
| 671 |
+
mf, mt = P.ltm(t)
|
| 672 |
+
prospects = list(cat_buyers - set(_sku_buyers(pids, mf, mt, team_id)))
|
| 673 |
+
if not prospects:
|
| 674 |
+
return {'category': category, 'rows': []}
|
| 675 |
+
g = O.read_group('sale.order', sales_mod.order_domain(mf, mt, team_id) + [('partner_id', 'in', prospects)],
|
| 676 |
+
['amount_untaxed:sum'], ['partner_id'], lazy=False)
|
| 677 |
+
rows = [{'pid': O.m2o_id(r['partner_id']), 'customer': O.m2o_name(r['partner_id']),
|
| 678 |
+
'total_spend': r.get('amount_untaxed') or 0.0} for r in g if r.get('partner_id')]
|
| 679 |
+
rows.sort(key=lambda x: -x['total_spend'])
|
| 680 |
+
return {'category': category, 'rows': rows[:top]}
|
| 681 |
+
|
| 682 |
+
|
| 683 |
+
def sku_drawer_bundle(code, t=None, team_id=None, allsku=None):
|
| 684 |
+
"""The whole SKU drawer's first paint in one cached unit: sku_detail first (the not-found gate),
|
| 685 |
+
then the other six pulls CONCURRENTLY (O.parallel). Cuts a ~6-call cold open to ~max(call)."""
|
| 686 |
+
detail = sku_detail(code, t=t, team_id=team_id, allsku=allsku)
|
| 687 |
+
if detail is None:
|
| 688 |
+
return {'detail': None}
|
| 689 |
+
bridge, conc, price, buyers, white, ca = O.parallel([
|
| 690 |
+
lambda: sku_buyer_bridge(code, t, team_id),
|
| 691 |
+
lambda: sku_concentration(code, t, team_id),
|
| 692 |
+
lambda: sku_price_dispersion(code, t, team_id),
|
| 693 |
+
lambda: sku_top_buyers(code, t, team_id),
|
| 694 |
+
lambda: sku_whitespace(code, t, team_id),
|
| 695 |
+
lambda: sku_customer_analysis(code, t, team_id),
|
| 696 |
+
])
|
| 697 |
+
return {'detail': detail, 'bridge': bridge, 'conc': conc, 'price': price,
|
| 698 |
+
'buyers': buyers, 'white': white, 'ca': ca}
|
| 699 |
+
|
| 700 |
+
|
| 701 |
+
def validate(t=None, team_id=None):
|
| 702 |
+
"""Reconcile SKU metrics to independent Odoo aggregates. When team_id is set (a single BU
|
| 703 |
+
selected) every check runs SCOPED to that BU, so the validation panel never reconciles
|
| 704 |
+
against β or exposes β the other BU's numbers. All checks here are BU-scopeable (no cross-BU
|
| 705 |
+
mirror), so team_id threads straight through."""
|
| 706 |
+
t = t or P.today()
|
| 707 |
+
yf, yt = P.ytd(t)
|
| 708 |
+
checks = []
|
| 709 |
+
sku = _sku_rev(yf, yt, team_id)
|
| 710 |
+
sku_sum = sum(v['rev'] for v in sku.values())
|
| 711 |
+
line_total = O.sum_field('sale.order.line',
|
| 712 |
+
O.sale_line_domain(yf, yt, team_id, extra=_NO_SVC), 'price_subtotal')
|
| 713 |
+
checks.append({'check': 'SKU rev: Ξ£(per-SKU) == total line revenue, ex-services (YTD)',
|
| 714 |
+
'a': round(sku_sum, 2), 'b': round(line_total, 2),
|
| 715 |
+
'gap': round(sku_sum - line_total, 2),
|
| 716 |
+
'ok': abs(sku_sum - line_total) <= 1.0})
|
| 717 |
+
|
| 718 |
+
m = yoy_movers(t, limit=10**9, team_id=team_id)
|
| 719 |
+
movers_sum = sum(r['change'] for r in m['risers']) + sum(r['change'] for r in m['decliners'])
|
| 720 |
+
lf, lt = P.ytd_last_year(t)
|
| 721 |
+
last_total = O.sum_field('sale.order.line',
|
| 722 |
+
O.sale_line_domain(lf, lt, team_id, extra=_NO_SVC), 'price_subtotal')
|
| 723 |
+
checks.append({'check': 'SKU movers: Ξ£(Ξ) == (YTD β LY) total',
|
| 724 |
+
'a': round(movers_sum, 2), 'b': round(line_total - last_total, 2),
|
| 725 |
+
'gap': round(movers_sum - (line_total - last_total), 2),
|
| 726 |
+
'ok': abs(movers_sum - (line_total - last_total)) <= 1.0})
|
| 727 |
+
|
| 728 |
+
# SKU drawer: buyer-bridge reconciles β retained + new buyer revenue == this-period SKU revenue
|
| 729 |
+
top_code = max(sku.items(), key=lambda kv: kv[1]['rev'])[0] if sku else None
|
| 730 |
+
if top_code:
|
| 731 |
+
bb = sku_buyer_bridge(top_code, t, team_id=team_id)
|
| 732 |
+
recon = bb['retained']['rev'] + bb['new']['rev']
|
| 733 |
+
checks.append({'check': 'SKU drawer: retained+new buyer rev == SKU YTD revenue',
|
| 734 |
+
'a': round(recon, 2), 'b': round(bb['this_total'], 2),
|
| 735 |
+
'gap': round(recon - bb['this_total'], 2),
|
| 736 |
+
'ok': abs(recon - bb['this_total']) <= 1.0})
|
| 737 |
+
return checks
|
web/public/sample_customers.json
CHANGED
|
@@ -1,558 +1,558 @@
|
|
| 1 |
-
{
|
| 2 |
-
"fields": [
|
| 3 |
-
{
|
| 4 |
-
"key": "customer",
|
| 5 |
-
"label": "Customer",
|
| 6 |
-
"type": "text",
|
| 7 |
-
"source": "odoo",
|
| 8 |
-
"pinned": true,
|
| 9 |
-
"default": true,
|
| 10 |
-
"description": "The customer's name in Odoo. One row per customer who ordered in the last 24 months."
|
| 11 |
-
},
|
| 12 |
-
{
|
| 13 |
-
"key": "partner_id",
|
| 14 |
-
"label": "Odoo ID",
|
| 15 |
-
"type": "int",
|
| 16 |
-
"source": "odoo",
|
| 17 |
-
"derived": true,
|
| 18 |
-
"default": false,
|
| 19 |
-
"description": "The Odoo res.partner id β the key every Odoo document joins on. DERIVED: this row's pid IS the partner id, so a stored copy would be a second source."
|
| 20 |
-
},
|
| 21 |
-
{
|
| 22 |
-
"key": "odoo_status",
|
| 23 |
-
"label": "Odoo record",
|
| 24 |
-
"type": "status",
|
| 25 |
-
"source": "odoo",
|
| 26 |
-
"default": false,
|
| 27 |
-
"options": [
|
| 28 |
-
"Active",
|
| 29 |
-
"Archived"
|
| 30 |
-
],
|
| 31 |
-
"description": "Whether this customer still exists in Odoo. Archived means deleted there."
|
| 32 |
-
},
|
| 33 |
-
{
|
| 34 |
-
"key": "agent",
|
| 35 |
-
"label": "Agent",
|
| 36 |
-
"type": "text",
|
| 37 |
-
"source": "odoo",
|
| 38 |
-
"default": true,
|
| 39 |
-
"description": "The sales agent who owns this account."
|
| 40 |
-
},
|
| 41 |
-
{
|
| 42 |
-
"key": "dba",
|
| 43 |
-
"label": "DBA",
|
| 44 |
-
"type": "select",
|
| 45 |
-
"source": "odoo",
|
| 46 |
-
"default": false,
|
| 47 |
-
"options": [
|
| 48 |
-
"Fisch",
|
| 49 |
-
"Royal",
|
| 50 |
-
"Both"
|
| 51 |
-
],
|
| 52 |
-
"description": "The brand this customer buys from - Fisch, Royal, or both. Amazon-channel orders are not a DBA."
|
| 53 |
-
},
|
| 54 |
-
{
|
| 55 |
-
"key": "salesperson",
|
| 56 |
-
"label": "Salesperson",
|
| 57 |
-
"type": "text",
|
| 58 |
-
"source": "odoo",
|
| 59 |
-
"default": false,
|
| 60 |
-
"description": "Who keyed in most of this customer's orders β not the Agent, who owns the account."
|
| 61 |
-
},
|
| 62 |
-
{
|
| 63 |
-
"key": "street",
|
| 64 |
-
"label": "Street",
|
| 65 |
-
"type": "text",
|
| 66 |
-
"source": "odoo",
|
| 67 |
-
"default": false,
|
| 68 |
-
"description": "First address line, from res.partner directly - not the geocoder, so a customer the map cannot place still shows its address."
|
| 69 |
-
},
|
| 70 |
-
{
|
| 71 |
-
"key": "street2",
|
| 72 |
-
"label": "Street 2",
|
| 73 |
-
"type": "text",
|
| 74 |
-
"source": "odoo",
|
| 75 |
-
"default": false,
|
| 76 |
-
"description": "Second address line (suite, unit, floor) on the customer's Odoo address."
|
| 77 |
-
},
|
| 78 |
-
{
|
| 79 |
-
"key": "city",
|
| 80 |
-
"label": "City",
|
| 81 |
-
"type": "text",
|
| 82 |
-
"source": "odoo",
|
| 83 |
-
"default": true,
|
| 84 |
-
"description": "City on the customer's Odoo address."
|
| 85 |
-
},
|
| 86 |
-
{
|
| 87 |
-
"key": "state",
|
| 88 |
-
"label": "State",
|
| 89 |
-
"type": "text",
|
| 90 |
-
"source": "odoo",
|
| 91 |
-
"default": true,
|
| 92 |
-
"description": "State or province on the customer's Odoo address."
|
| 93 |
-
},
|
| 94 |
-
{
|
| 95 |
-
"key": "country",
|
| 96 |
-
"label": "Country",
|
| 97 |
-
"type": "text",
|
| 98 |
-
"source": "odoo",
|
| 99 |
-
"default": false,
|
| 100 |
-
"description": "Country on the customer's Odoo address."
|
| 101 |
-
},
|
| 102 |
-
{
|
| 103 |
-
"key": "zip",
|
| 104 |
-
"label": "ZIP",
|
| 105 |
-
"type": "text",
|
| 106 |
-
"source": "odoo",
|
| 107 |
-
"default": false,
|
| 108 |
-
"description": "Postal code on the customer's Odoo address."
|
| 109 |
-
},
|
| 110 |
-
{
|
| 111 |
-
"key": "customer_since",
|
| 112 |
-
"label": "Customer since",
|
| 113 |
-
"type": "date",
|
| 114 |
-
"source": "odoo",
|
| 115 |
-
"default": false,
|
| 116 |
-
"description": "When this customer was first set up in Odoo."
|
| 117 |
-
},
|
| 118 |
-
{
|
| 119 |
-
"key": "tags",
|
| 120 |
-
"label": "Tags",
|
| 121 |
-
"type": "text",
|
| 122 |
-
"source": "odoo",
|
| 123 |
-
"default": false,
|
| 124 |
-
"description": "Odoo labels on this customer, comma-separated."
|
| 125 |
-
},
|
| 126 |
-
{
|
| 127 |
-
"key": "pricelist",
|
| 128 |
-
"label": "Price list",
|
| 129 |
-
"type": "text",
|
| 130 |
-
"source": "odoo",
|
| 131 |
-
"default": false,
|
| 132 |
-
"description": "The price list this customer buys on."
|
| 133 |
-
},
|
| 134 |
-
{
|
| 135 |
-
"key": "payment_terms",
|
| 136 |
-
"label": "Payment terms",
|
| 137 |
-
"type": "text",
|
| 138 |
-
"source": "odoo",
|
| 139 |
-
"default": false,
|
| 140 |
-
"description": "Payment terms on this customer's account β Net 30, for example."
|
| 141 |
-
},
|
| 142 |
-
{
|
| 143 |
-
"key": "last_order",
|
| 144 |
-
"label": "Last order",
|
| 145 |
-
"type": "date",
|
| 146 |
-
"source": "odoo",
|
| 147 |
-
"default": true,
|
| 148 |
-
"description": "Date of the most recent confirmed order."
|
| 149 |
-
},
|
| 150 |
-
{
|
| 151 |
-
"key": "overdue_days",
|
| 152 |
-
"label": "Overdue days",
|
| 153 |
-
"type": "int",
|
| 154 |
-
"source": "odoo",
|
| 155 |
-
"default": true,
|
| 156 |
-
"description": "How many days late this customer is running against their own usual ordering rhythm."
|
| 157 |
-
},
|
| 158 |
-
{
|
| 159 |
-
"_note": "filterable:false β DERIVED ANALYTIC: est_missed is min(cycles missed, 3) x AOV, a score we compute rather than an object the business has, so a condition on it would read as a fact about the customer when it is a fact about our arithmetic. It still displays and still sorts. Until wave 6 this flag also covered the frozen-window presets (revenue_ytd / revenue_ly / orders_24m / aov / yoy_pct); those are now DELETED outright under the owner's no-buildable-presets rule β see _comment. est_missed itself STAYS: no creatable measure or formula reproduces the cadence model behind it.",
|
| 160 |
-
"key": "est_missed",
|
| 161 |
-
"label": "Est. missed $",
|
| 162 |
-
"type": "currency",
|
| 163 |
-
"source": "odoo",
|
| 164 |
-
"default": true,
|
| 165 |
-
"agg": "sum",
|
| 166 |
-
"filterable": false,
|
| 167 |
-
"description": "Estimated sales missed while quiet: missed orders (capped at 3) times average order value. An estimate, not money owed."
|
| 168 |
-
},
|
| 169 |
-
{
|
| 170 |
-
"_note": "wave 21 R1 β KEY UNCHANGED, LABEL RENAMED. The computation is a DISJOINT split (ar.py credit_exposure): this column is only the not-yet-due residual, its sibling is the past-grace residual, and the two sum to the total. Under the label 'AR open $' the majority-late book read as 'Overdue > Open', which is nonsense in AR vocabulary β 'open' universally means the total. The label now says what the number is; the key stays so saved views and filters keep working.",
|
| 171 |
-
"key": "ar_open",
|
| 172 |
-
"label": "AR current $",
|
| 173 |
-
"type": "currency",
|
| 174 |
-
"source": "odoo",
|
| 175 |
-
"default": false,
|
| 176 |
-
"description": "Invoiced money owed but not yet due (a 5-day grace applies before it counts as overdue)."
|
| 177 |
-
},
|
| 178 |
-
{
|
| 179 |
-
"key": "ar_overdue",
|
| 180 |
-
"label": "AR overdue $",
|
| 181 |
-
"type": "currency",
|
| 182 |
-
"source": "odoo",
|
| 183 |
-
"default": false,
|
| 184 |
-
"description": "Invoiced money past due β same basis as the Collections page."
|
| 185 |
-
},
|
| 186 |
-
{
|
| 187 |
-
"_note": "wave 21 R1 β the TOTAL, added beside the rename above. AR current $ + AR overdue $, i.e. what most people mean by 'open AR'. Composed from the same ar.credit_exposure rows the siblings use, so it is transitively reconciled by ar.validate()'s residual read_group tie β no second oracle.",
|
| 188 |
-
"key": "ar_outstanding",
|
| 189 |
-
"label": "AR outstanding $",
|
| 190 |
-
"type": "currency",
|
| 191 |
-
"source": "odoo",
|
| 192 |
-
"default": false,
|
| 193 |
-
"description": "Total invoiced money owed right now: AR current $ plus AR overdue $."
|
| 194 |
-
},
|
| 195 |
-
{
|
| 196 |
-
"key": "ar_exposure",
|
| 197 |
-
"label": "Credit exposure $",
|
| 198 |
-
"type": "currency",
|
| 199 |
-
"source": "odoo",
|
| 200 |
-
"default": false,
|
| 201 |
-
"description": "The most you could be out if they stopped paying today: open, overdue, draft and not-yet-invoiced."
|
| 202 |
-
},
|
| 203 |
-
{
|
| 204 |
-
"key": "ar_aged_1_30",
|
| 205 |
-
"label": "1-30 days $",
|
| 206 |
-
"type": "currency",
|
| 207 |
-
"source": "odoo",
|
| 208 |
-
"default": false,
|
| 209 |
-
"description": "Overdue between 1 and 30 days. The four aging buckets sum to AR overdue $."
|
| 210 |
-
},
|
| 211 |
-
{
|
| 212 |
-
"key": "ar_aged_31_60",
|
| 213 |
-
"label": "31-60 days $",
|
| 214 |
-
"type": "currency",
|
| 215 |
-
"source": "odoo",
|
| 216 |
-
"default": false,
|
| 217 |
-
"description": "Overdue between 31 and 60 days. The four aging buckets sum to AR overdue $."
|
| 218 |
-
},
|
| 219 |
-
{
|
| 220 |
-
"key": "ar_aged_61_90",
|
| 221 |
-
"label": "61-90 days $",
|
| 222 |
-
"type": "currency",
|
| 223 |
-
"source": "odoo",
|
| 224 |
-
"default": false,
|
| 225 |
-
"description": "Overdue between 61 and 90 days. The four aging buckets sum to AR overdue $."
|
| 226 |
-
},
|
| 227 |
-
{
|
| 228 |
-
"key": "ar_aged_90_plus",
|
| 229 |
-
"label": "90+ days $",
|
| 230 |
-
"type": "currency",
|
| 231 |
-
"source": "odoo",
|
| 232 |
-
"default": false,
|
| 233 |
-
"description": "Overdue by more than 90 days. The four aging buckets sum to AR overdue $."
|
| 234 |
-
},
|
| 235 |
-
{
|
| 236 |
-
"key": "days_to_pay",
|
| 237 |
-
"label": "Days to pay",
|
| 238 |
-
"type": "int",
|
| 239 |
-
"source": "odoo",
|
| 240 |
-
"default": false,
|
| 241 |
-
"description": "Average days to pay an invoice in full. Blank means no fully paid invoice yet."
|
| 242 |
-
},
|
| 243 |
-
{
|
| 244 |
-
"key": "top_category",
|
| 245 |
-
"label": "Top category",
|
| 246 |
-
"type": "text",
|
| 247 |
-
"source": "odoo",
|
| 248 |
-
"default": false,
|
| 249 |
-
"description": "The category this customer spent the most on in the last 12 months."
|
| 250 |
-
},
|
| 251 |
-
{
|
| 252 |
-
"key": "top_category_pct",
|
| 253 |
-
"label": "Top category %",
|
| 254 |
-
"type": "pct",
|
| 255 |
-
"source": "odoo",
|
| 256 |
-
"default": false,
|
| 257 |
-
"description": "Share of last-12-months spend that went to the top category."
|
| 258 |
-
},
|
| 259 |
-
{
|
| 260 |
-
"key": "sku_count",
|
| 261 |
-
"label": "SKUs bought",
|
| 262 |
-
"type": "int",
|
| 263 |
-
"source": "odoo",
|
| 264 |
-
"default": false,
|
| 265 |
-
"description": "Distinct products bought in the last 12 months."
|
| 266 |
-
},
|
| 267 |
-
{
|
| 268 |
-
"key": "top_sku",
|
| 269 |
-
"label": "Top SKU",
|
| 270 |
-
"type": "text",
|
| 271 |
-
"source": "odoo",
|
| 272 |
-
"default": false,
|
| 273 |
-
"description": "The product this customer spent the most on in the last 12 months."
|
| 274 |
-
},
|
| 275 |
-
{
|
| 276 |
-
"key": "days_since",
|
| 277 |
-
"label": "Days since order",
|
| 278 |
-
"type": "int",
|
| 279 |
-
"source": "odoo",
|
| 280 |
-
"default": false,
|
| 281 |
-
"description": "Days since the last confirmed order."
|
| 282 |
-
},
|
| 283 |
-
{
|
| 284 |
-
"key": "typical_gap_days",
|
| 285 |
-
"label": "Typical gap days",
|
| 286 |
-
"type": "int",
|
| 287 |
-
"source": "odoo",
|
| 288 |
-
"default": false,
|
| 289 |
-
"description": "Days this customer usually goes between orders, from their own history."
|
| 290 |
-
},
|
| 291 |
-
{
|
| 292 |
-
"key": "notes",
|
| 293 |
-
"label": "Notes",
|
| 294 |
-
"type": "text",
|
| 295 |
-
"source": "overlay",
|
| 296 |
-
"default": false,
|
| 297 |
-
"description": "Your notes on this customer. Saved in this app only, visible only to you."
|
| 298 |
-
}
|
| 299 |
-
],
|
| 300 |
-
"rows": [
|
| 301 |
-
{
|
| 302 |
-
"pid": 101,
|
| 303 |
-
"customer": "Poppy Flowers",
|
| 304 |
-
"status": "New",
|
| 305 |
-
"agent": "Naomi Linnell Rivera",
|
| 306 |
-
"city": "Charlottesville",
|
| 307 |
-
"state": "Virginia (US)",
|
| 308 |
-
"last_order": "2026-07-21",
|
| 309 |
-
"est_missed": 0,
|
| 310 |
-
"notes": "",
|
| 311 |
-
"country": "United States",
|
| 312 |
-
"zip": "02720",
|
| 313 |
-
"payment_terms": "30 Days",
|
| 314 |
-
"pricelist": "Fisch 1 (USD)",
|
| 315 |
-
"tags": "Royal",
|
| 316 |
-
"customer_since": "2023-01-10",
|
| 317 |
-
"salesperson": "Jessica",
|
| 318 |
-
"ar_open": 0,
|
| 319 |
-
"ar_overdue": 0,
|
| 320 |
-
"ar_exposure": 0,
|
| 321 |
-
"top_category": "Styrofoam",
|
| 322 |
-
"top_category_pct": 0.47,
|
| 323 |
-
"sku_count": 93,
|
| 324 |
-
"top_sku": "AQUAFOAM FLORAL FOAM BRICK | 48-Piece per Pack",
|
| 325 |
-
"days_to_pay": 34,
|
| 326 |
-
"_created": "2023-01-15 09:10:00",
|
| 327 |
-
"lat": 25.7617,
|
| 328 |
-
"lon": -80.1918,
|
| 329 |
-
"odoo_status": "Archived",
|
| 330 |
-
"dba": "Royal",
|
| 331 |
-
"ar_outstanding": 0
|
| 332 |
-
},
|
| 333 |
-
{
|
| 334 |
-
"pid": 102,
|
| 335 |
-
"customer": "Meadow & Vine Wholesale",
|
| 336 |
-
"status": "Growing",
|
| 337 |
-
"agent": "Carla Jimenez",
|
| 338 |
-
"city": "Portland",
|
| 339 |
-
"state": "Oregon (US)",
|
| 340 |
-
"last_order": "2026-07-19",
|
| 341 |
-
"est_missed": 0,
|
| 342 |
-
"notes": "Expanding to a second storefront.",
|
| 343 |
-
"country": "United States",
|
| 344 |
-
"zip": "77041",
|
| 345 |
-
"payment_terms": "Immediate Payment",
|
| 346 |
-
"pricelist": "Royal 1 (USD)",
|
| 347 |
-
"tags": "Royal, Key account",
|
| 348 |
-
"customer_since": "2024-02-11",
|
| 349 |
-
"salesperson": "Naomi",
|
| 350 |
-
"ar_open": 1240.5,
|
| 351 |
-
"ar_overdue": 0,
|
| 352 |
-
"ar_exposure": 1740.5,
|
| 353 |
-
"top_category": "Ribbon",
|
| 354 |
-
"top_category_pct": 0.95,
|
| 355 |
-
"sku_count": 4,
|
| 356 |
-
"top_sku": "2\" X 24\" X 36\" GREEN STYROFOAM BOARD",
|
| 357 |
-
"days_to_pay": null,
|
| 358 |
-
"_created": "2023-02-15 09:11:00",
|
| 359 |
-
"lat": 27.9506,
|
| 360 |
-
"lon": -82.4572,
|
| 361 |
-
"odoo_status": "Active",
|
| 362 |
-
"dba": "Fisch",
|
| 363 |
-
"ar_outstanding": 1240.5
|
| 364 |
-
},
|
| 365 |
-
{
|
| 366 |
-
"pid": 103,
|
| 367 |
-
"customer": "Bluestem Floral Supply",
|
| 368 |
-
"status": "Growing",
|
| 369 |
-
"agent": "Naomi Linnell Rivera",
|
| 370 |
-
"city": "Kansas City",
|
| 371 |
-
"state": "Missouri (US)",
|
| 372 |
-
"last_order": "2026-07-17",
|
| 373 |
-
"est_missed": 0,
|
| 374 |
-
"notes": "",
|
| 375 |
-
"country": "United States",
|
| 376 |
-
"zip": "11219",
|
| 377 |
-
"payment_terms": "60 Days",
|
| 378 |
-
"pricelist": "Royal 1 (USD)",
|
| 379 |
-
"tags": "Fisch",
|
| 380 |
-
"customer_since": "2025-03-12",
|
| 381 |
-
"salesperson": "Karen",
|
| 382 |
-
"ar_open": 0,
|
| 383 |
-
"ar_overdue": 5120.25,
|
| 384 |
-
"ar_exposure": 5120.25,
|
| 385 |
-
"top_category": "Foams & Finishes",
|
| 386 |
-
"top_category_pct": 0.31,
|
| 387 |
-
"sku_count": 27,
|
| 388 |
-
"top_sku": "SATIN RIBBON 2IN",
|
| 389 |
-
"days_to_pay": 61,
|
| 390 |
-
"_created": "2023-03-15 09:12:00",
|
| 391 |
-
"lat": 28.5384,
|
| 392 |
-
"lon": -81.3789,
|
| 393 |
-
"odoo_status": "Active",
|
| 394 |
-
"dba": "Both",
|
| 395 |
-
"ar_outstanding": 5120.25
|
| 396 |
-
},
|
| 397 |
-
{
|
| 398 |
-
"pid": 104,
|
| 399 |
-
"customer": "Camellia Row Florist",
|
| 400 |
-
"status": "Declining",
|
| 401 |
-
"agent": "Devon Marsh",
|
| 402 |
-
"city": "Savannah",
|
| 403 |
-
"state": "Georgia (US)",
|
| 404 |
-
"last_order": "2026-05-30",
|
| 405 |
-
"est_missed": 41200,
|
| 406 |
-
"notes": "Switched some volume to a local grower.",
|
| 407 |
-
"country": "United States",
|
| 408 |
-
"zip": "07649",
|
| 409 |
-
"payment_terms": "30 Days",
|
| 410 |
-
"pricelist": "Fisch 1 (USD)",
|
| 411 |
-
"tags": "(none)",
|
| 412 |
-
"customer_since": "2026-04-13",
|
| 413 |
-
"salesperson": "(none)",
|
| 414 |
-
"ar_open": 8300,
|
| 415 |
-
"ar_overdue": 940,
|
| 416 |
-
"ar_exposure": 11440,
|
| 417 |
-
"top_category": "All",
|
| 418 |
-
"top_category_pct": 1.0,
|
| 419 |
-
"sku_count": 1,
|
| 420 |
-
"top_sku": "(none)",
|
| 421 |
-
"days_to_pay": 12,
|
| 422 |
-
"_created": "2023-04-15 09:13:00",
|
| 423 |
-
"lat": 30.3322,
|
| 424 |
-
"lon": -81.6557,
|
| 425 |
-
"odoo_status": "Active",
|
| 426 |
-
"dba": "Royal",
|
| 427 |
-
"ar_outstanding": 9240
|
| 428 |
-
},
|
| 429 |
-
{
|
| 430 |
-
"pid": 105,
|
| 431 |
-
"customer": "Harborlight Wholesale Blooms",
|
| 432 |
-
"status": "Growing",
|
| 433 |
-
"agent": "Carla Jimenez",
|
| 434 |
-
"city": "Seattle",
|
| 435 |
-
"state": "Washington (US)",
|
| 436 |
-
"last_order": "2026-07-22",
|
| 437 |
-
"est_missed": 0,
|
| 438 |
-
"notes": "Top-10 account.",
|
| 439 |
-
"country": "United States",
|
| 440 |
-
"zip": "33125",
|
| 441 |
-
"payment_terms": "Immediate Payment",
|
| 442 |
-
"pricelist": "Royal 1 (USD)",
|
| 443 |
-
"tags": "Royal",
|
| 444 |
-
"customer_since": "2023-05-14",
|
| 445 |
-
"salesperson": "Jessica",
|
| 446 |
-
"ar_open": 0,
|
| 447 |
-
"ar_overdue": 0,
|
| 448 |
-
"ar_exposure": 0,
|
| 449 |
-
"top_category": "Styrofoam",
|
| 450 |
-
"top_category_pct": 0.47,
|
| 451 |
-
"sku_count": 93,
|
| 452 |
-
"top_sku": "AQUAFOAM FLORAL FOAM BRICK | 48-Piece per Pack",
|
| 453 |
-
"days_to_pay": 34,
|
| 454 |
-
"_created": "2023-05-15 09:14:00",
|
| 455 |
-
"lat": 26.1224,
|
| 456 |
-
"lon": -80.1373,
|
| 457 |
-
"odoo_status": "Active",
|
| 458 |
-
"dba": "",
|
| 459 |
-
"ar_outstanding": 0
|
| 460 |
-
},
|
| 461 |
-
{
|
| 462 |
-
"pid": 106,
|
| 463 |
-
"customer": "Dogwood & Fern Co.",
|
| 464 |
-
"status": "Dormant",
|
| 465 |
-
"agent": "Devon Marsh",
|
| 466 |
-
"city": "Asheville",
|
| 467 |
-
"state": "North Carolina (US)",
|
| 468 |
-
"last_order": "2026-02-11",
|
| 469 |
-
"est_missed": 52400,
|
| 470 |
-
"notes": "No spring order this year.",
|
| 471 |
-
"country": "United States",
|
| 472 |
-
"zip": "90210",
|
| 473 |
-
"payment_terms": "60 Days",
|
| 474 |
-
"pricelist": "Royal 1 (USD)",
|
| 475 |
-
"tags": "Royal, Key account",
|
| 476 |
-
"customer_since": "2024-06-15",
|
| 477 |
-
"salesperson": "Naomi",
|
| 478 |
-
"ar_open": 1240.5,
|
| 479 |
-
"ar_overdue": 0,
|
| 480 |
-
"ar_exposure": 1740.5,
|
| 481 |
-
"top_category": "Ribbon",
|
| 482 |
-
"top_category_pct": 0.95,
|
| 483 |
-
"sku_count": 4,
|
| 484 |
-
"top_sku": "2\" X 24\" X 36\" GREEN STYROFOAM BOARD",
|
| 485 |
-
"days_to_pay": null,
|
| 486 |
-
"_created": "2023-06-15 09:15:00",
|
| 487 |
-
"lat": 27.3364,
|
| 488 |
-
"lon": -82.5307,
|
| 489 |
-
"odoo_status": "Active",
|
| 490 |
-
"dba": "Fisch",
|
| 491 |
-
"ar_outstanding": 1240.5
|
| 492 |
-
},
|
| 493 |
-
{
|
| 494 |
-
"pid": 107,
|
| 495 |
-
"customer": "Verbena Market Florals",
|
| 496 |
-
"status": "Lost",
|
| 497 |
-
"agent": "Naomi Linnell Rivera",
|
| 498 |
-
"city": "Austin",
|
| 499 |
-
"state": "Texas (US)",
|
| 500 |
-
"last_order": "2025-11-04",
|
| 501 |
-
"est_missed": 78300,
|
| 502 |
-
"notes": "Went with a competitor on freight terms.",
|
| 503 |
-
"country": "United States",
|
| 504 |
-
"zip": "08701",
|
| 505 |
-
"payment_terms": "30 Days",
|
| 506 |
-
"pricelist": "Fisch 1 (USD)",
|
| 507 |
-
"tags": "Fisch",
|
| 508 |
-
"customer_since": "2025-07-16",
|
| 509 |
-
"salesperson": "Karen",
|
| 510 |
-
"ar_open": 0,
|
| 511 |
-
"ar_overdue": 5120.25,
|
| 512 |
-
"ar_exposure": 5120.25,
|
| 513 |
-
"top_category": "Foams & Finishes",
|
| 514 |
-
"top_category_pct": 0.31,
|
| 515 |
-
"sku_count": 27,
|
| 516 |
-
"top_sku": "SATIN RIBBON 2IN",
|
| 517 |
-
"days_to_pay": 61,
|
| 518 |
-
"_created": "2023-07-15 09:16:00",
|
| 519 |
-
"lat": null,
|
| 520 |
-
"lon": null,
|
| 521 |
-
"odoo_status": "Active",
|
| 522 |
-
"dba": "Both",
|
| 523 |
-
"ar_outstanding": 5120.25
|
| 524 |
-
},
|
| 525 |
-
{
|
| 526 |
-
"pid": 108,
|
| 527 |
-
"customer": "Larkspur Lane Supply",
|
| 528 |
-
"status": "New",
|
| 529 |
-
"agent": "Carla Jimenez",
|
| 530 |
-
"city": "Denver",
|
| 531 |
-
"state": "Colorado (US)",
|
| 532 |
-
"last_order": "2026-07-14",
|
| 533 |
-
"est_missed": 0,
|
| 534 |
-
"notes": "First order in April.",
|
| 535 |
-
"country": "United States",
|
| 536 |
-
"zip": "60614",
|
| 537 |
-
"payment_terms": "Immediate Payment",
|
| 538 |
-
"pricelist": "Royal 1 (USD)",
|
| 539 |
-
"tags": "(none)",
|
| 540 |
-
"customer_since": "2026-08-17",
|
| 541 |
-
"salesperson": "(none)",
|
| 542 |
-
"ar_open": 8300,
|
| 543 |
-
"ar_overdue": 940,
|
| 544 |
-
"ar_exposure": 11440,
|
| 545 |
-
"top_category": "All",
|
| 546 |
-
"top_category_pct": 1.0,
|
| 547 |
-
"sku_count": 1,
|
| 548 |
-
"top_sku": "(none)",
|
| 549 |
-
"days_to_pay": 12,
|
| 550 |
-
"_created": "2023-08-15 09:17:00",
|
| 551 |
-
"lat": 33.749,
|
| 552 |
-
"lon": -84.388,
|
| 553 |
-
"odoo_status": "Active",
|
| 554 |
-
"dba": "Royal",
|
| 555 |
-
"ar_outstanding": 9240
|
| 556 |
-
}
|
| 557 |
-
]
|
| 558 |
-
}
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"fields": [
|
| 3 |
+
{
|
| 4 |
+
"key": "customer",
|
| 5 |
+
"label": "Customer",
|
| 6 |
+
"type": "text",
|
| 7 |
+
"source": "odoo",
|
| 8 |
+
"pinned": true,
|
| 9 |
+
"default": true,
|
| 10 |
+
"description": "The customer's name in Odoo. One row per customer who ordered in the last 24 months."
|
| 11 |
+
},
|
| 12 |
+
{
|
| 13 |
+
"key": "partner_id",
|
| 14 |
+
"label": "Odoo ID",
|
| 15 |
+
"type": "int",
|
| 16 |
+
"source": "odoo",
|
| 17 |
+
"derived": true,
|
| 18 |
+
"default": false,
|
| 19 |
+
"description": "The Odoo res.partner id β the key every Odoo document joins on. DERIVED: this row's pid IS the partner id, so a stored copy would be a second source."
|
| 20 |
+
},
|
| 21 |
+
{
|
| 22 |
+
"key": "odoo_status",
|
| 23 |
+
"label": "Odoo record",
|
| 24 |
+
"type": "status",
|
| 25 |
+
"source": "odoo",
|
| 26 |
+
"default": false,
|
| 27 |
+
"options": [
|
| 28 |
+
"Active",
|
| 29 |
+
"Archived"
|
| 30 |
+
],
|
| 31 |
+
"description": "Whether this customer still exists in Odoo. Archived means deleted there."
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"key": "agent",
|
| 35 |
+
"label": "Agent",
|
| 36 |
+
"type": "text",
|
| 37 |
+
"source": "odoo",
|
| 38 |
+
"default": true,
|
| 39 |
+
"description": "The sales agent who owns this account."
|
| 40 |
+
},
|
| 41 |
+
{
|
| 42 |
+
"key": "dba",
|
| 43 |
+
"label": "DBA",
|
| 44 |
+
"type": "select",
|
| 45 |
+
"source": "odoo",
|
| 46 |
+
"default": false,
|
| 47 |
+
"options": [
|
| 48 |
+
"Fisch",
|
| 49 |
+
"Royal",
|
| 50 |
+
"Both"
|
| 51 |
+
],
|
| 52 |
+
"description": "The brand this customer buys from - Fisch, Royal, or both. Amazon-channel orders are not a DBA."
|
| 53 |
+
},
|
| 54 |
+
{
|
| 55 |
+
"key": "salesperson",
|
| 56 |
+
"label": "Salesperson",
|
| 57 |
+
"type": "text",
|
| 58 |
+
"source": "odoo",
|
| 59 |
+
"default": false,
|
| 60 |
+
"description": "Who keyed in most of this customer's orders β not the Agent, who owns the account."
|
| 61 |
+
},
|
| 62 |
+
{
|
| 63 |
+
"key": "street",
|
| 64 |
+
"label": "Street",
|
| 65 |
+
"type": "text",
|
| 66 |
+
"source": "odoo",
|
| 67 |
+
"default": false,
|
| 68 |
+
"description": "First address line, from res.partner directly - not the geocoder, so a customer the map cannot place still shows its address."
|
| 69 |
+
},
|
| 70 |
+
{
|
| 71 |
+
"key": "street2",
|
| 72 |
+
"label": "Street 2",
|
| 73 |
+
"type": "text",
|
| 74 |
+
"source": "odoo",
|
| 75 |
+
"default": false,
|
| 76 |
+
"description": "Second address line (suite, unit, floor) on the customer's Odoo address."
|
| 77 |
+
},
|
| 78 |
+
{
|
| 79 |
+
"key": "city",
|
| 80 |
+
"label": "City",
|
| 81 |
+
"type": "text",
|
| 82 |
+
"source": "odoo",
|
| 83 |
+
"default": true,
|
| 84 |
+
"description": "City on the customer's Odoo address."
|
| 85 |
+
},
|
| 86 |
+
{
|
| 87 |
+
"key": "state",
|
| 88 |
+
"label": "State",
|
| 89 |
+
"type": "text",
|
| 90 |
+
"source": "odoo",
|
| 91 |
+
"default": true,
|
| 92 |
+
"description": "State or province on the customer's Odoo address."
|
| 93 |
+
},
|
| 94 |
+
{
|
| 95 |
+
"key": "country",
|
| 96 |
+
"label": "Country",
|
| 97 |
+
"type": "text",
|
| 98 |
+
"source": "odoo",
|
| 99 |
+
"default": false,
|
| 100 |
+
"description": "Country on the customer's Odoo address."
|
| 101 |
+
},
|
| 102 |
+
{
|
| 103 |
+
"key": "zip",
|
| 104 |
+
"label": "ZIP",
|
| 105 |
+
"type": "text",
|
| 106 |
+
"source": "odoo",
|
| 107 |
+
"default": false,
|
| 108 |
+
"description": "Postal code on the customer's Odoo address."
|
| 109 |
+
},
|
| 110 |
+
{
|
| 111 |
+
"key": "customer_since",
|
| 112 |
+
"label": "Customer since",
|
| 113 |
+
"type": "date",
|
| 114 |
+
"source": "odoo",
|
| 115 |
+
"default": false,
|
| 116 |
+
"description": "When this customer was first set up in Odoo."
|
| 117 |
+
},
|
| 118 |
+
{
|
| 119 |
+
"key": "tags",
|
| 120 |
+
"label": "Tags",
|
| 121 |
+
"type": "text",
|
| 122 |
+
"source": "odoo",
|
| 123 |
+
"default": false,
|
| 124 |
+
"description": "Odoo labels on this customer, comma-separated."
|
| 125 |
+
},
|
| 126 |
+
{
|
| 127 |
+
"key": "pricelist",
|
| 128 |
+
"label": "Price list",
|
| 129 |
+
"type": "text",
|
| 130 |
+
"source": "odoo",
|
| 131 |
+
"default": false,
|
| 132 |
+
"description": "The price list this customer buys on."
|
| 133 |
+
},
|
| 134 |
+
{
|
| 135 |
+
"key": "payment_terms",
|
| 136 |
+
"label": "Payment terms",
|
| 137 |
+
"type": "text",
|
| 138 |
+
"source": "odoo",
|
| 139 |
+
"default": false,
|
| 140 |
+
"description": "Payment terms on this customer's account β Net 30, for example."
|
| 141 |
+
},
|
| 142 |
+
{
|
| 143 |
+
"key": "last_order",
|
| 144 |
+
"label": "Last order",
|
| 145 |
+
"type": "date",
|
| 146 |
+
"source": "odoo",
|
| 147 |
+
"default": true,
|
| 148 |
+
"description": "Date of the most recent confirmed order."
|
| 149 |
+
},
|
| 150 |
+
{
|
| 151 |
+
"key": "overdue_days",
|
| 152 |
+
"label": "Overdue days",
|
| 153 |
+
"type": "int",
|
| 154 |
+
"source": "odoo",
|
| 155 |
+
"default": true,
|
| 156 |
+
"description": "How many days late this customer is running against their own usual ordering rhythm."
|
| 157 |
+
},
|
| 158 |
+
{
|
| 159 |
+
"_note": "filterable:false β DERIVED ANALYTIC: est_missed is min(cycles missed, 3) x AOV, a score we compute rather than an object the business has, so a condition on it would read as a fact about the customer when it is a fact about our arithmetic. It still displays and still sorts. Until wave 6 this flag also covered the frozen-window presets (revenue_ytd / revenue_ly / orders_24m / aov / yoy_pct); those are now DELETED outright under the owner's no-buildable-presets rule β see _comment. est_missed itself STAYS: no creatable measure or formula reproduces the cadence model behind it.",
|
| 160 |
+
"key": "est_missed",
|
| 161 |
+
"label": "Est. missed $",
|
| 162 |
+
"type": "currency",
|
| 163 |
+
"source": "odoo",
|
| 164 |
+
"default": true,
|
| 165 |
+
"agg": "sum",
|
| 166 |
+
"filterable": false,
|
| 167 |
+
"description": "Estimated sales missed while quiet: missed orders (capped at 3) times average order value. An estimate, not money owed."
|
| 168 |
+
},
|
| 169 |
+
{
|
| 170 |
+
"_note": "wave 21 R1 β KEY UNCHANGED, LABEL RENAMED. The computation is a DISJOINT split (ar.py credit_exposure): this column is only the not-yet-due residual, its sibling is the past-grace residual, and the two sum to the total. Under the label 'AR open $' the majority-late book read as 'Overdue > Open', which is nonsense in AR vocabulary β 'open' universally means the total. The label now says what the number is; the key stays so saved views and filters keep working.",
|
| 171 |
+
"key": "ar_open",
|
| 172 |
+
"label": "AR current $",
|
| 173 |
+
"type": "currency",
|
| 174 |
+
"source": "odoo",
|
| 175 |
+
"default": false,
|
| 176 |
+
"description": "Invoiced money owed but not yet due (a 5-day grace applies before it counts as overdue)."
|
| 177 |
+
},
|
| 178 |
+
{
|
| 179 |
+
"key": "ar_overdue",
|
| 180 |
+
"label": "AR overdue $",
|
| 181 |
+
"type": "currency",
|
| 182 |
+
"source": "odoo",
|
| 183 |
+
"default": false,
|
| 184 |
+
"description": "Invoiced money past due β same basis as the Collections page."
|
| 185 |
+
},
|
| 186 |
+
{
|
| 187 |
+
"_note": "wave 21 R1 β the TOTAL, added beside the rename above. AR current $ + AR overdue $, i.e. what most people mean by 'open AR'. Composed from the same ar.credit_exposure rows the siblings use, so it is transitively reconciled by ar.validate()'s residual read_group tie β no second oracle.",
|
| 188 |
+
"key": "ar_outstanding",
|
| 189 |
+
"label": "AR outstanding $",
|
| 190 |
+
"type": "currency",
|
| 191 |
+
"source": "odoo",
|
| 192 |
+
"default": false,
|
| 193 |
+
"description": "Total invoiced money owed right now: AR current $ plus AR overdue $."
|
| 194 |
+
},
|
| 195 |
+
{
|
| 196 |
+
"key": "ar_exposure",
|
| 197 |
+
"label": "Credit exposure $",
|
| 198 |
+
"type": "currency",
|
| 199 |
+
"source": "odoo",
|
| 200 |
+
"default": false,
|
| 201 |
+
"description": "The most you could be out if they stopped paying today: open, overdue, draft and not-yet-invoiced."
|
| 202 |
+
},
|
| 203 |
+
{
|
| 204 |
+
"key": "ar_aged_1_30",
|
| 205 |
+
"label": "1-30 days $",
|
| 206 |
+
"type": "currency",
|
| 207 |
+
"source": "odoo",
|
| 208 |
+
"default": false,
|
| 209 |
+
"description": "Overdue between 1 and 30 days. The four aging buckets sum to AR overdue $."
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"key": "ar_aged_31_60",
|
| 213 |
+
"label": "31-60 days $",
|
| 214 |
+
"type": "currency",
|
| 215 |
+
"source": "odoo",
|
| 216 |
+
"default": false,
|
| 217 |
+
"description": "Overdue between 31 and 60 days. The four aging buckets sum to AR overdue $."
|
| 218 |
+
},
|
| 219 |
+
{
|
| 220 |
+
"key": "ar_aged_61_90",
|
| 221 |
+
"label": "61-90 days $",
|
| 222 |
+
"type": "currency",
|
| 223 |
+
"source": "odoo",
|
| 224 |
+
"default": false,
|
| 225 |
+
"description": "Overdue between 61 and 90 days. The four aging buckets sum to AR overdue $."
|
| 226 |
+
},
|
| 227 |
+
{
|
| 228 |
+
"key": "ar_aged_90_plus",
|
| 229 |
+
"label": "90+ days $",
|
| 230 |
+
"type": "currency",
|
| 231 |
+
"source": "odoo",
|
| 232 |
+
"default": false,
|
| 233 |
+
"description": "Overdue by more than 90 days. The four aging buckets sum to AR overdue $."
|
| 234 |
+
},
|
| 235 |
+
{
|
| 236 |
+
"key": "days_to_pay",
|
| 237 |
+
"label": "Days to pay",
|
| 238 |
+
"type": "int",
|
| 239 |
+
"source": "odoo",
|
| 240 |
+
"default": false,
|
| 241 |
+
"description": "Average days to pay an invoice in full. Blank means no fully paid invoice yet."
|
| 242 |
+
},
|
| 243 |
+
{
|
| 244 |
+
"key": "top_category",
|
| 245 |
+
"label": "Top category",
|
| 246 |
+
"type": "text",
|
| 247 |
+
"source": "odoo",
|
| 248 |
+
"default": false,
|
| 249 |
+
"description": "The category this customer spent the most on in the last 12 months."
|
| 250 |
+
},
|
| 251 |
+
{
|
| 252 |
+
"key": "top_category_pct",
|
| 253 |
+
"label": "Top category %",
|
| 254 |
+
"type": "pct",
|
| 255 |
+
"source": "odoo",
|
| 256 |
+
"default": false,
|
| 257 |
+
"description": "Share of last-12-months spend that went to the top category."
|
| 258 |
+
},
|
| 259 |
+
{
|
| 260 |
+
"key": "sku_count",
|
| 261 |
+
"label": "SKUs bought",
|
| 262 |
+
"type": "int",
|
| 263 |
+
"source": "odoo",
|
| 264 |
+
"default": false,
|
| 265 |
+
"description": "Distinct products bought in the last 12 months."
|
| 266 |
+
},
|
| 267 |
+
{
|
| 268 |
+
"key": "top_sku",
|
| 269 |
+
"label": "Top SKU",
|
| 270 |
+
"type": "text",
|
| 271 |
+
"source": "odoo",
|
| 272 |
+
"default": false,
|
| 273 |
+
"description": "The product this customer spent the most on in the last 12 months."
|
| 274 |
+
},
|
| 275 |
+
{
|
| 276 |
+
"key": "days_since",
|
| 277 |
+
"label": "Days since order",
|
| 278 |
+
"type": "int",
|
| 279 |
+
"source": "odoo",
|
| 280 |
+
"default": false,
|
| 281 |
+
"description": "Days since the last confirmed order."
|
| 282 |
+
},
|
| 283 |
+
{
|
| 284 |
+
"key": "typical_gap_days",
|
| 285 |
+
"label": "Typical gap days",
|
| 286 |
+
"type": "int",
|
| 287 |
+
"source": "odoo",
|
| 288 |
+
"default": false,
|
| 289 |
+
"description": "Days this customer usually goes between orders, from their own history."
|
| 290 |
+
},
|
| 291 |
+
{
|
| 292 |
+
"key": "notes",
|
| 293 |
+
"label": "Notes",
|
| 294 |
+
"type": "text",
|
| 295 |
+
"source": "overlay",
|
| 296 |
+
"default": false,
|
| 297 |
+
"description": "Your notes on this customer. Saved in this app only, visible only to you."
|
| 298 |
+
}
|
| 299 |
+
],
|
| 300 |
+
"rows": [
|
| 301 |
+
{
|
| 302 |
+
"pid": 101,
|
| 303 |
+
"customer": "Poppy Flowers",
|
| 304 |
+
"status": "New",
|
| 305 |
+
"agent": "Naomi Linnell Rivera",
|
| 306 |
+
"city": "Charlottesville",
|
| 307 |
+
"state": "Virginia (US)",
|
| 308 |
+
"last_order": "2026-07-21",
|
| 309 |
+
"est_missed": 0,
|
| 310 |
+
"notes": "",
|
| 311 |
+
"country": "United States",
|
| 312 |
+
"zip": "02720",
|
| 313 |
+
"payment_terms": "30 Days",
|
| 314 |
+
"pricelist": "Fisch 1 (USD)",
|
| 315 |
+
"tags": "Royal",
|
| 316 |
+
"customer_since": "2023-01-10",
|
| 317 |
+
"salesperson": "Jessica",
|
| 318 |
+
"ar_open": 0,
|
| 319 |
+
"ar_overdue": 0,
|
| 320 |
+
"ar_exposure": 0,
|
| 321 |
+
"top_category": "Styrofoam",
|
| 322 |
+
"top_category_pct": 0.47,
|
| 323 |
+
"sku_count": 93,
|
| 324 |
+
"top_sku": "AQUAFOAM FLORAL FOAM BRICK | 48-Piece per Pack",
|
| 325 |
+
"days_to_pay": 34,
|
| 326 |
+
"_created": "2023-01-15 09:10:00",
|
| 327 |
+
"lat": 25.7617,
|
| 328 |
+
"lon": -80.1918,
|
| 329 |
+
"odoo_status": "Archived",
|
| 330 |
+
"dba": "Royal",
|
| 331 |
+
"ar_outstanding": 0
|
| 332 |
+
},
|
| 333 |
+
{
|
| 334 |
+
"pid": 102,
|
| 335 |
+
"customer": "Meadow & Vine Wholesale",
|
| 336 |
+
"status": "Growing",
|
| 337 |
+
"agent": "Carla Jimenez",
|
| 338 |
+
"city": "Portland",
|
| 339 |
+
"state": "Oregon (US)",
|
| 340 |
+
"last_order": "2026-07-19",
|
| 341 |
+
"est_missed": 0,
|
| 342 |
+
"notes": "Expanding to a second storefront.",
|
| 343 |
+
"country": "United States",
|
| 344 |
+
"zip": "77041",
|
| 345 |
+
"payment_terms": "Immediate Payment",
|
| 346 |
+
"pricelist": "Royal 1 (USD)",
|
| 347 |
+
"tags": "Royal, Key account",
|
| 348 |
+
"customer_since": "2024-02-11",
|
| 349 |
+
"salesperson": "Naomi",
|
| 350 |
+
"ar_open": 1240.5,
|
| 351 |
+
"ar_overdue": 0,
|
| 352 |
+
"ar_exposure": 1740.5,
|
| 353 |
+
"top_category": "Ribbon",
|
| 354 |
+
"top_category_pct": 0.95,
|
| 355 |
+
"sku_count": 4,
|
| 356 |
+
"top_sku": "2\" X 24\" X 36\" GREEN STYROFOAM BOARD",
|
| 357 |
+
"days_to_pay": null,
|
| 358 |
+
"_created": "2023-02-15 09:11:00",
|
| 359 |
+
"lat": 27.9506,
|
| 360 |
+
"lon": -82.4572,
|
| 361 |
+
"odoo_status": "Active",
|
| 362 |
+
"dba": "Fisch",
|
| 363 |
+
"ar_outstanding": 1240.5
|
| 364 |
+
},
|
| 365 |
+
{
|
| 366 |
+
"pid": 103,
|
| 367 |
+
"customer": "Bluestem Floral Supply",
|
| 368 |
+
"status": "Growing",
|
| 369 |
+
"agent": "Naomi Linnell Rivera",
|
| 370 |
+
"city": "Kansas City",
|
| 371 |
+
"state": "Missouri (US)",
|
| 372 |
+
"last_order": "2026-07-17",
|
| 373 |
+
"est_missed": 0,
|
| 374 |
+
"notes": "",
|
| 375 |
+
"country": "United States",
|
| 376 |
+
"zip": "11219",
|
| 377 |
+
"payment_terms": "60 Days",
|
| 378 |
+
"pricelist": "Royal 1 (USD)",
|
| 379 |
+
"tags": "Fisch",
|
| 380 |
+
"customer_since": "2025-03-12",
|
| 381 |
+
"salesperson": "Karen",
|
| 382 |
+
"ar_open": 0,
|
| 383 |
+
"ar_overdue": 5120.25,
|
| 384 |
+
"ar_exposure": 5120.25,
|
| 385 |
+
"top_category": "Foams & Finishes",
|
| 386 |
+
"top_category_pct": 0.31,
|
| 387 |
+
"sku_count": 27,
|
| 388 |
+
"top_sku": "SATIN RIBBON 2IN",
|
| 389 |
+
"days_to_pay": 61,
|
| 390 |
+
"_created": "2023-03-15 09:12:00",
|
| 391 |
+
"lat": 28.5384,
|
| 392 |
+
"lon": -81.3789,
|
| 393 |
+
"odoo_status": "Active",
|
| 394 |
+
"dba": "Both",
|
| 395 |
+
"ar_outstanding": 5120.25
|
| 396 |
+
},
|
| 397 |
+
{
|
| 398 |
+
"pid": 104,
|
| 399 |
+
"customer": "Camellia Row Florist",
|
| 400 |
+
"status": "Declining",
|
| 401 |
+
"agent": "Devon Marsh",
|
| 402 |
+
"city": "Savannah",
|
| 403 |
+
"state": "Georgia (US)",
|
| 404 |
+
"last_order": "2026-05-30",
|
| 405 |
+
"est_missed": 41200,
|
| 406 |
+
"notes": "Switched some volume to a local grower.",
|
| 407 |
+
"country": "United States",
|
| 408 |
+
"zip": "07649",
|
| 409 |
+
"payment_terms": "30 Days",
|
| 410 |
+
"pricelist": "Fisch 1 (USD)",
|
| 411 |
+
"tags": "(none)",
|
| 412 |
+
"customer_since": "2026-04-13",
|
| 413 |
+
"salesperson": "(none)",
|
| 414 |
+
"ar_open": 8300,
|
| 415 |
+
"ar_overdue": 940,
|
| 416 |
+
"ar_exposure": 11440,
|
| 417 |
+
"top_category": "All",
|
| 418 |
+
"top_category_pct": 1.0,
|
| 419 |
+
"sku_count": 1,
|
| 420 |
+
"top_sku": "(none)",
|
| 421 |
+
"days_to_pay": 12,
|
| 422 |
+
"_created": "2023-04-15 09:13:00",
|
| 423 |
+
"lat": 30.3322,
|
| 424 |
+
"lon": -81.6557,
|
| 425 |
+
"odoo_status": "Active",
|
| 426 |
+
"dba": "Royal",
|
| 427 |
+
"ar_outstanding": 9240
|
| 428 |
+
},
|
| 429 |
+
{
|
| 430 |
+
"pid": 105,
|
| 431 |
+
"customer": "Harborlight Wholesale Blooms",
|
| 432 |
+
"status": "Growing",
|
| 433 |
+
"agent": "Carla Jimenez",
|
| 434 |
+
"city": "Seattle",
|
| 435 |
+
"state": "Washington (US)",
|
| 436 |
+
"last_order": "2026-07-22",
|
| 437 |
+
"est_missed": 0,
|
| 438 |
+
"notes": "Top-10 account.",
|
| 439 |
+
"country": "United States",
|
| 440 |
+
"zip": "33125",
|
| 441 |
+
"payment_terms": "Immediate Payment",
|
| 442 |
+
"pricelist": "Royal 1 (USD)",
|
| 443 |
+
"tags": "Royal",
|
| 444 |
+
"customer_since": "2023-05-14",
|
| 445 |
+
"salesperson": "Jessica",
|
| 446 |
+
"ar_open": 0,
|
| 447 |
+
"ar_overdue": 0,
|
| 448 |
+
"ar_exposure": 0,
|
| 449 |
+
"top_category": "Styrofoam",
|
| 450 |
+
"top_category_pct": 0.47,
|
| 451 |
+
"sku_count": 93,
|
| 452 |
+
"top_sku": "AQUAFOAM FLORAL FOAM BRICK | 48-Piece per Pack",
|
| 453 |
+
"days_to_pay": 34,
|
| 454 |
+
"_created": "2023-05-15 09:14:00",
|
| 455 |
+
"lat": 26.1224,
|
| 456 |
+
"lon": -80.1373,
|
| 457 |
+
"odoo_status": "Active",
|
| 458 |
+
"dba": "",
|
| 459 |
+
"ar_outstanding": 0
|
| 460 |
+
},
|
| 461 |
+
{
|
| 462 |
+
"pid": 106,
|
| 463 |
+
"customer": "Dogwood & Fern Co.",
|
| 464 |
+
"status": "Dormant",
|
| 465 |
+
"agent": "Devon Marsh",
|
| 466 |
+
"city": "Asheville",
|
| 467 |
+
"state": "North Carolina (US)",
|
| 468 |
+
"last_order": "2026-02-11",
|
| 469 |
+
"est_missed": 52400,
|
| 470 |
+
"notes": "No spring order this year.",
|
| 471 |
+
"country": "United States",
|
| 472 |
+
"zip": "90210",
|
| 473 |
+
"payment_terms": "60 Days",
|
| 474 |
+
"pricelist": "Royal 1 (USD)",
|
| 475 |
+
"tags": "Royal, Key account",
|
| 476 |
+
"customer_since": "2024-06-15",
|
| 477 |
+
"salesperson": "Naomi",
|
| 478 |
+
"ar_open": 1240.5,
|
| 479 |
+
"ar_overdue": 0,
|
| 480 |
+
"ar_exposure": 1740.5,
|
| 481 |
+
"top_category": "Ribbon",
|
| 482 |
+
"top_category_pct": 0.95,
|
| 483 |
+
"sku_count": 4,
|
| 484 |
+
"top_sku": "2\" X 24\" X 36\" GREEN STYROFOAM BOARD",
|
| 485 |
+
"days_to_pay": null,
|
| 486 |
+
"_created": "2023-06-15 09:15:00",
|
| 487 |
+
"lat": 27.3364,
|
| 488 |
+
"lon": -82.5307,
|
| 489 |
+
"odoo_status": "Active",
|
| 490 |
+
"dba": "Fisch",
|
| 491 |
+
"ar_outstanding": 1240.5
|
| 492 |
+
},
|
| 493 |
+
{
|
| 494 |
+
"pid": 107,
|
| 495 |
+
"customer": "Verbena Market Florals",
|
| 496 |
+
"status": "Lost",
|
| 497 |
+
"agent": "Naomi Linnell Rivera",
|
| 498 |
+
"city": "Austin",
|
| 499 |
+
"state": "Texas (US)",
|
| 500 |
+
"last_order": "2025-11-04",
|
| 501 |
+
"est_missed": 78300,
|
| 502 |
+
"notes": "Went with a competitor on freight terms.",
|
| 503 |
+
"country": "United States",
|
| 504 |
+
"zip": "08701",
|
| 505 |
+
"payment_terms": "30 Days",
|
| 506 |
+
"pricelist": "Fisch 1 (USD)",
|
| 507 |
+
"tags": "Fisch",
|
| 508 |
+
"customer_since": "2025-07-16",
|
| 509 |
+
"salesperson": "Karen",
|
| 510 |
+
"ar_open": 0,
|
| 511 |
+
"ar_overdue": 5120.25,
|
| 512 |
+
"ar_exposure": 5120.25,
|
| 513 |
+
"top_category": "Foams & Finishes",
|
| 514 |
+
"top_category_pct": 0.31,
|
| 515 |
+
"sku_count": 27,
|
| 516 |
+
"top_sku": "SATIN RIBBON 2IN",
|
| 517 |
+
"days_to_pay": 61,
|
| 518 |
+
"_created": "2023-07-15 09:16:00",
|
| 519 |
+
"lat": null,
|
| 520 |
+
"lon": null,
|
| 521 |
+
"odoo_status": "Active",
|
| 522 |
+
"dba": "Both",
|
| 523 |
+
"ar_outstanding": 5120.25
|
| 524 |
+
},
|
| 525 |
+
{
|
| 526 |
+
"pid": 108,
|
| 527 |
+
"customer": "Larkspur Lane Supply",
|
| 528 |
+
"status": "New",
|
| 529 |
+
"agent": "Carla Jimenez",
|
| 530 |
+
"city": "Denver",
|
| 531 |
+
"state": "Colorado (US)",
|
| 532 |
+
"last_order": "2026-07-14",
|
| 533 |
+
"est_missed": 0,
|
| 534 |
+
"notes": "First order in April.",
|
| 535 |
+
"country": "United States",
|
| 536 |
+
"zip": "60614",
|
| 537 |
+
"payment_terms": "Immediate Payment",
|
| 538 |
+
"pricelist": "Royal 1 (USD)",
|
| 539 |
+
"tags": "(none)",
|
| 540 |
+
"customer_since": "2026-08-17",
|
| 541 |
+
"salesperson": "(none)",
|
| 542 |
+
"ar_open": 8300,
|
| 543 |
+
"ar_overdue": 940,
|
| 544 |
+
"ar_exposure": 11440,
|
| 545 |
+
"top_category": "All",
|
| 546 |
+
"top_category_pct": 1.0,
|
| 547 |
+
"sku_count": 1,
|
| 548 |
+
"top_sku": "(none)",
|
| 549 |
+
"days_to_pay": 12,
|
| 550 |
+
"_created": "2023-08-15 09:17:00",
|
| 551 |
+
"lat": 33.749,
|
| 552 |
+
"lon": -84.388,
|
| 553 |
+
"odoo_status": "Active",
|
| 554 |
+
"dba": "Royal",
|
| 555 |
+
"ar_outstanding": 9240
|
| 556 |
+
}
|
| 557 |
+
]
|
| 558 |
+
}
|
web/src/alerts/alertsModel.ts
CHANGED
|
@@ -1,418 +1,418 @@
|
|
| 1 |
-
// ---------------------------------------------------------------------------
|
| 2 |
-
// alerts/alertsModel.ts β WAVE 20 item 25 (contract C-ALERT): the inbox's PURE
|
| 3 |
-
// half. React-free and fetch-free, so `verify_alerts.py` runs it under node.
|
| 4 |
-
//
|
| 5 |
-
// An alert says "tell me when a record ENTERS this view". The client half is
|
| 6 |
-
// small, and every part of it fails silently when it is wrong:
|
| 7 |
-
//
|
| 8 |
-
// Β· an unread count taken from `items.length` rather than from the server's
|
| 9 |
-
// own `unread` disagrees with the badge the moment a page is capped or a
|
| 10 |
-
// read lands in another tab β and a badge that says 3 when the list shows 9
|
| 11 |
-
// teaches the reader to ignore the badge;
|
| 12 |
-
// Β· a notification whose `viewId` no longer resolves must open NOTHING rather
|
| 13 |
-
// than a wrong view β alerts outlive the views they were made from;
|
| 14 |
-
// Β· `topic` is the surface's scope key ("customer", "product", "ut_β¦"), and
|
| 15 |
-
// the ROUTE is a registry key ("customer_data") β mapping one to the other
|
| 16 |
-
// by guesswork sends every click to a page that does not exist.
|
| 17 |
-
// ---------------------------------------------------------------------------
|
| 18 |
-
|
| 19 |
-
/**
|
| 20 |
-
* ββ WAVE 32 Β· T20 Β· CONTRACT C3 β WHERE AN INBOX ITEM OPENS.
|
| 21 |
-
*
|
| 22 |
-
* `module` is the destination surface (`"database"` or `"automation"`) and `id` is what to open
|
| 23 |
-
* in it. `tab` is the SUB-SELECTION inside that module: the literal `"runs"` for an automation's
|
| 24 |
-
* run log, or the VIEW ID to select on a database. One optional key, two destinations.
|
| 25 |
-
*
|
| 26 |
-
* β EVERY FIELD IS A PLAIN `string`, NEVER A UNION, and that is alertsModel's wave-9 law rather
|
| 27 |
-
* than laziness: the vocabulary is the SERVER's, and a client union over it turns "the server grew
|
| 28 |
-
* a module" into "the client silently drops the row". An unknown module is refused by the
|
| 29 |
-
* FRAME's dispatcher, out loud, which is a different thing from never arriving.
|
| 30 |
-
*/
|
| 31 |
-
export interface NotificationTarget {
|
| 32 |
-
module: string;
|
| 33 |
-
id: string;
|
| 34 |
-
tab?: string;
|
| 35 |
-
}
|
| 36 |
-
|
| 37 |
-
/** C3's `kind` vocabulary, mirroring `routes_alerts.NOTIF_KIND_*`. Held as constants so the
|
| 38 |
-
* Inbox's branch and the gate's fixtures cannot disagree about the word. */
|
| 39 |
-
export const NOTIF_KIND_ALERT = "alert";
|
| 40 |
-
export const NOTIF_KIND_AUTOMATION = "automation";
|
| 41 |
-
export const NOTIF_KIND_SHARE = "share";
|
| 42 |
-
|
| 43 |
-
/** C3's `target.module` vocabulary, and the automation sub-selection. */
|
| 44 |
-
export const TARGET_MODULE_DATABASE = "database";
|
| 45 |
-
export const TARGET_MODULE_AUTOMATION = "automation";
|
| 46 |
-
export const TARGET_TAB_RUNS = "runs";
|
| 47 |
-
|
| 48 |
-
/** One notification, as `GET /api/v1/notifications` sends it. */
|
| 49 |
-
export interface Notification {
|
| 50 |
-
id: string;
|
| 51 |
-
alertId: string;
|
| 52 |
-
viewId: string;
|
| 53 |
-
topic: string;
|
| 54 |
-
rowId: string;
|
| 55 |
-
/** What entered β the record's own label. */
|
| 56 |
-
label: string;
|
| 57 |
-
/** What the alert is called, so a row reads without opening anything. */
|
| 58 |
-
alertLabel: string;
|
| 59 |
-
/** UTC WITH OFFSET (D-18). Kept as the server's STRING: re-formatting it here
|
| 60 |
-
* would re-introduce the browser-clock drift the offset exists to remove. */
|
| 61 |
-
at: string;
|
| 62 |
-
read: boolean;
|
| 63 |
-
/**
|
| 64 |
-
* WAVE 23 (contract C6) β WHAT KIND of notification this is.
|
| 65 |
-
*
|
| 66 |
-
* Absent (and every notification written before this wave) means the original one: a record
|
| 67 |
-
* ENTERED a watched view, routed by `topic` + `viewId`. `"automation_review"` means a card
|
| 68 |
-
* arrived at a review stage and routes by `autoId` instead β a different destination reached
|
| 69 |
-
* from the same list.
|
| 70 |
-
*
|
| 71 |
-
* β A STRING, NEVER A UNION, and that is the wave-9 law rather than laziness: the vocabulary
|
| 72 |
-
* is the SERVER's, and a client union over it turns "the server grew a kind" into "the client
|
| 73 |
-
* silently drops the row". Unknown kinds fall through to the view route, which is exactly what
|
| 74 |
-
* they did before this field existed.
|
| 75 |
-
*/
|
| 76 |
-
kind?: string;
|
| 77 |
-
/** `automation_review` only: the automation whose review stage a card reached. Absent on
|
| 78 |
-
* every other kind β and an `automation_review` row that arrives WITHOUT one opens nothing
|
| 79 |
-
* rather than guessing, the same posture `viewId` gets. */
|
| 80 |
-
autoId?: string;
|
| 81 |
-
/** Advisory. A surface that does not scroll to a stage simply selects the automation. */
|
| 82 |
-
stageId?: string;
|
| 83 |
-
/** How many cards arrived in the batch. C6 queues ONE notification per run naming the count,
|
| 84 |
-
* never one per record β so this is the number the row's own text is built from. */
|
| 85 |
-
count?: number;
|
| 86 |
-
/**
|
| 87 |
-
* β WAVE 32 Β· C3 β THE HEADER LINE, so the Inbox can be laid out like mail.
|
| 88 |
-
*
|
| 89 |
-
* `subject` is what the item is ABOUT (the alert's name, the automation's name, the database
|
| 90 |
-
* that was shared); `label` stays what HAPPENED (the record that entered, the run summary).
|
| 91 |
-
* They were one field, which is why the pane could only ever render a sentence with no sender.
|
| 92 |
-
* Absent on a server that predates this wave β {@link subjectOf} falls back rather than
|
| 93 |
-
* rendering a blank header.
|
| 94 |
-
*/
|
| 95 |
-
subject?: string;
|
| 96 |
-
/**
|
| 97 |
-
* ββ WAVE 33 Β· W33-T28 β WHO IT IS FROM. Mail has a sender; this list did not.
|
| 98 |
-
*
|
| 99 |
-
* A `verifier` reading the finished wave-32 Inbox found the row's sender POSITION occupied by
|
| 100 |
-
* `kindLabel(n.kind)` β the literals "Alert" / "Automation" / "Shared with you", a CATEGORY
|
| 101 |
-
* standing where a who belongs β and no sender anywhere on the wire. Derived by the server
|
| 102 |
-
* (`routes_alerts.notification_view`), never here: only it knows that a share has a person
|
| 103 |
-
* behind it and an alert has a machine.
|
| 104 |
-
* β Optional, so a payload from a server that predates this wave keeps today's shape;
|
| 105 |
-
* {@link senderOf} falls back rather than rendering a blank From column.
|
| 106 |
-
*/
|
| 107 |
-
sender?: string;
|
| 108 |
-
/** β WAVE 32 Β· C3 β where clicking it goes. ABSENT when this product cannot resolve a
|
| 109 |
-
* destination (an alert on a table this account can no longer route to), and that absence is
|
| 110 |
-
* load-bearing: the Inbox renders such a row as plainly unclickable rather than as a click
|
| 111 |
-
* that silently does nothing. */
|
| 112 |
-
target?: NotificationTarget;
|
| 113 |
-
}
|
| 114 |
-
|
| 115 |
-
/** One alert, as `GET /api/v1/alerts` sends it. */
|
| 116 |
-
export interface Alert {
|
| 117 |
-
id: string;
|
| 118 |
-
viewId: string;
|
| 119 |
-
topic: string;
|
| 120 |
-
owner: string;
|
| 121 |
-
label: string;
|
| 122 |
-
createdAt: string;
|
| 123 |
-
/** How many records are in its remembered set right now. */
|
| 124 |
-
matched: number;
|
| 125 |
-
seeded: boolean;
|
| 126 |
-
lastRunAt: string;
|
| 127 |
-
lastError: string;
|
| 128 |
-
}
|
| 129 |
-
|
| 130 |
-
export interface Inbox {
|
| 131 |
-
unread: number;
|
| 132 |
-
items: Notification[];
|
| 133 |
-
/**
|
| 134 |
-
* ββ WAVE 33 Β· W33-T28 / D-208 β THE SERVER'S CLOCK, so a stamp can be mail-shaped.
|
| 135 |
-
*
|
| 136 |
-
* β It exists so {@link stampText} never reads the BROWSER's clock, which is D-208's exit
|
| 137 |
-
* condition word for word. `at` is UTC with its offset (D-18) precisely so every reader sees
|
| 138 |
-
* the same instant; asking "is this today?" of a local clock would put the drift back.
|
| 139 |
-
* β Optional: absent, `stampText` returns the old absolute string rather than guessing.
|
| 140 |
-
*/
|
| 141 |
-
now?: string;
|
| 142 |
-
}
|
| 143 |
-
|
| 144 |
-
export const EMPTY_INBOX: Inbox = { unread: 0, items: [] };
|
| 145 |
-
|
| 146 |
-
/**
|
| 147 |
-
* ββ WAVE 31 Β· T23 (owner item 1) β WHAT THE PANE IS ENTITLED TO SAY, AS A FUNCTION.
|
| 148 |
-
*
|
| 149 |
-
* Owner, verbatim: *"Alerts shows notification, but when clicked it says nothing, and it doesn't
|
| 150 |
-
* remove the notification number."* Both halves are ONE mechanism. `AlertsPane` held its own
|
| 151 |
-
* `inbox` seeded to {@link EMPTY_INBOX} and had no pending state, so between opening the panel
|
| 152 |
-
* and `GET /notifications` answering β measured at **3,280 ms live** β and for ever after a
|
| 153 |
-
* failed fetch:
|
| 154 |
-
* Β· `items` was `[]`, so the pane printed **"Nothing new."** β a claim, not a wait;
|
| 155 |
-
* Β· `unread` was `0`, so **"Mark all read" was `disabled`**, and the badge the frame had
|
| 156 |
-
* already loaded could never be cleared.
|
| 157 |
-
* A confident sentence about somebody's inbox, and the one control that would fix it, both
|
| 158 |
-
* switched off by the same uninitialised state.
|
| 159 |
-
*
|
| 160 |
-
* β IT IS A FUNCTION BECAUSE THE PANE IS TSX AND TSX IS NOT UNDER TEST HERE. `verify_alerts`
|
| 161 |
-
* compiles and RUNS this module under node; markup it cannot reach is markup no control can
|
| 162 |
-
* mutate. Deciding here means the truth table is asserted and each branch is load-bearing.
|
| 163 |
-
*/
|
| 164 |
-
export type PaneView = "pending" | "rows" | "empty" | "error";
|
| 165 |
-
|
| 166 |
-
/**
|
| 167 |
-
* `pending` while the first read is in flight Β· `error` when it failed and we have nothing to
|
| 168 |
-
* show Β· `rows` when there is something Β· `empty` ONLY when a successful read returned nothing.
|
| 169 |
-
*
|
| 170 |
-
* β ROWS WIN OVER AN ERROR, and that is deliberate rather than lax: a refresh that fails while
|
| 171 |
-
* the pane already holds notifications should not blank them β the reader loses real information
|
| 172 |
-
* to a transient. The error still reaches them as the pane's message line.
|
| 173 |
-
*/
|
| 174 |
-
export function paneView(
|
| 175 |
-
phase: "pending" | "ready" | "error",
|
| 176 |
-
rowCount: number,
|
| 177 |
-
seededUnread = 0
|
| 178 |
-
): PaneView {
|
| 179 |
-
if (rowCount > 0) return "rows";
|
| 180 |
-
if (phase === "pending") return "pending";
|
| 181 |
-
if (phase === "error") return "error";
|
| 182 |
-
// β A SUCCESSFUL READ WITH NO ROWS AND A NON-ZERO COUNT IS NOT "nothing new". The badge says
|
| 183 |
-
// there is something; the page we were given does not contain it. Saying "Nothing new" there
|
| 184 |
-
// is the same false confidence in a different costume.
|
| 185 |
-
return seededUnread > 0 ? "error" : "empty";
|
| 186 |
-
}
|
| 187 |
-
|
| 188 |
-
/**
|
| 189 |
-
* May "Mark all read" be pressed?
|
| 190 |
-
*
|
| 191 |
-
* β NOT `unread === 0`, WHICH IS THE SHIPPED BUG. That test asked the PANE's own state β zero
|
| 192 |
-
* until its fetch lands, zero for ever if the fetch fails β so the control was dead in exactly
|
| 193 |
-
* the situations the owner hit. The question is about the ACCOUNT, so it is asked of the count
|
| 194 |
-
* the frame already holds, and a failed read does not take the verb away: `POST /notifications/
|
| 195 |
-
* read` with `ids: null` clears the account's inbox whether or not we managed to list it.
|
| 196 |
-
*/
|
| 197 |
-
export function canMarkAll(unread: number, busy = false): boolean {
|
| 198 |
-
return !busy && Math.max(0, Math.floor(unread)) > 0;
|
| 199 |
-
}
|
| 200 |
-
|
| 201 |
-
const str = (v: unknown): string => (typeof v === "string" ? v : "");
|
| 202 |
-
const num = (v: unknown): number => (typeof v === "number" && isFinite(v) ? v : 0);
|
| 203 |
-
|
| 204 |
-
/**
|
| 205 |
-
* β WAVE 32 Β· C3 β one wire `target` β a {@link NotificationTarget}, or `null`.
|
| 206 |
-
*
|
| 207 |
-
* β BOTH `module` AND `id` ARE REQUIRED, and dropping either test is the failure this guard is
|
| 208 |
-
* for: a target with a module and no id dispatches an open request naming NOTHING β a click that
|
| 209 |
-
* appears to work and silently does not, which is this repo's most-repeated shape. `null` is
|
| 210 |
-
* rendered as an unclickable row, which a reader can SEE.
|
| 211 |
-
*/
|
| 212 |
-
export function parseTarget(raw: unknown): NotificationTarget | null {
|
| 213 |
-
if (!raw || typeof raw !== "object") return null;
|
| 214 |
-
const t = raw as Record<string, unknown>;
|
| 215 |
-
const module = str(t.module).trim();
|
| 216 |
-
const id = str(t.id).trim();
|
| 217 |
-
if (!module || !id) return null;
|
| 218 |
-
const tab = str(t.tab).trim();
|
| 219 |
-
return { module, id, ...(tab ? { tab } : {}) };
|
| 220 |
-
}
|
| 221 |
-
|
| 222 |
-
/**
|
| 223 |
-
* `GET /notifications` β the inbox, fail-closed.
|
| 224 |
-
*
|
| 225 |
-
* β `unread` COMES FROM THE SERVER, and is not recounted from `items`. The two
|
| 226 |
-
* can legitimately differ β the list is what this page holds, the count is what
|
| 227 |
-
* the account has β and recomputing it here would make the badge a function of
|
| 228 |
-
* whatever the last fetch happened to include.
|
| 229 |
-
*/
|
| 230 |
-
export function parseInbox(body: unknown): Inbox {
|
| 231 |
-
const b = (body && typeof body === "object" ? body : {}) as Record<string, unknown>;
|
| 232 |
-
const raw = Array.isArray(b.items) ? b.items : [];
|
| 233 |
-
const items: Notification[] = [];
|
| 234 |
-
for (const item of raw) {
|
| 235 |
-
if (!item || typeof item !== "object") continue;
|
| 236 |
-
const n = item as Record<string, unknown>;
|
| 237 |
-
const id = str(n.id);
|
| 238 |
-
// An id-less notification cannot be marked read, so it would sit unread for
|
| 239 |
-
// ever and hold the badge up. Dropped, not rendered.
|
| 240 |
-
if (!id) continue;
|
| 241 |
-
items.push({
|
| 242 |
-
id,
|
| 243 |
-
alertId: str(n.alertId),
|
| 244 |
-
viewId: str(n.viewId),
|
| 245 |
-
topic: str(n.topic),
|
| 246 |
-
rowId: String(n.rowId ?? ""),
|
| 247 |
-
label: str(n.label) || String(n.rowId ?? ""),
|
| 248 |
-
alertLabel: str(n.alertLabel),
|
| 249 |
-
at: str(n.at),
|
| 250 |
-
read: n.read === true,
|
| 251 |
-
// WAVE 23 C6 β ADDITIVE and spread-conditional, exactly like `NavPage`'s flags: a payload
|
| 252 |
-
// that predates this wave keeps today's shape rather than gaining four `undefined` keys,
|
| 253 |
-
// and an `automation_review` row missing its `autoId` is left WITHOUT one rather than
|
| 254 |
-
// with an empty string that would render as a real destination.
|
| 255 |
-
...(str(n.kind) ? { kind: str(n.kind) } : {}),
|
| 256 |
-
...(str(n.autoId) ? { autoId: str(n.autoId) } : {}),
|
| 257 |
-
...(str(n.stageId) ? { stageId: str(n.stageId) } : {}),
|
| 258 |
-
...(num(n.count) > 0 ? { count: num(n.count) } : {}),
|
| 259 |
-
// β WAVE 32 Β· C3 β additive and spread-conditional, exactly like the four above.
|
| 260 |
-
...(str(n.subject) ? { subject: str(n.subject) } : {}),
|
| 261 |
-
// β WAVE 33 Β· W33-T28 β additive and spread-conditional, like every flag above it.
|
| 262 |
-
...(str(n.sender) ? { sender: str(n.sender) } : {}),
|
| 263 |
-
...(parseTarget(n.target) ? { target: parseTarget(n.target)! } : {}),
|
| 264 |
-
});
|
| 265 |
-
}
|
| 266 |
-
// β W33-T28 / D-208 β the server's clock rides through, spread-conditionally like every other
|
| 267 |
-
// additive key here, so a server that predates this wave yields the same object it always did.
|
| 268 |
-
return {
|
| 269 |
-
unread: Math.max(0, num(b.unread)),
|
| 270 |
-
items,
|
| 271 |
-
...(str(b.now) ? { now: str(b.now) } : {}),
|
| 272 |
-
};
|
| 273 |
-
}
|
| 274 |
-
|
| 275 |
-
/** `GET /alerts` β the alert list, fail-closed. */
|
| 276 |
-
export function parseAlerts(body: unknown): Alert[] {
|
| 277 |
-
const b = (body && typeof body === "object" ? body : {}) as Record<string, unknown>;
|
| 278 |
-
const raw = Array.isArray(b.alerts) ? b.alerts : [];
|
| 279 |
-
const out: Alert[] = [];
|
| 280 |
-
for (const item of raw) {
|
| 281 |
-
if (!item || typeof item !== "object") continue;
|
| 282 |
-
const a = item as Record<string, unknown>;
|
| 283 |
-
const id = str(a.id);
|
| 284 |
-
if (!id) continue;
|
| 285 |
-
out.push({
|
| 286 |
-
id,
|
| 287 |
-
viewId: str(a.viewId),
|
| 288 |
-
topic: str(a.topic),
|
| 289 |
-
owner: str(a.owner),
|
| 290 |
-
label: str(a.label) || "Untitled alert",
|
| 291 |
-
createdAt: str(a.createdAt),
|
| 292 |
-
matched: num(a.matched),
|
| 293 |
-
seeded: a.seeded === true,
|
| 294 |
-
lastRunAt: str(a.lastRunAt),
|
| 295 |
-
lastError: str(a.lastError),
|
| 296 |
-
});
|
| 297 |
-
}
|
| 298 |
-
return out;
|
| 299 |
-
}
|
| 300 |
-
|
| 301 |
-
/**
|
| 302 |
-
* A topic (the grid's scope key) β the hash route that renders it.
|
| 303 |
-
*
|
| 304 |
-
* The two built-ins are the only pair that differ, and they differ because the
|
| 305 |
-
* REGISTRY names the surface while the GRID names the scope; a user table is its
|
| 306 |
-
* own key in both. `null` for anything else: a notification for a topic this
|
| 307 |
-
* client cannot route to must do nothing, not navigate somewhere plausible.
|
| 308 |
-
*/
|
| 309 |
-
export function routeForTopic(topic: string): string | null {
|
| 310 |
-
const t = str(topic).trim();
|
| 311 |
-
if (t === "customer") return "customer_data";
|
| 312 |
-
if (t === "product") return "product_data";
|
| 313 |
-
if (/^ut_[A-Za-z0-9_]+$/.test(t)) return t;
|
| 314 |
-
return null;
|
| 315 |
-
}
|
| 316 |
-
|
| 317 |
-
|
| 318 |
-
|
| 319 |
-
|
| 320 |
-
|
| 321 |
-
|
| 322 |
-
/**
|
| 323 |
-
* The stamp, made readable WITHOUT touching a clock.
|
| 324 |
-
*
|
| 325 |
-
* β NO `new Date()`, NO `toLocaleString()`, and that is the whole design. The
|
| 326 |
-
* server sends UTC WITH ITS OFFSET (D-18) precisely so every reader sees the
|
| 327 |
-
* same instant; parsing it into a browser Date and formatting it back would
|
| 328 |
-
* re-introduce the drift the offset exists to remove β a tenant a day ahead
|
| 329 |
-
* being told an event happened tomorrow ([[date-window-vocabulary]]). This is
|
| 330 |
-
* STRING SURGERY: keep the date and the minutes, drop the seconds and the `T`.
|
| 331 |
-
* Anything that does not look like an ISO stamp passes through untouched, so a
|
| 332 |
-
* format this function has never seen is shown as sent rather than mangled.
|
| 333 |
-
*/
|
| 334 |
-
const MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun",
|
| 335 |
-
"Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
|
| 336 |
-
|
| 337 |
-
export function stampText(at: string, now?: string): string {
|
| 338 |
-
const m = /^(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2})/.exec(str(at));
|
| 339 |
-
if (!m) return str(at);
|
| 340 |
-
const [, day, hhmm] = m;
|
| 341 |
-
// ββ W33-T28 / D-208 β MAIL-SHAPED, AND STILL STRING SURGERY.
|
| 342 |
-
//
|
| 343 |
-
// β `now` COMES FROM THE SERVER (`inbox.now`), NEVER FROM `new Date()`. That is D-208's exit
|
| 344 |
-
// condition word for word β "without reading the browser clock" β and the reason is the same
|
| 345 |
-
// one this function has always carried: `at` is sent as UTC WITH its offset (D-18) so every
|
| 346 |
-
// reader sees the same instant, and deciding "is this today?" against a browser clock would
|
| 347 |
-
// re-introduce exactly the drift the offset removes. Both operands are the server's.
|
| 348 |
-
//
|
| 349 |
-
// β AND WITHOUT `now` IT DEGRADES TO THE OLD FORMAT RATHER THAN GUESSING. A caller that has no
|
| 350 |
-
// server clock gets `2026-08-13 09:41` β the pre-wave string, unambiguous and never wrong β
|
| 351 |
-
// instead of a relative stamp computed from something we do not trust.
|
| 352 |
-
const today = /^(\d{4}-\d{2}-\d{2})/.exec(str(now));
|
| 353 |
-
if (!today) return `${day} ${hhmm}`;
|
| 354 |
-
if (today[1] === day) return hhmm;
|
| 355 |
-
const [y, mo, d] = day.split("-");
|
| 356 |
-
const month = MONTHS[Number(mo) - 1] || mo;
|
| 357 |
-
// The year only when it differs β a mail client does not print "2026" on a message from March.
|
| 358 |
-
return today[1].slice(0, 4) === y
|
| 359 |
-
? `${month} ${Number(d)}`
|
| 360 |
-
: `${month} ${Number(d)}, ${y}`;
|
| 361 |
-
}
|
| 362 |
-
|
| 363 |
-
/**
|
| 364 |
-
* The badge's text. Never the raw number past 99: a nav row is 236px wide and a
|
| 365 |
-
* four-digit badge pushes the label out of it.
|
| 366 |
-
*/
|
| 367 |
-
export function badgeText(unread: number): string {
|
| 368 |
-
const n = Math.max(0, Math.floor(unread));
|
| 369 |
-
if (n <= 0) return "";
|
| 370 |
-
return n > 99 ? "99+" : String(n);
|
| 371 |
-
}
|
| 372 |
-
|
| 373 |
-
/**
|
| 374 |
-
* Apply a read/unread change LOCALLY, mirroring what the server just stored, and
|
| 375 |
-
* return the new inbox with the count corrected.
|
| 376 |
-
*
|
| 377 |
-
* `ids === null` is "all of them" (the API's own convention for mark-all). The
|
| 378 |
-
* count is derived from the ITEMS here β deliberately, and it is the one place
|
| 379 |
-
* that is right to do so: the server's answer is in flight, and the alternative
|
| 380 |
-
* is a badge that keeps its old number until the refetch lands.
|
| 381 |
-
*/
|
| 382 |
-
export function applyRead(inbox: Inbox, ids: string[] | null, read: boolean): Inbox {
|
| 383 |
-
const wanted = ids === null ? null : new Set(ids);
|
| 384 |
-
const items = inbox.items.map((n) =>
|
| 385 |
-
wanted === null || wanted.has(n.id) ? { ...n, read } : n
|
| 386 |
-
);
|
| 387 |
-
const seenUnread = items.filter((n) => !n.read).length;
|
| 388 |
-
// A page can hold fewer notifications than the account has, so a partial read
|
| 389 |
-
// must SUBTRACT from the server's count rather than replace it with this
|
| 390 |
-
// page's tally β except when marking everything, where zero is the answer.
|
| 391 |
-
if (wanted === null) return { unread: read ? 0 : items.length, items };
|
| 392 |
-
const changed = inbox.items.filter(
|
| 393 |
-
(n) => wanted.has(n.id) && n.read !== read
|
| 394 |
-
).length;
|
| 395 |
-
const delta = read ? -changed : changed;
|
| 396 |
-
return { unread: Math.max(seenUnread, inbox.unread + delta), items };
|
| 397 |
-
}
|
| 398 |
-
|
| 399 |
-
/**
|
| 400 |
-
* The shellβrail channel for "make an alert out of this view" β the view rail
|
| 401 |
-
* raises it, the frame (which knows the current route, and therefore the topic)
|
| 402 |
-
* answers. Same reason as C-SHARE's event: the rail is host-neutral and cannot
|
| 403 |
-
* import the shell, and it does not know its own scope key.
|
| 404 |
-
*/
|
| 405 |
-
export const ALERT_CREATE_EVENT = "aios:alert-create";
|
| 406 |
-
|
| 407 |
-
export interface AlertCreateRequest {
|
| 408 |
-
viewId: string;
|
| 409 |
-
label: string;
|
| 410 |
-
}
|
| 411 |
-
|
| 412 |
-
export function parseAlertCreate(detail: unknown): AlertCreateRequest | null {
|
| 413 |
-
if (!detail || typeof detail !== "object") return null;
|
| 414 |
-
const d = detail as Record<string, unknown>;
|
| 415 |
-
const viewId = str(d.viewId).trim();
|
| 416 |
-
if (!viewId) return null;
|
| 417 |
-
return { viewId, label: str(d.label).trim() || viewId };
|
| 418 |
-
}
|
|
|
|
| 1 |
+
// ---------------------------------------------------------------------------
|
| 2 |
+
// alerts/alertsModel.ts β WAVE 20 item 25 (contract C-ALERT): the inbox's PURE
|
| 3 |
+
// half. React-free and fetch-free, so `verify_alerts.py` runs it under node.
|
| 4 |
+
//
|
| 5 |
+
// An alert says "tell me when a record ENTERS this view". The client half is
|
| 6 |
+
// small, and every part of it fails silently when it is wrong:
|
| 7 |
+
//
|
| 8 |
+
// Β· an unread count taken from `items.length` rather than from the server's
|
| 9 |
+
// own `unread` disagrees with the badge the moment a page is capped or a
|
| 10 |
+
// read lands in another tab β and a badge that says 3 when the list shows 9
|
| 11 |
+
// teaches the reader to ignore the badge;
|
| 12 |
+
// Β· a notification whose `viewId` no longer resolves must open NOTHING rather
|
| 13 |
+
// than a wrong view β alerts outlive the views they were made from;
|
| 14 |
+
// Β· `topic` is the surface's scope key ("customer", "product", "ut_β¦"), and
|
| 15 |
+
// the ROUTE is a registry key ("customer_data") β mapping one to the other
|
| 16 |
+
// by guesswork sends every click to a page that does not exist.
|
| 17 |
+
// ---------------------------------------------------------------------------
|
| 18 |
+
|
| 19 |
+
/**
|
| 20 |
+
* ββ WAVE 32 Β· T20 Β· CONTRACT C3 β WHERE AN INBOX ITEM OPENS.
|
| 21 |
+
*
|
| 22 |
+
* `module` is the destination surface (`"database"` or `"automation"`) and `id` is what to open
|
| 23 |
+
* in it. `tab` is the SUB-SELECTION inside that module: the literal `"runs"` for an automation's
|
| 24 |
+
* run log, or the VIEW ID to select on a database. One optional key, two destinations.
|
| 25 |
+
*
|
| 26 |
+
* β EVERY FIELD IS A PLAIN `string`, NEVER A UNION, and that is alertsModel's wave-9 law rather
|
| 27 |
+
* than laziness: the vocabulary is the SERVER's, and a client union over it turns "the server grew
|
| 28 |
+
* a module" into "the client silently drops the row". An unknown module is refused by the
|
| 29 |
+
* FRAME's dispatcher, out loud, which is a different thing from never arriving.
|
| 30 |
+
*/
|
| 31 |
+
export interface NotificationTarget {
|
| 32 |
+
module: string;
|
| 33 |
+
id: string;
|
| 34 |
+
tab?: string;
|
| 35 |
+
}
|
| 36 |
+
|
| 37 |
+
/** C3's `kind` vocabulary, mirroring `routes_alerts.NOTIF_KIND_*`. Held as constants so the
|
| 38 |
+
* Inbox's branch and the gate's fixtures cannot disagree about the word. */
|
| 39 |
+
export const NOTIF_KIND_ALERT = "alert";
|
| 40 |
+
export const NOTIF_KIND_AUTOMATION = "automation";
|
| 41 |
+
export const NOTIF_KIND_SHARE = "share";
|
| 42 |
+
|
| 43 |
+
/** C3's `target.module` vocabulary, and the automation sub-selection. */
|
| 44 |
+
export const TARGET_MODULE_DATABASE = "database";
|
| 45 |
+
export const TARGET_MODULE_AUTOMATION = "automation";
|
| 46 |
+
export const TARGET_TAB_RUNS = "runs";
|
| 47 |
+
|
| 48 |
+
/** One notification, as `GET /api/v1/notifications` sends it. */
|
| 49 |
+
export interface Notification {
|
| 50 |
+
id: string;
|
| 51 |
+
alertId: string;
|
| 52 |
+
viewId: string;
|
| 53 |
+
topic: string;
|
| 54 |
+
rowId: string;
|
| 55 |
+
/** What entered β the record's own label. */
|
| 56 |
+
label: string;
|
| 57 |
+
/** What the alert is called, so a row reads without opening anything. */
|
| 58 |
+
alertLabel: string;
|
| 59 |
+
/** UTC WITH OFFSET (D-18). Kept as the server's STRING: re-formatting it here
|
| 60 |
+
* would re-introduce the browser-clock drift the offset exists to remove. */
|
| 61 |
+
at: string;
|
| 62 |
+
read: boolean;
|
| 63 |
+
/**
|
| 64 |
+
* WAVE 23 (contract C6) β WHAT KIND of notification this is.
|
| 65 |
+
*
|
| 66 |
+
* Absent (and every notification written before this wave) means the original one: a record
|
| 67 |
+
* ENTERED a watched view, routed by `topic` + `viewId`. `"automation_review"` means a card
|
| 68 |
+
* arrived at a review stage and routes by `autoId` instead β a different destination reached
|
| 69 |
+
* from the same list.
|
| 70 |
+
*
|
| 71 |
+
* β A STRING, NEVER A UNION, and that is the wave-9 law rather than laziness: the vocabulary
|
| 72 |
+
* is the SERVER's, and a client union over it turns "the server grew a kind" into "the client
|
| 73 |
+
* silently drops the row". Unknown kinds fall through to the view route, which is exactly what
|
| 74 |
+
* they did before this field existed.
|
| 75 |
+
*/
|
| 76 |
+
kind?: string;
|
| 77 |
+
/** `automation_review` only: the automation whose review stage a card reached. Absent on
|
| 78 |
+
* every other kind β and an `automation_review` row that arrives WITHOUT one opens nothing
|
| 79 |
+
* rather than guessing, the same posture `viewId` gets. */
|
| 80 |
+
autoId?: string;
|
| 81 |
+
/** Advisory. A surface that does not scroll to a stage simply selects the automation. */
|
| 82 |
+
stageId?: string;
|
| 83 |
+
/** How many cards arrived in the batch. C6 queues ONE notification per run naming the count,
|
| 84 |
+
* never one per record β so this is the number the row's own text is built from. */
|
| 85 |
+
count?: number;
|
| 86 |
+
/**
|
| 87 |
+
* β WAVE 32 Β· C3 β THE HEADER LINE, so the Inbox can be laid out like mail.
|
| 88 |
+
*
|
| 89 |
+
* `subject` is what the item is ABOUT (the alert's name, the automation's name, the database
|
| 90 |
+
* that was shared); `label` stays what HAPPENED (the record that entered, the run summary).
|
| 91 |
+
* They were one field, which is why the pane could only ever render a sentence with no sender.
|
| 92 |
+
* Absent on a server that predates this wave β {@link subjectOf} falls back rather than
|
| 93 |
+
* rendering a blank header.
|
| 94 |
+
*/
|
| 95 |
+
subject?: string;
|
| 96 |
+
/**
|
| 97 |
+
* ββ WAVE 33 Β· W33-T28 β WHO IT IS FROM. Mail has a sender; this list did not.
|
| 98 |
+
*
|
| 99 |
+
* A `verifier` reading the finished wave-32 Inbox found the row's sender POSITION occupied by
|
| 100 |
+
* `kindLabel(n.kind)` β the literals "Alert" / "Automation" / "Shared with you", a CATEGORY
|
| 101 |
+
* standing where a who belongs β and no sender anywhere on the wire. Derived by the server
|
| 102 |
+
* (`routes_alerts.notification_view`), never here: only it knows that a share has a person
|
| 103 |
+
* behind it and an alert has a machine.
|
| 104 |
+
* β Optional, so a payload from a server that predates this wave keeps today's shape;
|
| 105 |
+
* {@link senderOf} falls back rather than rendering a blank From column.
|
| 106 |
+
*/
|
| 107 |
+
sender?: string;
|
| 108 |
+
/** β WAVE 32 Β· C3 β where clicking it goes. ABSENT when this product cannot resolve a
|
| 109 |
+
* destination (an alert on a table this account can no longer route to), and that absence is
|
| 110 |
+
* load-bearing: the Inbox renders such a row as plainly unclickable rather than as a click
|
| 111 |
+
* that silently does nothing. */
|
| 112 |
+
target?: NotificationTarget;
|
| 113 |
+
}
|
| 114 |
+
|
| 115 |
+
/** One alert, as `GET /api/v1/alerts` sends it. */
|
| 116 |
+
export interface Alert {
|
| 117 |
+
id: string;
|
| 118 |
+
viewId: string;
|
| 119 |
+
topic: string;
|
| 120 |
+
owner: string;
|
| 121 |
+
label: string;
|
| 122 |
+
createdAt: string;
|
| 123 |
+
/** How many records are in its remembered set right now. */
|
| 124 |
+
matched: number;
|
| 125 |
+
seeded: boolean;
|
| 126 |
+
lastRunAt: string;
|
| 127 |
+
lastError: string;
|
| 128 |
+
}
|
| 129 |
+
|
| 130 |
+
export interface Inbox {
|
| 131 |
+
unread: number;
|
| 132 |
+
items: Notification[];
|
| 133 |
+
/**
|
| 134 |
+
* ββ WAVE 33 Β· W33-T28 / D-208 β THE SERVER'S CLOCK, so a stamp can be mail-shaped.
|
| 135 |
+
*
|
| 136 |
+
* β It exists so {@link stampText} never reads the BROWSER's clock, which is D-208's exit
|
| 137 |
+
* condition word for word. `at` is UTC with its offset (D-18) precisely so every reader sees
|
| 138 |
+
* the same instant; asking "is this today?" of a local clock would put the drift back.
|
| 139 |
+
* β Optional: absent, `stampText` returns the old absolute string rather than guessing.
|
| 140 |
+
*/
|
| 141 |
+
now?: string;
|
| 142 |
+
}
|
| 143 |
+
|
| 144 |
+
export const EMPTY_INBOX: Inbox = { unread: 0, items: [] };
|
| 145 |
+
|
| 146 |
+
/**
|
| 147 |
+
* ββ WAVE 31 Β· T23 (owner item 1) β WHAT THE PANE IS ENTITLED TO SAY, AS A FUNCTION.
|
| 148 |
+
*
|
| 149 |
+
* Owner, verbatim: *"Alerts shows notification, but when clicked it says nothing, and it doesn't
|
| 150 |
+
* remove the notification number."* Both halves are ONE mechanism. `AlertsPane` held its own
|
| 151 |
+
* `inbox` seeded to {@link EMPTY_INBOX} and had no pending state, so between opening the panel
|
| 152 |
+
* and `GET /notifications` answering β measured at **3,280 ms live** β and for ever after a
|
| 153 |
+
* failed fetch:
|
| 154 |
+
* Β· `items` was `[]`, so the pane printed **"Nothing new."** β a claim, not a wait;
|
| 155 |
+
* Β· `unread` was `0`, so **"Mark all read" was `disabled`**, and the badge the frame had
|
| 156 |
+
* already loaded could never be cleared.
|
| 157 |
+
* A confident sentence about somebody's inbox, and the one control that would fix it, both
|
| 158 |
+
* switched off by the same uninitialised state.
|
| 159 |
+
*
|
| 160 |
+
* β IT IS A FUNCTION BECAUSE THE PANE IS TSX AND TSX IS NOT UNDER TEST HERE. `verify_alerts`
|
| 161 |
+
* compiles and RUNS this module under node; markup it cannot reach is markup no control can
|
| 162 |
+
* mutate. Deciding here means the truth table is asserted and each branch is load-bearing.
|
| 163 |
+
*/
|
| 164 |
+
export type PaneView = "pending" | "rows" | "empty" | "error";
|
| 165 |
+
|
| 166 |
+
/**
|
| 167 |
+
* `pending` while the first read is in flight Β· `error` when it failed and we have nothing to
|
| 168 |
+
* show Β· `rows` when there is something Β· `empty` ONLY when a successful read returned nothing.
|
| 169 |
+
*
|
| 170 |
+
* β ROWS WIN OVER AN ERROR, and that is deliberate rather than lax: a refresh that fails while
|
| 171 |
+
* the pane already holds notifications should not blank them β the reader loses real information
|
| 172 |
+
* to a transient. The error still reaches them as the pane's message line.
|
| 173 |
+
*/
|
| 174 |
+
export function paneView(
|
| 175 |
+
phase: "pending" | "ready" | "error",
|
| 176 |
+
rowCount: number,
|
| 177 |
+
seededUnread = 0
|
| 178 |
+
): PaneView {
|
| 179 |
+
if (rowCount > 0) return "rows";
|
| 180 |
+
if (phase === "pending") return "pending";
|
| 181 |
+
if (phase === "error") return "error";
|
| 182 |
+
// β A SUCCESSFUL READ WITH NO ROWS AND A NON-ZERO COUNT IS NOT "nothing new". The badge says
|
| 183 |
+
// there is something; the page we were given does not contain it. Saying "Nothing new" there
|
| 184 |
+
// is the same false confidence in a different costume.
|
| 185 |
+
return seededUnread > 0 ? "error" : "empty";
|
| 186 |
+
}
|
| 187 |
+
|
| 188 |
+
/**
|
| 189 |
+
* May "Mark all read" be pressed?
|
| 190 |
+
*
|
| 191 |
+
* β NOT `unread === 0`, WHICH IS THE SHIPPED BUG. That test asked the PANE's own state β zero
|
| 192 |
+
* until its fetch lands, zero for ever if the fetch fails β so the control was dead in exactly
|
| 193 |
+
* the situations the owner hit. The question is about the ACCOUNT, so it is asked of the count
|
| 194 |
+
* the frame already holds, and a failed read does not take the verb away: `POST /notifications/
|
| 195 |
+
* read` with `ids: null` clears the account's inbox whether or not we managed to list it.
|
| 196 |
+
*/
|
| 197 |
+
export function canMarkAll(unread: number, busy = false): boolean {
|
| 198 |
+
return !busy && Math.max(0, Math.floor(unread)) > 0;
|
| 199 |
+
}
|
| 200 |
+
|
| 201 |
+
const str = (v: unknown): string => (typeof v === "string" ? v : "");
|
| 202 |
+
const num = (v: unknown): number => (typeof v === "number" && isFinite(v) ? v : 0);
|
| 203 |
+
|
| 204 |
+
/**
|
| 205 |
+
* β WAVE 32 Β· C3 β one wire `target` β a {@link NotificationTarget}, or `null`.
|
| 206 |
+
*
|
| 207 |
+
* β BOTH `module` AND `id` ARE REQUIRED, and dropping either test is the failure this guard is
|
| 208 |
+
* for: a target with a module and no id dispatches an open request naming NOTHING β a click that
|
| 209 |
+
* appears to work and silently does not, which is this repo's most-repeated shape. `null` is
|
| 210 |
+
* rendered as an unclickable row, which a reader can SEE.
|
| 211 |
+
*/
|
| 212 |
+
export function parseTarget(raw: unknown): NotificationTarget | null {
|
| 213 |
+
if (!raw || typeof raw !== "object") return null;
|
| 214 |
+
const t = raw as Record<string, unknown>;
|
| 215 |
+
const module = str(t.module).trim();
|
| 216 |
+
const id = str(t.id).trim();
|
| 217 |
+
if (!module || !id) return null;
|
| 218 |
+
const tab = str(t.tab).trim();
|
| 219 |
+
return { module, id, ...(tab ? { tab } : {}) };
|
| 220 |
+
}
|
| 221 |
+
|
| 222 |
+
/**
|
| 223 |
+
* `GET /notifications` β the inbox, fail-closed.
|
| 224 |
+
*
|
| 225 |
+
* β `unread` COMES FROM THE SERVER, and is not recounted from `items`. The two
|
| 226 |
+
* can legitimately differ β the list is what this page holds, the count is what
|
| 227 |
+
* the account has β and recomputing it here would make the badge a function of
|
| 228 |
+
* whatever the last fetch happened to include.
|
| 229 |
+
*/
|
| 230 |
+
export function parseInbox(body: unknown): Inbox {
|
| 231 |
+
const b = (body && typeof body === "object" ? body : {}) as Record<string, unknown>;
|
| 232 |
+
const raw = Array.isArray(b.items) ? b.items : [];
|
| 233 |
+
const items: Notification[] = [];
|
| 234 |
+
for (const item of raw) {
|
| 235 |
+
if (!item || typeof item !== "object") continue;
|
| 236 |
+
const n = item as Record<string, unknown>;
|
| 237 |
+
const id = str(n.id);
|
| 238 |
+
// An id-less notification cannot be marked read, so it would sit unread for
|
| 239 |
+
// ever and hold the badge up. Dropped, not rendered.
|
| 240 |
+
if (!id) continue;
|
| 241 |
+
items.push({
|
| 242 |
+
id,
|
| 243 |
+
alertId: str(n.alertId),
|
| 244 |
+
viewId: str(n.viewId),
|
| 245 |
+
topic: str(n.topic),
|
| 246 |
+
rowId: String(n.rowId ?? ""),
|
| 247 |
+
label: str(n.label) || String(n.rowId ?? ""),
|
| 248 |
+
alertLabel: str(n.alertLabel),
|
| 249 |
+
at: str(n.at),
|
| 250 |
+
read: n.read === true,
|
| 251 |
+
// WAVE 23 C6 β ADDITIVE and spread-conditional, exactly like `NavPage`'s flags: a payload
|
| 252 |
+
// that predates this wave keeps today's shape rather than gaining four `undefined` keys,
|
| 253 |
+
// and an `automation_review` row missing its `autoId` is left WITHOUT one rather than
|
| 254 |
+
// with an empty string that would render as a real destination.
|
| 255 |
+
...(str(n.kind) ? { kind: str(n.kind) } : {}),
|
| 256 |
+
...(str(n.autoId) ? { autoId: str(n.autoId) } : {}),
|
| 257 |
+
...(str(n.stageId) ? { stageId: str(n.stageId) } : {}),
|
| 258 |
+
...(num(n.count) > 0 ? { count: num(n.count) } : {}),
|
| 259 |
+
// β WAVE 32 Β· C3 β additive and spread-conditional, exactly like the four above.
|
| 260 |
+
...(str(n.subject) ? { subject: str(n.subject) } : {}),
|
| 261 |
+
// β WAVE 33 Β· W33-T28 β additive and spread-conditional, like every flag above it.
|
| 262 |
+
...(str(n.sender) ? { sender: str(n.sender) } : {}),
|
| 263 |
+
...(parseTarget(n.target) ? { target: parseTarget(n.target)! } : {}),
|
| 264 |
+
});
|
| 265 |
+
}
|
| 266 |
+
// β W33-T28 / D-208 β the server's clock rides through, spread-conditionally like every other
|
| 267 |
+
// additive key here, so a server that predates this wave yields the same object it always did.
|
| 268 |
+
return {
|
| 269 |
+
unread: Math.max(0, num(b.unread)),
|
| 270 |
+
items,
|
| 271 |
+
...(str(b.now) ? { now: str(b.now) } : {}),
|
| 272 |
+
};
|
| 273 |
+
}
|
| 274 |
+
|
| 275 |
+
/** `GET /alerts` β the alert list, fail-closed. */
|
| 276 |
+
export function parseAlerts(body: unknown): Alert[] {
|
| 277 |
+
const b = (body && typeof body === "object" ? body : {}) as Record<string, unknown>;
|
| 278 |
+
const raw = Array.isArray(b.alerts) ? b.alerts : [];
|
| 279 |
+
const out: Alert[] = [];
|
| 280 |
+
for (const item of raw) {
|
| 281 |
+
if (!item || typeof item !== "object") continue;
|
| 282 |
+
const a = item as Record<string, unknown>;
|
| 283 |
+
const id = str(a.id);
|
| 284 |
+
if (!id) continue;
|
| 285 |
+
out.push({
|
| 286 |
+
id,
|
| 287 |
+
viewId: str(a.viewId),
|
| 288 |
+
topic: str(a.topic),
|
| 289 |
+
owner: str(a.owner),
|
| 290 |
+
label: str(a.label) || "Untitled alert",
|
| 291 |
+
createdAt: str(a.createdAt),
|
| 292 |
+
matched: num(a.matched),
|
| 293 |
+
seeded: a.seeded === true,
|
| 294 |
+
lastRunAt: str(a.lastRunAt),
|
| 295 |
+
lastError: str(a.lastError),
|
| 296 |
+
});
|
| 297 |
+
}
|
| 298 |
+
return out;
|
| 299 |
+
}
|
| 300 |
+
|
| 301 |
+
/**
|
| 302 |
+
* A topic (the grid's scope key) β the hash route that renders it.
|
| 303 |
+
*
|
| 304 |
+
* The two built-ins are the only pair that differ, and they differ because the
|
| 305 |
+
* REGISTRY names the surface while the GRID names the scope; a user table is its
|
| 306 |
+
* own key in both. `null` for anything else: a notification for a topic this
|
| 307 |
+
* client cannot route to must do nothing, not navigate somewhere plausible.
|
| 308 |
+
*/
|
| 309 |
+
export function routeForTopic(topic: string): string | null {
|
| 310 |
+
const t = str(topic).trim();
|
| 311 |
+
if (t === "customer") return "customer_data";
|
| 312 |
+
if (t === "product") return "product_data";
|
| 313 |
+
if (/^ut_[A-Za-z0-9_]+$/.test(t)) return t;
|
| 314 |
+
return null;
|
| 315 |
+
}
|
| 316 |
+
|
| 317 |
+
|
| 318 |
+
|
| 319 |
+
|
| 320 |
+
|
| 321 |
+
|
| 322 |
+
/**
|
| 323 |
+
* The stamp, made readable WITHOUT touching a clock.
|
| 324 |
+
*
|
| 325 |
+
* β NO `new Date()`, NO `toLocaleString()`, and that is the whole design. The
|
| 326 |
+
* server sends UTC WITH ITS OFFSET (D-18) precisely so every reader sees the
|
| 327 |
+
* same instant; parsing it into a browser Date and formatting it back would
|
| 328 |
+
* re-introduce the drift the offset exists to remove β a tenant a day ahead
|
| 329 |
+
* being told an event happened tomorrow ([[date-window-vocabulary]]). This is
|
| 330 |
+
* STRING SURGERY: keep the date and the minutes, drop the seconds and the `T`.
|
| 331 |
+
* Anything that does not look like an ISO stamp passes through untouched, so a
|
| 332 |
+
* format this function has never seen is shown as sent rather than mangled.
|
| 333 |
+
*/
|
| 334 |
+
const MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun",
|
| 335 |
+
"Jul", "Aug", "Sep", "Oct", "Nov", "Dec"];
|
| 336 |
+
|
| 337 |
+
export function stampText(at: string, now?: string): string {
|
| 338 |
+
const m = /^(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2})/.exec(str(at));
|
| 339 |
+
if (!m) return str(at);
|
| 340 |
+
const [, day, hhmm] = m;
|
| 341 |
+
// ββ W33-T28 / D-208 β MAIL-SHAPED, AND STILL STRING SURGERY.
|
| 342 |
+
//
|
| 343 |
+
// β `now` COMES FROM THE SERVER (`inbox.now`), NEVER FROM `new Date()`. That is D-208's exit
|
| 344 |
+
// condition word for word β "without reading the browser clock" β and the reason is the same
|
| 345 |
+
// one this function has always carried: `at` is sent as UTC WITH its offset (D-18) so every
|
| 346 |
+
// reader sees the same instant, and deciding "is this today?" against a browser clock would
|
| 347 |
+
// re-introduce exactly the drift the offset removes. Both operands are the server's.
|
| 348 |
+
//
|
| 349 |
+
// β AND WITHOUT `now` IT DEGRADES TO THE OLD FORMAT RATHER THAN GUESSING. A caller that has no
|
| 350 |
+
// server clock gets `2026-08-13 09:41` β the pre-wave string, unambiguous and never wrong β
|
| 351 |
+
// instead of a relative stamp computed from something we do not trust.
|
| 352 |
+
const today = /^(\d{4}-\d{2}-\d{2})/.exec(str(now));
|
| 353 |
+
if (!today) return `${day} ${hhmm}`;
|
| 354 |
+
if (today[1] === day) return hhmm;
|
| 355 |
+
const [y, mo, d] = day.split("-");
|
| 356 |
+
const month = MONTHS[Number(mo) - 1] || mo;
|
| 357 |
+
// The year only when it differs β a mail client does not print "2026" on a message from March.
|
| 358 |
+
return today[1].slice(0, 4) === y
|
| 359 |
+
? `${month} ${Number(d)}`
|
| 360 |
+
: `${month} ${Number(d)}, ${y}`;
|
| 361 |
+
}
|
| 362 |
+
|
| 363 |
+
/**
|
| 364 |
+
* The badge's text. Never the raw number past 99: a nav row is 236px wide and a
|
| 365 |
+
* four-digit badge pushes the label out of it.
|
| 366 |
+
*/
|
| 367 |
+
export function badgeText(unread: number): string {
|
| 368 |
+
const n = Math.max(0, Math.floor(unread));
|
| 369 |
+
if (n <= 0) return "";
|
| 370 |
+
return n > 99 ? "99+" : String(n);
|
| 371 |
+
}
|
| 372 |
+
|
| 373 |
+
/**
|
| 374 |
+
* Apply a read/unread change LOCALLY, mirroring what the server just stored, and
|
| 375 |
+
* return the new inbox with the count corrected.
|
| 376 |
+
*
|
| 377 |
+
* `ids === null` is "all of them" (the API's own convention for mark-all). The
|
| 378 |
+
* count is derived from the ITEMS here β deliberately, and it is the one place
|
| 379 |
+
* that is right to do so: the server's answer is in flight, and the alternative
|
| 380 |
+
* is a badge that keeps its old number until the refetch lands.
|
| 381 |
+
*/
|
| 382 |
+
export function applyRead(inbox: Inbox, ids: string[] | null, read: boolean): Inbox {
|
| 383 |
+
const wanted = ids === null ? null : new Set(ids);
|
| 384 |
+
const items = inbox.items.map((n) =>
|
| 385 |
+
wanted === null || wanted.has(n.id) ? { ...n, read } : n
|
| 386 |
+
);
|
| 387 |
+
const seenUnread = items.filter((n) => !n.read).length;
|
| 388 |
+
// A page can hold fewer notifications than the account has, so a partial read
|
| 389 |
+
// must SUBTRACT from the server's count rather than replace it with this
|
| 390 |
+
// page's tally β except when marking everything, where zero is the answer.
|
| 391 |
+
if (wanted === null) return { unread: read ? 0 : items.length, items };
|
| 392 |
+
const changed = inbox.items.filter(
|
| 393 |
+
(n) => wanted.has(n.id) && n.read !== read
|
| 394 |
+
).length;
|
| 395 |
+
const delta = read ? -changed : changed;
|
| 396 |
+
return { unread: Math.max(seenUnread, inbox.unread + delta), items };
|
| 397 |
+
}
|
| 398 |
+
|
| 399 |
+
/**
|
| 400 |
+
* The shellβrail channel for "make an alert out of this view" β the view rail
|
| 401 |
+
* raises it, the frame (which knows the current route, and therefore the topic)
|
| 402 |
+
* answers. Same reason as C-SHARE's event: the rail is host-neutral and cannot
|
| 403 |
+
* import the shell, and it does not know its own scope key.
|
| 404 |
+
*/
|
| 405 |
+
export const ALERT_CREATE_EVENT = "aios:alert-create";
|
| 406 |
+
|
| 407 |
+
export interface AlertCreateRequest {
|
| 408 |
+
viewId: string;
|
| 409 |
+
label: string;
|
| 410 |
+
}
|
| 411 |
+
|
| 412 |
+
export function parseAlertCreate(detail: unknown): AlertCreateRequest | null {
|
| 413 |
+
if (!detail || typeof detail !== "object") return null;
|
| 414 |
+
const d = detail as Record<string, unknown>;
|
| 415 |
+
const viewId = str(d.viewId).trim();
|
| 416 |
+
if (!viewId) return null;
|
| 417 |
+
return { viewId, label: str(d.label).trim() || viewId };
|
| 418 |
+
}
|
web/src/assistant/AssistantPage.tsx
CHANGED
|
@@ -1,748 +1,748 @@
|
|
| 1 |
-
import { lazy, Suspense, useCallback, useEffect, useMemo, useRef, useState } from "react";
|
| 2 |
-
|
| 3 |
-
import { QUERY_OPEN_EVENT, signal } from "../apiContract";
|
| 4 |
-
import { FolderMark } from "../customer-grid/icons";
|
| 5 |
-
import { queryCitationLabel } from "../customer-grid/queryPreview";
|
| 6 |
-
import { retryEmit } from "../inbox/inboxModel";
|
| 7 |
-
import { queryGroups, QueryRail } from "../query/queryParts";
|
| 8 |
-
import { DbIcon } from "../shell/dbFrame";
|
| 9 |
-
import { ASSISTANT_ROUTE, databaseEntries, QUERY_ROUTE, routeKeyOf } from "../shell/nav";
|
| 10 |
-
import type { NavEntry } from "../shell/nav";
|
| 11 |
-
import {
|
| 12 |
-
deleteQuery, deleteThread, fetchQueries, rateMessage, submitChat, type QueryCitation,
|
| 13 |
-
type QueryIndex, type QueryMessage, type SavedQuery,
|
| 14 |
-
} from "../query/queryApi";
|
| 15 |
-
import "./assistant.css";
|
| 16 |
-
|
| 17 |
-
/**
|
| 18 |
-
* β LAZY ON PURPOSE. `QueryPage` imports `CustomerGrid`, the largest module in the client; a
|
| 19 |
-
* static import here would put the whole grid in the chat bundle, so opening the Assistant to ask
|
| 20 |
-
* one question would download the surface it is merged WITH but not showing. The list itself
|
| 21 |
-
* comes from `queryParts`, which imports no grid, so the left panel renders in Query mode before
|
| 22 |
-
* this chunk arrives.
|
| 23 |
-
*/
|
| 24 |
-
const QueryWorkspace = lazy(() => import("../query/QueryPage"));
|
| 25 |
-
|
| 26 |
-
export interface AssistantPageProps { granted: NavEntry[]; }
|
| 27 |
-
|
| 28 |
-
/**
|
| 29 |
-
* ββ R14 / CONTRACT C2 β ONE MODULE, ONE ROUTE, TWO LISTS. Owner, 2026-08-16: *"Combine the AI
|
| 30 |
-
* assistant module AND the Query module. In the secondary navigation at the top, just above
|
| 31 |
-
* '+ New chat', an easily accessible toggle between Chat / Query; the list below then shows either
|
| 32 |
-
* Chats or Queries."*
|
| 33 |
-
*
|
| 34 |
-
* The toggle changes WHICH LIST the left panel shows, never which route you are on: `#/assistant`
|
| 35 |
-
* serves both, and `#/query` redirects here carrying `?mode=query`.
|
| 36 |
-
*/
|
| 37 |
-
export type AssistantMode = "chat" | "query";
|
| 38 |
-
const MODE_KEY = "aios-assistant-mode";
|
| 39 |
-
const isMode = (value: unknown): value is AssistantMode => value === "chat" || value === "query";
|
| 40 |
-
|
| 41 |
-
/**
|
| 42 |
-
* The hash wins on arrival, the stored choice answers a reload.
|
| 43 |
-
*
|
| 44 |
-
* `?mode=query` is the REDIRECT'S INSTRUCTION β somebody followed a `#/query` link or bookmark β
|
| 45 |
-
* and a preference stored days ago must not override the link just clicked. Everything else reads
|
| 46 |
-
* `localStorage`, which is what makes the choice survive a reload.
|
| 47 |
-
*
|
| 48 |
-
* β THE TOGGLE STILL DOES NOT WRITE THE HASH, AND THE REASON HAS CHANGED β corrected here rather
|
| 49 |
-
* than left, because a stale comment that later readers quote is this wave's own worst defect.
|
| 50 |
-
* It USED to be a hard constraint: `Shell.tsx::read` took the whole remainder of the hash as the
|
| 51 |
-
* route key, so writing `#/assistant?mode=query` resolved to no route and landed on Home. B's
|
| 52 |
-
* `routeKeyOf` splits on `?` now (NOTE B-8), so writing it would be safe. It is still not written,
|
| 53 |
-
* for a smaller reason: which LIST a panel shows is a preference, not a location, and the hash is
|
| 54 |
-
* B's to own. The arrival parameter is consumed once, below, and stored.
|
| 55 |
-
* Pure and exported so the decision can be checked without a browser.
|
| 56 |
-
*/
|
| 57 |
-
export function arrivalMode(hash: string): AssistantMode | null {
|
| 58 |
-
const mark = hash.indexOf("?");
|
| 59 |
-
if (mark < 0) return null;
|
| 60 |
-
const asked = new URLSearchParams(hash.slice(mark + 1)).get("mode");
|
| 61 |
-
return isMode(asked) ? asked : null;
|
| 62 |
-
}
|
| 63 |
-
|
| 64 |
-
export function initialMode(hash: string, stored: string | null): AssistantMode {
|
| 65 |
-
const asked = arrivalMode(hash);
|
| 66 |
-
if (asked) return asked;
|
| 67 |
-
return isMode(stored) ? stored : "chat";
|
| 68 |
-
}
|
| 69 |
-
|
| 70 |
-
/** A private window refuses storage; that is a lost preference, never an error the reader sees. */
|
| 71 |
-
function readMode(): AssistantMode {
|
| 72 |
-
if (typeof window === "undefined") return "chat";
|
| 73 |
-
let stored: string | null = null;
|
| 74 |
-
try { stored = window.localStorage.getItem(MODE_KEY); } catch { stored = null; }
|
| 75 |
-
return initialMode(window.location.hash, stored);
|
| 76 |
-
}
|
| 77 |
-
|
| 78 |
-
function storeMode(mode: AssistantMode): void {
|
| 79 |
-
if (typeof window === "undefined") return;
|
| 80 |
-
try { window.localStorage.setItem(MODE_KEY, mode); } catch { /* see readMode */ }
|
| 81 |
-
}
|
| 82 |
-
|
| 83 |
-
const KIND_LABEL: Record<string, string> = {
|
| 84 |
-
grid: "table", list: "list", chart: "chart", kanban: "board", calendar: "calendar",
|
| 85 |
-
timeseries: "time series", map: "map",
|
| 86 |
-
};
|
| 87 |
-
|
| 88 |
-
/**
|
| 89 |
-
* ββ THE MODEL NAMES THE OWNER PICKS FROM (owner item 2, 2026-08-15). Verbatim: *"the user can
|
| 90 |
-
* choose which model to use since we have a selections of them."*
|
| 91 |
-
*
|
| 92 |
-
* β THE WIRE VALUE IS THE PROVIDER KEY AND MUST STAY THE PROVIDER KEY. `routes_query.model_choices`
|
| 93 |
-
* derives the list from `QUERY_PROVIDER_ORDER`, and `_call_model` looks the choice up in
|
| 94 |
-
* `harness.analyst.PROVIDERS` by that exact name. This map is a DISPLAY name only, applied to
|
| 95 |
-
* whatever the server sent β an unknown key falls through to itself rather than being hidden, so a
|
| 96 |
-
* provider added server-side appears here the day it is added instead of the day someone
|
| 97 |
-
* remembers to edit this file [[a-flag-can-ship-without-its-writer]].
|
| 98 |
-
*
|
| 99 |
-
* β `anthropic: "Claude"` AND `openai: "OpenAI"` ARE GONE, and the reason is worth keeping because
|
| 100 |
-
* the ticket that removed them described the defect backwards. They were never OFFERED: the picker
|
| 101 |
-
* renders `index.models`, which is `model_choices()` = `[auto, *QUERY_PROVIDER_ORDER]`, so no
|
| 102 |
-
* reader ever saw "Claude" in this control. They were labels for providers that do not exist in
|
| 103 |
-
* the platform ladder, i.e. dead weight, and the fall-through above already covers a provider
|
| 104 |
-
* added server-side. The REAL defect the ticket named lives one layer down and is fixed by
|
| 105 |
-
* `modelStatus`: a model that IS offered and whose key this deployment does not hold.
|
| 106 |
-
*/
|
| 107 |
-
const MODEL_LABEL: Record<string, string> = {
|
| 108 |
-
// β `anthropic` JOINS THE MAP (W36 Β· ASK E-11, ruling R4). Not a fifth option nobody wanted.
|
| 109 |
-
// E measured all three side rungs refusing live today (cerebras 402, groq 404, openrouter 402),
|
| 110 |
-
// so Anthropic is the only rung that answers at all, and C5 makes it the tool-calling path by
|
| 111 |
-
// ruling rather than by accident. The fall-through below still covers a provider added
|
| 112 |
-
// server-side; this row exists so the ONE rung that works is not the one rendered as a raw key.
|
| 113 |
-
// β NO `word` FOLLOWED BY A COLON IN THIS COMMENT, and that is not fussiness. `verify_query`
|
| 114 |
-
// reads this map with a regex over the block, comments included, so a phrase like "asked for"
|
| 115 |
-
// with a colon after it registered as a SIXTH key named `for` and reddened the check that
|
| 116 |
-
// exists to keep this map equal to the server's list [[prose-that-becomes-its-own-marker]].
|
| 117 |
-
auto: "Auto", anthropic: "Anthropic", cerebras: "Cerebras", groq: "Groq",
|
| 118 |
-
openrouter: "OpenRouter",
|
| 119 |
-
};
|
| 120 |
-
const modelLabel = (key: string) => MODEL_LABEL[key] ?? key;
|
| 121 |
-
|
| 122 |
-
/**
|
| 123 |
-
* Route to the exact Query-owned artefact. C owns the destination wiring, not this page.
|
| 124 |
-
*
|
| 125 |
-
* β `hosted` is R14's one change and it is a NAVIGATION suppressor, not a second path: when this
|
| 126 |
-
* surface is itself showing Query, there is nowhere to route TO β the workspace is already mounted
|
| 127 |
-
* in the main column β and setting the hash would send the reader to the standalone route we just
|
| 128 |
-
* merged away from. The qid contract is unchanged either way, which is the point: the same public
|
| 129 |
-
* hand-off answers an external link and an in-page click, so Query keeps ONE selection door.
|
| 130 |
-
* β The retry ladder still earns its keep here: the hosted workspace is lazy, so the event can
|
| 131 |
-
* fire before its chunk has finished loading and its listener exists.
|
| 132 |
-
*/
|
| 133 |
-
export function openBuiltView(qid: string, hosted = false): () => void {
|
| 134 |
-
if (typeof window === "undefined") return () => {};
|
| 135 |
-
// β `routeKeyOf`, NOT `hash.replace(...)` (B's NOTE B-14). Since T15 the hash is
|
| 136 |
-
// `#/assistant?mode=query`, so the whole-remainder idiom yields the key `assistant?mode=query`,
|
| 137 |
-
// which is never `query` β the guard was permanently true and its intent ("do not navigate, we
|
| 138 |
-
// are already there") had become unsatisfiable by any hash. Latent rather than live, because the
|
| 139 |
-
// one caller passes `hosted`, and fixed rather than left because a guard that cannot be
|
| 140 |
-
// satisfied is the kind a later reader trusts.
|
| 141 |
-
if (!hosted && routeKeyOf(window.location.hash) !== QUERY_ROUTE) {
|
| 142 |
-
window.location.hash = `#/${QUERY_ROUTE}`;
|
| 143 |
-
}
|
| 144 |
-
return retryEmit(() => signal(QUERY_OPEN_EVENT, { qid }));
|
| 145 |
-
}
|
| 146 |
-
|
| 147 |
-
/**
|
| 148 |
-
* R9's provenance, as a DISCLOSURE. Every clause the ruling requires is present β sources,
|
| 149 |
-
* fields, filters, aggregation, contributing record count, retrieval time, each a link to the
|
| 150 |
-
* exact artefact β but shut by default, because a chat that prints its own audit trail under
|
| 151 |
-
* every sentence is not the "plain look" the owner asked for and is not readable as an answer.
|
| 152 |
-
*/
|
| 153 |
-
export function CitationLine({ citation }: { citation: QueryCitation }) {
|
| 154 |
-
return <a className="as-citation" href={citation.href}>{queryCitationLabel(citation)}</a>;
|
| 155 |
-
}
|
| 156 |
-
|
| 157 |
-
/** One glyph, drawn once and turned over for the other. R20 asked for thumbs; these are thumbs. */
|
| 158 |
-
function ThumbIcon({ down = false }: { down?: boolean }) {
|
| 159 |
-
return (
|
| 160 |
-
<svg viewBox="0 0 16 16" aria-hidden="true" width="13" height="13"
|
| 161 |
-
{...(down ? { transform: "rotate(180)" } : {})}>
|
| 162 |
-
<path d="M5.6 14.2V6.9h1.5l2.2-4.3a1 1 0 0 1 1.8.9l-.7 2.6h2.4a1 1 0 0 1 1 1.3l-1.1 4.6a1.5 1.5 0 0 1-1.5 1.1H5.6z" />
|
| 163 |
-
<path d="M5.6 6.9H2.6v7.3h3" />
|
| 164 |
-
</svg>
|
| 165 |
-
);
|
| 166 |
-
}
|
| 167 |
-
|
| 168 |
-
export function AssistantMessage({ message, view, citations, onPreview, onRate }: {
|
| 169 |
-
message: QueryMessage; view?: SavedQuery; citations: QueryCitation[]; onPreview: (id: string) => void;
|
| 170 |
-
/** Optional so the message can be rendered outside a live conversation without a rating door. */
|
| 171 |
-
onRate?: (id: string, rating: "up" | "down" | null, reason?: string) => void;
|
| 172 |
-
}) {
|
| 173 |
-
const [openSources, setOpenSources] = useState(false);
|
| 174 |
-
const [askWhy, setAskWhy] = useState(false);
|
| 175 |
-
const [why, setWhy] = useState("");
|
| 176 |
-
const matching = citations.filter((citation) => (message.citationIds || []).includes(citation.id));
|
| 177 |
-
if (message.role === "user") {
|
| 178 |
-
return (
|
| 179 |
-
<article className="as-message as-message--user">
|
| 180 |
-
<p className="as-bubble">{message.content}</p>
|
| 181 |
-
</article>
|
| 182 |
-
);
|
| 183 |
-
}
|
| 184 |
-
const numeric = message.numeric;
|
| 185 |
-
return (
|
| 186 |
-
<article className={"as-message as-message--assistant" + (message.reason ? " is-refusal" : "")}>
|
| 187 |
-
<p className="as-message-text">{message.content}</p>
|
| 188 |
-
{numeric && numeric.value !== null && numeric.value !== undefined ? (
|
| 189 |
-
<p className="as-numeric">
|
| 190 |
-
<span className="as-numeric-label">{numeric.label}</span>
|
| 191 |
-
<strong className="as-numeric-value">{numeric.value.toLocaleString()}</strong>
|
| 192 |
-
</p>
|
| 193 |
-
) : null}
|
| 194 |
-
{view ? (
|
| 195 |
-
<button type="button" className="as-preview" onClick={() => onPreview(view.id)}>
|
| 196 |
-
<span className="as-preview-name">{view.name}</span>
|
| 197 |
-
{/* β ONE expression, not three text nodes. `Open the {x} in Query` renders under SSR as
|
| 198 |
-
`Open the<!-- -->table<!-- --> in Query`, so a check reading the sentence back sees
|
| 199 |
-
a string nobody wrote [[ui-invisible-to-assertions]]. */}
|
| 200 |
-
<span className="as-preview-meta">{`Open the ${KIND_LABEL[view.kind] ?? view.kind} in Query`}</span>
|
| 201 |
-
</button>
|
| 202 |
-
) : null}
|
| 203 |
-
{/* β The thumbs are NOT a survey: the server reads a thumbs-down back into the next turn of
|
| 204 |
-
this thread. That is why a down asks for one short reason and why both are clearable. */}
|
| 205 |
-
{onRate ? (
|
| 206 |
-
<div className="as-rate">
|
| 207 |
-
<button type="button" aria-label="Helpful" aria-pressed={message.rating === "up"}
|
| 208 |
-
className={"as-rate-btn" + (message.rating === "up" ? " is-on" : "")}
|
| 209 |
-
onClick={() => { setAskWhy(false); onRate(message.id, message.rating === "up" ? null : "up"); }}>
|
| 210 |
-
<ThumbIcon />
|
| 211 |
-
</button>
|
| 212 |
-
<button type="button" aria-label="Not helpful" aria-pressed={message.rating === "down"}
|
| 213 |
-
className={"as-rate-btn" + (message.rating === "down" ? " is-on" : "")}
|
| 214 |
-
onClick={() => {
|
| 215 |
-
if (message.rating === "down") { setAskWhy(false); onRate(message.id, null); return; }
|
| 216 |
-
setAskWhy(true); onRate(message.id, "down");
|
| 217 |
-
}}>
|
| 218 |
-
<ThumbIcon down />
|
| 219 |
-
</button>
|
| 220 |
-
{message.rating === "down" && !askWhy && message.ratingReason ? (
|
| 221 |
-
<span className="as-rate-why-said">{message.ratingReason}</span>
|
| 222 |
-
) : null}
|
| 223 |
-
</div>
|
| 224 |
-
) : null}
|
| 225 |
-
{onRate && askWhy && message.rating === "down" ? (
|
| 226 |
-
<form className="as-rate-why" onSubmit={(event) => {
|
| 227 |
-
event.preventDefault(); setAskWhy(false); onRate(message.id, "down", why.trim());
|
| 228 |
-
}}>
|
| 229 |
-
<input className="as-rate-input" value={why} aria-label="What was wrong"
|
| 230 |
-
placeholder="What was wrong? Optional."
|
| 231 |
-
onChange={(event) => setWhy(event.currentTarget.value)} />
|
| 232 |
-
<button type="submit" className="as-rate-send">Send</button>
|
| 233 |
-
</form>
|
| 234 |
-
) : null}
|
| 235 |
-
{matching.length ? (
|
| 236 |
-
<div className="as-sources">
|
| 237 |
-
<button type="button" className="as-sources-btn" aria-expanded={openSources}
|
| 238 |
-
onClick={() => setOpenSources(!openSources)}>
|
| 239 |
-
{openSources ? "Hide sources" : "Sources"}
|
| 240 |
-
</button>
|
| 241 |
-
{openSources ? (
|
| 242 |
-
<div className="as-sources-list">
|
| 243 |
-
{matching.map((citation) => <CitationLine key={citation.id} citation={citation} />)}
|
| 244 |
-
</div>
|
| 245 |
-
) : null}
|
| 246 |
-
</div>
|
| 247 |
-
) : null}
|
| 248 |
-
</article>
|
| 249 |
-
);
|
| 250 |
-
}
|
| 251 |
-
|
| 252 |
-
function initialIndex(): QueryIndex {
|
| 253 |
-
return { threads: [], messages: [], views: [], citations: [], models: ["auto"],
|
| 254 |
-
modelStatus: [], sources: [] };
|
| 255 |
-
}
|
| 256 |
-
|
| 257 |
-
/**
|
| 258 |
-
* ββ OWNER ITEM 2 (2026-08-15) β A PLAIN CHATBOT, IN THREE BORROWED SHAPES.
|
| 259 |
-
*
|
| 260 |
-
* Verbatim: *"AI assistant module should be a plain chatbot interface where the user can choose
|
| 261 |
-
* which model to use⦠I really like the plain look for Glean⦠Notice how the database you can
|
| 262 |
-
* select is at the bottom there with all the logos, that's how we should allow user to select
|
| 263 |
-
* databases too⦠The interface should exactly be like ChatGPT⦠ALSO I want a navigation for the
|
| 264 |
-
* chat EXACTLY like how Airtable does it."*
|
| 265 |
-
*
|
| 266 |
-
* So the surface is assembled from the three references the owner supplied, each for the part it
|
| 267 |
-
* is actually about:
|
| 268 |
-
* Β· `reference/ChatGPT 1.png` β the CONVERSATION: one centred column, the question as a bubble
|
| 269 |
-
* on the right, the answer as plain flowing text with no label and no bubble, the composer
|
| 270 |
-
* pinned to the bottom with its controls INSIDE the field.
|
| 271 |
-
* Β· `reference/Glean 1.png` β the SOURCES: a row of chips carrying each source's own mark,
|
| 272 |
-
* directly under the composer. Not a dropdown, not a settings panel.
|
| 273 |
-
* Β· `reference/Airtable AI 1.png` β the NAVIGATION: "New chat" at the top of a left panel, the
|
| 274 |
-
* thread list under it, and a centred "How can I help?" when the thread is empty.
|
| 275 |
-
*
|
| 276 |
-
* β WHAT IS DELIBERATELY NOT COPIED: ChatGPT's dark canvas and Glean's vendor logos. The palette
|
| 277 |
-
* is Loopable's own (`DESIGN.md`, and the font-token rule is gated app-wide over every CSS file
|
| 278 |
-
* in this tree), and the chips wear `FolderMark` β the SAME mark the nav rail draws for that
|
| 279 |
-
* database β because a brand logo for someone else's product is what those references have and
|
| 280 |
-
* this product's databases are not brands. "Exactly like" is about the interaction shape.
|
| 281 |
-
*
|
| 282 |
-
* β AND THE TOOL BOUNDARY IS NARROWER THAN THE SENTENCE "the chatbot can basically call the tools
|
| 283 |
-
* in our App". Under ruling R1 the assistant operates the caller's permitted database/view tools
|
| 284 |
-
* and nothing else: server-side there is exactly ONE tool, `build_view`. It cannot run an
|
| 285 |
-
* automation, touch a connector, or change a setting, and no part of this page implies it can.
|
| 286 |
-
*/
|
| 287 |
-
export default function AssistantPage({ granted }: AssistantPageProps) {
|
| 288 |
-
const [index, setIndex] = useState<QueryIndex>(initialIndex);
|
| 289 |
-
const [activeThread, setActiveThread] = useState("");
|
| 290 |
-
const [selected, setSelected] = useState<string[]>([]);
|
| 291 |
-
const [target, setTarget] = useState("");
|
| 292 |
-
const [model, setModel] = useState("auto");
|
| 293 |
-
const [question, setQuestion] = useState("");
|
| 294 |
-
const [busy, setBusy] = useState(false);
|
| 295 |
-
const [problem, setProblem] = useState("");
|
| 296 |
-
const [loaded, setLoaded] = useState(false);
|
| 297 |
-
const [confirmThread, setConfirmThread] = useState("");
|
| 298 |
-
const [mode, setMode] = useState<AssistantMode>(readMode);
|
| 299 |
-
const [activeQuery, setActiveQuery] = useState("");
|
| 300 |
-
/** Bumped on delete so the hosted workspace's own copy of the list cannot go stale behind us. */
|
| 301 |
-
const [queryRefresh, setQueryRefresh] = useState(0);
|
| 302 |
-
const openCancel = useRef<(() => void) | null>(null);
|
| 303 |
-
const foot = useRef<HTMLDivElement | null>(null);
|
| 304 |
-
/** A counter, not a random id: the optimistic message only has to be unique within this page. */
|
| 305 |
-
const pending = useRef(0);
|
| 306 |
-
|
| 307 |
-
const databases = useMemo(() => databaseEntries(granted).filter((entry) => entry.kind !== "group"), [granted]);
|
| 308 |
-
const labels = useMemo(() => new Map(databases.map((entry) => [entry.key, entry.label])), [databases]);
|
| 309 |
-
/**
|
| 310 |
-
* ββ WHICH CHIPS CAN ACTUALLY BE ASKED, and why the ones that cannot say so.
|
| 311 |
-
* The nav grants twelve databases on tenant #0 and the read boundary can answer for two: the
|
| 312 |
-
* ten `ut_odoo_*` grids are served through the connector mirror, which the Assistant refuses by
|
| 313 |
-
* design. Offering ten doors that cannot open β and letting the reader discover it one spent
|
| 314 |
-
* prompt at a time β is the same defect shape as a rail that paints two rows differently from
|
| 315 |
-
* five. β Absent from the map β TREATED AS ANSWERABLE: the server is the authority and its
|
| 316 |
-
* refusal is still the wall; a client that greyed out everything it had not heard about would
|
| 317 |
-
* hide a working source the moment this field failed to arrive.
|
| 318 |
-
*/
|
| 319 |
-
const sourceStatus = useMemo(
|
| 320 |
-
() => new Map(index.sources.map((row) => [row.database, row])), [index.sources]);
|
| 321 |
-
const blockedReason = useCallback(
|
| 322 |
-
(key: string) => (sourceStatus.get(key)?.answerable === false
|
| 323 |
-
? sourceStatus.get(key)?.reason || "this database cannot be asked yet" : ""),
|
| 324 |
-
[sourceStatus]);
|
| 325 |
-
const messages = index.messages.filter((row) => row.threadId === activeThread);
|
| 326 |
-
const viewById = useMemo(() => new Map(index.views.map((view) => [view.id, view])), [index.views]);
|
| 327 |
-
|
| 328 |
-
/** Offered is not callable: a model whose key this deployment does not hold says so, and why. */
|
| 329 |
-
const modelBlocked = useCallback((key: string) => {
|
| 330 |
-
const row = index.modelStatus.find((item) => item.model === key);
|
| 331 |
-
return row && row.available === false
|
| 332 |
-
? row.reason || "this model is not available on this deployment" : "";
|
| 333 |
-
}, [index.modelStatus]);
|
| 334 |
-
|
| 335 |
-
const reload = useCallback(async () => {
|
| 336 |
-
const result = await fetchQueries();
|
| 337 |
-
setLoaded(true);
|
| 338 |
-
if (result.ok) {
|
| 339 |
-
setIndex(result.value);
|
| 340 |
-
// β Falls back to Auto when the SELECTED model is offered but not callable, not only when it
|
| 341 |
-
// has left the list: a stored thread can restore a model whose key was removed since, and a
|
| 342 |
-
// disabled option the picker cannot clear is a control that refuses its own value.
|
| 343 |
-
const callable = (key: string) => result.value.models.includes(key)
|
| 344 |
-
&& result.value.modelStatus.every((row) => row.model !== key || row.available);
|
| 345 |
-
setModel((current) => callable(current) ? current : "auto");
|
| 346 |
-
}
|
| 347 |
-
}, []);
|
| 348 |
-
|
| 349 |
-
useEffect(() => { void reload(); }, [reload]);
|
| 350 |
-
useEffect(() => () => openCancel.current?.(), []);
|
| 351 |
-
useEffect(() => {
|
| 352 |
-
const permitted = databases.map((entry) => entry.key);
|
| 353 |
-
setSelected((current) => current.filter((key) => permitted.includes(key)));
|
| 354 |
-
setTarget((current) => permitted.includes(current) ? current : "");
|
| 355 |
-
}, [databases]);
|
| 356 |
-
// The conversation reads bottom-up, like every chat the owner named.
|
| 357 |
-
useEffect(() => { foot.current?.scrollIntoView({ block: "end" }); }, [messages.length, busy]);
|
| 358 |
-
|
| 359 |
-
/**
|
| 360 |
-
* The `?mode=` the redirect carried has been read into state; drop it so a toggle made later is
|
| 361 |
-
* not overridden by a stale parameter on the next reload. `replaceState` does NOT fire
|
| 362 |
-
* `hashchange`, so the shell never re-resolves the route and nothing flickers.
|
| 363 |
-
*
|
| 364 |
-
* β AND IT IS STORED IN THE SAME BREATH, WHICH IS THE HALF THAT WAS MISSING. `useState(readMode)`
|
| 365 |
-
* reads the hash and writes NOTHING; only `pickMode` persists. So a reader who arrived in Query
|
| 366 |
-
* by following a `#/query` link, never touching the toggle, would have the parameter stripped
|
| 367 |
-
* here and then land back in Chat on the next reload, restored from a choice made days earlier.
|
| 368 |
-
* Following the link IS choosing.
|
| 369 |
-
*
|
| 370 |
-
* ββ AND IT LISTENS FOR `hashchange`, WHICH IS THE DEFECT THE OWNER REPORTED (QA 2026-08-17,
|
| 371 |
-
* measured on the deployed build). `#/query` and `#/assistant` resolve to the SAME route, so
|
| 372 |
-
* following a `#/query` link from anywhere inside the running app is a same-document hash
|
| 373 |
-
* change: this component is already mounted, `useState(readMode)` does not run a second time,
|
| 374 |
-
* and the panel stayed on Chat while the address bar read `?mode=query`. Only a full reload
|
| 375 |
-
* ever showed Query. **A mount-time read cannot see an arrival that does not remount** β the
|
| 376 |
-
* listener is the whole fix, and `arrivalMode` is the ONE parser both paths call so a later
|
| 377 |
-
* change cannot make the two disagree.
|
| 378 |
-
*/
|
| 379 |
-
useEffect(() => {
|
| 380 |
-
if (typeof window === "undefined") return;
|
| 381 |
-
const consume = () => {
|
| 382 |
-
const asked = arrivalMode(window.location.hash);
|
| 383 |
-
if (!asked) return;
|
| 384 |
-
setMode(asked);
|
| 385 |
-
storeMode(asked);
|
| 386 |
-
window.history.replaceState(null, "", `#/${ASSISTANT_ROUTE}`);
|
| 387 |
-
};
|
| 388 |
-
consume();
|
| 389 |
-
window.addEventListener("hashchange", consume);
|
| 390 |
-
return () => window.removeEventListener("hashchange", consume);
|
| 391 |
-
}, []);
|
| 392 |
-
|
| 393 |
-
const pickMode = useCallback((next: AssistantMode) => {
|
| 394 |
-
setMode(next); storeMode(next); setProblem("");
|
| 395 |
-
}, []);
|
| 396 |
-
|
| 397 |
-
/** The saved Query views, grouped exactly as the standalone page groups them (ONE function). */
|
| 398 |
-
const queryList = useMemo(() => queryGroups(index.views, databases), [index.views, databases]);
|
| 399 |
-
|
| 400 |
-
const selectQuery = useCallback((qid: string) => {
|
| 401 |
-
setActiveQuery(qid);
|
| 402 |
-
openCancel.current?.();
|
| 403 |
-
openCancel.current = openBuiltView(qid, true);
|
| 404 |
-
}, []);
|
| 405 |
-
|
| 406 |
-
/** A renamed or duplicated artefact, merged in place, and the hosted workspace told to re-read. */
|
| 407 |
-
const upsertQuery = useCallback((view: SavedQuery) => {
|
| 408 |
-
setIndex((current) => ({
|
| 409 |
-
...current, views: [view, ...current.views.filter((row) => row.id !== view.id)],
|
| 410 |
-
}));
|
| 411 |
-
setQueryRefresh((count) => count + 1);
|
| 412 |
-
}, []);
|
| 413 |
-
|
| 414 |
-
const removeQuery = useCallback(async (qid: string) => {
|
| 415 |
-
const result = await deleteQuery(qid);
|
| 416 |
-
if (!result.ok) { setProblem(result.message); return; }
|
| 417 |
-
setIndex((current) => ({ ...current, views: current.views.filter((row) => row.id !== qid) }));
|
| 418 |
-
setActiveQuery((current) => (current === qid ? "" : current));
|
| 419 |
-
setQueryRefresh((count) => count + 1);
|
| 420 |
-
}, []);
|
| 421 |
-
|
| 422 |
-
/** Arriving in Query with nothing chosen selects the newest view of the first database. */
|
| 423 |
-
useEffect(() => {
|
| 424 |
-
if (mode !== "query" || activeQuery) return;
|
| 425 |
-
const first = queryList[0]?.views[0];
|
| 426 |
-
if (first) selectQuery(first.id);
|
| 427 |
-
}, [mode, activeQuery, queryList, selectQuery]);
|
| 428 |
-
|
| 429 |
-
const newChat = useCallback(() => {
|
| 430 |
-
setActiveThread(""); setQuestion(""); setProblem("");
|
| 431 |
-
}, []);
|
| 432 |
-
|
| 433 |
-
/** Opening a past chat restores the sources and the model it ran with (R6). */
|
| 434 |
-
const openThread = useCallback((id: string) => {
|
| 435 |
-
setActiveThread(id); setProblem("");
|
| 436 |
-
const row = index.threads.find((thread) => thread.id === id);
|
| 437 |
-
if (!row) return;
|
| 438 |
-
const permitted = databases.map((entry) => entry.key);
|
| 439 |
-
const sources = (row.sources || []).filter((key) => permitted.includes(key));
|
| 440 |
-
if (sources.length) { setSelected(sources); setTarget(sources[0]); }
|
| 441 |
-
if (row.model && index.models.includes(row.model) && !modelBlocked(row.model)) setModel(row.model);
|
| 442 |
-
}, [databases, index.models, index.threads, modelBlocked]);
|
| 443 |
-
|
| 444 |
-
const toggleSource = useCallback((key: string) => {
|
| 445 |
-
if (blockedReason(key)) return;
|
| 446 |
-
setSelected((current) => {
|
| 447 |
-
const next = current.includes(key) ? current.filter((item) => item !== key) : [...current, key];
|
| 448 |
-
setTarget((currentTarget) => next.includes(currentTarget) ? currentTarget : next[0] || "");
|
| 449 |
-
return next;
|
| 450 |
-
});
|
| 451 |
-
}, [blockedReason]);
|
| 452 |
-
|
| 453 |
-
/**
|
| 454 |
-
* ββ R16 β ENTER GOES TO THE CHAT, NOT TO A WAIT. Owner: *"Pressing enter goes immediately to
|
| 455 |
-
* the chat view showing that it is loading; there is a thinking animation."*
|
| 456 |
-
*
|
| 457 |
-
* This used to POST and only then touch `index`, so for the whole model call the reader sat on
|
| 458 |
-
* the opening screen looking at their own unsent sentence with nothing moving. The message is
|
| 459 |
-
* appended FIRST, which is what flips `empty` and hands the surface to the stream; `busy` then
|
| 460 |
-
* draws the app's one loading mark under it.
|
| 461 |
-
*
|
| 462 |
-
* β THE INVERSE IS CAPTURED BEFORE THE WRITE. An optimistic append that is not reconciled on
|
| 463 |
-
* failure leaves a message on screen the server never received, and the next turn replays it to
|
| 464 |
-
* the model as if it had [[undo-capture-before-the-write]]. The typed text goes back in the
|
| 465 |
-
* composer too: losing somebody's sentence to a failed request is the second injury.
|
| 466 |
-
*/
|
| 467 |
-
const ask = useCallback(async () => {
|
| 468 |
-
const text = question.trim();
|
| 469 |
-
if (!text || !target || !selected.includes(target) || busy) return;
|
| 470 |
-
const pendingId = `pending_${(pending.current += 1)}`;
|
| 471 |
-
const optimistic: QueryMessage = {
|
| 472 |
-
id: pendingId, threadId: activeThread, role: "user", content: text,
|
| 473 |
-
createdAt: new Date().toISOString(), targetDatabase: target,
|
| 474 |
-
sources: selected, requestedModel: model,
|
| 475 |
-
};
|
| 476 |
-
const withoutPending = (rows: QueryMessage[]) => rows.filter((row) => row.id !== pendingId);
|
| 477 |
-
setBusy(true); setProblem(""); setQuestion("");
|
| 478 |
-
setIndex((current) => ({ ...current, messages: [...current.messages, optimistic] }));
|
| 479 |
-
const result = await submitChat({ question: text, database: target, sources: selected,
|
| 480 |
-
...(activeThread ? { threadId: activeThread } : {}), model });
|
| 481 |
-
setBusy(false);
|
| 482 |
-
if (!result.ok) {
|
| 483 |
-
setIndex((current) => ({ ...current, messages: withoutPending(current.messages) }));
|
| 484 |
-
setQuestion(text);
|
| 485 |
-
setProblem(result.message);
|
| 486 |
-
return;
|
| 487 |
-
}
|
| 488 |
-
setActiveThread(result.value.thread.id);
|
| 489 |
-
setIndex((current) => ({
|
| 490 |
-
...current,
|
| 491 |
-
threads: [result.value.thread, ...current.threads.filter((row) => row.id !== result.value.thread.id)],
|
| 492 |
-
// The server's own copy of the question replaces the optimistic one, so a thread never holds
|
| 493 |
-
// two of the same turn and the id the rest of the page keys on is the durable one.
|
| 494 |
-
messages: [...withoutPending(current.messages), result.value.userMessage, result.value.message],
|
| 495 |
-
views: result.value.view ? [result.value.view, ...current.views.filter((row) => row.id !== result.value.view?.id)] : current.views,
|
| 496 |
-
citations: [...current.citations, ...result.value.citations],
|
| 497 |
-
}));
|
| 498 |
-
}, [activeThread, busy, model, question, selected, target]);
|
| 499 |
-
|
| 500 |
-
/** β The chat goes; the views it built STAY (they live in Query). Two-click, in the row. */
|
| 501 |
-
const removeThread = useCallback(async (id: string) => {
|
| 502 |
-
const result = await deleteThread(id);
|
| 503 |
-
if (!result.ok) { setProblem(result.message); return; }
|
| 504 |
-
setIndex((current) => ({
|
| 505 |
-
...current,
|
| 506 |
-
threads: current.threads.filter((row) => row.id !== id),
|
| 507 |
-
messages: current.messages.filter((row) => row.threadId !== id),
|
| 508 |
-
}));
|
| 509 |
-
setActiveThread((current) => (current === id ? "" : current));
|
| 510 |
-
setConfirmThread("");
|
| 511 |
-
}, []);
|
| 512 |
-
|
| 513 |
-
/**
|
| 514 |
-
* The rating goes to the server and the SERVER'S copy comes back, rather than the page patching
|
| 515 |
-
* its own guess: the reason is normalised and cleared there, and a rating that only looked
|
| 516 |
-
* applied is exactly the shape this module keeps being bitten by.
|
| 517 |
-
*/
|
| 518 |
-
const rate = useCallback(async (id: string, rating: "up" | "down" | null, reason?: string) => {
|
| 519 |
-
const result = await rateMessage(id, rating, reason);
|
| 520 |
-
if (!result.ok) { setProblem(result.message); return; }
|
| 521 |
-
setIndex((current) => ({
|
| 522 |
-
...current,
|
| 523 |
-
messages: current.messages.map((row) => (row.id === id ? result.value : row)),
|
| 524 |
-
}));
|
| 525 |
-
}, []);
|
| 526 |
-
|
| 527 |
-
/** A built view opens WHERE THE READER ALREADY IS: the toggle flips, the list follows. */
|
| 528 |
-
const preview = useCallback((qid: string) => {
|
| 529 |
-
pickMode("query");
|
| 530 |
-
selectQuery(qid);
|
| 531 |
-
}, [pickMode, selectQuery]);
|
| 532 |
-
|
| 533 |
-
const empty = messages.length === 0;
|
| 534 |
-
const composer = (
|
| 535 |
-
<div className="as-composer">
|
| 536 |
-
<div className="as-field">
|
| 537 |
-
<textarea
|
| 538 |
-
className="as-prompt"
|
| 539 |
-
value={question}
|
| 540 |
-
rows={1}
|
| 541 |
-
placeholder={target ? `Ask about ${labels.get(target) || target}β¦` : "Pick a database below, then askβ¦"}
|
| 542 |
-
aria-label="Ask the assistant"
|
| 543 |
-
onChange={(event) => setQuestion(event.currentTarget.value)}
|
| 544 |
-
onKeyDown={(event) => {
|
| 545 |
-
if (event.key === "Enter" && !event.shiftKey) { event.preventDefault(); void ask(); }
|
| 546 |
-
}}
|
| 547 |
-
/>
|
| 548 |
-
<div className="as-field-foot">
|
| 549 |
-
{/* ChatGPT puts its model control inside the field, as text rather than a boxed
|
| 550 |
-
form control. A native `select` keeps the keyboard and screen-reader behaviour
|
| 551 |
-
a hand-rolled popover would have to re-earn. */}
|
| 552 |
-
<label className="as-pick">
|
| 553 |
-
<span className="as-pick-label">Model</span>
|
| 554 |
-
<select className="as-pick-select" value={model} aria-label="Model"
|
| 555 |
-
onChange={(event) => setModel(event.currentTarget.value)}>
|
| 556 |
-
{index.models.map((choice) => {
|
| 557 |
-
const blocked = modelBlocked(choice);
|
| 558 |
-
return (
|
| 559 |
-
<option key={choice} value={choice} disabled={!!blocked}
|
| 560 |
-
title={blocked || undefined}>
|
| 561 |
-
{blocked ? `${modelLabel(choice)} (not configured)` : modelLabel(choice)}
|
| 562 |
-
</option>
|
| 563 |
-
);
|
| 564 |
-
})}
|
| 565 |
-
</select>
|
| 566 |
-
</label>
|
| 567 |
-
{/* β ONLY WHEN THE ANSWER IS AMBIGUOUS. Every generated view names ONE database (R3);
|
| 568 |
-
with a single source selected that database is not a question, so asking it would
|
| 569 |
-
be chrome for its own sake. */}
|
| 570 |
-
{selected.length > 1 ? (
|
| 571 |
-
<label className="as-pick">
|
| 572 |
-
<span className="as-pick-label">Build on</span>
|
| 573 |
-
<select className="as-pick-select" value={target} aria-label="Target database"
|
| 574 |
-
onChange={(event) => setTarget(event.currentTarget.value)}>
|
| 575 |
-
{selected.map((key) => <option key={key} value={key}>{labels.get(key) || key}</option>)}
|
| 576 |
-
</select>
|
| 577 |
-
</label>
|
| 578 |
-
) : null}
|
| 579 |
-
<button type="button" className="as-send" aria-label="Send"
|
| 580 |
-
disabled={busy || !question.trim() || !target} onClick={() => void ask()}>
|
| 581 |
-
<svg viewBox="0 0 16 16" aria-hidden="true" width="15" height="15">
|
| 582 |
-
<path d="M8 13V3.6M4 7.4 8 3.4l4 4" />
|
| 583 |
-
</svg>
|
| 584 |
-
</button>
|
| 585 |
-
</div>
|
| 586 |
-
</div>
|
| 587 |
-
{/* Glean's row: the sources, with their marks, under the field. */}
|
| 588 |
-
<div className="as-chips" role="group" aria-label="Databases the assistant may read">
|
| 589 |
-
{databases.map((database) => {
|
| 590 |
-
const on = selected.includes(database.key);
|
| 591 |
-
const blocked = blockedReason(database.key);
|
| 592 |
-
return (
|
| 593 |
-
<button type="button" key={database.key}
|
| 594 |
-
className={"as-chip" + (on ? " is-selected" : "")
|
| 595 |
-
+ (database.key === target ? " is-target" : "") + (blocked ? " is-blocked" : "")}
|
| 596 |
-
aria-pressed={on}
|
| 597 |
-
aria-disabled={blocked ? true : undefined}
|
| 598 |
-
title={blocked || undefined}
|
| 599 |
-
onClick={() => toggleSource(database.key)}>
|
| 600 |
-
<span className="as-chip-mark" aria-hidden="true">
|
| 601 |
-
{database.icon ? <FolderMark icon={database.icon} size={14} /> : <DbIcon />}
|
| 602 |
-
</span>
|
| 603 |
-
<span className="as-chip-name">{database.label}</span>
|
| 604 |
-
</button>
|
| 605 |
-
);
|
| 606 |
-
})}
|
| 607 |
-
{/* β ONE sentence for the whole row, not one per chip: with ten blocked sources the
|
| 608 |
-
per-chip copy would be the loudest thing on a surface whose whole instruction was
|
| 609 |
-
"plain". The chip carries the exact cause in its `title`. */}
|
| 610 |
-
{databases.some((database) => blockedReason(database.key)) ? (
|
| 611 |
-
<p className="as-chips-note">
|
| 612 |
-
Greyed databases are served live from a connector and cannot be asked yet.
|
| 613 |
-
</p>
|
| 614 |
-
) : null}
|
| 615 |
-
{databases.length === 0 && loaded ? (
|
| 616 |
-
<p className="as-chips-empty">No databases are shared with your account yet.</p>
|
| 617 |
-
) : null}
|
| 618 |
-
</div>
|
| 619 |
-
</div>
|
| 620 |
-
);
|
| 621 |
-
|
| 622 |
-
// β `aria-label` is "Assistant", not "AI assistant" (W34-T13, corrected at QA 2026-08-17).
|
| 623 |
-
// An aria-label IS a label β it is what a screen reader announces this region as β so the rail
|
| 624 |
-
// reading Assistant while this said AI assistant left the rename half-done in the one place
|
| 625 |
-
// sighted QA never looks.
|
| 626 |
-
return (
|
| 627 |
-
<div className="as-chat" aria-label="Assistant">
|
| 628 |
-
{/* Airtable's panel: one door to a new chat, then the history. */}
|
| 629 |
-
<aside className="as-history">
|
| 630 |
-
{/* C2: two values, directly ABOVE "+ New chat". It changes which LIST shows, not the route. */}
|
| 631 |
-
<div className="as-mode-toggle" role="tablist" aria-label="Chat or Query">
|
| 632 |
-
{(["chat", "query"] as const).map((value) => (
|
| 633 |
-
<button type="button" key={value} role="tab" aria-selected={mode === value}
|
| 634 |
-
className={"as-mode" + (mode === value ? " is-on" : "")}
|
| 635 |
-
onClick={() => pickMode(value)}>
|
| 636 |
-
{value === "chat" ? "Chat" : "Query"}
|
| 637 |
-
</button>
|
| 638 |
-
))}
|
| 639 |
-
</div>
|
| 640 |
-
{mode === "query" ? (
|
| 641 |
-
/* The SAME rail the standalone page renders; only its host changed, so the differences
|
| 642 |
-
are CSS in assistant.css and nothing here. */
|
| 643 |
-
<QueryRail groups={queryList} activeId={activeQuery} onSelect={selectQuery}
|
| 644 |
-
onDelete={(id) => void removeQuery(id)} onChanged={upsertQuery} />
|
| 645 |
-
) : (
|
| 646 |
-
<>
|
| 647 |
-
<button type="button" className="as-new" onClick={newChat}>
|
| 648 |
-
<svg viewBox="0 0 16 16" aria-hidden="true" width="14" height="14">
|
| 649 |
-
<path d="M8 3.5v9M3.5 8h9" />
|
| 650 |
-
</svg>
|
| 651 |
-
New chat
|
| 652 |
-
</button>
|
| 653 |
-
<nav className="as-thread-list" aria-label="Chat history">
|
| 654 |
-
{index.threads.map((row) => (
|
| 655 |
-
<div className={"as-thread-row" + (row.id === activeThread ? " is-active" : "")} key={row.id}>
|
| 656 |
-
<button type="button" className="as-thread"
|
| 657 |
-
onClick={() => { setConfirmThread(""); openThread(row.id); }}
|
| 658 |
-
title={row.title}>{row.title}</button>
|
| 659 |
-
{confirmThread === row.id ? (
|
| 660 |
-
<button type="button" className="as-thread-confirm"
|
| 661 |
-
onClick={() => void removeThread(row.id)}>Delete</button>
|
| 662 |
-
) : (
|
| 663 |
-
<button type="button" className="as-thread-del" aria-label={`Delete chat: ${row.title}`}
|
| 664 |
-
onClick={() => setConfirmThread(row.id)}>
|
| 665 |
-
<svg viewBox="0 0 16 16" aria-hidden="true" width="13" height="13">
|
| 666 |
-
<path d="M3.5 4.5h9M6.5 4.5V3.2h3v1.3M5 4.5l.6 8h4.8l.6-8" />
|
| 667 |
-
</svg>
|
| 668 |
-
</button>
|
| 669 |
-
)}
|
| 670 |
-
</div>
|
| 671 |
-
))}
|
| 672 |
-
{loaded && index.threads.length === 0 ? (
|
| 673 |
-
<p className="as-thread-empty">No chats yet.</p>
|
| 674 |
-
) : null}
|
| 675 |
-
</nav>
|
| 676 |
-
</>
|
| 677 |
-
)}
|
| 678 |
-
</aside>
|
| 679 |
-
<main className="as-conversation">
|
| 680 |
-
{mode === "query" ? (
|
| 681 |
-
queryList.length ? (
|
| 682 |
-
<Suspense fallback={<p className="as-working" role="status">
|
| 683 |
-
<span className="lp-spin" aria-hidden="true" />
|
| 684 |
-
</p>}>
|
| 685 |
-
<QueryWorkspace granted={granted} hostedRail refreshToken={queryRefresh}
|
| 686 |
-
selectedId={activeQuery} />
|
| 687 |
-
</Suspense>
|
| 688 |
-
) : !loaded ? (
|
| 689 |
-
/* β W35-T22, the SECOND state that paints something untrue for a frame. `queryList` is
|
| 690 |
-
empty until `GET /query` lands, so arriving here β a `#/query` link, a bookmark, a
|
| 691 |
-
reload β asserted "No Query views yet" to somebody who has twenty, then replaced it.
|
| 692 |
-
`loaded` is the flag the chat history one screen down already waits on; the two
|
| 693 |
-
empty states now agree about when they are entitled to speak. */
|
| 694 |
-
<p className="as-working" role="status" aria-label="Loading your Query views">
|
| 695 |
-
<span className="lp-spin" aria-hidden="true" />
|
| 696 |
-
</p>
|
| 697 |
-
) : (
|
| 698 |
-
<div className="as-opening">
|
| 699 |
-
<h1 className="as-opening-h">No Query views yet</h1>
|
| 700 |
-
<p className="as-opening-p">
|
| 701 |
-
Ask a question in Chat. When the assistant builds a view, it is saved here.
|
| 702 |
-
</p>
|
| 703 |
-
</div>
|
| 704 |
-
)
|
| 705 |
-
) : empty ? (
|
| 706 |
-
<div className="as-opening">
|
| 707 |
-
<h1 className="as-opening-h">How can I help?</h1>
|
| 708 |
-
<p className="as-opening-p">
|
| 709 |
-
Ask about a database you can already open. The answer arrives as a view in Query.
|
| 710 |
-
</p>
|
| 711 |
-
{composer}
|
| 712 |
-
{problem ? <p className="as-problem" role="status">{problem}</p> : null}
|
| 713 |
-
</div>
|
| 714 |
-
) : (
|
| 715 |
-
<>
|
| 716 |
-
<div className="as-stream">
|
| 717 |
-
<div className="as-column">
|
| 718 |
-
{messages.map((message) => (
|
| 719 |
-
<AssistantMessage key={message.id} message={message}
|
| 720 |
-
view={message.viewId ? viewById.get(message.viewId) : undefined}
|
| 721 |
-
citations={index.citations} onPreview={preview}
|
| 722 |
-
onRate={message.role === "assistant" && !message.id.startsWith("pending_")
|
| 723 |
-
? rate : undefined} />
|
| 724 |
-
))}
|
| 725 |
-
{/* R16's thinking animation is the app's ONE loading mark, not a second spinner
|
| 726 |
-
vocabulary: `.lp-spin` is the same mark every surface uses and C is repainting
|
| 727 |
-
it in the brand purple this wave. The label is for screen readers only, since
|
| 728 |
-
an aria-hidden spinner inside a status region announces nothing. */}
|
| 729 |
-
{busy ? (
|
| 730 |
-
<p className="as-working" role="status" aria-label="The assistant is thinking">
|
| 731 |
-
<span className="lp-spin" aria-hidden="true" />
|
| 732 |
-
</p>
|
| 733 |
-
) : null}
|
| 734 |
-
<div ref={foot} />
|
| 735 |
-
</div>
|
| 736 |
-
</div>
|
| 737 |
-
<div className="as-dock">
|
| 738 |
-
<div className="as-column">
|
| 739 |
-
{problem ? <p className="as-problem" role="status">{problem}</p> : null}
|
| 740 |
-
{composer}
|
| 741 |
-
</div>
|
| 742 |
-
</div>
|
| 743 |
-
</>
|
| 744 |
-
)}
|
| 745 |
-
</main>
|
| 746 |
-
</div>
|
| 747 |
-
);
|
| 748 |
-
}
|
|
|
|
| 1 |
+
import { lazy, Suspense, useCallback, useEffect, useMemo, useRef, useState } from "react";
|
| 2 |
+
|
| 3 |
+
import { QUERY_OPEN_EVENT, signal } from "../apiContract";
|
| 4 |
+
import { FolderMark } from "../customer-grid/icons";
|
| 5 |
+
import { queryCitationLabel } from "../customer-grid/queryPreview";
|
| 6 |
+
import { retryEmit } from "../inbox/inboxModel";
|
| 7 |
+
import { queryGroups, QueryRail } from "../query/queryParts";
|
| 8 |
+
import { DbIcon } from "../shell/dbFrame";
|
| 9 |
+
import { ASSISTANT_ROUTE, databaseEntries, QUERY_ROUTE, routeKeyOf } from "../shell/nav";
|
| 10 |
+
import type { NavEntry } from "../shell/nav";
|
| 11 |
+
import {
|
| 12 |
+
deleteQuery, deleteThread, fetchQueries, rateMessage, submitChat, type QueryCitation,
|
| 13 |
+
type QueryIndex, type QueryMessage, type SavedQuery,
|
| 14 |
+
} from "../query/queryApi";
|
| 15 |
+
import "./assistant.css";
|
| 16 |
+
|
| 17 |
+
/**
|
| 18 |
+
* β LAZY ON PURPOSE. `QueryPage` imports `CustomerGrid`, the largest module in the client; a
|
| 19 |
+
* static import here would put the whole grid in the chat bundle, so opening the Assistant to ask
|
| 20 |
+
* one question would download the surface it is merged WITH but not showing. The list itself
|
| 21 |
+
* comes from `queryParts`, which imports no grid, so the left panel renders in Query mode before
|
| 22 |
+
* this chunk arrives.
|
| 23 |
+
*/
|
| 24 |
+
const QueryWorkspace = lazy(() => import("../query/QueryPage"));
|
| 25 |
+
|
| 26 |
+
export interface AssistantPageProps { granted: NavEntry[]; }
|
| 27 |
+
|
| 28 |
+
/**
|
| 29 |
+
* ββ R14 / CONTRACT C2 β ONE MODULE, ONE ROUTE, TWO LISTS. Owner, 2026-08-16: *"Combine the AI
|
| 30 |
+
* assistant module AND the Query module. In the secondary navigation at the top, just above
|
| 31 |
+
* '+ New chat', an easily accessible toggle between Chat / Query; the list below then shows either
|
| 32 |
+
* Chats or Queries."*
|
| 33 |
+
*
|
| 34 |
+
* The toggle changes WHICH LIST the left panel shows, never which route you are on: `#/assistant`
|
| 35 |
+
* serves both, and `#/query` redirects here carrying `?mode=query`.
|
| 36 |
+
*/
|
| 37 |
+
export type AssistantMode = "chat" | "query";
|
| 38 |
+
const MODE_KEY = "aios-assistant-mode";
|
| 39 |
+
const isMode = (value: unknown): value is AssistantMode => value === "chat" || value === "query";
|
| 40 |
+
|
| 41 |
+
/**
|
| 42 |
+
* The hash wins on arrival, the stored choice answers a reload.
|
| 43 |
+
*
|
| 44 |
+
* `?mode=query` is the REDIRECT'S INSTRUCTION β somebody followed a `#/query` link or bookmark β
|
| 45 |
+
* and a preference stored days ago must not override the link just clicked. Everything else reads
|
| 46 |
+
* `localStorage`, which is what makes the choice survive a reload.
|
| 47 |
+
*
|
| 48 |
+
* β THE TOGGLE STILL DOES NOT WRITE THE HASH, AND THE REASON HAS CHANGED β corrected here rather
|
| 49 |
+
* than left, because a stale comment that later readers quote is this wave's own worst defect.
|
| 50 |
+
* It USED to be a hard constraint: `Shell.tsx::read` took the whole remainder of the hash as the
|
| 51 |
+
* route key, so writing `#/assistant?mode=query` resolved to no route and landed on Home. B's
|
| 52 |
+
* `routeKeyOf` splits on `?` now (NOTE B-8), so writing it would be safe. It is still not written,
|
| 53 |
+
* for a smaller reason: which LIST a panel shows is a preference, not a location, and the hash is
|
| 54 |
+
* B's to own. The arrival parameter is consumed once, below, and stored.
|
| 55 |
+
* Pure and exported so the decision can be checked without a browser.
|
| 56 |
+
*/
|
| 57 |
+
export function arrivalMode(hash: string): AssistantMode | null {
|
| 58 |
+
const mark = hash.indexOf("?");
|
| 59 |
+
if (mark < 0) return null;
|
| 60 |
+
const asked = new URLSearchParams(hash.slice(mark + 1)).get("mode");
|
| 61 |
+
return isMode(asked) ? asked : null;
|
| 62 |
+
}
|
| 63 |
+
|
| 64 |
+
export function initialMode(hash: string, stored: string | null): AssistantMode {
|
| 65 |
+
const asked = arrivalMode(hash);
|
| 66 |
+
if (asked) return asked;
|
| 67 |
+
return isMode(stored) ? stored : "chat";
|
| 68 |
+
}
|
| 69 |
+
|
| 70 |
+
/** A private window refuses storage; that is a lost preference, never an error the reader sees. */
|
| 71 |
+
function readMode(): AssistantMode {
|
| 72 |
+
if (typeof window === "undefined") return "chat";
|
| 73 |
+
let stored: string | null = null;
|
| 74 |
+
try { stored = window.localStorage.getItem(MODE_KEY); } catch { stored = null; }
|
| 75 |
+
return initialMode(window.location.hash, stored);
|
| 76 |
+
}
|
| 77 |
+
|
| 78 |
+
function storeMode(mode: AssistantMode): void {
|
| 79 |
+
if (typeof window === "undefined") return;
|
| 80 |
+
try { window.localStorage.setItem(MODE_KEY, mode); } catch { /* see readMode */ }
|
| 81 |
+
}
|
| 82 |
+
|
| 83 |
+
const KIND_LABEL: Record<string, string> = {
|
| 84 |
+
grid: "table", list: "list", chart: "chart", kanban: "board", calendar: "calendar",
|
| 85 |
+
timeseries: "time series", map: "map",
|
| 86 |
+
};
|
| 87 |
+
|
| 88 |
+
/**
|
| 89 |
+
* ββ THE MODEL NAMES THE OWNER PICKS FROM (owner item 2, 2026-08-15). Verbatim: *"the user can
|
| 90 |
+
* choose which model to use since we have a selections of them."*
|
| 91 |
+
*
|
| 92 |
+
* β THE WIRE VALUE IS THE PROVIDER KEY AND MUST STAY THE PROVIDER KEY. `routes_query.model_choices`
|
| 93 |
+
* derives the list from `QUERY_PROVIDER_ORDER`, and `_call_model` looks the choice up in
|
| 94 |
+
* `harness.analyst.PROVIDERS` by that exact name. This map is a DISPLAY name only, applied to
|
| 95 |
+
* whatever the server sent β an unknown key falls through to itself rather than being hidden, so a
|
| 96 |
+
* provider added server-side appears here the day it is added instead of the day someone
|
| 97 |
+
* remembers to edit this file [[a-flag-can-ship-without-its-writer]].
|
| 98 |
+
*
|
| 99 |
+
* β `anthropic: "Claude"` AND `openai: "OpenAI"` ARE GONE, and the reason is worth keeping because
|
| 100 |
+
* the ticket that removed them described the defect backwards. They were never OFFERED: the picker
|
| 101 |
+
* renders `index.models`, which is `model_choices()` = `[auto, *QUERY_PROVIDER_ORDER]`, so no
|
| 102 |
+
* reader ever saw "Claude" in this control. They were labels for providers that do not exist in
|
| 103 |
+
* the platform ladder, i.e. dead weight, and the fall-through above already covers a provider
|
| 104 |
+
* added server-side. The REAL defect the ticket named lives one layer down and is fixed by
|
| 105 |
+
* `modelStatus`: a model that IS offered and whose key this deployment does not hold.
|
| 106 |
+
*/
|
| 107 |
+
const MODEL_LABEL: Record<string, string> = {
|
| 108 |
+
// β `anthropic` JOINS THE MAP (W36 Β· ASK E-11, ruling R4). Not a fifth option nobody wanted.
|
| 109 |
+
// E measured all three side rungs refusing live today (cerebras 402, groq 404, openrouter 402),
|
| 110 |
+
// so Anthropic is the only rung that answers at all, and C5 makes it the tool-calling path by
|
| 111 |
+
// ruling rather than by accident. The fall-through below still covers a provider added
|
| 112 |
+
// server-side; this row exists so the ONE rung that works is not the one rendered as a raw key.
|
| 113 |
+
// β NO `word` FOLLOWED BY A COLON IN THIS COMMENT, and that is not fussiness. `verify_query`
|
| 114 |
+
// reads this map with a regex over the block, comments included, so a phrase like "asked for"
|
| 115 |
+
// with a colon after it registered as a SIXTH key named `for` and reddened the check that
|
| 116 |
+
// exists to keep this map equal to the server's list [[prose-that-becomes-its-own-marker]].
|
| 117 |
+
auto: "Auto", anthropic: "Anthropic", cerebras: "Cerebras", groq: "Groq",
|
| 118 |
+
openrouter: "OpenRouter",
|
| 119 |
+
};
|
| 120 |
+
const modelLabel = (key: string) => MODEL_LABEL[key] ?? key;
|
| 121 |
+
|
| 122 |
+
/**
|
| 123 |
+
* Route to the exact Query-owned artefact. C owns the destination wiring, not this page.
|
| 124 |
+
*
|
| 125 |
+
* β `hosted` is R14's one change and it is a NAVIGATION suppressor, not a second path: when this
|
| 126 |
+
* surface is itself showing Query, there is nowhere to route TO β the workspace is already mounted
|
| 127 |
+
* in the main column β and setting the hash would send the reader to the standalone route we just
|
| 128 |
+
* merged away from. The qid contract is unchanged either way, which is the point: the same public
|
| 129 |
+
* hand-off answers an external link and an in-page click, so Query keeps ONE selection door.
|
| 130 |
+
* β The retry ladder still earns its keep here: the hosted workspace is lazy, so the event can
|
| 131 |
+
* fire before its chunk has finished loading and its listener exists.
|
| 132 |
+
*/
|
| 133 |
+
export function openBuiltView(qid: string, hosted = false): () => void {
|
| 134 |
+
if (typeof window === "undefined") return () => {};
|
| 135 |
+
// β `routeKeyOf`, NOT `hash.replace(...)` (B's NOTE B-14). Since T15 the hash is
|
| 136 |
+
// `#/assistant?mode=query`, so the whole-remainder idiom yields the key `assistant?mode=query`,
|
| 137 |
+
// which is never `query` β the guard was permanently true and its intent ("do not navigate, we
|
| 138 |
+
// are already there") had become unsatisfiable by any hash. Latent rather than live, because the
|
| 139 |
+
// one caller passes `hosted`, and fixed rather than left because a guard that cannot be
|
| 140 |
+
// satisfied is the kind a later reader trusts.
|
| 141 |
+
if (!hosted && routeKeyOf(window.location.hash) !== QUERY_ROUTE) {
|
| 142 |
+
window.location.hash = `#/${QUERY_ROUTE}`;
|
| 143 |
+
}
|
| 144 |
+
return retryEmit(() => signal(QUERY_OPEN_EVENT, { qid }));
|
| 145 |
+
}
|
| 146 |
+
|
| 147 |
+
/**
|
| 148 |
+
* R9's provenance, as a DISCLOSURE. Every clause the ruling requires is present β sources,
|
| 149 |
+
* fields, filters, aggregation, contributing record count, retrieval time, each a link to the
|
| 150 |
+
* exact artefact β but shut by default, because a chat that prints its own audit trail under
|
| 151 |
+
* every sentence is not the "plain look" the owner asked for and is not readable as an answer.
|
| 152 |
+
*/
|
| 153 |
+
export function CitationLine({ citation }: { citation: QueryCitation }) {
|
| 154 |
+
return <a className="as-citation" href={citation.href}>{queryCitationLabel(citation)}</a>;
|
| 155 |
+
}
|
| 156 |
+
|
| 157 |
+
/** One glyph, drawn once and turned over for the other. R20 asked for thumbs; these are thumbs. */
|
| 158 |
+
function ThumbIcon({ down = false }: { down?: boolean }) {
|
| 159 |
+
return (
|
| 160 |
+
<svg viewBox="0 0 16 16" aria-hidden="true" width="13" height="13"
|
| 161 |
+
{...(down ? { transform: "rotate(180)" } : {})}>
|
| 162 |
+
<path d="M5.6 14.2V6.9h1.5l2.2-4.3a1 1 0 0 1 1.8.9l-.7 2.6h2.4a1 1 0 0 1 1 1.3l-1.1 4.6a1.5 1.5 0 0 1-1.5 1.1H5.6z" />
|
| 163 |
+
<path d="M5.6 6.9H2.6v7.3h3" />
|
| 164 |
+
</svg>
|
| 165 |
+
);
|
| 166 |
+
}
|
| 167 |
+
|
| 168 |
+
export function AssistantMessage({ message, view, citations, onPreview, onRate }: {
|
| 169 |
+
message: QueryMessage; view?: SavedQuery; citations: QueryCitation[]; onPreview: (id: string) => void;
|
| 170 |
+
/** Optional so the message can be rendered outside a live conversation without a rating door. */
|
| 171 |
+
onRate?: (id: string, rating: "up" | "down" | null, reason?: string) => void;
|
| 172 |
+
}) {
|
| 173 |
+
const [openSources, setOpenSources] = useState(false);
|
| 174 |
+
const [askWhy, setAskWhy] = useState(false);
|
| 175 |
+
const [why, setWhy] = useState("");
|
| 176 |
+
const matching = citations.filter((citation) => (message.citationIds || []).includes(citation.id));
|
| 177 |
+
if (message.role === "user") {
|
| 178 |
+
return (
|
| 179 |
+
<article className="as-message as-message--user">
|
| 180 |
+
<p className="as-bubble">{message.content}</p>
|
| 181 |
+
</article>
|
| 182 |
+
);
|
| 183 |
+
}
|
| 184 |
+
const numeric = message.numeric;
|
| 185 |
+
return (
|
| 186 |
+
<article className={"as-message as-message--assistant" + (message.reason ? " is-refusal" : "")}>
|
| 187 |
+
<p className="as-message-text">{message.content}</p>
|
| 188 |
+
{numeric && numeric.value !== null && numeric.value !== undefined ? (
|
| 189 |
+
<p className="as-numeric">
|
| 190 |
+
<span className="as-numeric-label">{numeric.label}</span>
|
| 191 |
+
<strong className="as-numeric-value">{numeric.value.toLocaleString()}</strong>
|
| 192 |
+
</p>
|
| 193 |
+
) : null}
|
| 194 |
+
{view ? (
|
| 195 |
+
<button type="button" className="as-preview" onClick={() => onPreview(view.id)}>
|
| 196 |
+
<span className="as-preview-name">{view.name}</span>
|
| 197 |
+
{/* β ONE expression, not three text nodes. `Open the {x} in Query` renders under SSR as
|
| 198 |
+
`Open the<!-- -->table<!-- --> in Query`, so a check reading the sentence back sees
|
| 199 |
+
a string nobody wrote [[ui-invisible-to-assertions]]. */}
|
| 200 |
+
<span className="as-preview-meta">{`Open the ${KIND_LABEL[view.kind] ?? view.kind} in Query`}</span>
|
| 201 |
+
</button>
|
| 202 |
+
) : null}
|
| 203 |
+
{/* β The thumbs are NOT a survey: the server reads a thumbs-down back into the next turn of
|
| 204 |
+
this thread. That is why a down asks for one short reason and why both are clearable. */}
|
| 205 |
+
{onRate ? (
|
| 206 |
+
<div className="as-rate">
|
| 207 |
+
<button type="button" aria-label="Helpful" aria-pressed={message.rating === "up"}
|
| 208 |
+
className={"as-rate-btn" + (message.rating === "up" ? " is-on" : "")}
|
| 209 |
+
onClick={() => { setAskWhy(false); onRate(message.id, message.rating === "up" ? null : "up"); }}>
|
| 210 |
+
<ThumbIcon />
|
| 211 |
+
</button>
|
| 212 |
+
<button type="button" aria-label="Not helpful" aria-pressed={message.rating === "down"}
|
| 213 |
+
className={"as-rate-btn" + (message.rating === "down" ? " is-on" : "")}
|
| 214 |
+
onClick={() => {
|
| 215 |
+
if (message.rating === "down") { setAskWhy(false); onRate(message.id, null); return; }
|
| 216 |
+
setAskWhy(true); onRate(message.id, "down");
|
| 217 |
+
}}>
|
| 218 |
+
<ThumbIcon down />
|
| 219 |
+
</button>
|
| 220 |
+
{message.rating === "down" && !askWhy && message.ratingReason ? (
|
| 221 |
+
<span className="as-rate-why-said">{message.ratingReason}</span>
|
| 222 |
+
) : null}
|
| 223 |
+
</div>
|
| 224 |
+
) : null}
|
| 225 |
+
{onRate && askWhy && message.rating === "down" ? (
|
| 226 |
+
<form className="as-rate-why" onSubmit={(event) => {
|
| 227 |
+
event.preventDefault(); setAskWhy(false); onRate(message.id, "down", why.trim());
|
| 228 |
+
}}>
|
| 229 |
+
<input className="as-rate-input" value={why} aria-label="What was wrong"
|
| 230 |
+
placeholder="What was wrong? Optional."
|
| 231 |
+
onChange={(event) => setWhy(event.currentTarget.value)} />
|
| 232 |
+
<button type="submit" className="as-rate-send">Send</button>
|
| 233 |
+
</form>
|
| 234 |
+
) : null}
|
| 235 |
+
{matching.length ? (
|
| 236 |
+
<div className="as-sources">
|
| 237 |
+
<button type="button" className="as-sources-btn" aria-expanded={openSources}
|
| 238 |
+
onClick={() => setOpenSources(!openSources)}>
|
| 239 |
+
{openSources ? "Hide sources" : "Sources"}
|
| 240 |
+
</button>
|
| 241 |
+
{openSources ? (
|
| 242 |
+
<div className="as-sources-list">
|
| 243 |
+
{matching.map((citation) => <CitationLine key={citation.id} citation={citation} />)}
|
| 244 |
+
</div>
|
| 245 |
+
) : null}
|
| 246 |
+
</div>
|
| 247 |
+
) : null}
|
| 248 |
+
</article>
|
| 249 |
+
);
|
| 250 |
+
}
|
| 251 |
+
|
| 252 |
+
function initialIndex(): QueryIndex {
|
| 253 |
+
return { threads: [], messages: [], views: [], citations: [], models: ["auto"],
|
| 254 |
+
modelStatus: [], sources: [] };
|
| 255 |
+
}
|
| 256 |
+
|
| 257 |
+
/**
|
| 258 |
+
* ββ OWNER ITEM 2 (2026-08-15) β A PLAIN CHATBOT, IN THREE BORROWED SHAPES.
|
| 259 |
+
*
|
| 260 |
+
* Verbatim: *"AI assistant module should be a plain chatbot interface where the user can choose
|
| 261 |
+
* which model to use⦠I really like the plain look for Glean⦠Notice how the database you can
|
| 262 |
+
* select is at the bottom there with all the logos, that's how we should allow user to select
|
| 263 |
+
* databases too⦠The interface should exactly be like ChatGPT⦠ALSO I want a navigation for the
|
| 264 |
+
* chat EXACTLY like how Airtable does it."*
|
| 265 |
+
*
|
| 266 |
+
* So the surface is assembled from the three references the owner supplied, each for the part it
|
| 267 |
+
* is actually about:
|
| 268 |
+
* Β· `reference/ChatGPT 1.png` β the CONVERSATION: one centred column, the question as a bubble
|
| 269 |
+
* on the right, the answer as plain flowing text with no label and no bubble, the composer
|
| 270 |
+
* pinned to the bottom with its controls INSIDE the field.
|
| 271 |
+
* Β· `reference/Glean 1.png` β the SOURCES: a row of chips carrying each source's own mark,
|
| 272 |
+
* directly under the composer. Not a dropdown, not a settings panel.
|
| 273 |
+
* Β· `reference/Airtable AI 1.png` β the NAVIGATION: "New chat" at the top of a left panel, the
|
| 274 |
+
* thread list under it, and a centred "How can I help?" when the thread is empty.
|
| 275 |
+
*
|
| 276 |
+
* β WHAT IS DELIBERATELY NOT COPIED: ChatGPT's dark canvas and Glean's vendor logos. The palette
|
| 277 |
+
* is Loopable's own (`DESIGN.md`, and the font-token rule is gated app-wide over every CSS file
|
| 278 |
+
* in this tree), and the chips wear `FolderMark` β the SAME mark the nav rail draws for that
|
| 279 |
+
* database β because a brand logo for someone else's product is what those references have and
|
| 280 |
+
* this product's databases are not brands. "Exactly like" is about the interaction shape.
|
| 281 |
+
*
|
| 282 |
+
* β AND THE TOOL BOUNDARY IS NARROWER THAN THE SENTENCE "the chatbot can basically call the tools
|
| 283 |
+
* in our App". Under ruling R1 the assistant operates the caller's permitted database/view tools
|
| 284 |
+
* and nothing else: server-side there is exactly ONE tool, `build_view`. It cannot run an
|
| 285 |
+
* automation, touch a connector, or change a setting, and no part of this page implies it can.
|
| 286 |
+
*/
|
| 287 |
+
export default function AssistantPage({ granted }: AssistantPageProps) {
|
| 288 |
+
const [index, setIndex] = useState<QueryIndex>(initialIndex);
|
| 289 |
+
const [activeThread, setActiveThread] = useState("");
|
| 290 |
+
const [selected, setSelected] = useState<string[]>([]);
|
| 291 |
+
const [target, setTarget] = useState("");
|
| 292 |
+
const [model, setModel] = useState("auto");
|
| 293 |
+
const [question, setQuestion] = useState("");
|
| 294 |
+
const [busy, setBusy] = useState(false);
|
| 295 |
+
const [problem, setProblem] = useState("");
|
| 296 |
+
const [loaded, setLoaded] = useState(false);
|
| 297 |
+
const [confirmThread, setConfirmThread] = useState("");
|
| 298 |
+
const [mode, setMode] = useState<AssistantMode>(readMode);
|
| 299 |
+
const [activeQuery, setActiveQuery] = useState("");
|
| 300 |
+
/** Bumped on delete so the hosted workspace's own copy of the list cannot go stale behind us. */
|
| 301 |
+
const [queryRefresh, setQueryRefresh] = useState(0);
|
| 302 |
+
const openCancel = useRef<(() => void) | null>(null);
|
| 303 |
+
const foot = useRef<HTMLDivElement | null>(null);
|
| 304 |
+
/** A counter, not a random id: the optimistic message only has to be unique within this page. */
|
| 305 |
+
const pending = useRef(0);
|
| 306 |
+
|
| 307 |
+
const databases = useMemo(() => databaseEntries(granted).filter((entry) => entry.kind !== "group"), [granted]);
|
| 308 |
+
const labels = useMemo(() => new Map(databases.map((entry) => [entry.key, entry.label])), [databases]);
|
| 309 |
+
/**
|
| 310 |
+
* ββ WHICH CHIPS CAN ACTUALLY BE ASKED, and why the ones that cannot say so.
|
| 311 |
+
* The nav grants twelve databases on tenant #0 and the read boundary can answer for two: the
|
| 312 |
+
* ten `ut_odoo_*` grids are served through the connector mirror, which the Assistant refuses by
|
| 313 |
+
* design. Offering ten doors that cannot open β and letting the reader discover it one spent
|
| 314 |
+
* prompt at a time β is the same defect shape as a rail that paints two rows differently from
|
| 315 |
+
* five. β Absent from the map β TREATED AS ANSWERABLE: the server is the authority and its
|
| 316 |
+
* refusal is still the wall; a client that greyed out everything it had not heard about would
|
| 317 |
+
* hide a working source the moment this field failed to arrive.
|
| 318 |
+
*/
|
| 319 |
+
const sourceStatus = useMemo(
|
| 320 |
+
() => new Map(index.sources.map((row) => [row.database, row])), [index.sources]);
|
| 321 |
+
const blockedReason = useCallback(
|
| 322 |
+
(key: string) => (sourceStatus.get(key)?.answerable === false
|
| 323 |
+
? sourceStatus.get(key)?.reason || "this database cannot be asked yet" : ""),
|
| 324 |
+
[sourceStatus]);
|
| 325 |
+
const messages = index.messages.filter((row) => row.threadId === activeThread);
|
| 326 |
+
const viewById = useMemo(() => new Map(index.views.map((view) => [view.id, view])), [index.views]);
|
| 327 |
+
|
| 328 |
+
/** Offered is not callable: a model whose key this deployment does not hold says so, and why. */
|
| 329 |
+
const modelBlocked = useCallback((key: string) => {
|
| 330 |
+
const row = index.modelStatus.find((item) => item.model === key);
|
| 331 |
+
return row && row.available === false
|
| 332 |
+
? row.reason || "this model is not available on this deployment" : "";
|
| 333 |
+
}, [index.modelStatus]);
|
| 334 |
+
|
| 335 |
+
const reload = useCallback(async () => {
|
| 336 |
+
const result = await fetchQueries();
|
| 337 |
+
setLoaded(true);
|
| 338 |
+
if (result.ok) {
|
| 339 |
+
setIndex(result.value);
|
| 340 |
+
// β Falls back to Auto when the SELECTED model is offered but not callable, not only when it
|
| 341 |
+
// has left the list: a stored thread can restore a model whose key was removed since, and a
|
| 342 |
+
// disabled option the picker cannot clear is a control that refuses its own value.
|
| 343 |
+
const callable = (key: string) => result.value.models.includes(key)
|
| 344 |
+
&& result.value.modelStatus.every((row) => row.model !== key || row.available);
|
| 345 |
+
setModel((current) => callable(current) ? current : "auto");
|
| 346 |
+
}
|
| 347 |
+
}, []);
|
| 348 |
+
|
| 349 |
+
useEffect(() => { void reload(); }, [reload]);
|
| 350 |
+
useEffect(() => () => openCancel.current?.(), []);
|
| 351 |
+
useEffect(() => {
|
| 352 |
+
const permitted = databases.map((entry) => entry.key);
|
| 353 |
+
setSelected((current) => current.filter((key) => permitted.includes(key)));
|
| 354 |
+
setTarget((current) => permitted.includes(current) ? current : "");
|
| 355 |
+
}, [databases]);
|
| 356 |
+
// The conversation reads bottom-up, like every chat the owner named.
|
| 357 |
+
useEffect(() => { foot.current?.scrollIntoView({ block: "end" }); }, [messages.length, busy]);
|
| 358 |
+
|
| 359 |
+
/**
|
| 360 |
+
* The `?mode=` the redirect carried has been read into state; drop it so a toggle made later is
|
| 361 |
+
* not overridden by a stale parameter on the next reload. `replaceState` does NOT fire
|
| 362 |
+
* `hashchange`, so the shell never re-resolves the route and nothing flickers.
|
| 363 |
+
*
|
| 364 |
+
* β AND IT IS STORED IN THE SAME BREATH, WHICH IS THE HALF THAT WAS MISSING. `useState(readMode)`
|
| 365 |
+
* reads the hash and writes NOTHING; only `pickMode` persists. So a reader who arrived in Query
|
| 366 |
+
* by following a `#/query` link, never touching the toggle, would have the parameter stripped
|
| 367 |
+
* here and then land back in Chat on the next reload, restored from a choice made days earlier.
|
| 368 |
+
* Following the link IS choosing.
|
| 369 |
+
*
|
| 370 |
+
* ββ AND IT LISTENS FOR `hashchange`, WHICH IS THE DEFECT THE OWNER REPORTED (QA 2026-08-17,
|
| 371 |
+
* measured on the deployed build). `#/query` and `#/assistant` resolve to the SAME route, so
|
| 372 |
+
* following a `#/query` link from anywhere inside the running app is a same-document hash
|
| 373 |
+
* change: this component is already mounted, `useState(readMode)` does not run a second time,
|
| 374 |
+
* and the panel stayed on Chat while the address bar read `?mode=query`. Only a full reload
|
| 375 |
+
* ever showed Query. **A mount-time read cannot see an arrival that does not remount** β the
|
| 376 |
+
* listener is the whole fix, and `arrivalMode` is the ONE parser both paths call so a later
|
| 377 |
+
* change cannot make the two disagree.
|
| 378 |
+
*/
|
| 379 |
+
useEffect(() => {
|
| 380 |
+
if (typeof window === "undefined") return;
|
| 381 |
+
const consume = () => {
|
| 382 |
+
const asked = arrivalMode(window.location.hash);
|
| 383 |
+
if (!asked) return;
|
| 384 |
+
setMode(asked);
|
| 385 |
+
storeMode(asked);
|
| 386 |
+
window.history.replaceState(null, "", `#/${ASSISTANT_ROUTE}`);
|
| 387 |
+
};
|
| 388 |
+
consume();
|
| 389 |
+
window.addEventListener("hashchange", consume);
|
| 390 |
+
return () => window.removeEventListener("hashchange", consume);
|
| 391 |
+
}, []);
|
| 392 |
+
|
| 393 |
+
const pickMode = useCallback((next: AssistantMode) => {
|
| 394 |
+
setMode(next); storeMode(next); setProblem("");
|
| 395 |
+
}, []);
|
| 396 |
+
|
| 397 |
+
/** The saved Query views, grouped exactly as the standalone page groups them (ONE function). */
|
| 398 |
+
const queryList = useMemo(() => queryGroups(index.views, databases), [index.views, databases]);
|
| 399 |
+
|
| 400 |
+
const selectQuery = useCallback((qid: string) => {
|
| 401 |
+
setActiveQuery(qid);
|
| 402 |
+
openCancel.current?.();
|
| 403 |
+
openCancel.current = openBuiltView(qid, true);
|
| 404 |
+
}, []);
|
| 405 |
+
|
| 406 |
+
/** A renamed or duplicated artefact, merged in place, and the hosted workspace told to re-read. */
|
| 407 |
+
const upsertQuery = useCallback((view: SavedQuery) => {
|
| 408 |
+
setIndex((current) => ({
|
| 409 |
+
...current, views: [view, ...current.views.filter((row) => row.id !== view.id)],
|
| 410 |
+
}));
|
| 411 |
+
setQueryRefresh((count) => count + 1);
|
| 412 |
+
}, []);
|
| 413 |
+
|
| 414 |
+
const removeQuery = useCallback(async (qid: string) => {
|
| 415 |
+
const result = await deleteQuery(qid);
|
| 416 |
+
if (!result.ok) { setProblem(result.message); return; }
|
| 417 |
+
setIndex((current) => ({ ...current, views: current.views.filter((row) => row.id !== qid) }));
|
| 418 |
+
setActiveQuery((current) => (current === qid ? "" : current));
|
| 419 |
+
setQueryRefresh((count) => count + 1);
|
| 420 |
+
}, []);
|
| 421 |
+
|
| 422 |
+
/** Arriving in Query with nothing chosen selects the newest view of the first database. */
|
| 423 |
+
useEffect(() => {
|
| 424 |
+
if (mode !== "query" || activeQuery) return;
|
| 425 |
+
const first = queryList[0]?.views[0];
|
| 426 |
+
if (first) selectQuery(first.id);
|
| 427 |
+
}, [mode, activeQuery, queryList, selectQuery]);
|
| 428 |
+
|
| 429 |
+
const newChat = useCallback(() => {
|
| 430 |
+
setActiveThread(""); setQuestion(""); setProblem("");
|
| 431 |
+
}, []);
|
| 432 |
+
|
| 433 |
+
/** Opening a past chat restores the sources and the model it ran with (R6). */
|
| 434 |
+
const openThread = useCallback((id: string) => {
|
| 435 |
+
setActiveThread(id); setProblem("");
|
| 436 |
+
const row = index.threads.find((thread) => thread.id === id);
|
| 437 |
+
if (!row) return;
|
| 438 |
+
const permitted = databases.map((entry) => entry.key);
|
| 439 |
+
const sources = (row.sources || []).filter((key) => permitted.includes(key));
|
| 440 |
+
if (sources.length) { setSelected(sources); setTarget(sources[0]); }
|
| 441 |
+
if (row.model && index.models.includes(row.model) && !modelBlocked(row.model)) setModel(row.model);
|
| 442 |
+
}, [databases, index.models, index.threads, modelBlocked]);
|
| 443 |
+
|
| 444 |
+
const toggleSource = useCallback((key: string) => {
|
| 445 |
+
if (blockedReason(key)) return;
|
| 446 |
+
setSelected((current) => {
|
| 447 |
+
const next = current.includes(key) ? current.filter((item) => item !== key) : [...current, key];
|
| 448 |
+
setTarget((currentTarget) => next.includes(currentTarget) ? currentTarget : next[0] || "");
|
| 449 |
+
return next;
|
| 450 |
+
});
|
| 451 |
+
}, [blockedReason]);
|
| 452 |
+
|
| 453 |
+
/**
|
| 454 |
+
* ββ R16 β ENTER GOES TO THE CHAT, NOT TO A WAIT. Owner: *"Pressing enter goes immediately to
|
| 455 |
+
* the chat view showing that it is loading; there is a thinking animation."*
|
| 456 |
+
*
|
| 457 |
+
* This used to POST and only then touch `index`, so for the whole model call the reader sat on
|
| 458 |
+
* the opening screen looking at their own unsent sentence with nothing moving. The message is
|
| 459 |
+
* appended FIRST, which is what flips `empty` and hands the surface to the stream; `busy` then
|
| 460 |
+
* draws the app's one loading mark under it.
|
| 461 |
+
*
|
| 462 |
+
* β THE INVERSE IS CAPTURED BEFORE THE WRITE. An optimistic append that is not reconciled on
|
| 463 |
+
* failure leaves a message on screen the server never received, and the next turn replays it to
|
| 464 |
+
* the model as if it had [[undo-capture-before-the-write]]. The typed text goes back in the
|
| 465 |
+
* composer too: losing somebody's sentence to a failed request is the second injury.
|
| 466 |
+
*/
|
| 467 |
+
const ask = useCallback(async () => {
|
| 468 |
+
const text = question.trim();
|
| 469 |
+
if (!text || !target || !selected.includes(target) || busy) return;
|
| 470 |
+
const pendingId = `pending_${(pending.current += 1)}`;
|
| 471 |
+
const optimistic: QueryMessage = {
|
| 472 |
+
id: pendingId, threadId: activeThread, role: "user", content: text,
|
| 473 |
+
createdAt: new Date().toISOString(), targetDatabase: target,
|
| 474 |
+
sources: selected, requestedModel: model,
|
| 475 |
+
};
|
| 476 |
+
const withoutPending = (rows: QueryMessage[]) => rows.filter((row) => row.id !== pendingId);
|
| 477 |
+
setBusy(true); setProblem(""); setQuestion("");
|
| 478 |
+
setIndex((current) => ({ ...current, messages: [...current.messages, optimistic] }));
|
| 479 |
+
const result = await submitChat({ question: text, database: target, sources: selected,
|
| 480 |
+
...(activeThread ? { threadId: activeThread } : {}), model });
|
| 481 |
+
setBusy(false);
|
| 482 |
+
if (!result.ok) {
|
| 483 |
+
setIndex((current) => ({ ...current, messages: withoutPending(current.messages) }));
|
| 484 |
+
setQuestion(text);
|
| 485 |
+
setProblem(result.message);
|
| 486 |
+
return;
|
| 487 |
+
}
|
| 488 |
+
setActiveThread(result.value.thread.id);
|
| 489 |
+
setIndex((current) => ({
|
| 490 |
+
...current,
|
| 491 |
+
threads: [result.value.thread, ...current.threads.filter((row) => row.id !== result.value.thread.id)],
|
| 492 |
+
// The server's own copy of the question replaces the optimistic one, so a thread never holds
|
| 493 |
+
// two of the same turn and the id the rest of the page keys on is the durable one.
|
| 494 |
+
messages: [...withoutPending(current.messages), result.value.userMessage, result.value.message],
|
| 495 |
+
views: result.value.view ? [result.value.view, ...current.views.filter((row) => row.id !== result.value.view?.id)] : current.views,
|
| 496 |
+
citations: [...current.citations, ...result.value.citations],
|
| 497 |
+
}));
|
| 498 |
+
}, [activeThread, busy, model, question, selected, target]);
|
| 499 |
+
|
| 500 |
+
/** β The chat goes; the views it built STAY (they live in Query). Two-click, in the row. */
|
| 501 |
+
const removeThread = useCallback(async (id: string) => {
|
| 502 |
+
const result = await deleteThread(id);
|
| 503 |
+
if (!result.ok) { setProblem(result.message); return; }
|
| 504 |
+
setIndex((current) => ({
|
| 505 |
+
...current,
|
| 506 |
+
threads: current.threads.filter((row) => row.id !== id),
|
| 507 |
+
messages: current.messages.filter((row) => row.threadId !== id),
|
| 508 |
+
}));
|
| 509 |
+
setActiveThread((current) => (current === id ? "" : current));
|
| 510 |
+
setConfirmThread("");
|
| 511 |
+
}, []);
|
| 512 |
+
|
| 513 |
+
/**
|
| 514 |
+
* The rating goes to the server and the SERVER'S copy comes back, rather than the page patching
|
| 515 |
+
* its own guess: the reason is normalised and cleared there, and a rating that only looked
|
| 516 |
+
* applied is exactly the shape this module keeps being bitten by.
|
| 517 |
+
*/
|
| 518 |
+
const rate = useCallback(async (id: string, rating: "up" | "down" | null, reason?: string) => {
|
| 519 |
+
const result = await rateMessage(id, rating, reason);
|
| 520 |
+
if (!result.ok) { setProblem(result.message); return; }
|
| 521 |
+
setIndex((current) => ({
|
| 522 |
+
...current,
|
| 523 |
+
messages: current.messages.map((row) => (row.id === id ? result.value : row)),
|
| 524 |
+
}));
|
| 525 |
+
}, []);
|
| 526 |
+
|
| 527 |
+
/** A built view opens WHERE THE READER ALREADY IS: the toggle flips, the list follows. */
|
| 528 |
+
const preview = useCallback((qid: string) => {
|
| 529 |
+
pickMode("query");
|
| 530 |
+
selectQuery(qid);
|
| 531 |
+
}, [pickMode, selectQuery]);
|
| 532 |
+
|
| 533 |
+
const empty = messages.length === 0;
|
| 534 |
+
const composer = (
|
| 535 |
+
<div className="as-composer">
|
| 536 |
+
<div className="as-field">
|
| 537 |
+
<textarea
|
| 538 |
+
className="as-prompt"
|
| 539 |
+
value={question}
|
| 540 |
+
rows={1}
|
| 541 |
+
placeholder={target ? `Ask about ${labels.get(target) || target}β¦` : "Pick a database below, then askβ¦"}
|
| 542 |
+
aria-label="Ask the assistant"
|
| 543 |
+
onChange={(event) => setQuestion(event.currentTarget.value)}
|
| 544 |
+
onKeyDown={(event) => {
|
| 545 |
+
if (event.key === "Enter" && !event.shiftKey) { event.preventDefault(); void ask(); }
|
| 546 |
+
}}
|
| 547 |
+
/>
|
| 548 |
+
<div className="as-field-foot">
|
| 549 |
+
{/* ChatGPT puts its model control inside the field, as text rather than a boxed
|
| 550 |
+
form control. A native `select` keeps the keyboard and screen-reader behaviour
|
| 551 |
+
a hand-rolled popover would have to re-earn. */}
|
| 552 |
+
<label className="as-pick">
|
| 553 |
+
<span className="as-pick-label">Model</span>
|
| 554 |
+
<select className="as-pick-select" value={model} aria-label="Model"
|
| 555 |
+
onChange={(event) => setModel(event.currentTarget.value)}>
|
| 556 |
+
{index.models.map((choice) => {
|
| 557 |
+
const blocked = modelBlocked(choice);
|
| 558 |
+
return (
|
| 559 |
+
<option key={choice} value={choice} disabled={!!blocked}
|
| 560 |
+
title={blocked || undefined}>
|
| 561 |
+
{blocked ? `${modelLabel(choice)} (not configured)` : modelLabel(choice)}
|
| 562 |
+
</option>
|
| 563 |
+
);
|
| 564 |
+
})}
|
| 565 |
+
</select>
|
| 566 |
+
</label>
|
| 567 |
+
{/* β ONLY WHEN THE ANSWER IS AMBIGUOUS. Every generated view names ONE database (R3);
|
| 568 |
+
with a single source selected that database is not a question, so asking it would
|
| 569 |
+
be chrome for its own sake. */}
|
| 570 |
+
{selected.length > 1 ? (
|
| 571 |
+
<label className="as-pick">
|
| 572 |
+
<span className="as-pick-label">Build on</span>
|
| 573 |
+
<select className="as-pick-select" value={target} aria-label="Target database"
|
| 574 |
+
onChange={(event) => setTarget(event.currentTarget.value)}>
|
| 575 |
+
{selected.map((key) => <option key={key} value={key}>{labels.get(key) || key}</option>)}
|
| 576 |
+
</select>
|
| 577 |
+
</label>
|
| 578 |
+
) : null}
|
| 579 |
+
<button type="button" className="as-send" aria-label="Send"
|
| 580 |
+
disabled={busy || !question.trim() || !target} onClick={() => void ask()}>
|
| 581 |
+
<svg viewBox="0 0 16 16" aria-hidden="true" width="15" height="15">
|
| 582 |
+
<path d="M8 13V3.6M4 7.4 8 3.4l4 4" />
|
| 583 |
+
</svg>
|
| 584 |
+
</button>
|
| 585 |
+
</div>
|
| 586 |
+
</div>
|
| 587 |
+
{/* Glean's row: the sources, with their marks, under the field. */}
|
| 588 |
+
<div className="as-chips" role="group" aria-label="Databases the assistant may read">
|
| 589 |
+
{databases.map((database) => {
|
| 590 |
+
const on = selected.includes(database.key);
|
| 591 |
+
const blocked = blockedReason(database.key);
|
| 592 |
+
return (
|
| 593 |
+
<button type="button" key={database.key}
|
| 594 |
+
className={"as-chip" + (on ? " is-selected" : "")
|
| 595 |
+
+ (database.key === target ? " is-target" : "") + (blocked ? " is-blocked" : "")}
|
| 596 |
+
aria-pressed={on}
|
| 597 |
+
aria-disabled={blocked ? true : undefined}
|
| 598 |
+
title={blocked || undefined}
|
| 599 |
+
onClick={() => toggleSource(database.key)}>
|
| 600 |
+
<span className="as-chip-mark" aria-hidden="true">
|
| 601 |
+
{database.icon ? <FolderMark icon={database.icon} size={14} /> : <DbIcon />}
|
| 602 |
+
</span>
|
| 603 |
+
<span className="as-chip-name">{database.label}</span>
|
| 604 |
+
</button>
|
| 605 |
+
);
|
| 606 |
+
})}
|
| 607 |
+
{/* β ONE sentence for the whole row, not one per chip: with ten blocked sources the
|
| 608 |
+
per-chip copy would be the loudest thing on a surface whose whole instruction was
|
| 609 |
+
"plain". The chip carries the exact cause in its `title`. */}
|
| 610 |
+
{databases.some((database) => blockedReason(database.key)) ? (
|
| 611 |
+
<p className="as-chips-note">
|
| 612 |
+
Greyed databases are served live from a connector and cannot be asked yet.
|
| 613 |
+
</p>
|
| 614 |
+
) : null}
|
| 615 |
+
{databases.length === 0 && loaded ? (
|
| 616 |
+
<p className="as-chips-empty">No databases are shared with your account yet.</p>
|
| 617 |
+
) : null}
|
| 618 |
+
</div>
|
| 619 |
+
</div>
|
| 620 |
+
);
|
| 621 |
+
|
| 622 |
+
// β `aria-label` is "Assistant", not "AI assistant" (W34-T13, corrected at QA 2026-08-17).
|
| 623 |
+
// An aria-label IS a label β it is what a screen reader announces this region as β so the rail
|
| 624 |
+
// reading Assistant while this said AI assistant left the rename half-done in the one place
|
| 625 |
+
// sighted QA never looks.
|
| 626 |
+
return (
|
| 627 |
+
<div className="as-chat" aria-label="Assistant">
|
| 628 |
+
{/* Airtable's panel: one door to a new chat, then the history. */}
|
| 629 |
+
<aside className="as-history">
|
| 630 |
+
{/* C2: two values, directly ABOVE "+ New chat". It changes which LIST shows, not the route. */}
|
| 631 |
+
<div className="as-mode-toggle" role="tablist" aria-label="Chat or Query">
|
| 632 |
+
{(["chat", "query"] as const).map((value) => (
|
| 633 |
+
<button type="button" key={value} role="tab" aria-selected={mode === value}
|
| 634 |
+
className={"as-mode" + (mode === value ? " is-on" : "")}
|
| 635 |
+
onClick={() => pickMode(value)}>
|
| 636 |
+
{value === "chat" ? "Chat" : "Query"}
|
| 637 |
+
</button>
|
| 638 |
+
))}
|
| 639 |
+
</div>
|
| 640 |
+
{mode === "query" ? (
|
| 641 |
+
/* The SAME rail the standalone page renders; only its host changed, so the differences
|
| 642 |
+
are CSS in assistant.css and nothing here. */
|
| 643 |
+
<QueryRail groups={queryList} activeId={activeQuery} onSelect={selectQuery}
|
| 644 |
+
onDelete={(id) => void removeQuery(id)} onChanged={upsertQuery} />
|
| 645 |
+
) : (
|
| 646 |
+
<>
|
| 647 |
+
<button type="button" className="as-new" onClick={newChat}>
|
| 648 |
+
<svg viewBox="0 0 16 16" aria-hidden="true" width="14" height="14">
|
| 649 |
+
<path d="M8 3.5v9M3.5 8h9" />
|
| 650 |
+
</svg>
|
| 651 |
+
New chat
|
| 652 |
+
</button>
|
| 653 |
+
<nav className="as-thread-list" aria-label="Chat history">
|
| 654 |
+
{index.threads.map((row) => (
|
| 655 |
+
<div className={"as-thread-row" + (row.id === activeThread ? " is-active" : "")} key={row.id}>
|
| 656 |
+
<button type="button" className="as-thread"
|
| 657 |
+
onClick={() => { setConfirmThread(""); openThread(row.id); }}
|
| 658 |
+
title={row.title}>{row.title}</button>
|
| 659 |
+
{confirmThread === row.id ? (
|
| 660 |
+
<button type="button" className="as-thread-confirm"
|
| 661 |
+
onClick={() => void removeThread(row.id)}>Delete</button>
|
| 662 |
+
) : (
|
| 663 |
+
<button type="button" className="as-thread-del" aria-label={`Delete chat: ${row.title}`}
|
| 664 |
+
onClick={() => setConfirmThread(row.id)}>
|
| 665 |
+
<svg viewBox="0 0 16 16" aria-hidden="true" width="13" height="13">
|
| 666 |
+
<path d="M3.5 4.5h9M6.5 4.5V3.2h3v1.3M5 4.5l.6 8h4.8l.6-8" />
|
| 667 |
+
</svg>
|
| 668 |
+
</button>
|
| 669 |
+
)}
|
| 670 |
+
</div>
|
| 671 |
+
))}
|
| 672 |
+
{loaded && index.threads.length === 0 ? (
|
| 673 |
+
<p className="as-thread-empty">No chats yet.</p>
|
| 674 |
+
) : null}
|
| 675 |
+
</nav>
|
| 676 |
+
</>
|
| 677 |
+
)}
|
| 678 |
+
</aside>
|
| 679 |
+
<main className="as-conversation">
|
| 680 |
+
{mode === "query" ? (
|
| 681 |
+
queryList.length ? (
|
| 682 |
+
<Suspense fallback={<p className="as-working" role="status">
|
| 683 |
+
<span className="lp-spin" aria-hidden="true" />
|
| 684 |
+
</p>}>
|
| 685 |
+
<QueryWorkspace granted={granted} hostedRail refreshToken={queryRefresh}
|
| 686 |
+
selectedId={activeQuery} />
|
| 687 |
+
</Suspense>
|
| 688 |
+
) : !loaded ? (
|
| 689 |
+
/* β W35-T22, the SECOND state that paints something untrue for a frame. `queryList` is
|
| 690 |
+
empty until `GET /query` lands, so arriving here β a `#/query` link, a bookmark, a
|
| 691 |
+
reload β asserted "No Query views yet" to somebody who has twenty, then replaced it.
|
| 692 |
+
`loaded` is the flag the chat history one screen down already waits on; the two
|
| 693 |
+
empty states now agree about when they are entitled to speak. */
|
| 694 |
+
<p className="as-working" role="status" aria-label="Loading your Query views">
|
| 695 |
+
<span className="lp-spin" aria-hidden="true" />
|
| 696 |
+
</p>
|
| 697 |
+
) : (
|
| 698 |
+
<div className="as-opening">
|
| 699 |
+
<h1 className="as-opening-h">No Query views yet</h1>
|
| 700 |
+
<p className="as-opening-p">
|
| 701 |
+
Ask a question in Chat. When the assistant builds a view, it is saved here.
|
| 702 |
+
</p>
|
| 703 |
+
</div>
|
| 704 |
+
)
|
| 705 |
+
) : empty ? (
|
| 706 |
+
<div className="as-opening">
|
| 707 |
+
<h1 className="as-opening-h">How can I help?</h1>
|
| 708 |
+
<p className="as-opening-p">
|
| 709 |
+
Ask about a database you can already open. The answer arrives as a view in Query.
|
| 710 |
+
</p>
|
| 711 |
+
{composer}
|
| 712 |
+
{problem ? <p className="as-problem" role="status">{problem}</p> : null}
|
| 713 |
+
</div>
|
| 714 |
+
) : (
|
| 715 |
+
<>
|
| 716 |
+
<div className="as-stream">
|
| 717 |
+
<div className="as-column">
|
| 718 |
+
{messages.map((message) => (
|
| 719 |
+
<AssistantMessage key={message.id} message={message}
|
| 720 |
+
view={message.viewId ? viewById.get(message.viewId) : undefined}
|
| 721 |
+
citations={index.citations} onPreview={preview}
|
| 722 |
+
onRate={message.role === "assistant" && !message.id.startsWith("pending_")
|
| 723 |
+
? rate : undefined} />
|
| 724 |
+
))}
|
| 725 |
+
{/* R16's thinking animation is the app's ONE loading mark, not a second spinner
|
| 726 |
+
vocabulary: `.lp-spin` is the same mark every surface uses and C is repainting
|
| 727 |
+
it in the brand purple this wave. The label is for screen readers only, since
|
| 728 |
+
an aria-hidden spinner inside a status region announces nothing. */}
|
| 729 |
+
{busy ? (
|
| 730 |
+
<p className="as-working" role="status" aria-label="The assistant is thinking">
|
| 731 |
+
<span className="lp-spin" aria-hidden="true" />
|
| 732 |
+
</p>
|
| 733 |
+
) : null}
|
| 734 |
+
<div ref={foot} />
|
| 735 |
+
</div>
|
| 736 |
+
</div>
|
| 737 |
+
<div className="as-dock">
|
| 738 |
+
<div className="as-column">
|
| 739 |
+
{problem ? <p className="as-problem" role="status">{problem}</p> : null}
|
| 740 |
+
{composer}
|
| 741 |
+
</div>
|
| 742 |
+
</div>
|
| 743 |
+
</>
|
| 744 |
+
)}
|
| 745 |
+
</main>
|
| 746 |
+
</div>
|
| 747 |
+
);
|
| 748 |
+
}
|
web/src/automation/AutomationBuilder.tsx
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|