Spaces:
Paused
Paused
| title: IG Costing Server | |
| emoji: 💸 | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| # IG Costing — ingest + query server | |
| Collects LLM cost events and Lambda invocations from the InterviewGod backend and | |
| spreads them across **multiple Atlas clusters**. Stateless: MongoDB is external, | |
| so the container holds nothing and restarts cost nothing. | |
| Stable base URL: `https://<user>-<space>.hf.space` — no tunnel, never rotates. | |
| ## Secrets | |
| Set by CI (`.github/workflows/deploy-space.yml` mirrors them from GitHub secrets), | |
| or by hand in Settings → Variables and secrets. | |
| | Secret | Required | Purpose | | |
| |---|---|---| | |
| | `CostingDBURL1`, `CostingDBURL2`, … | yes | Atlas URIs for `llm_events` + `budget_config` | | |
| | `LambdaDBURL1`, `LambdaDBURL2`, … | yes | Atlas URIs for `lambda_events` | | |
| | `REPORTING_CLIENT_ID` | yes | HMAC client id — must match prod | | |
| | `REPORTING_CLIENT_SECRET` | yes | HMAC secret — must match prod | | |
| | `SPACE_URL` | for keep-alive | This Space's own public URL | | |
| | `MONGO_DB_NAME` | no | Database name on every cluster (default `costingDB`) | | |
| | `SHARD_ROLLOVER_AT` | no | Quota fraction at which writes roll to the next shard (default `0.9`) | | |
| | `KEEPALIVE_MINUTES` | no | Self-ping interval (default `30`) | | |
| Numbering is **1-based and sorted numerically**, so `LambdaDBURL10` comes after | |
| `LambdaDBURL9`. Order is the write policy — see below. | |
| Each shard is opened with an explicit database name. Connection strings ending in | |
| `/` would otherwise land in `test` on some clusters and `costingDB` on others. | |
| ## Sharding | |
| **Writes — fill-and-spill.** Everything goes to the first shard under | |
| `SHARD_ROLLOVER_AT`; when it fills, writes roll to the next. Sizes are cached for | |
| 5 minutes, and an Atlas quota error forces an immediate re-probe and retry rather | |
| than dropping the batch. | |
| Hashing by `func_name` was the obvious alternative and is wrong here: one function | |
| is ~99% of Lambda traffic, so it would pin nearly every write to one shard while | |
| the rest sat empty. | |
| Note this multiplies **storage**, not **throughput** — all writes still land on one | |
| cluster at a time, and a free cluster caps at 100 ops/sec. | |
| **Reads — fan out and merge.** Older data lives on earlier shards, so every read | |
| queries all of them. `$sort`/`$limit` were removed from the `by_provider` / | |
| `by_model` / `by_func` facets and applied after merging: a top-10 taken per shard | |
| can miss an entry ranking 11th everywhere but first overall. | |
| ## Endpoints | |
| ``` | |
| POST /v1/llm-calls ingest, HMAC-signed | |
| POST /v1/lambda-executions ingest, HMAC-signed | |
| GET /v1/stats /v1/overview … dashboard queries | |
| GET /v1/setup-indexes creates indexes on EVERY shard — run once | |
| GET /v1/admin/shards per-shard usage + which one is taking writes | |
| GET /v1/admin/health liveness, does not touch Mongo | |
| ``` | |
| ## Keep-alive | |
| `instrumentation.ts` pings `SPACE_URL/api/v1/admin/health` every | |
| `KEEPALIVE_MINUTES`. It hits the public URL deliberately — HF's idle timer counts | |
| requests through their router, not traffic inside the container. | |
| It keeps a *running* Space awake. It cannot restart a *stopped* one; nothing is | |
| left to send the ping. Pair it with an external cron if that matters. | |
| ## Going live | |
| 1. Push to `master` — CI syncs secrets, uploads this folder, Space rebuilds | |
| 2. `GET /v1/admin/health` | |
| 3. `GET /v1/setup-indexes` — every shard needs indexes, including empty ones | |
| 4. `GET /v1/admin/shards` — confirm all clusters reachable | |
| 5. Repoint prod's reporting base URL here | |
| ## Provisioning new clusters | |
| Atlas allows one free cluster per project, so each new shard means a new project, | |
| cluster, user and IP rule. `scripts/atlas-provision.mjs` does all of it. | |
| One-time setup in the Atlas UI: | |
| 1. **Organization → Access Manager → Applications → Add Service Account** | |
| 2. Role: **Organization Project Creator** (needs to create projects, not just read) | |
| 3. Copy the **Client ID** and **Client Secret** — the secret is shown once | |
| 4. **Organization → Settings** for the Organization ID | |
| 5. Put all three in `backend/.env`: | |
| ``` | |
| ATLAS_CLIENT_ID=... | |
| ATLAS_CLIENT_SECRET=... | |
| ATLAS_ORG_ID=... | |
| ``` | |
| Then: | |
| ```bash | |
| node scripts/atlas-provision.mjs lambda 5 6 # LambdaDBURL5, LambdaDBURL6 | |
| node scripts/atlas-provision.mjs costing 3 | |
| ``` | |
| Each cluster takes ~3-5 minutes. The script skips numbers already in `.env`, so | |
| re-running after a failure is safe. | |
| Each connection string is written to `backend/.env` **and** pushed to GitHub | |
| secrets, because that is where the Space reads its config from. Then: | |
| ```bash | |
| gh workflow run deploy-space.yml | |
| ``` | |
| The `ATLAS_*` service-account credentials stay on your machine. Only this script | |
| uses them, and they are absent from the deploy workflow's secret allowlist, so | |
| they cannot reach the Space even if added to GitHub by hand. | |
| ## Watch | |
| `/v1/admin/shards`. Raw Lambda events run ~47 MB/day, so each 512 MB cluster | |
| fills in **about 11 days**. | |
| ```bash | |
| npm run check:shards # read-only preflight against every configured cluster | |
| npm test # cross-shard merge checks | |
| ``` | |