README.md

metadata

title: IG Costing Server
emoji: 💸
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false

IG Costing — ingest + query server

Collects LLM cost events and Lambda invocations from the InterviewGod backend and spreads them across multiple Atlas clusters. Stateless: MongoDB is external, so the container holds nothing and restarts cost nothing.

Stable base URL: https://<user>-<space>.hf.space — no tunnel, never rotates.

Secrets

Set by CI (.github/workflows/deploy-space.yml mirrors them from GitHub secrets), or by hand in Settings → Variables and secrets.

Secret	Required	Purpose
CostingDBURL1, CostingDBURL2, …	yes	Atlas URIs for llm_events + budget_config
LambdaDBURL1, LambdaDBURL2, …	yes	Atlas URIs for lambda_events
REPORTING_CLIENT_ID	yes	HMAC client id — must match prod
REPORTING_CLIENT_SECRET	yes	HMAC secret — must match prod
SPACE_URL	for keep-alive	This Space's own public URL
MONGO_DB_NAME	no	Database name on every cluster (default costingDB)
SHARD_ROLLOVER_AT	no	Quota fraction at which writes roll to the next shard (default 0.9)
KEEPALIVE_MINUTES	no	Self-ping interval (default 30)

Numbering is 1-based and sorted numerically, so LambdaDBURL10 comes after LambdaDBURL9. Order is the write policy — see below.

Each shard is opened with an explicit database name. Connection strings ending in / would otherwise land in test on some clusters and costingDB on others.

Sharding

Writes — fill-and-spill. Everything goes to the first shard under SHARD_ROLLOVER_AT; when it fills, writes roll to the next. Sizes are cached for 5 minutes, and an Atlas quota error forces an immediate re-probe and retry rather than dropping the batch.

Hashing by func_name was the obvious alternative and is wrong here: one function is ~99% of Lambda traffic, so it would pin nearly every write to one shard while the rest sat empty.

Note this multiplies storage, not throughput — all writes still land on one cluster at a time, and a free cluster caps at 100 ops/sec.

Reads — fan out and merge. Older data lives on earlier shards, so every read queries all of them. $sort/$limit were removed from the by_provider / by_model / by_func facets and applied after merging: a top-10 taken per shard can miss an entry ranking 11th everywhere but first overall.

Endpoints

POST /v1/llm-calls            ingest, HMAC-signed
POST /v1/lambda-executions    ingest, HMAC-signed
GET  /v1/stats /v1/overview … dashboard queries
GET  /v1/setup-indexes        creates indexes on EVERY shard — run once
GET  /v1/admin/shards         per-shard usage + which one is taking writes
GET  /v1/admin/health         liveness, does not touch Mongo

Keep-alive

instrumentation.ts pings SPACE_URL/api/v1/admin/health every KEEPALIVE_MINUTES. It hits the public URL deliberately — HF's idle timer counts requests through their router, not traffic inside the container.

It keeps a running Space awake. It cannot restart a stopped one; nothing is left to send the ping. Pair it with an external cron if that matters.

Going live

1. Push to master — CI syncs secrets, uploads this folder, Space rebuilds
2. GET /v1/admin/health
3. GET /v1/setup-indexes — every shard needs indexes, including empty ones
4. GET /v1/admin/shards — confirm all clusters reachable
5. Repoint prod's reporting base URL here

Provisioning new clusters

Atlas allows one free cluster per project, so each new shard means a new project, cluster, user and IP rule. scripts/atlas-provision.mjs does all of it.

One-time setup in the Atlas UI:

1. Organization → Access Manager → Applications → Add Service Account
2. Role: Organization Project Creator (needs to create projects, not just read)
3. Copy the Client ID and Client Secret — the secret is shown once
4. Organization → Settings for the Organization ID
5. Put all three in backend/.env:

ATLAS_CLIENT_ID=...
ATLAS_CLIENT_SECRET=...
ATLAS_ORG_ID=...

Then:

node scripts/atlas-provision.mjs lambda 5 6     # LambdaDBURL5, LambdaDBURL6
node scripts/atlas-provision.mjs costing 3

Each cluster takes ~3-5 minutes. The script skips numbers already in .env, so re-running after a failure is safe.

Each connection string is written to backend/.env and pushed to GitHub secrets, because that is where the Space reads its config from. Then:

gh workflow run deploy-space.yml

The ATLAS_* service-account credentials stay on your machine. Only this script uses them, and they are absent from the deploy workflow's secret allowlist, so they cannot reach the Space even if added to GitHub by hand.

Watch

/v1/admin/shards. Raw Lambda events run ~47 MB/day, so each 512 MB cluster fills in about 11 days.

npm run check:shards    # read-only preflight against every configured cluster
npm test                # cross-shard merge checks