---
title: IG Costing Server
emoji: 💸
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
---

# IG Costing — ingest + query server

Collects LLM cost events and Lambda invocations from the InterviewGod backend and
spreads them across **multiple Atlas clusters**. Stateless: MongoDB is external,
so the container holds nothing and restarts cost nothing.

Stable base URL: `https://<user>-<space>.hf.space` — no tunnel, never rotates.

## Secrets

Set by CI (`.github/workflows/deploy-space.yml` mirrors them from GitHub secrets),
or by hand in Settings → Variables and secrets.

| Secret | Required | Purpose |
|---|---|---|
| `CostingDBURL1`, `CostingDBURL2`, … | yes | Atlas URIs for `llm_events` + `budget_config` |
| `LambdaDBURL1`, `LambdaDBURL2`, … | yes | Atlas URIs for `lambda_events` |
| `REPORTING_CLIENT_ID` | yes | HMAC client id — must match prod |
| `REPORTING_CLIENT_SECRET` | yes | HMAC secret — must match prod |
| `SPACE_URL` | for keep-alive | This Space's own public URL |
| `MONGO_DB_NAME` | no | Database name on every cluster (default `costingDB`) |
| `SHARD_ROLLOVER_AT` | no | Quota fraction at which writes roll to the next shard (default `0.9`) |
| `KEEPALIVE_MINUTES` | no | Self-ping interval (default `30`) |

Numbering is **1-based and sorted numerically**, so `LambdaDBURL10` comes after
`LambdaDBURL9`. Order is the write policy — see below.

Each shard is opened with an explicit database name. Connection strings ending in
`/` would otherwise land in `test` on some clusters and `costingDB` on others.

## Sharding

**Writes — fill-and-spill.** Everything goes to the first shard under
`SHARD_ROLLOVER_AT`; when it fills, writes roll to the next. Sizes are cached for
5 minutes, and an Atlas quota error forces an immediate re-probe and retry rather
than dropping the batch.

Hashing by `func_name` was the obvious alternative and is wrong here: one function
is ~99% of Lambda traffic, so it would pin nearly every write to one shard while
the rest sat empty.

Note this multiplies **storage**, not **throughput** — all writes still land on one
cluster at a time, and a free cluster caps at 100 ops/sec.

**Reads — fan out and merge.** Older data lives on earlier shards, so every read
queries all of them. `$sort`/`$limit` were removed from the `by_provider` /
`by_model` / `by_func` facets and applied after merging: a top-10 taken per shard
can miss an entry ranking 11th everywhere but first overall.

## Endpoints

```
POST /v1/llm-calls            ingest, HMAC-signed
POST /v1/lambda-executions    ingest, HMAC-signed
GET  /v1/stats /v1/overview … dashboard queries
GET  /v1/setup-indexes        creates indexes on EVERY shard — run once
GET  /v1/admin/shards         per-shard usage + which one is taking writes
GET  /v1/admin/health         liveness, does not touch Mongo
```

## Keep-alive

`instrumentation.ts` pings `SPACE_URL/api/v1/admin/health` every
`KEEPALIVE_MINUTES`. It hits the public URL deliberately — HF's idle timer counts
requests through their router, not traffic inside the container.

It keeps a *running* Space awake. It cannot restart a *stopped* one; nothing is
left to send the ping. Pair it with an external cron if that matters.

## Going live

1. Push to `master` — CI syncs secrets, uploads this folder, Space rebuilds
2. `GET /v1/admin/health`
3. `GET /v1/setup-indexes` — every shard needs indexes, including empty ones
4. `GET /v1/admin/shards` — confirm all clusters reachable
5. Repoint prod's reporting base URL here

## Provisioning new clusters

Atlas allows one free cluster per project, so each new shard means a new project,
cluster, user and IP rule. `scripts/atlas-provision.mjs` does all of it.

One-time setup in the Atlas UI:

1. **Organization → Access Manager → Applications → Add Service Account**
2. Role: **Organization Project Creator** (needs to create projects, not just read)
3. Copy the **Client ID** and **Client Secret** — the secret is shown once
4. **Organization → Settings** for the Organization ID
5. Put all three in `backend/.env`:

```
ATLAS_CLIENT_ID=...
ATLAS_CLIENT_SECRET=...
ATLAS_ORG_ID=...
```

Then:

```bash
node scripts/atlas-provision.mjs lambda 5 6     # LambdaDBURL5, LambdaDBURL6
node scripts/atlas-provision.mjs costing 3
```

Each cluster takes ~3-5 minutes. The script skips numbers already in `.env`, so
re-running after a failure is safe.

Each connection string is written to `backend/.env` **and** pushed to GitHub
secrets, because that is where the Space reads its config from. Then:

```bash
gh workflow run deploy-space.yml
```

The `ATLAS_*` service-account credentials stay on your machine. Only this script
uses them, and they are absent from the deploy workflow's secret allowlist, so
they cannot reach the Space even if added to GitHub by hand.

## Watch

`/v1/admin/shards`. Raw Lambda events run ~47 MB/day, so each 512 MB cluster
fills in **about 11 days**.

```bash
npm run check:shards    # read-only preflight against every configured cluster
npm test                # cross-shard merge checks
```