docs/PROCESS.md

# x402 Demand — how the model was built

This is the full record of how the `x402-demand` checkpoints were made: the source
data, the label and why it is shaped that way, the splits, the baselines a checkpoint
had to beat, the training recipe, the results, and what was published. Written so the
run can be repeated, extended, or challenged.

Owner: Ryan Price (Spark Intelligence Station). Started 2026-09-23.

---

## 0. Publication policy

| Artifact | Published? | Where |
|---|---|---|
| Checkpoints (weights, tokenizer, config) | **yes** | HF model repo |
| Metrics (`results/*.json`) and figures (`results/*.png`) | **yes** | HF model card |
| Demo app | **yes** | HF Space |
| This document, the training/eval code | on the model repo (as docs), **no public git** | — |
| **The dataset** (snapshots, `data/rows.parquet`, per-row predictions) | **NO** | stays on the Spark box |

The dataset is deliberately withheld. The daily snapshots are the lab's own longitudinal
record of the CDP Bazaar and are the durable asset; the model card describes the data
in aggregate only. `data/` and `runs/` are gitignored so nothing row-level can be
committed by accident. The figures show aggregate rates and scores, never rows. The
demo's example listings are invented, not drawn from the data.

---

## 1. Source data

- **What:** the Coinbase CDP x402 discovery index (the "Bazaar"),
  `GET https://api.cdp.coinbase.com/platform/v2/x402/discovery/resources`, fetched once
  a day, fully paginated (1,000/page), by the lab's `Scout:Bazaar`
  (`spark-lab/agents/scouts.py`). Each daily file is `vault/raw/bazaar/YYYY-MM-DD.json`;
  this project reads only its `cdp` list.
- **Per listing:** `resource` (URL), `type`, `description`, `price` (USDC atomic units,
  6 decimals), `network`, `lastUpdated`, and `quality` = {`l30DaysTotalCalls`,
  `l30DaysUniquePayers`, `lastCalledAt`}.
- **Window used for v1:** 31 snapshots, 2026-08-24 → 2026-09-23. 14.2k-16.2k listings per
  day; 26,642 distinct resources seen; 2,464 hosts in the modelled rows.
- **Completeness check:** daily counts are stable (no truncated pages). 1,297 resources
  (4.9%) disappear and reappear — "flicker". The label below is built from
  `lastCalledAt` timestamps rather than from presence, so it is robust to that.

### The fact that shaped the label

The index **evicts a listing exactly 30 days after its `quality.lastCalledAt`** (measured
to the second on the lab's own listing on 2026-09-12; `lastUpdated` does not matter). So
"will this listing still be in the index in N days?" is mostly *already decided* by a
timestamp we can read: anything called in the last 30-N days survives by rule. A model
trained on survival would mostly learn that arithmetic.

## 2. The label

> **`paid_Hd` = 1 iff the listing receives at least one paid call in the H days after the
> anchor snapshot**, i.e. some later snapshot shows `lastCalledAt` ∈ (ref_t, ref_t + H].

- `ref_t` is the snapshot's reference time = the newest `lastCalledAt` in it (≈ fetch time).
- A listing that is evicted, or stays listed without a new call, is 0.
- H ∈ {7, 14}. A horizon is labelled only if the whole window (+1 day of slack) is observed.
- Result: **330,035 labelled rows** (paid_7d, anchors 0-21), base rate **0.240**;
  241,825 rows for paid_14d (anchors 0-15), base rate **0.324**.
  In words: **only a quarter of listed x402 services get paid even once in a week.**

What the label is not: it is not "is this a legitimate service". A self-call to keep a
listing alive counts as a paid call. See §7.

## 3. Features (all computed from snapshots ≤ t — no look-ahead)

Listing: URL, host, path depth, path parameter present, description (+ length), price in
USD, network.
Telemetry: 30-day calls and unique payers, calls per payer, days since last paid call,
days since listing updated, days observed in the index (with a left-censoring flag for
listings already present on day 0), new-listing flag, presence and payer/call deltas vs 7
days earlier.
Host aggregates at t: number of listings, share of the host's listings called in the
last 7 days, share with exactly one call per payer, median payers, networks used.

## 4. Splits — two walls, both enforced in `common.py`

1. **Host wall.** Hosts are hashed (sha256) into 100 buckets. Buckets 80-99 are **test
   hosts**, never seen in training. Many hosts mass-list hundreds of near-identical
   endpoints (one lists 840); without this wall a model memorises the host.
2. **Time wall.** A training row's label window must end before the first test anchor, so
   no training label observes the test period.
   - paid_7d: train anchors 0-9, validation anchor 10, **test anchors 17-21**.
   - paid_14d: train anchor 0, validation anchor 1, **test anchor 15** (31 days only
     allows one clean 14-day window; this horizon is secondary until more history exists).
3. **Calibration hosts.** For the neural models, train-side buckets 70-79 are held out
   from training and used for model selection and temperature fitting, so calibration is
   measured on hosts the model has not seen.

Test set for paid_7d: 13,644 rows over 442 unseen hosts.

Segments reported everywhere: `all`, `new_listings` (first seen at the anchor: the cold
start), `small_hosts(<=20)` (hosts with ≤ 20 listings: the long tail, less factory noise),
`stale(>7d since call)`.

## 5. Baselines (`baselines.py`) — what a checkpoint has to beat

| paid_7d, test | all AUC | new listings AUC | small hosts AUC | ECE (all) |
|---|---|---|---|---|
| Recency rule (isotonic on days since last call) | 0.635 | 0.416 | 0.696 | 0.095 |
| Logistic regression, numeric features | **0.678** | 0.500 | 0.731 | **0.052** |
| LightGBM, numeric + network | 0.672 | 0.514 | **0.737** | 0.076 |

Top LightGBM features by gain: 30-day calls, host listing count, host 7-day activity,
days since update, host one-call-per-payer share, price.

Reading: telemetry gets you to AUC ~0.68 on unseen hosts. Brand-new listings are at
chance for every telemetry model: they have no history, and their first "call" is often
the settle that listed them.

## 6. Neural model (`train_laya.py`)

- **Base:** `convaiinnovations/laya` (ModernBERT-large encoder + decision head, 421M,
  Apache-2.0). Output is a Laya checkpoint: it loads with `laya.load()` and answers typed
  questions.
- **Questions trained:** two `noul` (yes/no probability) questions, `paid_7d` and
  `paid_14d`, with the instructions in `train_laya.py:QUESTIONS`. Use the same wording at
  inference.
- **Variants** (only the input state differs):
  - `text`: `{resource, description, price_usd, network}`. Works for any listing,
    including one not yet listed. The text is identical across a resource's anchors, so
    each resource is one item whose **soft target** = the fraction of its training anchors
    that were paid.
  - `full`: the listing plus telemetry as text (`paid_calls_30d`, `unique_payers_30d`,
    `days_since_last_paid_call`, `days_since_listing_updated`, `host_listing_count`,
    `host_share_called_in_last_7d`). Two anchors are sampled per resource, with hard targets.
- **Objective:** a single-GPU port of Laya's upstream RLCD recipe. REINFORCE on a strictly
  proper scoring rule (log + 0.75 spherical) with a group-mean baseline (G=4, σ 0.4→0.1),
  plus a soft cross-entropy term. AdamW with encoder lr 2.5e-5 and head lr 1e-4, weight
  decay 0.01, warmup and cosine, micro-batch 16 × accumulation 4, bf16 autocast, gradient
  checkpointing, max_len 384 (head budget 192), 3 epochs. The best epoch by held-out
  calibration-host AUC is kept.
- **Calibration:** one temperature for `noul`, fitted by LBFGS on the calibration hosts,
  written to `rl_agent_config.json` where `laya.Agent` applies it.
- **Hardware:** DGX Spark GB10, ~8.5 GB of GPU memory per run.

## 7. Results

All numbers are on the held-out test set (future anchors × unseen hosts). Source files:
`runs/laya-*/eval.json`, `runs/bootstrap_full_vs_baselines.json`, published as `results/metrics.json`.

### paid_7d, ROC AUC

| Model | All (n=13,644) | New listings (n=221) | Small hosts ≤20 (n=6,135) | ECE (all) |
|---|---|---|---|---|
| Recency rule | 0.635 | 0.416 | 0.696 | 0.095 |
| Logistic regression | 0.678 | 0.500 | 0.731 | 0.052 |
| LightGBM | 0.672 | 0.514 | 0.737 | 0.076 |
| **Laya · listing only** (`text`) | 0.639 | **0.575** | 0.668 | 0.116 |
| **Laya · listing + telemetry** (`full`) | **0.732** | 0.581 | **0.754** | 0.110 → **0.079** isotonic |

paid_14d (one test anchor, n=2,631): LightGBM 0.652, `text` 0.653, `full` **0.733**.

### Significance (paired bootstrap, resampling HOSTS, 1,000 draws)

Rows within a host are strongly correlated, so resampling rows would overstate certainty.

| Comparison | median ΔAUC | 95% CI | P(Δ ≤ 0) |
|---|---|---|---|
| `full` vs LightGBM | +0.058 | [+0.010, +0.110] | 0.014 |
| `full` vs `text` | +0.092 | [+0.051, +0.141] | 0.000 |
| LightGBM + `text` score vs LightGBM, same calibration-host rows | +0.027 | [−0.008, +0.073] | 0.076 |
| … new listings only | +0.080 | [−0.002, +0.153] | 0.026 |
| LightGBM + `full` score vs LightGBM, same rows | +0.017 | [−0.017, +0.068] | 0.202 |

### Reading

- **`full` is the best single model** and clears the strongest baseline, significant at the
  host level. It becomes the paid API.
- **`text` alone is weaker than telemetry**, but it is the only model above chance on
  cold-start listings and adds a consistent, not yet conclusive, lift when stacked. It is
  the open "shop window" checkpoint.
- **Both Laya models are over-confident above ~0.5.** One temperature cannot fix a
  top-end-only miscalibration. The API applies an isotonic map fitted on calibration hosts
  (`recalibrate.py` → `isotonic_paid7d.json`): ECE 0.110 → 0.079, Brier 0.180 → 0.164, AUC
  unchanged by construction. The open checkpoint ships temperature-only, and its card says so.
- The held-out calibration-host AUC seen during training (0.85 `text`, 0.91 `full`) is much
  higher than test. Those rows share training *dates*. Only the time + host wall is honest.

### Decision (2026-09-23)

| Checkpoint | Release |
|---|---|
| `text` | **public** HF model repo + Space (CPU) |
| `full` + isotonic map | **private** — served as a pay-per-call x402 endpoint from the SentEdge server; weights backed up to a private HF repo |

## 8. Limitations

- **31 days of history.** One clean 14-day test window, and a single month of seasonality.
- **Paid ≠ legitimate.** Keepalive self-calls and wash traffic count as paid. Roughly 9.8k
  listings show ~1 call per payer, a pattern consistent with scripted callers. A "wash"
  label needs payer-level data the index does not expose.
- **Index-specific.** Trained on the CDP Bazaar only; other x402 directories may differ.
- **Distribution drift.** The x402 market is weeks old and changing fast; retrain monthly.

## 9. Reproduce

```bash
python -m venv .venv && .venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cu130
.venv/bin/pip install -r requirements.txt
.venv/bin/python build_dataset.py --snap-dir /path/to/bazaar/snapshots   # -> data/rows.parquet
.venv/bin/python baselines.py                                           # -> runs/baselines.json
USE_TF=0 .venv/bin/python train_laya.py --variant text --epochs 3
USE_TF=0 .venv/bin/python train_laya.py --variant full --epochs 3 --anchors-per-resource 2
USE_TF=0 .venv/bin/python evaluate.py --ckpt runs/laya-text
USE_TF=0 .venv/bin/python evaluate.py --ckpt runs/laya-full
.venv/bin/python make_figures.py                                        # -> results/
```