docs/PROCESS.md

x402 Demand — how the model was built

This is the full record of how the x402-demand checkpoints were made: the source data, the label and why it is shaped that way, the splits, the baselines a checkpoint had to beat, the training recipe, the results, and what was published. Written so the run can be repeated, extended, or challenged.

Owner: Ryan Price (Spark Intelligence Station). Started 2026-09-23.

---

0. Publication policy

Artifact	Published?	Where
Checkpoints (weights, tokenizer, config)	yes	HF model repo
Metrics (results/*.json) and figures (results/*.png)	yes	HF model card
Demo app	yes	HF Space
This document, the training/eval code	on the model repo (as docs), no public git	—
The dataset (snapshots, data/rows.parquet, per-row predictions)	NO	stays on the Spark box

The dataset is deliberately withheld. The daily snapshots are the lab's own longitudinal record of the CDP Bazaar and are the durable asset; the model card describes the data in aggregate only. data/ and runs/ are gitignored so nothing row-level can be committed by accident. The figures show aggregate rates and scores, never rows. The demo's example listings are invented, not drawn from the data.

---

1. Source data

• What: the Coinbase CDP x402 discovery index (the "Bazaar"), GET https://api.cdp.coinbase.com/platform/v2/x402/discovery/resources, fetched once a day, fully paginated (1,000/page), by the lab's Scout:Bazaar (spark-lab/agents/scouts.py). Each daily file is vault/raw/bazaar/YYYY-MM-DD.json; this project reads only its cdp list.
• Per listing: resource (URL), type, description, price (USDC atomic units, 6 decimals), network, lastUpdated, and quality = {l30DaysTotalCalls, l30DaysUniquePayers, lastCalledAt}.
• Window used for v1: 31 snapshots, 2026-08-24 → 2026-09-23. 14.2k-16.2k listings per day; 26,642 distinct resources seen; 2,464 hosts in the modelled rows.
• Completeness check: daily counts are stable (no truncated pages). 1,297 resources (4.9%) disappear and reappear — "flicker". The label below is built from lastCalledAt timestamps rather than from presence, so it is robust to that.

The fact that shaped the label

The index evicts a listing exactly 30 days after its quality.lastCalledAt (measured to the second on the lab's own listing on 2026-09-12; lastUpdated does not matter). So "will this listing still be in the index in N days?" is mostly already decided by a timestamp we can read: anything called in the last 30-N days survives by rule. A model trained on survival would mostly learn that arithmetic.

2. The label

paid_Hd = 1 iff the listing receives at least one paid call in the H days after the anchor snapshot, i.e. some later snapshot shows lastCalledAt ∈ (ref_t, ref_t + H].

• ref_t is the snapshot's reference time = the newest lastCalledAt in it (≈ fetch time).
• A listing that is evicted, or stays listed without a new call, is 0.
• H ∈ {7, 14}. A horizon is labelled only if the whole window (+1 day of slack) is observed.
• Result: 330,035 labelled rows (paid_7d, anchors 0-21), base rate 0.240; 241,825 rows for paid_14d (anchors 0-15), base rate 0.324. In words: only a quarter of listed x402 services get paid even once in a week.

What the label is not: it is not "is this a legitimate service". A self-call to keep a listing alive counts as a paid call. See §7.

3. Features (all computed from snapshots ≤ t — no look-ahead)

Listing: URL, host, path depth, path parameter present, description (+ length), price in USD, network. Telemetry: 30-day calls and unique payers, calls per payer, days since last paid call, days since listing updated, days observed in the index (with a left-censoring flag for listings already present on day 0), new-listing flag, presence and payer/call deltas vs 7 days earlier. Host aggregates at t: number of listings, share of the host's listings called in the last 7 days, share with exactly one call per payer, median payers, networks used.

4. Splits — two walls, both enforced in common.py

1. Host wall. Hosts are hashed (sha256) into 100 buckets. Buckets 80-99 are test hosts, never seen in training. Many hosts mass-list hundreds of near-identical endpoints (one lists 840); without this wall a model memorises the host.
2. Time wall. A training row's label window must end before the first test anchor, so no training label observes the test period.
   • paid_7d: train anchors 0-9, validation anchor 10, test anchors 17-21.
   • paid_14d: train anchor 0, validation anchor 1, test anchor 15 (31 days only allows one clean 14-day window; this horizon is secondary until more history exists).
3. Calibration hosts. For the neural models, train-side buckets 70-79 are held out from training and used for model selection and temperature fitting, so calibration is measured on hosts the model has not seen.

Test set for paid_7d: 13,644 rows over 442 unseen hosts.

Segments reported everywhere: all, new_listings (first seen at the anchor: the cold start), small_hosts(<=20) (hosts with ≤ 20 listings: the long tail, less factory noise), stale(>7d since call).

5. Baselines (baselines.py) — what a checkpoint has to beat

paid_7d, test	all AUC	new listings AUC	small hosts AUC	ECE (all)
Recency rule (isotonic on days since last call)	0.635	0.416	0.696	0.095
Logistic regression, numeric features	0.678	0.500	0.731	0.052
LightGBM, numeric + network	0.672	0.514	0.737	0.076

Top LightGBM features by gain: 30-day calls, host listing count, host 7-day activity, days since update, host one-call-per-payer share, price.

Reading: telemetry gets you to AUC ~0.68 on unseen hosts. Brand-new listings are at chance for every telemetry model: they have no history, and their first "call" is often the settle that listed them.

6. Neural model (train_laya.py)

• Base: convaiinnovations/laya (ModernBERT-large encoder + decision head, 421M, Apache-2.0). Output is a Laya checkpoint: it loads with laya.load() and answers typed questions.
• Questions trained: two noul (yes/no probability) questions, paid_7d and paid_14d, with the instructions in train_laya.py:QUESTIONS. Use the same wording at inference.
• Variants (only the input state differs):
  • text: {resource, description, price_usd, network}. Works for any listing, including one not yet listed. The text is identical across a resource's anchors, so each resource is one item whose soft target = the fraction of its training anchors that were paid.
  • full: the listing plus telemetry as text (paid_calls_30d, unique_payers_30d, days_since_last_paid_call, days_since_listing_updated, host_listing_count, host_share_called_in_last_7d). Two anchors are sampled per resource, with hard targets.
• Objective: a single-GPU port of Laya's upstream RLCD recipe. REINFORCE on a strictly proper scoring rule (log + 0.75 spherical) with a group-mean baseline (G=4, σ 0.4→0.1), plus a soft cross-entropy term. AdamW with encoder lr 2.5e-5 and head lr 1e-4, weight decay 0.01, warmup and cosine, micro-batch 16 × accumulation 4, bf16 autocast, gradient checkpointing, max_len 384 (head budget 192), 3 epochs. The best epoch by held-out calibration-host AUC is kept.
• Calibration: one temperature for noul, fitted by LBFGS on the calibration hosts, written to rl_agent_config.json where laya.Agent applies it.
• Hardware: DGX Spark GB10, ~8.5 GB of GPU memory per run.

7. Results

All numbers are on the held-out test set (future anchors × unseen hosts). Source files: runs/laya-*/eval.json, runs/bootstrap_full_vs_baselines.json, published as results/metrics.json.

paid_7d, ROC AUC

Model	All (n=13,644)	New listings (n=221)	Small hosts ≤20 (n=6,135)	ECE (all)
Recency rule	0.635	0.416	0.696	0.095
Logistic regression	0.678	0.500	0.731	0.052
LightGBM	0.672	0.514	0.737	0.076
Laya · listing only (text)	0.639	0.575	0.668	0.116
Laya · listing + telemetry (full)	0.732	0.581	0.754	0.110 → 0.079 isotonic

paid_14d (one test anchor, n=2,631): LightGBM 0.652, text 0.653, full 0.733.

Significance (paired bootstrap, resampling HOSTS, 1,000 draws)

Rows within a host are strongly correlated, so resampling rows would overstate certainty.

Comparison	median ΔAUC	95% CI	P(Δ ≤ 0)
full vs LightGBM	+0.058	[+0.010, +0.110]	0.014
full vs text	+0.092	[+0.051, +0.141]	0.000
LightGBM + text score vs LightGBM, same calibration-host rows	+0.027	[−0.008, +0.073]	0.076
… new listings only	+0.080	[−0.002, +0.153]	0.026
LightGBM + full score vs LightGBM, same rows	+0.017	[−0.017, +0.068]	0.202

Reading

• full is the best single model and clears the strongest baseline, significant at the host level. It becomes the paid API.
• text alone is weaker than telemetry, but it is the only model above chance on cold-start listings and adds a consistent, not yet conclusive, lift when stacked. It is the open "shop window" checkpoint.
• Both Laya models are over-confident above ~0.5. One temperature cannot fix a top-end-only miscalibration. The API applies an isotonic map fitted on calibration hosts (recalibrate.py → isotonic_paid7d.json): ECE 0.110 → 0.079, Brier 0.180 → 0.164, AUC unchanged by construction. The open checkpoint ships temperature-only, and its card says so.
• The held-out calibration-host AUC seen during training (0.85 text, 0.91 full) is much higher than test. Those rows share training dates. Only the time + host wall is honest.

Decision (2026-09-23)

Checkpoint	Release
text	public HF model repo + Space (CPU)
full + isotonic map	private — served as a pay-per-call x402 endpoint from the SentEdge server; weights backed up to a private HF repo

8. Limitations

• 31 days of history. One clean 14-day test window, and a single month of seasonality.
• Paid ≠ legitimate. Keepalive self-calls and wash traffic count as paid. Roughly 9.8k listings show ~1 call per payer, a pattern consistent with scripted callers. A "wash" label needs payer-level data the index does not expose.
• Index-specific. Trained on the CDP Bazaar only; other x402 directories may differ.
• Distribution drift. The x402 market is weeks old and changing fast; retrain monthly.

9. Reproduce

python -m venv .venv && .venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cu130
.venv/bin/pip install -r requirements.txt
.venv/bin/python build_dataset.py --snap-dir /path/to/bazaar/snapshots   # -> data/rows.parquet
.venv/bin/python baselines.py                                           # -> runs/baselines.json
USE_TF=0 .venv/bin/python train_laya.py --variant text --epochs 3
USE_TF=0 .venv/bin/python train_laya.py --variant full --epochs 3 --anchors-per-resource 2
USE_TF=0 .venv/bin/python evaluate.py --ckpt runs/laya-text
USE_TF=0 .venv/bin/python evaluate.py --ckpt runs/laya-full
.venv/bin/python make_figures.py                                        # -> results/