keel / docs /COMPUTE.md
Tobi
Upload folder using huggingface_hub
32a505c verified
|
Raw History Blame Contribute Delete
8.69 kB

Compute and providers

Status: historical planning document. For what actually exists today, see STATUS.md.

Three credits, three different things. Only one of them runs a GPU you control.

credit what it actually is what it buys what it cannot do
Baseten $100 inference on one hosted model tokens on meta-models/muse-spark-open-1026, up to 2M TPM training, weights, activations, Glimmer, or any other model
Meta Model API $150 tokens Muse Spark 1.1–1.3, Muse Wonder, Muse Image, SAM 3.1 weights
Modal $30 GPU seconds ~12 A100-80GB hours, or ~7 H100 hours a comfortable 30B training run

The Baseten key is scoped to one model name. It is a black-box inference grant, not a compute grant: no from_pretrained, no hooks, no residuals. That is fine for the black-box channel and useless for the white-box one.


1. Answering "is it enough"

Enough for: the black-box channel on Spark Open, the judge, and a small-model white-box channel. Approximately $65 of the $280 in credits, no out-of-pocket spend.

Not enough for: the white-box channel on Glimmer 30B. That needs a 30B model on a GPU you control with hooks in the same process, about $35 of Vast, which none of these credits cover.

So there are two honest paths.

Path A, fully funded (recommended)

Run the white-box channel on a small Meta checkpoint (Llama-3.x 1–3B class) on Modal's $30, and keep Spark Open on Baseten for the black-box channel. The method is identical; the model is smaller. Cost fits inside the credits with margin.

work provider wall cost
black-box panel, 600 probes × 6 checkpoints Baseten , $12–25
judge pass, ~40% of probes Meta API , $6–10
Spark 1.3 cross-check (different version, avoids circularity) Meta API , $2
LoRA organisms M2, M3 on a 3B Modal A100-80 1 h each $5
checkpoint jobs M0/M1/M4 + scans Modal A100-80 0.5 h × 5 $6.25
attribution, 2 positives Modal A100-80 2 h $5
demo endpoint Modal A100-80 2 h $5
total ~$49

Path B, Glimmer 30B (stretch)

Same design with the 30B as the white-box subject. Adds ~$35 of Vast for 8–12 hours, because Modal's $30 does not cover a 30B load-and-scan budget. Worth it only if the extra scale is the story you want to tell.

A useful compromise: build the adapter against Glimmer, run the funded experiments on the 3B, and upgrade if time and money allow. The adapter is architecture-generic; the science does not depend on the parameter count.

2. The constraint that shapes execution

Dedicated inference returns tokens, not residuals. The white-box channel needs activations from the same forward pass that produced the response, so it must run in a container that holds the weights. Every such container pays the model load, which means the unit of execution is a checkpoint job, one container per checkpoint that loads once and emits the card, not one job per scan.

3. Baseten: black-box on Spark Open

base_url  https://inference.baseten.co/v1
model     meta-models/muse-spark-open-1026

OpenAI-compatible, so keel/api.py works unchanged with a base-URL and model override. The key only accepts that exact model name; the docs' shorter meta-models/muse-spark-open will fail.

Measured prices on this key: $1.25/M prompt, $4.25/M completion, $0.15/M cached prompt. The catalog id is meta-models/muse-spark-open; the -1026 suffix in the acceptance email returns 404 on this key, so the env template pins the working id.

Reasoning effort: a real lever, a noisy one

Spark Open always reasons. Baseten's table says the model has no reasoning control, but reasoning_effort demonstrably changes output length. Three runs per setting, same prompt:

effort completion tokens (3 runs) mean
default 311, 112, 404 276
minimal 202, 231, 38 157
low 77, 68, 97 81

low is reliably cheapest; minimal overlaps the default range and is not worth using. The variance is large, treat any single measurement as noise.

Four constraints that shape the black-box channel

  1. logprobs and top_logprobs are accepted but ignored, so there is no token-distribution drift to measure. The black-box channel scores behaviour only.
  2. n > 1 returns HTTP 400. One response per request; sample by issuing separate requests.
  3. Neither seed nor temperature: 0 guarantees reproducibility. Sampling noise is a real variance component, which is why the panel needs enough probes rather than enough repetitions of a few.
  4. No structured outputs and no JSON mode, Spark Open is the only model on the platform without them. The judge scores through tool calling (all models support it) or a strict parser, never a schema it cannot enforce.

Cost, and the two-tier panel

The measured figure is about $11.50 for a 600-probe sweep, not the $2.40 extrapolated from a one-line answer. Five full sweeps would consume most of the $100, so the panel runs in two tiers.

pass panel checkpoints cost
baseline 200 probes M0 through M4 5 x ~$4 = ~$20
confirmatory 600 probes M0, M1, M2, M3 4 x ~$11.50 = ~$46
re-runs and failures ~$15
Baseten total ~$81 of $100

The 200-probe pass finds out whether anything moved. The 600-probe pass is for the checkpoints whose numbers get published, and for tightening an interval that came back ambiguous. Shrinking the baseline pass is the lever that preserves the credit.

White-box attribution does not come out of the Baseten credit. Baseten is inference on Spark Open only. Attribution and white-box scans run on Modal's $30.

work hours cost
white-box scans, M0, M1, M2 1.5 h A100 ~$3.75
attribution, M1 and M2 3 h A100 ~$7.50
demo endpoint 2 h A100 ~$5
Modal total ~$16 of $30

That leaves about $14 of Modal headroom for a failed job or a longer attribution search, and keeps each credit doing the job it was granted for.

Training is $0 on Kaggle, so the credit pressure is entirely on the scanning side.

4. Meta Model API: the judge

item tokens cost
judge calls on the subjective axes (~40% of 3,600 responses) ~1.4M ~$6
Spark 1.3 as the cross-model judge (different version from the target) ~0.3M ~$1.50
buffer for re-runs , ~$20
committed ~$28

Judging with the same model and version you are testing is circular. Judging Spark Open 1026 with Spark 1.3 is defensible: different checkpoint, same family. Say so in the card.

5. Modal: the GPU

Verified 26 Oct 2026: the token launches a Tesla T4, 15360 MiB in workspace iamtrillion11, and the billing summary reported $0.00 metered before that run. The CLI does not expose a credit balance, so entitlement is proven by launching, not by reading a number.

$30 is ~12 A100-80GB hours at $2.50, ~50 T4-hours at $0.59, or ~37 L4-hours at $0.80. That is enough for a 3B white-box run and a demo, not for a 30B training run. A T4's 15 GB fits a 3B in BF16 or a 3B in 4-bit for LoRA; it is slow, so use it for scans and dev loops rather than training.

Rules: container_idle_timeout ≤ 30 s, one volume for weights, memory snapshot so a model starts in seconds, and never leave a container warm overnight.

6. What a scan costs

scenario GPU tokens total
black-box, 600 probes, Spark Open $0 ~$2.30 ~$2.30
white-box 3B, cold job $1.25 ~$2.30 ~$3.55
white-box 30B, cold job $2.00 ~$2.30 ~$4.30

The honest product claim is under $5 per checkpoint. The card prints GPU and token cost separately so the number stays auditable.

7. Day-0 checks

  1. Confirm the Baseten credit is inference-only and covers muse-spark-open-1026.
  2. Confirm the key works from your region.
  3. Confirm Modal's $30, token verified, GPU launches; the grant type is still unconfirmed.
  4. Export a spend cap on all three accounts before the first job.
  5. Run python scripts/muse_smoke.py --meta and --baseten before any task.

8. Fallbacks

failure response
Baseten quota exhausted shift the black-box channel to Meta API Spark 1.3; the panel is portable
Modal grant is monthly run the 3B white-box on a single Vast A100 for ~$3
3B too weak to show drift go to Path B and spend ~$35 on Vast for Glimmer
Everything slower than planned cut harden, then M5, then composition, never M3/M4