keel / docs /COMPUTE.md
Tobi
Upload folder using huggingface_hub
32a505c verified
|
Raw History Blame Contribute Delete
8.69 kB
# Compute and providers
> **Status:** historical planning document. For what actually exists today, see
> [STATUS.md](STATUS.md).
Three credits, three different things. Only one of them runs a GPU you control.
| credit | what it actually is | what it buys | what it cannot do |
|---|---|---|---|
| **Baseten** $100 | inference on one hosted model | tokens on `meta-models/muse-spark-open-1026`, up to 2M TPM | training, weights, activations, Glimmer, or any other model |
| **Meta Model API** $150 | tokens | Muse Spark 1.1–1.3, Muse Wonder, Muse Image, SAM 3.1 | weights |
| **Modal** $30 | GPU seconds | ~12 A100-80GB hours, or ~7 H100 hours | a comfortable 30B training run |
The Baseten key is scoped to one model name. It is a **black-box inference grant**,
not a compute grant: no `from_pretrained`, no hooks, no residuals. That is fine for
the black-box channel and useless for the white-box one.
---
## 1. Answering "is it enough"
**Enough for:** the black-box channel on Spark Open, the judge, and a *small-model*
white-box channel. Approximately $65 of the $280 in credits, no out-of-pocket spend.
**Not enough for:** the white-box channel on Glimmer 30B. That needs a 30B model on
a GPU you control with hooks in the same process, about $35 of Vast, which none of
these credits cover.
So there are two honest paths.
### Path A, fully funded (recommended)
Run the white-box channel on a small Meta checkpoint (Llama-3.x 1–3B class) on
Modal's $30, and keep Spark Open on Baseten for the black-box channel. The method
is identical; the model is smaller. Cost fits inside the credits with margin.
| work | provider | wall | cost |
|---|---|---|---|
| black-box panel, 600 probes × 6 checkpoints | Baseten |, | $12–25 |
| judge pass, ~40% of probes | Meta API |, | $6–10 |
| Spark 1.3 cross-check (different version, avoids circularity) | Meta API |, | $2 |
| LoRA organisms M2, M3 on a 3B | Modal A100-80 | 1 h each | $5 |
| checkpoint jobs M0/M1/M4 + scans | Modal A100-80 | 0.5 h × 5 | $6.25 |
| attribution, 2 positives | Modal A100-80 | 2 h | $5 |
| demo endpoint | Modal A100-80 | 2 h | $5 |
| **total** | | | **~$49** |
### Path B, Glimmer 30B (stretch)
Same design with the 30B as the white-box subject. Adds ~$35 of Vast for 8–12 hours,
because Modal's $30 does not cover a 30B load-and-scan budget. Worth it only if the
extra scale is the story you want to tell.
A useful compromise: build the adapter against Glimmer, run the funded experiments on
the 3B, and upgrade if time and money allow. The adapter is architecture-generic; the
science does not depend on the parameter count.
## 2. The constraint that shapes execution
**Dedicated inference returns tokens, not residuals.** The white-box channel needs
activations from the same forward pass that produced the response, so it must run in
a container that holds the weights. Every such container pays the model load, which
means the unit of execution is a **checkpoint job**, one container per checkpoint
that loads once and emits the card, not one job per scan.
## 3. Baseten: black-box on Spark Open
```
base_url https://inference.baseten.co/v1
model meta-models/muse-spark-open-1026
```
OpenAI-compatible, so `keel/api.py` works unchanged with a base-URL and model
override. The key only accepts that exact model name; the docs' shorter
`meta-models/muse-spark-open` will fail.
Measured prices on this key: **$1.25/M prompt, $4.25/M completion, $0.15/M cached
prompt**. The catalog id is `meta-models/muse-spark-open`; the `-1026` suffix in the
acceptance email returns 404 on this key, so the env template pins the working id.
### Reasoning effort: a real lever, a noisy one
Spark Open always reasons. Baseten's table says the model has no reasoning control,
but `reasoning_effort` demonstrably changes output length. Three runs per setting,
same prompt:
| effort | completion tokens (3 runs) | mean |
|---|---|---|
| default | 311, 112, 404 | 276 |
| minimal | 202, 231, 38 | 157 |
| low | 77, 68, 97 | **81** |
`low` is reliably cheapest; `minimal` overlaps the default range and is not worth
using. The variance is large, treat any single measurement as noise.
### Four constraints that shape the black-box channel
1. **`logprobs` and `top_logprobs` are accepted but ignored**, so there is no
token-distribution drift to measure. The black-box channel scores behaviour only.
2. **`n > 1` returns HTTP 400.** One response per request; sample by issuing separate
requests.
3. **Neither `seed` nor `temperature: 0` guarantees reproducibility.** Sampling noise
is a real variance component, which is why the panel needs enough probes rather
than enough repetitions of a few.
4. **No structured outputs and no JSON mode**, Spark Open is the only model on the
platform without them. The judge scores through tool calling (all models support
it) or a strict parser, never a schema it cannot enforce.
### Cost, and the two-tier panel
The measured figure is about **$11.50 for a 600-probe sweep**, not the $2.40 extrapolated
from a one-line answer. Five full sweeps would consume most of the $100, so the panel
runs in two tiers.
| pass | panel | checkpoints | cost |
|---|---|---|---|
| baseline | 200 probes | M0 through M4 | 5 x ~$4 = ~$20 |
| confirmatory | 600 probes | M0, M1, M2, M3 | 4 x ~$11.50 = ~$46 |
| re-runs and failures | | | ~$15 |
| **Baseten total** | | | **~$81 of $100** |
The 200-probe pass finds out whether anything moved. The 600-probe pass is for the
checkpoints whose numbers get published, and for tightening an interval that came back
ambiguous. Shrinking the baseline pass is the lever that preserves the credit.
**White-box attribution does not come out of the Baseten credit.** Baseten is inference
on Spark Open only. Attribution and white-box scans run on Modal's $30.
| work | hours | cost |
|---|---|---|
| white-box scans, M0, M1, M2 | 1.5 h A100 | ~$3.75 |
| attribution, M1 and M2 | 3 h A100 | ~$7.50 |
| demo endpoint | 2 h A100 | ~$5 |
| **Modal total** | | **~$16 of $30** |
That leaves about $14 of Modal headroom for a failed job or a longer attribution search,
and keeps each credit doing the job it was granted for.
Training is $0 on Kaggle, so the credit pressure is entirely on the scanning side.
## 4. Meta Model API: the judge
| item | tokens | cost |
|---|---|---|
| judge calls on the subjective axes (~40% of 3,600 responses) | ~1.4M | ~$6 |
| Spark 1.3 as the cross-model judge (different version from the target) | ~0.3M | ~$1.50 |
| buffer for re-runs |, | ~$20 |
| **committed** | | **~$28** |
Judging with the same model and version you are testing is circular. Judging Spark
Open 1026 with Spark 1.3 is defensible: different checkpoint, same family. Say so in
the card.
## 5. Modal: the GPU
Verified 26 Oct 2026: the token launches a `Tesla T4, 15360 MiB` in workspace
`iamtrillion11`, and the billing summary reported `$0.00` metered before that run.
The CLI does not expose a credit balance, so entitlement is proven by launching,
not by reading a number.
$30 is ~12 A100-80GB hours at $2.50, ~50 T4-hours at $0.59, or ~37 L4-hours at $0.80.
That is enough for a 3B white-box run and a demo, not for a 30B training run.
A T4's 15 GB fits a 3B in BF16 or a 3B in 4-bit for LoRA; it is slow, so use it for
scans and dev loops rather than training.
Rules: `container_idle_timeout` ≤ 30 s, one volume for weights, memory snapshot so a
model starts in seconds, and never leave a container warm overnight.
## 6. What a scan costs
| scenario | GPU | tokens | total |
|---|---|---|---|
| black-box, 600 probes, Spark Open | $0 | ~$2.30 | **~$2.30** |
| white-box 3B, cold job | $1.25 | ~$2.30 | **~$3.55** |
| white-box 30B, cold job | $2.00 | ~$2.30 | **~$4.30** |
The honest product claim is **under $5 per checkpoint**. The card prints GPU and
token cost separately so the number stays auditable.
## 7. Day-0 checks
1. Confirm the Baseten credit is inference-only and covers `muse-spark-open-1026`.
2. Confirm the key works from your region.
3. ~~Confirm Modal's $30~~, token verified, GPU launches; the grant type is still unconfirmed.
4. Export a spend cap on all three accounts before the first job.
5. Run `python scripts/muse_smoke.py --meta` and `--baseten` before any task.
## 8. Fallbacks
| failure | response |
|---|---|
| Baseten quota exhausted | shift the black-box channel to Meta API Spark 1.3; the panel is portable |
| Modal grant is monthly | run the 3B white-box on a single Vast A100 for ~$3 |
| 3B too weak to show drift | go to Path B and spend ~$35 on Vast for Glimmer |
| Everything slower than planned | cut harden, then M5, then composition, never M3/M4 |