BaseDecision-Dyno / SERVING.md
onlyaady's picture
Publish calibrated BaseDecision-Dyno with portable HTTP serving
3e88dcc verified
|
Raw History Blame Contribute Delete
3.07 kB
# Serve BaseDecision-Dyno
The server is included in the model download. It needs Python and a compatible
NVIDIA CUDA/PyTorch environment; no HPC account, project checkout, or project
environment variables are required. The tested stack used Python 3.12,
PyTorch 2.14.0+cu130, Transformers 5.17.0 and PEFT 0.20.0. The BF16 checkpoint
also needs accelerate and safetensors. This is the CUDA serving profile.
Download and start:
```bash
hf download onlyaady/BaseDecision-Dyno --local-dir BaseDecision-Dyno
python BaseDecision-Dyno/serve_typesafe.py
```
The model loads from the directory containing `serve_typesafe.py`, regardless
of the current working directory. Default: read twice, with its fitted
temperature applied internally. The local endpoint is http://127.0.0.1:8000.
Send a request from another terminal:
```bash
curl http://127.0.0.1:8000/v1/systemone \
-H 'Content-Type: application/json' \
-d '{"model":"onlyaady/BaseDecision-Dyno","state":"The invoice has a duplicated charge.","questions":{"decision":{"type":"choice","instructions":"Which team should handle this?","criteria":{"billing":"Invoices and refunds","technical":"Integration problems"}}}}'
```
Single-read flash mode requires explicit selection:
```bash
python BaseDecision-Dyno/serve_typesafe.py --mode flash
```
Flash uses its own fitted temperature and identifies itself as
`onlyaady/BaseDecision-Dyno-flash`. A server process has a fixed mode.
The request may use `onlyaady/BaseDecision-Dyno` or `BaseDecision-Dyno` as an
alias for the model that process serves; the response reports the actual mode.
Optional arguments: `--model /path/to/snapshot`, `--model onlyaady/BaseDecision-Dyno`,
`--revision COMMIT_SHA`, `--gpu 0`, `--host 0.0.0.0`, `--port 8000`.
`GET /health` shows readiness, mode, calibration status and release id.
`GET /v1/models` lists the served model.
The server supports choice (2–255 options), noul, and score (2–10 ordered levels).
It returns the model's probabilities; noul is the probability under `true`, and
score is the expected zero-based level. It does not generate answer text.
The limit is 16,384 rendered tokens per question, including repetition. Over-limit
requests return HTTP 400/max_tokens_exceeded. There is no truncation or automatic
flash fallback. Questions are independent forward passes. One request is processed
at a time; concurrent requests receive HTTP 529. JSON and gzip JSON are accepted.
For remote access, bind an appropriate host, supply your HTTPS reverse proxy, and
optionally set DYNO_API_KEY before startup; clients then send an Authorization:
Bearer header. This optional API key is the only project-named environment setting.
The server does not log request bodies or provision public hosting.
Benchmark Heaven can use its published `typesafe` adapter, base endpoint
http://127.0.0.1:8000, model `onlyaady/BaseDecision-Dyno`, and no key for a local
server without authentication. Pin the Hugging Face commit for an evaluation.
The evaluator still needs to review the adapter and run its official evaluation.