hans00 commited on
Commit
9ae1ae8
·
verified ·
1 Parent(s): b28c529

card: document the /v1/systemone path and its build requirement

Browse files
Files changed (1) hide show
  1. README.md +36 -1
README.md CHANGED
@@ -26,11 +26,46 @@ build/bin/llama-system-one -m jevling-e2b-v1-q8_0-embf16.gguf \
26
  --noul "refund:Is the customer asking for a refund?" \
27
  --score "urgency:How urgent is this?:routine,soon,urgent,critical" --json
28
  ```
29
- Answers come back as probabilities per option (`--json` is the same shape the server returns: `noul` P(true), `choice` with a distribution, `score` as the expected level). Pass option **descriptions** — the model was trained with them. Use `--system-one-request request.json` for anything with colons or many questions; `llama-system-one-batch` for throughput. The prompt template and readout live inside the GGUF (`tokenizer.chat_template.system_one`, `system_one.readout`); do not supply a chat template of your own.
30
 
31
  Measured: ≈1.7 s per 5-question request on 16 CPU threads, ≈2.9 GB RSS; run with `--swa-full`. Keep flash-attention off on CPU and threads = physical cores.
32
 
33
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
34
  ## Evaluation
35
  All numbers are accuracy on datasets the model was **not** trained on, asked in the System One format (all questions of an item in one request; option descriptions given where the dataset has them). Items per dataset: 120–150 unless noted.
36
 
 
26
  --noul "refund:Is the customer asking for a refund?" \
27
  --score "urgency:How urgent is this?:routine,soon,urgent,critical" --json
28
  ```
29
+ Answers come back as probabilities per option (`--json` is the same shape the server returns: `noul` P(true), `choice` with a distribution, `score` as the expected level). Pass option **descriptions** — the model was trained with them. Use `--system-one-request request.json` for anything with colons or many questions; `llama-system-one-batch` for throughput. The prompt template and readout live inside the GGUF (`tokenizer.chat_template.system_one`; the readout is derived from the model, not declared); do not supply a chat template of your own.
30
 
31
  Measured: ≈1.7 s per 5-question request on 16 CPU threads, ≈2.9 GB RSS; run with `--swa-full`. Keep flash-attention off on CPU and threads = physical cores.
32
 
33
 
34
+ ## Or the `/v1/systemone` endpoint
35
+
36
+ This file also answers upstream llama.cpp's decision-model endpoint, which needs a build that
37
+ tokenizes the prompt **piece by piece** — the boundaries this model was trained on. That is one
38
+ commit on top of upstream master:
39
+
40
+ ```bash
41
+ git clone -b system-one/decision-segments https://github.com/mybigday/system-one-llama.cpp
42
+ cd system-one-llama.cpp && cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build --target llama-server -j
43
+ build/bin/llama-server -m jevling-e2b-v1-q8_0-embf16.gguf -c 2048 -t $(nproc) --n-outputs-max-per-seq 16
44
+ ```
45
+
46
+ ```bash
47
+ curl http://localhost:8080/v1/systemone -H "Content-Type: application/json" -d '{
48
+ "state": "Customer: I was charged twice and nobody replied.",
49
+ "questions": {
50
+ "queue": {"type": "choice", "instructions": "Which team should handle this?",
51
+ "criteria": {"billing": "payments and refunds", "technical": "a product fault"}},
52
+ "refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"},
53
+ "urgency": {"type": "score", "instructions": "How urgent is this?",
54
+ "criteria": ["routine", "soon", "urgent", "critical"]}
55
+ }
56
+ }'
57
+ ```
58
+
59
+ **Stock llama.cpp must not be used for this.** It will load the file and answer, and its answers
60
+ will be wrong without saying so: a single pass over the whole prompt merges across one of the
61
+ training boundaries and comes out one token shorter, which moves a probability by up to **0.28**
62
+ and changes a few decisions in a thousand. Nothing is raised, because the separator the template
63
+ writes is simply undefined there and renders as empty. The branch above is a requirement, not a
64
+ suggestion.
65
+
66
+ Measured against the fp32 reference these weights were validated on, 255 states / 1275 answers:
67
+ its f32 weights read 1.5e-03 worst-case probability deviation from that reference with **0 of 1275** argmax differences — the figure is larger than the 0.8B's because gemma-4 uses GELU and llama.cpp's CPU GELU goes through an fp16 table, which changes no decision and is upstream's own accepted behaviour, and this file's decision agreement is noul 1.0000 · choice 1.0000 · score 1.0000, accuracy unchanged on all three, **0** flips.
68
+
69
  ## Evaluation
70
  All numbers are accuracy on datasets the model was **not** trained on, asked in the System One format (all questions of an item in one request; option descriptions given where the dataset has them). Items per dataset: 120–150 unless noted.
71