Instructions to use BricksDisplay/jevling-e2b-v1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use BricksDisplay/jevling-e2b-v1-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf BricksDisplay/jevling-e2b-v1-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf BricksDisplay/jevling-e2b-v1-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf BricksDisplay/jevling-e2b-v1-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf BricksDisplay/jevling-e2b-v1-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf BricksDisplay/jevling-e2b-v1-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf BricksDisplay/jevling-e2b-v1-GGUF:Q4_K_M
Use Docker
docker model run hf.co/BricksDisplay/jevling-e2b-v1-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use BricksDisplay/jevling-e2b-v1-GGUF with Ollama:
ollama run hf.co/BricksDisplay/jevling-e2b-v1-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use BricksDisplay/jevling-e2b-v1-GGUF with Docker Model Runner:
docker model run hf.co/BricksDisplay/jevling-e2b-v1-GGUF:Q4_K_M
- Lemonade
How to use BricksDisplay/jevling-e2b-v1-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull BricksDisplay/jevling-e2b-v1-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.jevling-e2b-v1-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
card: document the /v1/systemone path and its build requirement
Browse files
README.md
CHANGED
|
@@ -26,11 +26,46 @@ build/bin/llama-system-one -m jevling-e2b-v1-q8_0-embf16.gguf \
|
|
| 26 |
--noul "refund:Is the customer asking for a refund?" \
|
| 27 |
--score "urgency:How urgent is this?:routine,soon,urgent,critical" --json
|
| 28 |
```
|
| 29 |
-
Answers come back as probabilities per option (`--json` is the same shape the server returns: `noul` P(true), `choice` with a distribution, `score` as the expected level). Pass option **descriptions** — the model was trained with them. Use `--system-one-request request.json` for anything with colons or many questions; `llama-system-one-batch` for throughput. The prompt template and readout live inside the GGUF (`tokenizer.chat_template.system_one`
|
| 30 |
|
| 31 |
Measured: ≈1.7 s per 5-question request on 16 CPU threads, ≈2.9 GB RSS; run with `--swa-full`. Keep flash-attention off on CPU and threads = physical cores.
|
| 32 |
|
| 33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
## Evaluation
|
| 35 |
All numbers are accuracy on datasets the model was **not** trained on, asked in the System One format (all questions of an item in one request; option descriptions given where the dataset has them). Items per dataset: 120–150 unless noted.
|
| 36 |
|
|
|
|
| 26 |
--noul "refund:Is the customer asking for a refund?" \
|
| 27 |
--score "urgency:How urgent is this?:routine,soon,urgent,critical" --json
|
| 28 |
```
|
| 29 |
+
Answers come back as probabilities per option (`--json` is the same shape the server returns: `noul` P(true), `choice` with a distribution, `score` as the expected level). Pass option **descriptions** — the model was trained with them. Use `--system-one-request request.json` for anything with colons or many questions; `llama-system-one-batch` for throughput. The prompt template and readout live inside the GGUF (`tokenizer.chat_template.system_one`; the readout is derived from the model, not declared); do not supply a chat template of your own.
|
| 30 |
|
| 31 |
Measured: ≈1.7 s per 5-question request on 16 CPU threads, ≈2.9 GB RSS; run with `--swa-full`. Keep flash-attention off on CPU and threads = physical cores.
|
| 32 |
|
| 33 |
|
| 34 |
+
## Or the `/v1/systemone` endpoint
|
| 35 |
+
|
| 36 |
+
This file also answers upstream llama.cpp's decision-model endpoint, which needs a build that
|
| 37 |
+
tokenizes the prompt **piece by piece** — the boundaries this model was trained on. That is one
|
| 38 |
+
commit on top of upstream master:
|
| 39 |
+
|
| 40 |
+
```bash
|
| 41 |
+
git clone -b system-one/decision-segments https://github.com/mybigday/system-one-llama.cpp
|
| 42 |
+
cd system-one-llama.cpp && cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build --target llama-server -j
|
| 43 |
+
build/bin/llama-server -m jevling-e2b-v1-q8_0-embf16.gguf -c 2048 -t $(nproc) --n-outputs-max-per-seq 16
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
```bash
|
| 47 |
+
curl http://localhost:8080/v1/systemone -H "Content-Type: application/json" -d '{
|
| 48 |
+
"state": "Customer: I was charged twice and nobody replied.",
|
| 49 |
+
"questions": {
|
| 50 |
+
"queue": {"type": "choice", "instructions": "Which team should handle this?",
|
| 51 |
+
"criteria": {"billing": "payments and refunds", "technical": "a product fault"}},
|
| 52 |
+
"refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"},
|
| 53 |
+
"urgency": {"type": "score", "instructions": "How urgent is this?",
|
| 54 |
+
"criteria": ["routine", "soon", "urgent", "critical"]}
|
| 55 |
+
}
|
| 56 |
+
}'
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
**Stock llama.cpp must not be used for this.** It will load the file and answer, and its answers
|
| 60 |
+
will be wrong without saying so: a single pass over the whole prompt merges across one of the
|
| 61 |
+
training boundaries and comes out one token shorter, which moves a probability by up to **0.28**
|
| 62 |
+
and changes a few decisions in a thousand. Nothing is raised, because the separator the template
|
| 63 |
+
writes is simply undefined there and renders as empty. The branch above is a requirement, not a
|
| 64 |
+
suggestion.
|
| 65 |
+
|
| 66 |
+
Measured against the fp32 reference these weights were validated on, 255 states / 1275 answers:
|
| 67 |
+
its f32 weights read 1.5e-03 worst-case probability deviation from that reference with **0 of 1275** argmax differences — the figure is larger than the 0.8B's because gemma-4 uses GELU and llama.cpp's CPU GELU goes through an fp16 table, which changes no decision and is upstream's own accepted behaviour, and this file's decision agreement is noul 1.0000 · choice 1.0000 · score 1.0000, accuracy unchanged on all three, **0** flips.
|
| 68 |
+
|
| 69 |
## Evaluation
|
| 70 |
All numbers are accuracy on datasets the model was **not** trained on, asked in the System One format (all questions of an item in one request; option descriptions given where the dataset has them). Items per dataset: 120–150 unless noted.
|
| 71 |
|