Add knowline_engine.py (one-command Decision Index runs); --backend hf without accelerate

INFERENCE.md

@@ -95,9 +95,33 @@ decision-index score --suite-dir suite-0.2 --results results.jsonl --engine http
 - NVIDIA H20 (96 GB), driver 590.48.01, one server per GPU.
 - Latency was not measured on the board's reference hardware (1x RTX PRO 6000, latency-v1).
 
+## One-command Decision Index run
+
+`knowline_engine.py` (in this repository) is a Decision Index engine. It runs the same front end in process, so there
+is no server to start by hand:
+
+```bash
+pip install "sglang==0.5.21" "transformers==5.12.1" requests        # plus the decision-index kit
+git clone https://huggingface.co/PelaAI/KnowLine-4B-Gen1 && cd KnowLine-4B-Gen1   # puts both files on the path
+python -m decision_index pipeline --engine knowline_engine:KnowLine \
+    --option model=PelaAI/KnowLine-4B-Gen1 --option revision=<commit> --out runs/KnowLine-4B-Gen1
+```
+
+- **Default backend (the setting of our runs):** the engine starts SGLang with the flags above (FP8 at load) on a free
+  local port and stops it when the run ends. Use `--option gpu=<n>` to pick a GPU, and
+  `--option sglang_python=<python>` if SGLang lives in another environment.
+- **`--option backend=hf`:** transformers only, bf16, no server; slower, and not the setting of our runs.
+- **Limits:** up to 64 questions per request and 255 options per question. Larger requests are reported as
+  unsupported; nothing is truncated.
+- **Checked on Gen2:** on 300 random Decision Index 0.3 rows, the default backend gave the same answer as our published
+  Gen2 run on 706 of 709 questions. The 3 differences are near-ties (top two options within 0.06), from FP8 numerical
+  noise. Requests took 30.7 ms median, one at a time on one H20.
+
 ## Changes
 
 - 2026-10-08: `knowline_server.py` now accepts chats whose roles the chat template rejects (for example `customer` /
   `agent`): such a state is rendered as one user message, like any other structured state, instead of the request
   failing. Any other unexpected error now returns HTTP 500 instead of closing the connection. Requests that worked
   before give the same answers; our Decision Index runs had none that failed.
+- 2026-10-08: added `knowline_engine.py` (one-command Decision Index runs). `--backend hf` of `knowline_server.py` no
+  longer needs `accelerate`: without it, the model is loaded and moved to one device.

SHA256SUMS

@@ -1,8 +1,9 @@
 a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  chat_template.jinja
 ddc63e1c717afa86c865bb5e01313d89d72bb53b97ad4a8a03ba8510c0621670  config.json
 f888421726665e8a84b738eed42a64875aed79de8be7daade851ac8bf4c0cef9  configuration.json
-62be3575c09145de8c8a824831ede917d4138b6c585e4092278fa5888f7aa8cf  INFERENCE.md
-8f2f3b944722ec4a8c1de68e0da9f12eb1feb1d2b02d5d82a152f9652fc70046  knowline_server.py
+176afd91aa41f4b053b3265e523cbdf4945d3601c03843cd8970ea7b527de05c  INFERENCE.md
+6e4e32cfb44747202aa39030f7ac6d0893d2ad5de3e2413c2b8f4788c275132d  knowline_engine.py
+4bf7c0392ff679948012a8db2876b86d7159c83e5c918bbee28c696c0cfb85b4  knowline_server.py
 50cbab8a892c5f2993b8c7351a99182507472def3b1374558308605d99b86b32  LICENSE
 a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d  merges.txt
 77d5d691c4c13ae08eacbddf9d145b2ebdaa0f6e69f4917630eb5a87691f15e5  model-00001-of-00002.safetensors
@@ -11,7 +12,7 @@ e5fc485d419f2c554169c1fc441f57a1a45a50673bdb4e41d3dd0261b21abe7c  model-extra-fr
 48b56300c2b19dcd9a43001c9098098e7eaf4c6bc9f5cdd108148611fdfdca56  model.safetensors.index.json
 27225450ac9c6529872ee1924fcb0962ff5634834f817040f444118116f4e516  preprocessor_config.json
 fd774ff020c02f7c5e032d94be9ae0ea9f139e895a077515a06b4ca509f9afac  serve_knowline.sh
-316230d6a809701f4db5ea8f8fc862bc3a6f3229c937c174e674ff3ca0a64ac8  tokenizer_config.json
 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42  tokenizer.json
+316230d6a809701f4db5ea8f8fc862bc3a6f3229c937c174e674ff3ca0a64ac8  tokenizer_config.json
 7768af27c1fafa9cc9011c1dc20067e03f8915e03b63504550e11d5066986d13  video_preprocessor_config.json
 ce99b4cb2983d118806ce0a8b777a35b093e2000a503ebde25853284c9dfa003  vocab.json

knowline_engine.py

@@ -0,0 +1,138 @@
+"""Decision Index engine for KnowLine (PelaAI): the model repo's `knowline_server.py` front end, in process.
+
+One command, no server to start by hand:
+
+    # default: the setting of our submitted runs. Starts SGLang with the flags of serve_knowline.sh (FP8 at load) on a
+    # free local port, scores through it, and stops it when the run ends.
+    python -m decision_index pipeline --engine knowline_engine:KnowLine \\
+        --option model=PelaAI/KnowLine-4B-Gen2 --option revision=<commit> --out runs/KnowLine-4B-Gen2
+
+    # without SGLang: transformers only, bf16 (slower; not the setting of our runs)
+    ... --option backend=hf
+
+Put this file and `knowline_server.py` (both in the model repo) on PYTHONPATH, e.g. run from a clone of the model repo.
+Requirements: `transformers` and `requests`; `sglang==0.5.21` for the default backend, `torch` for backend=hf.
+Options: model, revision, backend (sglang | hf), gpu (CUDA device for SGLang; default: as CUDA_VISIBLE_DEVICES), mem (0.72), port (free one),
+temperature (1.0), workers (16), startup_timeout (s, 1800), sglang_python (interpreter with SGLang installed, if it is
+not the one running the kit), device (backend=hf: torch device or device map, default auto / cuda).
+Licence of this file: MIT.
+"""
+
+import atexit
+import os
+import signal
+import socket
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+from decision_index.engines.base import Engine, Unsupported
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import knowline_server as ks  # noqa: E402
+
+SGLANG_FLAGS = ["--served-model-name", "m", "--tp", "1", "--quantization", "fp8", "--mamba-radix-cache-strategy",
+                "extra_buffer", "--enable-fp32-lm-head"]
+
+
+def _free_port():
+    with socket.socket() as s:
+        s.bind(("127.0.0.1", 0))
+        return s.getsockname()[1]
+
+
+class KnowLine(Engine):
+    name = "knowline"
+    latency = ("In-process request wall time through knowline_server's KnowLine engine (rendering + one prefill per "
+               "question), against a local SGLang server for backend=sglang; excludes model loading and server startup.")
+
+    def __init__(self, model, revision=None, backend="sglang", gpu=None, mem=0.72, port=None, temperature=1.0,
+                 workers=16, startup_timeout=1800, sglang_python=None, device=None, **options):
+        super().__init__(**options)
+        import requests
+        import transformers
+        from transformers import AutoTokenizer
+
+        path = model
+        if not Path(model).exists():  # a Hub repo id: pin the files once so SGLang and the tokenizer read the same ones
+            from huggingface_hub import snapshot_download
+            path = snapshot_download(model, revision=revision)
+        self.model_id, self.backend_name, self.server = model, backend, None
+        tok = AutoTokenizer.from_pretrained(path)
+        if backend == "sglang":
+            port = int(port or _free_port())
+            env = dict(os.environ)
+            if gpu is not None:
+                env["CUDA_VISIBLE_DEVICES"] = str(gpu)
+            cmd = [sglang_python or sys.executable, "-m", "sglang.launch_server", "--model-path", path, *SGLANG_FLAGS,
+                   "--mem-fraction-static", str(mem), "--port", str(port)]
+            self.server = subprocess.Popen(cmd, env=env, start_new_session=True)
+            atexit.register(self.close)
+            url, deadline = f"http://127.0.0.1:{port}", time.time() + float(startup_timeout)
+            while True:
+                if self.server.poll() is not None:
+                    raise RuntimeError(f"SGLang exited with code {self.server.returncode} before becoming healthy")
+                try:
+                    if requests.get(f"{url}/health", timeout=3).status_code == 200:
+                        break
+                except requests.RequestException:
+                    pass
+                if time.time() > deadline:
+                    raise RuntimeError("SGLang did not become healthy in time")
+                time.sleep(5)
+            back = ks.SGLang(url)
+        elif backend == "hf":
+            back = ks.HF(path, device=device)
+        else:
+            raise ValueError("backend must be 'sglang' or 'hf'")
+        self.engine = ks.KnowLine(tok, back, float(temperature), int(workers), {})
+        self.provenance = {
+            "kind": f"knowline_server.KnowLine in process, backend {backend}",
+            "repo": model, "revision": revision, "front_end": "knowline_server.py (chat style, label-token softmax, "
+            f"temperature {temperature}, {workers} scoring threads, no calibration file)",
+            "sglang": " ".join(SGLANG_FLAGS + ["--mem-fraction-static", str(mem)]) if backend == "sglang" else None,
+            "transformers": transformers.__version__,
+            "policy": f"Unmodified state and questions; up to {ks.MAX_QUESTIONS} questions per request and "
+                      f"{ks.MAX_LABELS} options per question, larger requests are unsupported, nothing is truncated.",
+        }
+
+    def __call__(self, state, questions):
+        if len(questions) > ks.MAX_QUESTIONS:
+            raise Unsupported(f"{len(questions)} questions; at most {ks.MAX_QUESTIONS} per request")
+        try:
+            answers, usage = self.engine.run(state, questions)
+        except ValueError as exc:
+            if any(k in str(exc) for k in ("criteria", "levels", "options", "questions")):
+                raise Unsupported(str(exc)) from exc
+            raise
+        return {"model": self.model_id, "answers": answers, "usage": usage}, None
+
+    def runtime(self):
+        info = {"backend": self.backend_name}
+        try:
+            import torch
+            info.update(torch=torch.__version__, cuda=torch.version.cuda)
+            if torch.cuda.is_available():
+                info["gpu"] = torch.cuda.get_device_name()
+        except ImportError:
+            pass
+        if self.backend_name == "sglang":
+            try:
+                import sglang
+                info["sglang"] = sglang.__version__
+            except Exception:  # noqa: BLE001 - version is informational
+                pass
+        return info
+
+    def close(self):
+        if self.server is not None and self.server.poll() is None:
+            try:
+                os.killpg(self.server.pid, signal.SIGTERM)
+                self.server.wait(timeout=60)
+            except Exception:  # noqa: BLE001 - make sure the server does not outlive the run
+                try:
+                    os.killpg(self.server.pid, signal.SIGKILL)
+                except ProcessLookupError:
+                    pass
+        self.server = None

knowline_server.py

@@ -223,7 +223,12 @@ class HF:
         multimodal = hasattr(cfg, "vision_config") or hasattr(cfg, "audio_config")
         cls = getattr(transformers, "AutoModelForMultimodalLM", transformers.AutoModelForImageTextToText) if multimodal \
             else AutoModelForCausalLM
-        self.model = cls.from_pretrained(model, dtype=getattr(torch, dtype), device_map=device or "auto").eval()
+        try:
+            import accelerate  # noqa: F401  (transformers needs it for device_map)
+            self.model = cls.from_pretrained(model, dtype=getattr(torch, dtype), device_map=device or "auto").eval()
+        except ImportError:  # without accelerate: load, then move to one device
+            device = device or ("cuda" if torch.cuda.is_available() else "cpu")
+            self.model = cls.from_pretrained(model, dtype=getattr(torch, dtype)).to(device).eval()
         self.tok = AutoTokenizer.from_pretrained(model)
         self.lock = threading.Lock()