kvrun-relay-v1 / examples /predict_onnx.py
JoeKVRun's picture
Initial Commit
5e39d13 verified
Raw History Blame Contribute Delete
1.16 kB
"""Classify prompts with the ONNX export (ONNX Runtime; no PyTorch at inference).
pip install "laya[onnx]" huggingface_hub
python predict_onnx.py "Summarize this: the council voted to expand bike lanes"
`laya.onnx_agent.ONNXAgent` reproduces the exact input layout (question header, option markers,
state) and temperature scaling the graph was trained with. To call the graph directly, see its
`_encode_state` / `_decode_answers`. The Router app's Rust port (crates/router-core/src/router/laya.rs)
is a second reference implementation.
"""
import json
import sys
from huggingface_hub import snapshot_download
from laya.onnx_agent import ONNXAgent
REPO = "[org]/router-laya-v1"
local = snapshot_download(REPO, allow_patterns=["onnx/*", "tokenizer/*", "rl_agent_config.json", "questions.json", "encoder/*"])
agent = ONNXAgent(local, onnx_path=f"{local}/onnx/router.onnx")
questions = json.load(open(f"{local}/questions.json"))
for prompt in sys.argv[1:] or ["hi there", "Write a haiku about rain"]:
a = agent.predict(prompt, questions)["answers"]
print(f"{a['category']['choice']:<18} complexity={a['complexity']['score']:.2f} {prompt[:60]}")