Release prep: ensemble-aware CLI/server, clean-install validation, compact Run locally section and local inference doc
5f15b97 verified | # Local inference | |
| Pluto-1 is a two-encoder ensemble. The supported runtime is the PyTorch package | |
| under `code/`; it loads both encoders, the per-question heads, and the ensemble | |
| calibration. | |
| ## Tested environment | |
| - **CPU, clean virtual environment:** Python 3.13.13, `torch 2.15.0+cpu`, | |
| `transformers 5.22.0`, `huggingface_hub 2.4.0`, `fastapi 0.162.0`, | |
| `uvicorn 0.54.0`. | |
| - **CUDA:** Windows, NVIDIA GeForce RTX 5060 Ti 16GB, driver 610.74, | |
| Python 3.13.13, `torch 2.11.0+cu128`, `transformers 5.21.0`, | |
| `huggingface_hub 2.3.0`. | |
| No performance or memory figures are claimed here. | |
| ## Install | |
| The model is private during release preparation. Set `HF_TOKEN` to a token with | |
| read access to `omida-ai/pluto-1`. Once the repository is public, the token and | |
| `HF_TOKEN` lines can be omitted. | |
| ### Windows PowerShell (tested) | |
| ```powershell | |
| python -m venv .venv | |
| .venv\Scripts\python -m pip install --upgrade pip | |
| .venv\Scripts\python -m pip install huggingface_hub | |
| $env:HF_TOKEN="hf_..." | |
| .venv\Scripts\hf.exe download omida-ai/pluto-1 --local-dir pluto-1 | |
| .venv\Scripts\python -m pip install -e "pluto-1/code[serve]" | |
| .venv\Scripts\pluto.exe --model pluto-1 info | |
| ``` | |
| ### Linux/macOS | |
| These are the same package commands using the venv’s `bin/` paths; they were not | |
| exercised on the Windows test host. | |
| ```bash | |
| python3 -m venv .venv | |
| .venv/bin/python -m pip install --upgrade pip | |
| .venv/bin/python -m pip install huggingface_hub | |
| export HF_TOKEN="hf_..." | |
| .venv/bin/hf download omida-ai/pluto-1 --local-dir pluto-1 | |
| .venv/bin/python -m pip install -e "pluto-1/code[serve]" | |
| .venv/bin/pluto --model pluto-1 info | |
| ``` | |
| ## Device selection | |
| The loader chooses CUDA automatically when a working CUDA PyTorch build is | |
| available. Force CPU with `--device cpu`; force CUDA with `--device cuda`. | |
| ```powershell | |
| .venv\Scripts\pluto.exe --model pluto-1 --device cpu info | |
| .venv\Scripts\pluto.exe --model pluto-1 --device cuda info | |
| ``` | |
| For CUDA, use a CUDA PyTorch build before installing the runtime. The build used | |
| in testing was: | |
| ```powershell | |
| .venv\Scripts\python -m pip install torch --index-url https://download.pytorch.org/whl/cu128 | |
| ``` | |
| The default PyPI wheel in the clean CPU test was `torch 2.15.0+cpu`. | |
| ## Python | |
| From the parent directory that contains `pluto-1/`: | |
| ```python | |
| from pluto.ensemble import EnsembleEngine | |
| engine = EnsembleEngine("pluto-1") | |
| result = engine.predict( | |
| "Invoice #4411 was charged twice.", | |
| { | |
| "action": { | |
| "type": "choice", | |
| "instructions": "What should happen?", | |
| "criteria": { | |
| "continue": "Proceed.", | |
| "human_review": "Escalate.", | |
| "stop": "Stop.", | |
| }, | |
| }, | |
| "needs_review": {"type": "noul", "instructions": "Requires review."}, | |
| "urgency": { | |
| "type": "score", | |
| "instructions": "How urgent?", | |
| "criteria": ["low", "medium", "high"], | |
| }, | |
| }, | |
| ) | |
| print(result["answers"]) | |
| ``` | |
| Loading directly from the Hub, without a local checkout: | |
| ```python | |
| from pluto.ensemble import EnsembleEngine | |
| engine = EnsembleEngine.from_pretrained("omida-ai/pluto-1") | |
| ``` | |
| `from_pretrained` accepts `token=` when the repository is private. | |
| ## CLI | |
| Create `questions.json`: | |
| ```json | |
| { | |
| "action": { | |
| "type": "choice", | |
| "instructions": "What should the system do?", | |
| "criteria": {"continue": "Proceed.", "human_review": "Escalate.", "stop": "Stop."} | |
| }, | |
| "needs_review": {"type": "noul", "instructions": "Requires review."}, | |
| "urgency": { | |
| "type": "score", | |
| "instructions": "How urgent?", | |
| "criteria": ["low", "medium", "high"] | |
| } | |
| } | |
| ``` | |
| Run a decision: | |
| ```powershell | |
| .venv\Scripts\pluto.exe --model pluto-1 --device cpu predict ` | |
| --state "Trace: tool error 1, no irreversible actions, 7 steps." ` | |
| --questions questions.json | |
| ``` | |
| The output contains `answers.action` (choice), `answers.needs_review.noul` | |
| (probability of true), and `answers.urgency.score` (expected level). | |
| ## HTTP server | |
| Start the Jev-compatible server: | |
| ```powershell | |
| .venv\Scripts\pluto.exe --model pluto-1 --device cpu serve --host 127.0.0.1 --port 8000 | |
| ``` | |
| Check it: | |
| ```powershell | |
| curl.exe -s http://127.0.0.1:8000/health | |
| ``` | |
| Send a decision request (`request.json` contains `state` plus the | |
| `questions` object shown above): | |
| ```powershell | |
| curl.exe -s -H "Content-Type: application/json" --data-binary "@request.json" ` | |
| http://127.0.0.1:8000/v1/systemone | |
| ``` | |
| ## Hub-only CLI load | |
| ```powershell | |
| .venv\Scripts\pluto.exe --model omida-ai/pluto-1 --device cpu info | |
| ``` | |
| The first Hub-only load downloads the repository. Later loads reuse the local | |
| Hugging Face cache. | |
| ## Supported question names | |
| Pluto-1 is fitted for these fixed Typed Decisions question names: | |
| `action`, `needs_review`, `outcome`, `risk`, `urgency`, `category`, | |
| `churn_risk`, `needs_human`, `discrepancy_severity`, `duplicate`, | |
| `matches_order`, `credential_compromise`, `severity`, `true_positive`. | |
| Arbitrary new question names are not supported by this specialist checkpoint. | |