docs/LOCAL_INFERENCE.md

Local inference

Pluto-1 is a two-encoder ensemble. The supported runtime is the PyTorch package under code/; it loads both encoders, the per-question heads, and the ensemble calibration.

Tested environment

• CPU, clean virtual environment: Python 3.13.13, torch 2.15.0+cpu, transformers 5.22.0, huggingface_hub 2.4.0, fastapi 0.162.0, uvicorn 0.54.0.
• CUDA: Windows, NVIDIA GeForce RTX 5060 Ti 16GB, driver 610.74, Python 3.13.13, torch 2.11.0+cu128, transformers 5.21.0, huggingface_hub 2.3.0.

No performance or memory figures are claimed here.

Install

The model is private during release preparation. Set HF_TOKEN to a token with read access to omida-ai/pluto-1. Once the repository is public, the token and HF_TOKEN lines can be omitted.

Windows PowerShell (tested)

python -m venv .venv
.venv\Scripts\python -m pip install --upgrade pip
.venv\Scripts\python -m pip install huggingface_hub
$env:HF_TOKEN="hf_..."
.venv\Scripts\hf.exe download omida-ai/pluto-1 --local-dir pluto-1
.venv\Scripts\python -m pip install -e "pluto-1/code[serve]"
.venv\Scripts\pluto.exe --model pluto-1 info

Linux/macOS

These are the same package commands using the venv’s bin/ paths; they were not exercised on the Windows test host.

python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install huggingface_hub
export HF_TOKEN="hf_..."
.venv/bin/hf download omida-ai/pluto-1 --local-dir pluto-1
.venv/bin/python -m pip install -e "pluto-1/code[serve]"
.venv/bin/pluto --model pluto-1 info

Device selection

The loader chooses CUDA automatically when a working CUDA PyTorch build is available. Force CPU with --device cpu; force CUDA with --device cuda.

.venv\Scripts\pluto.exe --model pluto-1 --device cpu info
.venv\Scripts\pluto.exe --model pluto-1 --device cuda info

For CUDA, use a CUDA PyTorch build before installing the runtime. The build used in testing was:

.venv\Scripts\python -m pip install torch --index-url https://download.pytorch.org/whl/cu128

The default PyPI wheel in the clean CPU test was torch 2.15.0+cpu.

Python

From the parent directory that contains pluto-1/:

from pluto.ensemble import EnsembleEngine

engine = EnsembleEngine("pluto-1")
result = engine.predict(
    "Invoice #4411 was charged twice.",
    {
        "action": {
            "type": "choice",
            "instructions": "What should happen?",
            "criteria": {
                "continue": "Proceed.",
                "human_review": "Escalate.",
                "stop": "Stop.",
            },
        },
        "needs_review": {"type": "noul", "instructions": "Requires review."},
        "urgency": {
            "type": "score",
            "instructions": "How urgent?",
            "criteria": ["low", "medium", "high"],
        },
    },
)
print(result["answers"])

Loading directly from the Hub, without a local checkout:

from pluto.ensemble import EnsembleEngine

engine = EnsembleEngine.from_pretrained("omida-ai/pluto-1")

from_pretrained accepts token= when the repository is private.

CLI

Create questions.json:

{
  "action": {
    "type": "choice",
    "instructions": "What should the system do?",
    "criteria": {"continue": "Proceed.", "human_review": "Escalate.", "stop": "Stop."}
  },
  "needs_review": {"type": "noul", "instructions": "Requires review."},
  "urgency": {
    "type": "score",
    "instructions": "How urgent?",
    "criteria": ["low", "medium", "high"]
  }
}

Run a decision:

.venv\Scripts\pluto.exe --model pluto-1 --device cpu predict `
  --state "Trace: tool error 1, no irreversible actions, 7 steps." `
  --questions questions.json

The output contains answers.action (choice), answers.needs_review.noul (probability of true), and answers.urgency.score (expected level).

HTTP server

Start the Jev-compatible server:

.venv\Scripts\pluto.exe --model pluto-1 --device cpu serve --host 127.0.0.1 --port 8000

Check it:

curl.exe -s http://127.0.0.1:8000/health

Send a decision request (request.json contains state plus the questions object shown above):

curl.exe -s -H "Content-Type: application/json" --data-binary "@request.json" `
  http://127.0.0.1:8000/v1/systemone

Hub-only CLI load

.venv\Scripts\pluto.exe --model omida-ai/pluto-1 --device cpu info

The first Hub-only load downloads the repository. Later loads reuse the local Hugging Face cache.

Supported question names

Pluto-1 is fitted for these fixed Typed Decisions question names:

action, needs_review, outcome, risk, urgency, category, churn_risk, needs_human, discrepancy_severity, duplicate, matches_order, credential_compromise, severity, true_positive.

Arbitrary new question names are not supported by this specialist checkpoint.