fastino/GLiDE-9B

GLiDE 9B

Inference provider: Fastino — use GLiDE through Fastino's hosted API (POST https://api.fastino.ai/v1/systemone, model fastino/GLiDE). Get an API key · API documentation

GLiDE 9B is a GLiDE decision model with 7.94B parameters. It scores 68.77 on the Decision Index 0.3.1 public suite, measured with the official kit on one RTX PRO 6000, at a median latency of 38.6 ms per request on that GPU. Weights and compute are bf16.

GLiDE is a decision model from Fastino. Given a state (text or JSON) and one or more typed questions, it returns a probability for every option of every question.

Fastino API

Use GLiDE through Fastino’s hosted API for classification, routing, and scoring. Send decision requests to POST https://api.fastino.ai/v1/systemone with model fastino/GLiDE. Create an account and get an API key, or follow the API quickstart. The instructions below cover running this repository locally.

Overview

• Typed decisions: choice (one of up to 255 named options), noul (yes / no, returns P(yes)) and score (ordered levels).
• Probabilities for every option, read from a 255-way decision readout. Nothing is generated.
• One forward pass per question. All questions of a request (and of concurrent requests) are scored together in micro-batches.
• Text and JSON: the state can be a string or any JSON value. This model has no vision encoder; requests with images are rejected.
• Long contexts: up to 40,000 tokens per question prompt by default (state, question and options).
• Interfaces: a Python API and an HTTP server speaking the System One format (POST /v1/systemone).

Hardware and memory

Parameters	7.94 B (7.94 B language model, of which 1017 M token embeddings; 1.04 M decision readout)
Weights	bf16, 15.90 GB on disk
GPU memory	15.88 GB after loading; 18.92 GB peak over the mixed requests of the latency test below
Tested GPUs	NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB)
Load time	6 s from network storage

Weights are copied to the GPU tensor by tensor, so host memory use during loading stays low.

Install

pip install huggingface_hub
hf download fastino/GLiDE-9B --local-dir GLiDE-9B
cd GLiDE-9B
pip install -r requirements.txt

• Tested with Python 3.12, torch==2.14.0 (CUDA 13.0 wheels), triton==3.8.0 and transformers==5.17.0 (all pins are in requirements.txt).
• flash-linear-attention==0.5.2 is optional but recommended: it provides fast kernels for the linear-attention layers. Without it transformers falls back to a much slower PyTorch implementation, which was not part of the validation. All numbers below were measured with it installed.
• At load, transformers logs that causal_conv1d is not installed and uses its PyTorch path. This is expected: the validated setup does not use that package.
• The glide/ package lives at the repository root: run from the repository directory or add it to PYTHONPATH.

Quickstart

Python

from glide import Glide

model = Glide.from_pretrained("GLiDE-9B")   # local directory, or the Hub repo id "fastino/GLiDE-9B"

result = model.decide(
    state={"ticket": "I was charged twice for order #4821 and the second charge is still pending.", "customer_tier": "gold"},
    questions={
        "intent": {"type": "choice", "instructions": "What does the customer want?",
                   "criteria": {"refund": "Get money back", "cancel": "Cancel the order",
                                "track": "Track a delivery", "other": "Something else"}},
        "urgent": {"type": "noul", "instructions": "Should this ticket be escalated today?"},
        "priority": {"type": "score", "instructions": "What support priority should this ticket get?",
                     "criteria": ["low: no impact on the customer", "medium: the customer is inconvenienced",
                                  "high: the customer has lost money or cannot use the service"]},
    },
)

Result on an RTX PRO 6000 (rounded):

{
  "model": "glide-9b",
  "answers": {
    "intent": {
      "type": "choice",
      "choice": "refund",
      "confidence": 0.89,
      "probabilities": {
        "refund": 0.938,
        "cancel": 0.012,
        "track": 0.002,
        "other": 0.048
      }
    },
    "urgent": {
      "type": "noul",
      "noul": 0.775,
      "confidence": 0.549
    },
    "priority": {
      "type": "score",
      "score": 2,
      "expected_level": 1.861,
      "confidence": 0.734,
      "probabilities": {
        "0": 0.004,
        "1": 0.131,
        "2": 0.865
      }
    }
  },
  "usage": {
    "input_tokens": 316,
    "output_tokens": 0
  }
}

model.decide_batch([{"state": ..., "questions": ...}, ...]) scores several requests together and returns one item per request: a response dict, or a glide.RequestError for a request that cannot be served (decide raises it instead).

HTTP server

GLIDE_MODEL=/models/GLiDE-9B uvicorn glide.server:app --host 0.0.0.0 --port 8000
# or
python -m glide.server --model /models/GLiDE-9B --port 8000

The model loads at startup; GET /health answers once it is ready.

curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "glide-9b",
  "state": "Order #4821 was delivered to the wrong address and the customer wants it re-sent today.",
  "questions": {
    "action": {"type": "choice", "instructions": "What should support do next?",
               "criteria": {"resend": "Re-ship the order", "refund": "Refund the order", "investigate": "Open a carrier investigation"}},
    "urgent": {"type": "noul", "instructions": "Is the request time-sensitive?"}
  }
}'

Request

Field	Type	Description
model	string	Optional. Ignored; the response reports the served model name (GLIDE_MODEL_NAME, default glide-9b).
state	string or JSON value	The context to decide on. Strings are used as is; other JSON values are serialized.
questions	object	{key: question}. Every question is answered independently, one forward pass each.

Question fields:

Field	Description
type	choice, noul or score.
instructions	The question. A string; objects are rendered as key: value lines.
criteria	choice: object {option_key: description}, 1 to 255 options, shown in the given order; descriptions may be strings or JSON (an empty description shows only the key). noul: optional {"true": "...", "false": "..."} descriptions (default Yes / No). score: a list of level descriptions (level i is index i), or an object keyed by numeric level (sorted ascending and re-indexed from 0).

Response

For the curl request above (RTX PRO 6000, rounded; the HTTP response also carries latency_ms):

{
  "model": "glide-9b",
  "answers": {
    "action": {
      "type": "choice",
      "choice": "investigate",
      "confidence": 0.257,
      "probabilities": {
        "resend": 0.366,
        "refund": 0.011,
        "investigate": 0.623
      }
    },
    "urgent": {
      "type": "noul",
      "noul": 0.986,
      "confidence": 0.972
    }
  },
  "usage": {
    "input_tokens": 172,
    "output_tokens": 0
  }
}

Answer type	Fields
choice	choice: most probable key. probabilities: {key: p} over all options, summing to 1. confidence: top minus second probability.
noul	noul: P(true). confidence: |2 P(true) - 1|.
score	score: most probable level index. expected_level: sum of index x probability. probabilities: {"0": p, "1": p, ...}. confidence: top minus second probability.

usage.input_tokens is the prompt length summed over the request's questions. latency_ms is the server-side time for the request (queueing plus inference).

Errors return {"error": "..."}: 422 for requests the model cannot serve (see below), 401 for a missing or wrong API key, 503 when the queue is full, 504 on timeout, 500 for an internal error.

Production notes

Process model. Run one server process per GPU (workers=1) and scale out with replicas behind a load balancer. A single worker thread owns the GPU; HTTP handler threads only queue requests and wait for results.

Batching and concurrency (environment variables):

Variable	Default	Meaning
GLIDE_MODEL	required	Model directory or Hub repo id.
GLIDE_MODEL_NAME	glide-9b	model field in responses.
GLIDE_DEVICE	cuda	Torch device, e.g. cuda:1.
GLIDE_MAX_BATCH	16	Max requests scored together in one micro-batch.
GLIDE_BATCH_WAIT_MS	5	How long the worker waits to fill a micro-batch.
GLIDE_TOKEN_BUDGET	16384	Max padded tokens per forward pass; question prompts are length-sorted and packed under it (a longer prompt runs alone).
GLIDE_MAX_LEN	40000	Max tokens per question prompt.
GLIDE_MAX_QUEUE	1024	Queued requests before new ones get 503.
GLIDE_REQUEST_TIMEOUT_S	600	Max wait for a result before 504.
GLIDE_THREADS	1024	HTTP handler threads (concurrent open requests).
GLIDE_API_KEY	unset	If set, require Authorization: Bearer <key>.

Max context. 40,000 tokens per question prompt by default. Full-attention cost grows quadratically with length, so long prompts dominate a micro-batch; set client timeouts with headroom. Raising GLIDE_MAX_LEN is possible but untested.

Health. GET /health returns {"status": "ok", ...} with the queue length and batch counters once the model is loaded.

Latency (one RTX PRO 6000 Blackwell, warm, one request at a time; 750 requests of mixed length with 2.3 questions per request on average):

Interface	Median	Mean	p90	p99
Python API (in-process)	38.6 ms	116.0 ms	330.2 ms	1198.4 ms
HTTP server (client wall time)	40.8 ms	118.7 ms	332.3 ms	1208.4 ms

Latency grows with prompt length and the number of questions.

Throughput. Scoring 1,500 mixed requests (6,596 questions) in token-budget micro-batches (16,384 tokens) on one RTX PRO 6000: 15 questions/s. Run one process per GPU and add replicas for more throughput.

Cold start and the kernel cache. The linear-attention kernels are compiled and autotuned by Triton on first use of each new shape, so the first requests of a fresh process are slow. Compiled kernels and autotuning results are cached on disk (TRITON_CACHE_DIR, default ~/.triton/cache). In production:

1. point TRITON_CACHE_DIR at persistent storage shared by replicas with the same GPU type and software versions;
2. after the first deployment (or an upgrade), replay a few hundred representative requests through one replica to fill the cache;
3. keep a replica out of rotation until /health is up and its first warm-up requests have returned.

Unsupported requests get HTTP 422 with a message:

• more than 255 options: options per choice exceeds the label set (N > 255)
• a question prompt over GLIDE_MAX_LEN: prompt exceeds the maximum context length
• images: images are not supported by this deployment
• an unknown question type, missing state, or malformed questions / criteria

A request with an empty questions object returns 200 with empty answers.

Determinism. There is no sampling. The same request scored alone gives identical results on the same GPU type and software stack. The questions of one request always share forward passes; when concurrent requests are also micro-batched, padding changes the shape of the computation and probabilities can move slightly. Set GLIDE_MAX_BATCH=1 when results must not depend on other traffic.

Precision. Weights and compute are bf16 with PyTorch SDPA attention (Glide.from_pretrained disables the cuDNN SDPA backend process-wide, so the memory-efficient kernel is used).

Logging. Request contents are not logged.

Performance

Decision Index 0.3.1 public suite: 68.76, measured with the official kit on one RTX PRO 6000.

Limitations

• It chooses among the options it is given; it does not generate text or explanations.
• Probabilities are model estimates; set decision thresholds on your own data.
• At most 255 options per question and 40,000 tokens per question prompt (default); larger requests are rejected with 422.
• Text and JSON only: no images.
• Cost grows with the number of questions: each question is a separate forward pass over the full state.

License

GLiDE’s Fastino contributions are licensed under the Fastino Community License. Third-party components retain their original licenses; see the NOTICE for attribution.