laya-typed-decisions on Tenstorrent p150 and p150x4

Serves convaiinnovations/laya-typed-decisions by Convai Innovations at revision e929ae5c, the authors' fine-tune of Laya built on answerdotai/ModernBERT-large. See the original model card for training and evaluation details.

Laya typed-decisions is the authors' fine-tune of Laya on the LocalLLaMA/typed-decisions train split: a 421M-parameter ModernBERT-large encoder (28 layers, 1024 hidden, GeGLU, dual-theta RoPE, one global attention layer per three) with a 2-layer decision head that has an option-marker scorer and an act head, with a 1024-token context and a 256-token head budget. One forward pass answers typed noul, choice and score questions with calibrated probabilities; it never generates text. This package serves the Jev-compatible /v1/systemone API of the authors' laya-serve, with a demo page at /demo/.

Runs on Tenstorrent p150 or p150x4 β€” see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware p150, p150x4
License apache-2.0

Intended use

Calibrated typed decisions for the four typed-decisions workflows: agent-trace observability, customer service, invoice processing and security incidents.

Prerequisites

  • The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
  • Docker
  • Tenstorrent hardware: p150 or p150x4

Quickstart

tt serve tt-hous/laya-typed-decisions

tt serve downloads the Docker image and the convaiinnovations/laya-typed-decisions at e929ae5cf69bc34259cd2f95c9e91145b818b1f0 weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve tt-hous/laya-typed-decisions.

The demo page is at /demo/ on the port your serve command reported. Existing laya and Jev clients work by changing their base URL.

Serving profiles

Select a different serving profile using --profile.

profile hardware mesh description
p150 (default) p150 P150 One Blackhole p150 chip; bf8w_hifi3_erf; 37 traces (176.8 MiB) captured before the server reports ready: 1, 2, 4, 5, 8, 10, 16, 32, 50 and 64 rows at 128, 256 and 512 tokens and 1, 2, 4, 5, 8, 10 and 16 rows at 1024 tokens; at most 16384 padded tokens per forward (16 rows at 1024 tokens, 32 at 512, 64 at 256 and 128).
p150x4 p150x4 P150x4 1x4 mesh, data parallel over rows, no fabric; measured from the packaged image (build 2): 12.7 / 16.1 / 23.8 / 68.5 ms for 1 / 5 / 10 / 50 questions and 309 to 936 questions per second batched; parity smoke 100 of 100 decisions; a single question is slower than one chip (12.7 versus 10.8 ms) by the mesh host overhead.

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API β€” its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /v1/systemone takes {"state": ..., "questions": {id: {"type": "choice"|"score"|"noul", "instructions": ..., "criteria": ...}}, "max_len", "head_max_len", "min_confidence"} and returns {"model", "answers": {id: {...}}, "usage": {"input_tokens", "output_tokens", "state_tokens", "state_tokens_dropped", "truncated", "truncated_questions"}}. Use max_len up to 1024 and head_max_len up to 256. POST /v1/systemone/batch takes {"states": [...], "questions": {...}} (1 to 64 states sharing one question set) and returns {"results": [...], "total_usage": {...}}. GET /v1/health reports backend, precision, mesh, buckets and the startup sanity check. Every /v1 response carries X-Inference-Time-Ms, X-Laya-Device-Ms and X-Laya-Batch. shim/laya_tt_backend.py under code/ installs a TtBackend on a pip laya Agent that routes agent.model.forward to this server's /v1/forward.

Expected performance

Evaluations

eval this package reference (convaiinnovations/laya-typed-decisions card) items
LocalLLaMA/typed-decisions test set, accuracy 0.764 0.766 400 cases, 2,000 decisions
typed-decisions, soft accuracy 0.469 0.471 2,000 decisions
typed-decisions, Brier 0.062 0.062 2,000 decisions
typed-decisions, ECE 0.214 0.213 2,000 decisions
typed-decisions, score MAE 0.244 0.242 2,000 decisions
AG News (4 labels), accuracy 0.953 β€” 400 decisions
DAIR Emotion (6 labels), accuracy 0.595 β€” 400 decisions

From this package on one p150 at max_len 1024 and head_max_len 256. The reference figures are the authors' on the model card; the authors publish AG News and DAIR Emotion for the English root checkpoint only (0.950 / 0.595), not for this fine-tune.

Parity against the CPU fp32 reference of this checkpoint:

parity this package items
typed-decisions accuracy / soft accuracy / Brier / ECE / score MAE, CPU fp32 of the same checkpoint 0.766 / 0.471 / 0.061 / 0.213 / 0.242 2,000 decisions
confident decisions (reference margin at least 0.10) that agree 383 of 383 488-decision parity corpus
argmax agreement 485 of 488 488 decisions
max abs delta p, median / max 0.004 / 0.040 488 decisions
p150x4 parity smoke 100 of 100 decisions, 81 of 81 confident 100 decisions

Latency

Client-side medians per call, warm, with the authors' bench_latency protocol; device forward time in parentheses. The reference column is the authors' Tesla T4 figure on the model card for the English root checkpoint (same architecture and size), for the same number of questions; the authors publish no latency for this fine-tune.

profile request N median (ms) p95 (ms) reference, Tesla T4, English checkpoint (ms)
p150 (default) 1 question β€” 10.8 (device 9.2) β€” 39.5
p150 5 questions β€” 24.5 (22.7) β€” 84.5
p150 10 questions β€” 42.9 (40.3) β€” 158.6
p150 50 questions β€” 197.6 (191.9) β€” 771
p150x4 1 question β€” 12.7 β€” 39.5
p150x4 5 questions β€” 16.1 β€” 84.5
p150x4 10 questions β€” 23.8 β€” 158.6
p150x4 50 questions β€” 68.5 β€” 771

Batched /v1/systemone/batch: 203 to 249 questions per second on p150 (the authors' T4, English checkpoint: 103 to 332) and 309 to 936 on p150x4. A single question is slower on p150x4 than on one chip (12.7 against 10.8 ms). Long states that reach the 1024-token bucket (750-token rows): 23.9 / 99.0 / 183.2 / 305.6 ms for 1 / 5 / 10 / 16 questions, measured with the server run on the host rather than from the image.

Measured from this package served on one Blackhole p150 chip (p150) and on the four-chip p150x4 mesh, client on the same host.

Limitations

  • Fine-tuned for the four typed-decisions workflows (agent-trace observability, customer service, invoice processing, security incidents). Not for workflows far from that data without your own evaluation: use the base checkpoint (see Related packages). Not for text generation. English only.
  • Tested on one p150 chip and on the four-chip p150x4 mesh; no two-chip profile is offered.
  • max_len 1024, head_max_len 256. Requests are padded to 128, 256, 512 or 1024 tokens with 1 to 64 rows at 128 to 512 tokens (exact buckets at 5, 10 and 50 rows) and 1 to 16 rows at 1024 tokens (a longer 1024-token batch is split into chunks of 16 rows; at most 16,384 padded tokens per forward); longer sequences are refused with 422. Long option texts are truncated to 48 tokens silently, as in the reference.
  • Temperatures are clamped to [0.5, 5.0] as pip laya 0.3.27 does, so choice questions with more than 10 options use 0.5 instead of the shipped 0.1006.
  • action.act_probability carries no usable signal (per the authors); gate on answer_confidence. The shipped temperatures are over-confident per the authors (ECE 0.214 here); refit on your own data before trusting the probabilities.
  • Against the CPU fp32 reference, 13 of the 2,000 typed-decisions decisions flip, all with a reference margin under 0.10 (max abs delta p 0.031); on the 488-decision parity corpus 3 flip, all with a margin under 0.015, and none of the 383 confident decisions. Probabilities move by up to 0.040 (median 0.004) on the parity corpus.
  • One device process serialises requests. p150x4 is data parallel over rows and bit-identical to one chip at equal per-chip batch; it reports healthy in 20 s.
  • Not reproduced from the authors' cards: MASSIVE, XNLI, the 45 of 51 languages, and post-temperature ECE.

Licensing

Laya typed-decisions weights and the vendored rl_common.py, rl_agent_api.py and email_utils.py: Apache-2.0 (Convai Innovations); ModernBERT-large: Apache-2.0 (Answer.AI); TTNN port and serving code: Apache-2.0 (Tenstorrent AI ULC); demo feed cases: LocalLLaMA/typed-decisions, Apache-2.0.

Related packages

  • tt-hous/laya: the English root checkpoint (max_len 512), for decisions outside the four typed-decisions workflows; same server and API.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/laya-typed-decisions/discussions β€” that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from β€” code/ in this repo is byte-identical to the model code inside the image:

component built from
tt-metal a local checkout β€” commit not published
code/ digest 42ab6916a82d7364 (sha256, first 16 hex digits)
built 2026-10-06T13:50:55+00:00 by tt-model 0.1.0
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for tt-hous/laya-typed-decisions

Finetuned
(438)
this model

Collection including tt-hous/laya-typed-decisions