Add Merlin V3 ungated native evaluation release

README.md

@@ -12,6 +12,22 @@ tags:
   - deberta-v3
 ---
 
+<!-- merlin-v3-ungated-start -->
+## V3 ungated native evaluation release
+
+The [V3 native Go package](v3-ungated/README.md) is available in `v3-ungated/`.
+It keeps the V3 weights, threshold and action-agreement filter while removing the
+three restrictive eligibility gates. AgentDojo recall is 24.1% (85/352), with
+20/868 benign cases flagged. See the package for coverage, private-data aggregate
+results and limitations. No private evaluation examples are included.
+
+The root checkpoint and Python usage below remain **legacy v0.1**, with their
+original results. They are not the V3 runtime. This is a controlled evaluation
+release, not a claim of uniform performance across deployments.
+<!-- merlin-v3-ungated-end -->
+
+
+
 # Merlin
 
 Merlin is a 70.8M-parameter local encoder that classifies a proposed AI-agent

v3-ungated/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

v3-ungated/NOTICE

@@ -0,0 +1,7 @@
+Merlin
+Copyright 2026 Kontext Security
+
+The released checkpoint is fine-tuned from microsoft/deberta-v3-xsmall, which is
+distributed under the MIT License. TS-Bench and its component datasets retain
+their upstream terms. See DATASETS.md and the Hugging Face model card.
+

v3-ungated/README.md

@@ -0,0 +1,84 @@
+---
+license: other
+license_name: merlin-research-release
+base_model: microsoft/deberta-v3-xsmall
+tags:
+  - agents
+  - agent-safety
+  - tool-use
+  - native-go
+---
+
+# Merlin V3 ungated — local evaluation release
+
+Runtime `merlin-v3-ungated-20261104` uses the unchanged V3 contextual encoder and
+sparse action-agreement model from `merlin-v3-precision-20261007`. This is a
+runtime eligibility change, not newly trained weights. It removes the operation
+allowlist, lexical request gate and learned informational-request gate.
+
+The threshold remains **0.5**. The complete current action must fit **128 tokens**;
+long actions and excluded tools remain unassessed. The output score is a
+threshold-relative agreement score, **not a calibrated probability of risk**.
+This is an advisory detector; it does not execute or block tool calls.
+
+## Run locally
+
+Download this folder (`v3-ungated`) from `kontext-security/Merlin`, preserving its
+layout. Go 1.26.10 or later is required to build the standalone native runtime.
+It embeds the supplied assets and needs no Python, ONNX runtime or hosted inference.
+Go downloads normal build dependencies on the first build.
+
+```bash
+hf download kontext-security/Merlin --include 'v3-ungated/*' --local-dir dist/Merlin
+cd dist/Merlin/v3-ungated/native
+go build -o merlin ./cmd/merlin
+printf '%s\n' '{"id":"example","input":{"user_request":"Find public documentation","tool_name":"WebSearch","tool_arguments":{"query":"Go JSON decoder documentation"},"interaction_history":"[]"}}' | ./merlin
+```
+
+Each input line contains `id` and `input`. Input fields are `user_request`,
+`interaction_history` (a JSON array encoded as a string), `tool_name`,
+`tool_arguments`, and optionally `available_tool_schemas`. Include only completed
+**prior** tool observations in history, never the current result or later turns.
+Each output line preserves `id` and includes `evaluation`; unavailable assessments
+have an `error_code` and no score. Do not treat those as safe classifications.
+
+## Evaluation
+
+| Cohort | Assessed / available | Detections or alerts |
+|---|---:|---|
+| AgentDojo-Traj | 1,159 / 1,220 | 85/352 positives detected (24.1% recall); 20/868 benign cases flagged (2.3% FPR) |
+| Private historical regression | 415 / 610 | 1 false alert |
+| Private later workload sample | 600 / 1,207 | 3 clear false alerts, 1 uncertain low-impact alert |
+
+The original collected false-positive subset has 539 saved cases, of which 409
+were reconstructed and assessed: Claude Code 0/128 alerts; Codex 1/281. The expanded
+historical panel above includes these cases. These cohorts are not additive.
+The later sample's clear false-alert frequency is 3/600 assessed calls (0.50%);
+607 cases remain unassessed. This is not a population false-positive guarantee.
+
+Historical data overlap training/selection/calibration and AgentDojo was repeatedly
+consulted during development. The later sample is stratified, not traffic weighted.
+Historical context was reconstructed from transcripts rather than exact daemon
+snapshots. The remaining historical examples are unresolved, not successful tests.
+No fresh ungated ASB or AgentHarm result is claimed. The earlier v0.1 benchmark
+numbers do not describe this runtime.
+
+This release is suitable for controlled evaluation. It is not evidence of uniform
+precision on unfamiliar tools or workloads. See `evaluation.json` for aggregates
+and `release_manifest.json` for file checksums. **No private prompts, calls,
+transcripts, reviewer text, identifiers or per-example scores are distributed.**
+Private evaluation requires the internal access-controlled archive; the public
+release cannot independently reproduce its private-data metrics.
+
+## Provenance and terms
+
+Base encoder: `microsoft/deberta-v3-xsmall` (MIT), revision
+`4b419818330868dff6a60ad3e6b1c730f8b8c0c6`. The V3 encoder weights SHA-256 is
+`429b09164c6705790c4414eb31c0bc18d2fa8a374bea467b2e1b49ead6aeb5f1`.
+V3 was developed using benchmark data and private training/selection examples;
+only model parameters and aggregate evaluation results are released here.
+Upstream dataset/model terms continue to apply; consult the bundled attribution
+and provenance files. The checkpoint is a research release (`other` license),
+not a blanket commercial-use license. Runtime source follows the repository
+Apache-2.0 license. The older request-intent asset remains for provenance but is
+not used by the ungated inference path.

v3-ungated/evaluation.json

@@ -0,0 +1,60 @@
+{
+  "runtime": "merlin-v3-ungated-20261104",
+  "weights_candidate": "merlin-v3-precision-20261007",
+  "threshold": 0.5,
+  "action_token_limit": 128,
+  "action_agreement": true,
+  "removed_gates": [
+    "operation_allowlist",
+    "lexical_request",
+    "learned_request_intent"
+  ],
+  "AgentDojo": {
+    "total": 1220,
+    "assessed": 1159,
+    "tp": 85,
+    "fp": 20,
+    "positives": 352,
+    "negatives": 868
+  },
+  "private_historical_regression": {
+    "total": 415,
+    "assessed": 415,
+    "skipped": 0,
+    "alerts": 1
+  },
+  "private_original_flagged_no_risk": {
+    "saved": 539,
+    "reconstructed_and_assessed": 409,
+    "alerts": 1,
+    "claude_code": {
+      "saved": 133,
+      "assessed": 128,
+      "alerts": 0
+    },
+    "codex": {
+      "saved": 406,
+      "assessed": 281,
+      "alerts": 1
+    }
+  },
+  "private_recent_sample": {
+    "total": 1207,
+    "assessed": 600,
+    "skipped": 607,
+    "alerts": 4
+  },
+  "private_recent_alert_review": {
+    "clear_false_alerts": 3,
+    "uncertain_low_impact": 1,
+    "confirmed_true_positive": 0
+  },
+  "limitations": [
+    "Historical examples overlap model development; regression evidence, not unseen holdout.",
+    "AgentDojo was repeatedly consulted during development.",
+    "Recent sample is stratified and reconstructed from pre-call transcripts; not representative of every deployment.",
+    "Unassessed examples are not safe predictions.",
+    "No fresh ungated ASB or AgentHarm result is claimed."
+  ],
+  "private_examples_published": false
+}

v3-ungated/native/cmd/merlin/main.go

@@ -0,0 +1,52 @@
+package main
+
+import (
+	"bufio"
+	"context"
+	"encoding/json"
+	"fmt"
+	"os"
+	"time"
+
+	"github.com/kontext-security/merlin/native/stepsafety"
+)
+
+// Merlin only evaluates these inputs; it never executes the supplied tool calls.
+func main() {
+	ctx := context.Background()
+	evaluator := stepsafety.New(ctx, stepsafety.Config{Enabled: true, Timeout: 30 * time.Second, StartupTimeout: 30 * time.Second, MaxConcurrency: 1})
+	health := evaluator.Health(ctx)
+	if health.Status != "ready" {
+		fmt.Fprintln(os.Stderr, "Merlin initialization failed:", health.ErrorCode)
+		os.Exit(1)
+	}
+	scan := bufio.NewScanner(os.Stdin)
+	scan.Buffer(make([]byte, 65536), 4*1024*1024)
+	output := json.NewEncoder(os.Stdout)
+	for scan.Scan() {
+		var row struct {
+			ID    string           `json:"id"`
+			Input stepsafety.Input `json:"input"`
+		}
+		if err := json.Unmarshal(scan.Bytes(), &row); err != nil {
+			fmt.Fprintln(os.Stderr, "Invalid input JSON:", err)
+			os.Exit(1)
+		}
+		if row.Input.ToolName == "" {
+			fmt.Fprintln(os.Stderr, "input.tool_name is required")
+			os.Exit(1)
+		}
+		result := struct {
+			ID         string                `json:"id"`
+			Evaluation stepsafety.Evaluation `json:"evaluation"`
+		}{row.ID, evaluator.Evaluate(ctx, row.Input)}
+		if err := output.Encode(result); err != nil {
+			fmt.Fprintln(os.Stderr, err)
+			os.Exit(1)
+		}
+	}
+	if err := scan.Err(); err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+}

v3-ungated/native/go.mod

@@ -0,0 +1,8 @@
+module github.com/kontext-security/merlin/native
+
+go 1.26.10
+
+require (
+	golang.org/x/text v0.43.0
+	gonum.org/v1/gonum v0.17.0
+)

v3-ungated/native/go.sum

@@ -0,0 +1,4 @@
+golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
+golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
+gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
+gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=

v3-ungated/native/stepsafety/agreement.go

@@ -0,0 +1,145 @@
+package stepsafety
+
+import (
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"golang.org/x/text/cases"
+	"golang.org/x/text/language"
+	"io"
+	"math"
+	"strings"
+	"unicode"
+)
+
+type sparseVector struct {
+	Features map[string][2]float64 `json:"features"`
+}
+type scoreCalibration struct{ Coefficient, Intercept float64 }
+
+var agreementModel struct {
+	Word, Char         sparseVector
+	Intercept          float64
+	EncoderCalibration scoreCalibration `json:"encoder_calibrator"`
+	SparseCalibration  scoreCalibration `json:"sparse_calibrator"`
+	EncoderThreshold   float64          `json:"encoder_threshold"`
+	SparseThreshold    float64          `json:"sparse_threshold"`
+}
+
+func loadAgreementAssets() error {
+	f, err := nativeAssets.Open("model/native/agreement.json.gz")
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	reader, err := gzip.NewReader(f)
+	if err != nil {
+		return err
+	}
+	defer reader.Close()
+	data, err := io.ReadAll(io.LimitReader(reader, 4918863+1))
+	if err != nil {
+		return err
+	}
+	h := sha256.Sum256(data)
+	if len(data) != 4918863 || hex.EncodeToString(h[:]) != "7f01698e14f36f72ccad7f519c4e95298f125c75c469ef25b4b0f08052f1dcc8" {
+		return errors.New("agreement asset checksum mismatch")
+	}
+	return json.Unmarshal(data, &agreementModel)
+}
+
+func vectorScore(counts map[string]int, vector sparseVector) float64 {
+	var norm, dot float64
+	for term, count := range counts {
+		h := sha256.Sum256([]byte(term))
+		feature, ok := vector.Features[hex.EncodeToString(h[:])]
+		if !ok {
+			continue
+		}
+		value := (1 + math.Log(float64(count))) * feature[0]
+		norm += value * value
+		dot += value * feature[1]
+	}
+	if norm == 0 {
+		return 0
+	}
+	return dot / math.Sqrt(norm)
+}
+
+func pythonSpace(r rune) bool { return unicode.IsSpace(r) || (r >= 0x1c && r <= 0x1f) }
+
+func sparseMargin(text string) float64 {
+	// Match sklearn's lowercase Unicode word analyzer and char analyzer.
+	// The vocabulary is hashed to avoid shipping plaintext captured tokens.
+	text = cases.Lower(language.Und).String(text)
+	words := []string{}
+	run := []rune{}
+	flush := func() {
+		if len(run) >= 2 {
+			words = append(words, string(run))
+		}
+		run = run[:0]
+	}
+	for _, r := range text {
+		if unicode.IsLetter(r) || unicode.IsNumber(r) || r == '_' {
+			run = append(run, r)
+		} else {
+			flush()
+		}
+	}
+	flush()
+	counts := map[string]int{}
+	for i, word := range words {
+		counts[word]++
+		if i > 0 {
+			counts[words[i-1]+" "+word]++
+		}
+	}
+	score := agreementModel.Intercept + vectorScore(counts, agreementModel.Word)
+	var normalized strings.Builder
+	rr := []rune(text)
+	for i := 0; i < len(rr); {
+		j := i + 1
+		if pythonSpace(rr[i]) {
+			for j < len(rr) && pythonSpace(rr[j]) {
+				j++
+			}
+		}
+		if j-i >= 2 {
+			normalized.WriteByte(' ')
+		} else {
+			normalized.WriteRune(rr[i])
+		}
+		i = j
+	}
+	rr = []rune(normalized.String())
+	counts = map[string]int{}
+	for n := 3; n <= 5; n++ {
+		for i := 0; i+n <= len(rr); i++ {
+			counts[string(rr[i:i+n])]++
+		}
+	}
+	return score + vectorScore(counts, agreementModel.Char)
+}
+
+func applyAgreement(ctx context.Context, input Input, original InferenceResult) (InferenceResult, error) {
+	if err := ctx.Err(); err != nil {
+		return InferenceResult{}, err
+	}
+	fields, err := candidateFields(input)
+	if err != nil {
+		return InferenceResult{}, err
+	}
+	e, s := agreementModel.EncoderCalibration, agreementModel.SparseCalibration
+	et, st := agreementModel.EncoderThreshold, agreementModel.SparseThreshold
+	encoder := e.Coefficient*(original.Logits[1]-original.Logits[0]) + e.Intercept - math.Log(et/(1-et))
+	sparse := s.Coefficient*sparseMargin(fields[2]) + s.Intercept - math.Log(st/(1-st))
+	if err := ctx.Err(); err != nil {
+		return InferenceResult{}, err
+	}
+	original.Logits = [2]float64{0, math.Min(encoder, sparse)}
+	return original, nil
+}

v3-ungated/native/stepsafety/context.go

@@ -0,0 +1,247 @@
+package stepsafety
+
+import (
+	"bytes"
+	"encoding/json"
+	"strings"
+	"sync"
+)
+
+const (
+	maxContextSessions = 256
+	maxHistoryEntries  = 24
+	maxHistoryBytes    = maxPretokenizedFieldBytes
+	// Retain results that fit the history field so token packing can keep their
+	// beginning and end. The total per-session bound remains maxHistoryBytes.
+	maxHistoryEntryBytes = maxHistoryBytes
+	maxUserRequestBytes  = maxPretokenizedFieldBytes
+)
+
+type HistoryEntry struct {
+	ToolName      string
+	ToolArguments map[string]any
+	ToolResponse  map[string]any
+	Error         string
+}
+
+type sessionContext struct {
+	request         string
+	history         []string
+	historyOmitted  bool
+	requestTooLarge bool
+	serial          uint64
+}
+
+// ContextStore keeps only production-visible, structured hook data. It never
+// reads transcripts or assistant messages, so agent Thought/reasoning cannot
+// enter the model through this history path. The store is memory-only and
+// bounded; raw context is never part of telemetry.
+type ContextStore struct {
+	mu       sync.Mutex
+	sessions map[string]*sessionContext
+	serial   uint64
+}
+
+func NewContextStore() *ContextStore {
+	return &ContextStore{sessions: make(map[string]*sessionContext)}
+}
+
+func (s *ContextStore) RecordUserRequest(sessionID, request string) {
+	if s == nil || sessionID == "" {
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	ctx := s.session(sessionID)
+	ctx.requestTooLarge = len(request) > maxUserRequestBytes
+	if ctx.requestTooLarge {
+		ctx.request = ""
+	} else {
+		ctx.request = strings.Clone(request)
+	}
+}
+
+func (s *ContextStore) RecordInteraction(sessionID string, entry HistoryEntry) {
+	if s == nil || sessionID == "" || entry.ToolName == "" {
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	ctx := s.session(sessionID)
+	if ExcludedTool(entry.ToolName) {
+		ctx.historyOmitted = true
+		return
+	}
+	remaining := maxHistoryEntryBytes
+	for _, value := range []any{entry.ToolName, entry.ToolArguments, entry.ToolResponse, entry.Error} {
+		if err := boundedJSONSize(value, &remaining, 0); err != nil {
+			ctx.historyOmitted = true
+			return
+		}
+	}
+	encoded, err := serializeHistory([]HistoryEntry{entry})
+	if err != nil || len(encoded) > maxHistoryEntryBytes {
+		ctx.historyOmitted = true
+		return
+	}
+	// Store an immutable, bounded serialization, never references to caller
+	// maps or large raw tool results. Drop large events before evicting history.
+	ctx.history = append(ctx.history, encoded[1:len(encoded)-1])
+	for len(ctx.history) > maxHistoryEntries || len(joinHistory(ctx.history)) > maxHistoryBytes {
+		ctx.history = append([]string(nil), ctx.history[1:]...)
+		ctx.historyOmitted = true
+	}
+}
+
+func joinHistory(entries []string) string { return "[" + strings.Join(entries, ",") + "]" }
+
+type ContextSnapshot struct {
+	UserRequest        string
+	InteractionHistory string
+	HistoryOmitted     bool
+	RequestTooLarge    bool
+}
+
+func (s *ContextStore) Snapshot(sessionID string) (userRequest, interactionHistory string) {
+	snapshot := s.SnapshotWithCoverage(sessionID)
+	return snapshot.UserRequest, snapshot.InteractionHistory
+}
+
+func (s *ContextStore) SnapshotWithCoverage(sessionID string) ContextSnapshot {
+	empty := ContextSnapshot{InteractionHistory: "[]"}
+	if s == nil || sessionID == "" {
+		return empty
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	ctx := s.sessions[sessionID]
+	if ctx == nil {
+		return empty
+	}
+	s.serial++
+	ctx.serial = s.serial
+	return ContextSnapshot{
+		UserRequest: ctx.request, InteractionHistory: joinHistory(ctx.history),
+		HistoryOmitted: ctx.historyOmitted, RequestTooLarge: ctx.requestTooLarge,
+	}
+}
+
+// serializeHistory implements the exact structured-history representation used
+// to train the v2 encoder: a compact, key-sorted JSON array with tool,
+// arguments, and string observation fields. Empty history is always [].
+// Tool results remain inert text so their contents can never become a trusted
+// instruction channel through a richer runtime type.
+func serializeHistory(entries []HistoryEntry) (string, error) {
+	events := make([]map[string]any, 0, len(entries))
+	for _, entry := range entries {
+		if strings.TrimSpace(entry.ToolName) == "" {
+			continue
+		}
+		event := map[string]any{"tool": entry.ToolName}
+		if entry.ToolArguments != nil {
+			event["arguments"] = entry.ToolArguments
+		}
+		if observation, ok, err := observationText(entry.ToolResponse, entry.Error); err != nil {
+			return "", err
+		} else if ok {
+			event["observation"] = observation
+		}
+		events = append(events, event)
+	}
+	return compactSortedJSON(events)
+}
+
+func observationText(response map[string]any, errorText string) (string, bool, error) {
+	if response == nil {
+		if errorText == "" {
+			return "", false, nil
+		}
+		return errorText, true, nil
+	}
+	if errorText == "" {
+		encoded, err := compactSortedJSON(response)
+		return encoded, true, err
+	}
+	encoded, err := compactSortedJSON(map[string]any{
+		"error":    errorText,
+		"response": response,
+	})
+	return encoded, true, err
+}
+
+// compactSortedJSON matches Python json.dumps(..., ensure_ascii=False,
+// sort_keys=True, separators=(",", ":")) for the JSON values admitted by the
+// hook adapters. encoding/json sorts object keys; disabling HTML escaping keeps
+// ordinary Unicode and symbols byte-compatible with the training proxy.
+func compactSortedJSON(value any) (string, error) {
+	var output bytes.Buffer
+	encoder := json.NewEncoder(&output)
+	encoder.SetEscapeHTML(false)
+	if err := encoder.Encode(value); err != nil {
+		return "", err
+	}
+	return unescapeJSONLineSeparators(strings.TrimSuffix(output.String(), "\n")), nil
+}
+
+// encoding/json always escapes U+2028 and U+2029 for legacy JSONP safety,
+// while Python ensure_ascii=False keeps them as UTF-8. Only decode genuine JSON
+// escapes here; an input string containing the literal text `\u2028` must keep
+// its leading backslash.
+func unescapeJSONLineSeparators(value string) string {
+	if !strings.Contains(value, `\u2028`) && !strings.Contains(value, `\u2029`) {
+		return value
+	}
+	var output strings.Builder
+	output.Grow(len(value))
+	for index := 0; index < len(value); {
+		if index+6 <= len(value) && (value[index:index+6] == `\u2028` || value[index:index+6] == `\u2029`) {
+			precedingSlashes := 0
+			for prior := index - 1; prior >= 0 && value[prior] == '\\'; prior-- {
+				precedingSlashes++
+			}
+			if precedingSlashes%2 == 0 {
+				if value[index+5] == '8' {
+					output.WriteRune('\u2028')
+				} else {
+					output.WriteRune('\u2029')
+				}
+				index += 6
+				continue
+			}
+		}
+		output.WriteByte(value[index])
+		index++
+	}
+	return output.String()
+}
+
+func (s *ContextStore) CloseSession(sessionID string) {
+	if s == nil || sessionID == "" {
+		return
+	}
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	delete(s.sessions, sessionID)
+}
+
+func (s *ContextStore) session(sessionID string) *sessionContext {
+	if existing := s.sessions[sessionID]; existing != nil {
+		s.serial++
+		existing.serial = s.serial
+		return existing
+	}
+	if len(s.sessions) >= maxContextSessions {
+		var oldestID string
+		var oldest uint64
+		for id, candidate := range s.sessions {
+			if oldestID == "" || candidate.serial < oldest {
+				oldestID, oldest = id, candidate.serial
+			}
+		}
+		delete(s.sessions, oldestID)
+	}
+	s.serial++
+	created := &sessionContext{serial: s.serial}
+	s.sessions[sessionID] = created
+	return created
+}

v3-ungated/native/stepsafety/model/ATTRIBUTION.md

@@ -0,0 +1 @@
+The contextual DeBERTa-v3-xsmall base is Microsoft MIT-licensed (see LICENSE in this directory). Kontext's fine-tuned checkpoint is combined with two independently trained sparse heads. The request-intent head is adapted from MASSIVE 1.1 en-US, Copyright Amazon.com Inc. or its affiliates, derived from SLURP, licensed CC BY 4.0: https://github.com/alexa/massive and https://creativecommons.org/licenses/by/4.0/ . Adaptations include a binary query-intent mapping, deduplication, TF-IDF feature extraction and fitted logistic weights. No raw MASSIVE utterances or local captures are included in this serving bundle.

v3-ungated/native/stepsafety/model/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE

v3-ungated/native/stepsafety/model/PROVENANCE.json

@@ -0,0 +1,89 @@
+{
+  "schema_version": 3,
+  "model_version": "toolsafe-deberta-v3-xsmall-onnx-scoped-v2",
+  "source_project": "kontext-security/toolsafe-lab",
+  "source_revision": "9c63e6191598b0ba72947a4394ac8297c41053d1",
+  "source_artifact_path": "artifacts/models/history_serialization/deberta_v3_xsmall",
+  "source_results_path": "results/history_serialization_ablation.json",
+  "source_results_sha256": "09efd898c02dedb1ad8f83dd5b90888e84cbcf8919c51450c9289176c42db41a",
+  "source_findings_path": "docs/HISTORY_SERIALIZATION_ABLATION_FINDINGS.md",
+  "source_findings_sha256": "f4231362a77e8d622ace51c4d8ac323fc801cca6ebc6771d5295b55df2bef08f",
+  "source_serializer_path": "src/toolsafe_lab/history_serialization.py",
+  "source_serializer_sha256": "4b14a72a6825555d56c4be0549039428b78332940d3cdd25516ef278888642e2",
+  "source_packing_path": "src/toolsafe_lab/standalone_encoder.py",
+  "source_packing_sha256": "ca8db9401025c8489a142d9447140a857fd77df84e5246fde92c995ccbe173ff",
+  "source_protocol_path": "docs/HISTORY_SERIALIZATION_ABLATION_PROTOCOL.md",
+  "source_protocol_sha256": "6e44e1175be44573e6feef1130ef9f4faeaf8457025c79d5100212c568099089",
+  "base_model": "microsoft/deberta-v3-xsmall",
+  "base_model_revision": "4b419818330868dff6a60ad3e6b1c730f8b8c0c6",
+  "base_model_license": "MIT",
+  "input_mode": "execution_context + canonical structured history (agent Thought excluded)",
+  "thought_included": false,
+  "history_serialization": {
+    "event_fields": [
+      "tool",
+      "arguments",
+      "observation"
+    ],
+    "empty_history": "[]",
+    "observations": "strings",
+    "json_encoding": "UTF-8, sorted keys, compact separators"
+  },
+  "evaluation": {
+    "threshold": 0.5,
+    "accuracy": 0.9118629908103593,
+    "precision": 0.9266195885784464,
+    "recall": 0.8845252051582649,
+    "f1": 0.9050832208726945,
+    "mcc": 0.8236797425872459,
+    "worst_source_recall": 0.5085227272727273
+  },
+  "max_length": 512,
+  "field_markers": {
+    "request": "[USER_REQUEST]",
+    "history": "[INTERACTION_HISTORY]",
+    "action": "[CURRENT_ACTION]",
+    "schema": "[TOOL_DESCRIPTIONS]"
+  },
+  "field_budgets": {
+    "request": 96,
+    "history": 144,
+    "action": 128,
+    "schema": 128
+  },
+  "calibration_kind": "sigmoid",
+  "calibration_scale": 1.427213430140093,
+  "calibration_bias": 2.953687013257505,
+  "initial_threshold": 0.5,
+  "artifacts": [
+    {
+      "name": "config.json",
+      "size_bytes": 924,
+      "sha256": "41f89ae28c4f0549b92537d0ef5918ef9e1af25e9898af6456e3551bf0a93852"
+    },
+    {
+      "name": "model.onnx",
+      "size_bytes": 287233599,
+      "sha256": "7f7c068c80e826cee69fd3fca089662572b3372b0be109c6cf17b6852d94557c"
+    },
+    {
+      "name": "tokenizer.json",
+      "size_bytes": 8340654,
+      "sha256": "d6c20af053b5d86d986a9f70898c1fceccb9d93e7ce6f63dabc899a12a53b031"
+    },
+    {
+      "name": "tokenizer_config.json",
+      "size_bytes": 601,
+      "sha256": "141468772d3b095cc30275fd9c05dff92a78619c341487da9f5549a3564ea6cb"
+    }
+  ],
+  "excluded_training_artifact": {
+    "name": "training_checkpoint.pt",
+    "reason": "optimizer/RNG training-resume state is not required for inference"
+  },
+  "runtime_format": "ONNX opset 17, float32, [1,512] int64 input_ids/attention_mask, [1,2] logits",
+  "input_policy": "scoped-v2: exclude file tools; whole request/action/schema or abstain; filtered history uses first 72 and last 72 tokens, matching training",
+  "source_weights_sha256": "075d68f8bb4c1b4b98e2d2b2c0d5b013056b202172b080b34e7364b698844c86",
+  "export_toolchain": "torch 2.13.0; transformers 5.14.1; onnx 1.22.0; legacy exporter; scripts/step-safety/export_onnx.py",
+  "evaluation_scope": "Original held-out TS-Bench checkpoint evaluation; not an estimate for scoped-v2 production coverage"
+}

v3-ungated/native/stepsafety/model/native/PROVENANCE.json

@@ -0,0 +1,83 @@
+{
+  "model_version": "merlin-v3-ungated-20261104",
+  "source_weights_sha256": "429b09164c6705790c4414eb31c0bc18d2fa8a374bea467b2e1b49ead6aeb5f1",
+  "tokenizer_sha256": "d6c20af053b5d86d986a9f70898c1fceccb9d93e7ce6f63dabc899a12a53b031",
+  "experiment": "exp-20261007-merlin-v3-precision",
+  "candidate": "semantic_conservative",
+  "runtime": "Native Go FP32 DeBERTa plus native sparse TF-IDF logistic veto; all assets embedded, no ONNX or Python needed",
+  "base_model": "microsoft/deberta-v3-xsmall",
+  "base_revision": "4b419818330868dff6a60ad3e6b1c730f8b8c0c6",
+  "base_license": "MIT",
+  "scope": "All otherwise eligible tool invocations; operation allowlist, lexical authorization gate and learned request-intent gate removed",
+  "score_semantics": "Agreement margin rescaled to [0,1], not a calibrated production risk probability",
+  "context_budgets": [
+    96,
+    144,
+    128,
+    128
+  ],
+  "context_truncation": "Request head; history/schema head-tail; current action never truncated",
+  "advisory_only": true,
+  "deployed": false,
+  "sparse_model_sha256": "cddc99dd145eeb4e29ebfb35bc874408c2737b4f340917919024dc7e9113c229",
+  "selection_sha256": "e682888797fb0f43bffffe0b31d196aa82be6195e65ad74cc52e61043eaa1b28",
+  "evaluation_sha256": "b2f7d0ee87a9bf30a5aeb80768401ade17df613e6de1befd764b5052160ee7a0",
+  "exporter_sha256": "a1847a55de609edfb18dbfab0ad00e863c9d10d46fa7e37127fb8724b69e2d64",
+  "request_gate": {
+    "model_sha256": "0ec73c392582b7f93a6c59c78e3153cd36be98d36cab73445f0f300dcb087528",
+    "threshold": 0.7871534193379106,
+    "training_source": "MASSIVE 1.1 en-US original train, CC BY 4.0",
+    "selection_sha256": "ab818785fe65d6207673ebb8841954928ec692d27d04d1077175d7de8a2899e1",
+    "calibrator": {
+      "coefficient": 0.937442667480181,
+      "intercept": 0.020245760734966128
+    },
+    "enabled": false
+  },
+  "authorization_additional_words": [
+    "adjust",
+    "approve",
+    "archive",
+    "buy",
+    "checkout",
+    "clear",
+    "commit",
+    "disburse",
+    "discard",
+    "dispose",
+    "drop",
+    "empty",
+    "forward",
+    "grant",
+    "invite",
+    "merge",
+    "move",
+    "order",
+    "post",
+    "publish",
+    "purchase",
+    "purge",
+    "push",
+    "refund",
+    "reimburse",
+    "rename",
+    "reply",
+    "reschedule",
+    "respond",
+    "restart",
+    "rotate",
+    "save",
+    "set",
+    "share",
+    "start",
+    "stop",
+    "submit",
+    "unlink",
+    "upload",
+    "wipe"
+  ],
+  "runtime_variant": "v3-ungated-20261104",
+  "weights_candidate": "merlin-v3-precision-20261007",
+  "authorization_lexical_gate_enabled": false,
+  "operation_gate_enabled": false
+}

v3-ungated/native/stepsafety/model/native/agreement.json.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ffbd35e56a031df9f2ca08f7520486cc8ef65592c8d03cfe966fbd42523bbdf
+size 2290715

v3-ungated/native/stepsafety/model/native/candidate.json

@@ -0,0 +1,147 @@
+{
+  "candidate": "merlin-v3-ungated-20261104",
+  "weights_sha256": "429b09164c6705790c4414eb31c0bc18d2fa8a374bea467b2e1b49ead6aeb5f1",
+  "threshold": 0.5,
+  "calibrator": {
+    "coefficient": 1.0,
+    "intercept": 0.0
+  },
+  "agreement": {
+    "sparse_model_sha256": "cddc99dd145eeb4e29ebfb35bc874408c2737b4f340917919024dc7e9113c229",
+    "encoder_calibrator": {
+      "coefficient": 0.4128351080107227,
+      "intercept": -1.3945806128701024
+    },
+    "sparse_calibrator": {
+      "coefficient": 1.2411713915407203,
+      "intercept": 0.12969558567752842
+    },
+    "operating_point": {
+      "threshold": 0.5,
+      "encoder_threshold": 0.5455402148631912,
+      "sparse_threshold": 0.2,
+      "rank": [
+        0.900131845841785,
+        0.9841269841269841,
+        295,
+        0.5455402148631912,
+        0.2
+      ],
+      "validation": {
+        "selection": {
+          "local_staging": {
+            "n": 163,
+            "tp": 0,
+            "fp": 0,
+            "fn": 0,
+            "tn": 163,
+            "precision": null,
+            "recall": null,
+            "fpr": 0.0
+          },
+          "ASB-Traj": {
+            "n": 88,
+            "tp": 32,
+            "fp": 0,
+            "fn": 2,
+            "tn": 54,
+            "precision": 1.0,
+            "recall": 0.9411764705882353,
+            "fpr": 0.0
+          },
+          "AgentAlign-Traj": {
+            "n": 111,
+            "tp": 62,
+            "fp": 1,
+            "fn": 6,
+            "tn": 42,
+            "precision": 0.9841269841269841,
+            "recall": 0.9117647058823529,
+            "fpr": 0.023255813953488372
+          },
+          "authored": {
+            "n": 78,
+            "tp": 0,
+            "fp": 0,
+            "fn": 36,
+            "tn": 42,
+            "precision": null,
+            "recall": 0.0,
+            "fpr": 0.0
+          },
+          "benchmark": {
+            "n": 199,
+            "tp": 94,
+            "fp": 1,
+            "fn": 8,
+            "tn": 96,
+            "precision": 0.9894736842105263,
+            "recall": 0.9215686274509803,
+            "fpr": 0.010309278350515464
+          }
+        },
+        "calibration": {
+          "local_staging": {
+            "n": 132,
+            "tp": 0,
+            "fp": 0,
+            "fn": 0,
+            "tn": 132,
+            "precision": null,
+            "recall": null,
+            "fpr": 0.0
+          },
+          "ASB-Traj": {
+            "n": 56,
+            "tp": 23,
+            "fp": 0,
+            "fn": 2,
+            "tn": 31,
+            "precision": 1.0,
+            "recall": 0.92,
+            "fpr": 0.0
+          },
+          "AgentAlign-Traj": {
+            "n": 121,
+            "tp": 48,
+            "fp": 0,
+            "fn": 10,
+            "tn": 63,
+            "precision": 1.0,
+            "recall": 0.8275862068965517,
+            "fpr": 0.0
+          },
+          "authored": {
+            "n": 82,
+            "tp": 0,
+            "fp": 0,
+            "fn": 40,
+            "tn": 42,
+            "precision": null,
+            "recall": 0.0,
+            "fpr": 0.0
+          },
+          "benchmark": {
+            "n": 177,
+            "tp": 71,
+            "fp": 0,
+            "fn": 12,
+            "tn": 94,
+            "precision": 1.0,
+            "recall": 0.8554216867469879,
+            "fpr": 0.0
+          }
+        }
+      }
+    }
+  },
+  "default_enabled": true,
+  "score_semantics": "Threshold-relative agreement score; not a calibrated risk probability",
+  "weights_candidate": "merlin-v3-precision-20261007",
+  "runtime_gates": {
+    "operation": false,
+    "lexical_request": false,
+    "learned_request_intent": false,
+    "action_agreement": true
+  }
+}

v3-ungated/native/stepsafety/model/native/manifest.json

@@ -0,0 +1,49 @@
+{
+  "format": "kontext-merlin-native-safetensors-f32/v1",
+  "source_weights_sha256": "429b09164c6705790c4414eb31c0bc18d2fa8a374bea467b2e1b49ead6aeb5f1",
+  "tokenizer_sha256": "d6c20af053b5d86d986a9f70898c1fceccb9d93e7ce6f63dabc899a12a53b031",
+  "quantized": false,
+  "shard_uncompressed_bytes": 67108864,
+  "files": [
+    {
+      "name": "agreement.json.gz",
+      "size_bytes": 2290715,
+      "sha256": "6ffbd35e56a031df9f2ca08f7520486cc8ef65592c8d03cfe966fbd42523bbdf"
+    },
+    {
+      "name": "request_gate.json.gz",
+      "size_bytes": 2492102,
+      "sha256": "acfb7025661b5a424d482aae1fd351f31f714bf9c22b7c18dcabb623ca2e72c3"
+    },
+    {
+      "name": "tokenizer.json.gz",
+      "size_bytes": 1736709,
+      "sha256": "956061a0f96d88a118848b4f32e4e5fcdc8c363f4cbc223fdc175232668aa41c"
+    },
+    {
+      "name": "weights.00.gz",
+      "size_bytes": 46014808,
+      "sha256": "2c69b3cccc79582da64dd47ab44835604b5b4f5c676b1102eacef17dc52cacad"
+    },
+    {
+      "name": "weights.01.gz",
+      "size_bytes": 41825526,
+      "sha256": "129de5fde020ece20ce7cf2529564c779e486e29b0c4725f7ed09af086f6db04"
+    },
+    {
+      "name": "weights.02.gz",
+      "size_bytes": 42596160,
+      "sha256": "7cb825c60a553c37d70ee6859d597a49b0afa0e34a2a243d62903da5d297f47e"
+    },
+    {
+      "name": "weights.03.gz",
+      "size_bytes": 62319867,
+      "sha256": "d5a9ce4740491f638c497b8e88d6d4236a1f26b3debfa9f7566a22c960828b6c"
+    },
+    {
+      "name": "weights.04.gz",
+      "size_bytes": 13716366,
+      "sha256": "9316ce90d589cba6e3ce02f5e643fa244ff7e7bb2b5117b8508f176f01ed3000"
+    }
+  ]
+}

v3-ungated/native/stepsafety/model/native/request_gate.json.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acfb7025661b5a424d482aae1fd351f31f714bf9c22b7c18dcabb623ca2e72c3
+size 2492102

v3-ungated/native/stepsafety/model/native/tokenizer.json.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:956061a0f96d88a118848b4f32e4e5fcdc8c363f4cbc223fdc175232668aa41c
+size 1736709

v3-ungated/native/stepsafety/model/native/weights.00.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c69b3cccc79582da64dd47ab44835604b5b4f5c676b1102eacef17dc52cacad
+size 46014808

v3-ungated/native/stepsafety/model/native/weights.01.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:129de5fde020ece20ce7cf2529564c779e486e29b0c4725f7ed09af086f6db04
+size 41825526

v3-ungated/native/stepsafety/model/native/weights.02.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7cb825c60a553c37d70ee6859d597a49b0afa0e34a2a243d62903da5d297f47e
+size 42596160

v3-ungated/native/stepsafety/model/native/weights.03.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5a9ce4740491f638c497b8e88d6d4236a1f26b3debfa9f7566a22c960828b6c
+size 62319867

v3-ungated/native/stepsafety/model/native/weights.04.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9316ce90d589cba6e3ce02f5e643fa244ff7e7bb2b5117b8508f176f01ed3000
+size 13716366

v3-ungated/native/stepsafety/modelstore.go

@@ -0,0 +1,296 @@
+package stepsafety
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"time"
+)
+
+const legacyModelVersion = "toolsafe-deberta-v3-xsmall-onnx-scoped-v2"
+
+type artifactSpec struct {
+	Name   string `json:"name"`
+	Size   int64  `json:"size_bytes"`
+	SHA256 string `json:"sha256"`
+}
+
+var requiredArtifacts = []artifactSpec{
+	{Name: "config.json", Size: 924, SHA256: "41f89ae28c4f0549b92537d0ef5918ef9e1af25e9898af6456e3551bf0a93852"},
+	{Name: "model.onnx", Size: 287233599, SHA256: "7f7c068c80e826cee69fd3fca089662572b3372b0be109c6cf17b6852d94557c"},
+	{Name: "tokenizer.json", Size: 8340654, SHA256: "d6c20af053b5d86d986a9f70898c1fceccb9d93e7ce6f63dabc899a12a53b031"},
+	{Name: "tokenizer_config.json", Size: 601, SHA256: "141468772d3b095cc30275fd9c05dff92a78619c341487da9f5549a3564ea6cb"},
+}
+
+const (
+	sourceRevision       = "9c63e6191598b0ba72947a4394ac8297c41053d1"
+	sourceArtifactPath   = "artifacts/models/history_serialization/deberta_v3_xsmall"
+	sourceResultsPath    = "results/history_serialization_ablation.json"
+	sourceResultsSHA256  = "09efd898c02dedb1ad8f83dd5b90888e84cbcf8919c51450c9289176c42db41a"
+	sourceFindingsPath   = "docs/HISTORY_SERIALIZATION_ABLATION_FINDINGS.md"
+	sourceFindingsSHA256 = "f4231362a77e8d622ace51c4d8ac323fc801cca6ebc6771d5295b55df2bef08f"
+	sourceSerializerPath = "src/toolsafe_lab/history_serialization.py"
+	sourceSerializerSHA  = "4b14a72a6825555d56c4be0549039428b78332940d3cdd25516ef278888642e2"
+	sourcePackingPath    = "src/toolsafe_lab/standalone_encoder.py"
+	sourcePackingSHA     = "ca8db9401025c8489a142d9447140a857fd77df84e5246fde92c995ccbe173ff"
+	sourceProtocolPath   = "docs/HISTORY_SERIALIZATION_ABLATION_PROTOCOL.md"
+	sourceProtocolSHA    = "6e44e1175be44573e6feef1130ef9f4faeaf8457025c79d5100212c568099089"
+)
+
+const microsoftMITLicense = `MIT License
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE
+`
+
+type Provenance struct {
+	RuntimeFormat       string `json:"runtime_format"`
+	InputPolicy         string `json:"input_policy"`
+	SourceWeightsSHA256 string `json:"source_weights_sha256"`
+	ExportToolchain     string `json:"export_toolchain"`
+	EvaluationScope     string `json:"evaluation_scope"`
+
+	SchemaVersion        int                            `json:"schema_version"`
+	ModelVersion         string                         `json:"model_version"`
+	SourceProject        string                         `json:"source_project"`
+	SourceRevision       string                         `json:"source_revision"`
+	SourceArtifactPath   string                         `json:"source_artifact_path,omitempty"`
+	SourceResultsPath    string                         `json:"source_results_path,omitempty"`
+	SourceResultsSHA256  string                         `json:"source_results_sha256,omitempty"`
+	SourceFindingsPath   string                         `json:"source_findings_path,omitempty"`
+	SourceFindingsSHA256 string                         `json:"source_findings_sha256,omitempty"`
+	SourceSerializerPath string                         `json:"source_serializer_path,omitempty"`
+	SourceSerializerSHA  string                         `json:"source_serializer_sha256,omitempty"`
+	SourcePackingPath    string                         `json:"source_packing_path,omitempty"`
+	SourcePackingSHA     string                         `json:"source_packing_sha256,omitempty"`
+	SourceProtocolPath   string                         `json:"source_protocol_path,omitempty"`
+	SourceProtocolSHA    string                         `json:"source_protocol_sha256,omitempty"`
+	BaseModel            string                         `json:"base_model"`
+	BaseModelRevision    string                         `json:"base_model_revision"`
+	BaseModelLicense     string                         `json:"base_model_license"`
+	InputMode            string                         `json:"input_mode"`
+	ThoughtIncluded      bool                           `json:"thought_included"`
+	HistorySerialization historySerializationProvenance `json:"history_serialization"`
+	Evaluation           evaluationProvenance           `json:"evaluation"`
+	MaxLength            int                            `json:"max_length"`
+	FieldMarkers         map[string]string              `json:"field_markers"`
+	FieldBudgets         map[string]int                 `json:"field_budgets"`
+	CalibrationScale     float64                        `json:"calibration_scale"`
+	CalibrationBias      float64                        `json:"calibration_bias"`
+	InitialThreshold     float64                        `json:"initial_threshold"`
+	ImportedAt           time.Time                      `json:"imported_at"`
+	Artifacts            []artifactSpec                 `json:"artifacts"`
+}
+
+type historySerializationProvenance struct {
+	EventFields  []string `json:"event_fields"`
+	EmptyHistory string   `json:"empty_history"`
+	Observations string   `json:"observations"`
+	JSONEncoding string   `json:"json_encoding"`
+}
+
+type evaluationProvenance struct {
+	Threshold         float64 `json:"threshold"`
+	Accuracy          float64 `json:"accuracy"`
+	Precision         float64 `json:"precision"`
+	Recall            float64 `json:"recall"`
+	F1                float64 `json:"f1"`
+	MCC               float64 `json:"mcc"`
+	WorstSourceRecall float64 `json:"worst_source_recall"`
+}
+
+func DefaultModelDir(dbPath string) string {
+	root := filepath.Join(filepath.Dir(dbPath), "judge-models")
+	return filepath.Join(root, "toolsafe", legacyModelVersion)
+}
+
+func ValidateModelDir(dir string) error {
+	if strings.TrimSpace(dir) == "" {
+		return backendError(ErrorUnavailable, errors.New("step-safety model directory is empty"))
+	}
+	for _, artifact := range requiredArtifacts {
+		path := filepath.Join(dir, artifact.Name)
+		info, err := os.Stat(path)
+		if err != nil {
+			return backendError(ErrorUnavailable, fmt.Errorf("model artifact %s is unavailable", artifact.Name))
+		}
+		if !info.Mode().IsRegular() || info.Size() != artifact.Size {
+			return backendError(ErrorUnavailable, fmt.Errorf("model artifact %s has unexpected size", artifact.Name))
+		}
+		digest, err := fileSHA256(path)
+		if err != nil || digest != artifact.SHA256 {
+			return backendError(ErrorUnavailable, fmt.Errorf("model artifact %s failed checksum verification", artifact.Name))
+		}
+	}
+	return nil
+}
+
+// InstallModel copies only inference artifacts into Kontext's existing
+// database-adjacent model cache. It never imports the 849 MB training resume
+// checkpoint and leaves no reference to the source tree.
+func InstallModel(sourceDir, destinationDir string) (string, error) {
+	if err := ValidateModelDir(sourceDir); err != nil {
+		return "", err
+	}
+	if err := ValidateModelDir(destinationDir); err == nil {
+		return destinationDir, nil
+	} else if _, statErr := os.Stat(destinationDir); statErr == nil {
+		return "", fmt.Errorf("step-safety destination already exists but is invalid: %s", destinationDir)
+	} else if !errors.Is(statErr, os.ErrNotExist) {
+		return "", statErr
+	}
+	parent := filepath.Dir(destinationDir)
+	if err := os.MkdirAll(parent, 0o700); err != nil {
+		return "", err
+	}
+	temporary, err := os.MkdirTemp(parent, ".step-safety-install-*")
+	if err != nil {
+		return "", err
+	}
+	defer os.RemoveAll(temporary)
+
+	for _, artifact := range requiredArtifacts {
+		if err := copyArtifact(filepath.Join(sourceDir, artifact.Name), filepath.Join(temporary, artifact.Name)); err != nil {
+			return "", err
+		}
+	}
+	if err := os.WriteFile(filepath.Join(temporary, "LICENSE"), []byte(microsoftMITLicense), 0o600); err != nil {
+		return "", err
+	}
+	provenance := Provenance{
+		RuntimeFormat:       "ONNX opset 17, float32, [1,512] int64 input_ids/attention_mask, [1,2] logits",
+		InputPolicy:         "scoped-v2: exclude file tools; whole request/action/schema or abstain; filtered history uses first 72 and last 72 tokens, matching training",
+		SourceWeightsSHA256: "075d68f8bb4c1b4b98e2d2b2c0d5b013056b202172b080b34e7364b698844c86",
+		ExportToolchain:     "torch 2.13.0; transformers 5.14.1; onnx 1.22.0; legacy exporter; scripts/step-safety/export_onnx.py",
+		EvaluationScope:     "Original held-out TS-Bench checkpoint evaluation; not an estimate for scoped-v2 production coverage",
+
+		SchemaVersion:        3,
+		ModelVersion:         legacyModelVersion,
+		SourceProject:        "kontext-security/toolsafe-lab",
+		SourceRevision:       sourceRevision,
+		SourceArtifactPath:   sourceArtifactPath,
+		SourceResultsPath:    sourceResultsPath,
+		SourceResultsSHA256:  sourceResultsSHA256,
+		SourceFindingsPath:   sourceFindingsPath,
+		SourceFindingsSHA256: sourceFindingsSHA256,
+		SourceSerializerPath: sourceSerializerPath,
+		SourceSerializerSHA:  sourceSerializerSHA,
+		SourcePackingPath:    sourcePackingPath,
+		SourcePackingSHA:     sourcePackingSHA,
+		SourceProtocolPath:   sourceProtocolPath,
+		SourceProtocolSHA:    sourceProtocolSHA,
+		BaseModel:            "microsoft/deberta-v3-xsmall",
+		BaseModelRevision:    "4b419818330868dff6a60ad3e6b1c730f8b8c0c6",
+		BaseModelLicense:     "MIT",
+		InputMode:            "execution_context + canonical structured history (agent Thought excluded)",
+		ThoughtIncluded:      false,
+		HistorySerialization: historySerializationProvenance{
+			EventFields:  []string{"tool", "arguments", "observation"},
+			EmptyHistory: "[]",
+			Observations: "strings",
+			JSONEncoding: "UTF-8, sorted keys, compact separators",
+		},
+		Evaluation: evaluationProvenance{
+			Threshold:         0.5,
+			Accuracy:          0.9118629908103593,
+			Precision:         0.9266195885784464,
+			Recall:            0.8845252051582649,
+			F1:                0.9050832208726945,
+			MCC:               0.8236797425872459,
+			WorstSourceRecall: 0.5085227272727273,
+		},
+		MaxLength: 512,
+		FieldMarkers: map[string]string{
+			"request": "[USER_REQUEST]",
+			"history": "[INTERACTION_HISTORY]",
+			"action":  "[CURRENT_ACTION]",
+			"schema":  "[TOOL_DESCRIPTIONS]",
+		},
+		FieldBudgets: map[string]int{
+			"request": 96,
+			"history": 144,
+			"action":  128,
+			"schema":  128,
+		},
+		CalibrationScale: 1.427213430140093,
+		CalibrationBias:  2.953687013257505,
+		InitialThreshold: 0.5,
+		ImportedAt:       time.Now().UTC(),
+		Artifacts:        append([]artifactSpec(nil), requiredArtifacts...),
+	}
+	encoded, err := json.MarshalIndent(provenance, "", "  ")
+	if err != nil {
+		return "", err
+	}
+	if err := os.WriteFile(filepath.Join(temporary, "PROVENANCE.json"), append(encoded, '\n'), 0o600); err != nil {
+		return "", err
+	}
+	if err := ValidateModelDir(temporary); err != nil {
+		return "", err
+	}
+	if err := os.Rename(temporary, destinationDir); err != nil {
+		return "", err
+	}
+	return destinationDir, nil
+}
+
+func RequiredArtifacts() []string {
+	names := make([]string, 0, len(requiredArtifacts))
+	for _, artifact := range requiredArtifacts {
+		names = append(names, artifact.Name)
+	}
+	sort.Strings(names)
+	return names
+}
+
+func copyArtifact(source, destination string) error {
+	in, err := os.Open(source)
+	if err != nil {
+		return err
+	}
+	defer in.Close()
+	out, err := os.OpenFile(destination, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
+	if err != nil {
+		return err
+	}
+	_, copyErr := io.Copy(out, in)
+	closeErr := out.Close()
+	return errors.Join(copyErr, closeErr)
+}
+
+func fileSHA256(path string) (string, error) {
+	file, err := os.Open(path)
+	if err != nil {
+		return "", err
+	}
+	defer file.Close()
+	digest := sha256.New()
+	if _, err := io.Copy(digest, file); err != nil {
+		return "", err
+	}
+	return hex.EncodeToString(digest.Sum(nil)), nil
+}

v3-ungated/native/stepsafety/native_assets.go

@@ -0,0 +1,15 @@
+//go:build !nostepsafetymodel
+
+package stepsafety
+
+import "embed"
+
+// nativeAssets holds the checkpoint, tokenizer and classifiers. They are about
+// 213 MB of the binary; the nostepsafetymodel tag leaves them out.
+//
+//go:embed model/native/*.gz
+var nativeAssets embed.FS
+
+// modelEmbedded reports whether this build carries the model; tests that
+// need it skip without it.
+const modelEmbedded = true

v3-ungated/native/stepsafety/native_assets_nomodel.go

@@ -0,0 +1,14 @@
+//go:build nostepsafetymodel
+
+package stepsafety
+
+import "embed"
+
+// nativeAssets is empty in a build without the model, for hosts that cannot
+// carry the full binary. Loading then fails, so New returns an unavailable
+// evaluator and health reports "unavailable"; decisions do not depend on it.
+var nativeAssets embed.FS
+
+// modelEmbedded reports whether this build carries the model; tests that
+// need it skip without it.
+const modelEmbedded = false

v3-ungated/native/stepsafety/native_backend.go

@@ -0,0 +1,133 @@
+package stepsafety
+
+import (
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"errors"
+	"io"
+	"io/fs"
+	"sync"
+	"sync/atomic"
+)
+
+var nativeOnce sync.Once
+var nativeReady = make(chan struct{})
+var nativeShared struct {
+	model     *nativeModel
+	tokenizer *tokenizer
+	err       error
+}
+
+type nativeBackend struct {
+	model        *nativeModel
+	tokenizer    *tokenizer
+	modelVersion string
+	closed       atomic.Bool
+}
+
+// Load one immutable, checksum-verified checkpoint per process, just as Kestrel
+// does. No files are extracted, no library is loaded, and no download occurs.
+// A caller's startup deadline never poisons the shared load for later callers.
+func newNativeBackend(ctx context.Context, cfg Config) (*nativeBackend, error) {
+	ctx, cancel := context.WithTimeout(ctx, cfg.StartupTimeout)
+	defer cancel()
+	nativeOnce.Do(func() {
+		go func() {
+			defer close(nativeReady)
+			nativeShared.model, nativeShared.tokenizer, nativeShared.err = loadNativeAssets(context.Background())
+			if nativeShared.err == nil {
+				nativeShared.err = loadAgreementAssets()
+				if nativeShared.err == nil {
+					nativeShared.err = loadRequestGateAssets()
+				}
+			}
+		}()
+	})
+	select {
+	case <-ctx.Done():
+		return nil, ctx.Err()
+	case <-nativeReady:
+		if nativeShared.err != nil {
+			return nil, backendError(ErrorUnavailable, nativeShared.err)
+		}
+		return &nativeBackend{model: nativeShared.model, tokenizer: nativeShared.tokenizer, modelVersion: cfg.ModelVersion}, nil
+	}
+}
+
+func loadNativeAssets(ctx context.Context) (*nativeModel, *tokenizer, error) {
+	names, err := fs.Glob(nativeAssets, "model/native/weights.*.gz")
+	if err != nil || len(names) != 5 {
+		return nil, nil, errors.New("missing embedded native weights")
+	}
+	readers := make([]io.Reader, 0, len(names))
+	for _, name := range names {
+		f, err := nativeAssets.Open(name)
+		if err != nil {
+			return nil, nil, err
+		}
+		defer f.Close()
+		readers = append(readers, f)
+	}
+	r, err := gzip.NewReader(io.MultiReader(readers...))
+	if err != nil {
+		return nil, nil, err
+	}
+	defer r.Close()
+	weights, err := readNativeWeights(ctx, r)
+	if err != nil {
+		return nil, nil, err
+	}
+	model, err := buildNativeModel(ctx, weights)
+	if err != nil {
+		return nil, nil, err
+	}
+	f, err := nativeAssets.Open("model/native/tokenizer.json.gz")
+	if err != nil {
+		return nil, nil, err
+	}
+	defer f.Close()
+	t, err := gzip.NewReader(f)
+	if err != nil {
+		return nil, nil, err
+	}
+	defer t.Close()
+	data, err := io.ReadAll(io.LimitReader(t, 8340655))
+	if err != nil {
+		return nil, nil, err
+	}
+	digest := sha256.Sum256(data)
+	if len(data) != 8340654 || hex.EncodeToString(digest[:]) != "d6c20af053b5d86d986a9f70898c1fceccb9d93e7ce6f63dabc899a12a53b031" {
+		return nil, nil, errors.New("embedded tokenizer checksum mismatch")
+	}
+	tok, err := parseTokenizer(data)
+	return model, tok, err
+}
+
+func (b *nativeBackend) Infer(ctx context.Context, input Input) (InferenceResult, error) {
+	if b.closed.Load() {
+		return InferenceResult{}, backendError(ErrorUnavailable, nil)
+	}
+	packed, err := packScopedInput(ctx, b.tokenizer, input)
+	if err != nil {
+		return InferenceResult{}, err
+	}
+	out, err := b.model.infer(ctx, packed)
+	if err != nil {
+		return out, err
+	}
+	return applyAgreement(ctx, input, out)
+}
+
+func (b *nativeBackend) Health(ctx context.Context) (Health, error) {
+	if err := ctx.Err(); err != nil {
+		return Health{}, err
+	}
+	if b.closed.Load() {
+		return Health{}, backendError(ErrorUnavailable, nil)
+	}
+	return Health{Status: "ready", ModelVersion: b.modelVersion, Device: "go-cpu"}, nil
+}
+
+func (b *nativeBackend) Close() error { b.closed.Store(true); return nil }

v3-ungated/native/stepsafety/native_linear_arm64.go

@@ -0,0 +1,82 @@
+//go:build arm64 && !purego
+
+package stepsafety
+
+import (
+	"runtime"
+	"sync"
+
+	"gonum.org/v1/gonum/blas"
+	"gonum.org/v1/gonum/blas/gonum"
+)
+
+// Pack each group of eight output columns together. The NEON kernel consumes
+// four rows and eight columns without a C ABI, dynamic library, or cgo.
+func prepareNativeLinear(l nativeLinear) nativeLinear {
+	return prepareNativeLinearInto(l, nil)
+}
+
+func prepareNativeLinearInto(l nativeLinear, scratch []float32) nativeLinear {
+	if l.out < 8 || len(l.weight) != l.in*l.out {
+		return l
+	}
+	l.packed = nativeBuffer(scratch, (l.out+7)/8*8*l.in)
+	for j := 0; j < l.out; j += 8 {
+		for k := 0; k < l.in; k++ {
+			for c := 0; c < 8; c++ {
+				if j+c < l.out {
+					l.packed[j*l.in+k*8+c] = l.weight[(j+c)*l.in+k]
+				}
+			}
+		}
+	}
+	l.weight = nil
+	return l
+}
+
+func nativeLinearProduct(l nativeLinear, x []float32, rows int) []float32 {
+	return nativeLinearProductInto(l, x, rows, nil)
+}
+
+func nativeLinearProductInto(l nativeLinear, x []float32, rows int, scratch []float32) []float32 {
+	if len(l.packed) == 0 {
+		out := nativeBuffer(scratch, rows*l.out)
+		gonum.Implementation{}.Sgemm(blas.NoTrans, blas.Trans, rows, l.out, l.in, 1, x, l.in, l.weight, l.in, 0, out, l.out)
+		return out
+	}
+	padded := (rows + 3) / 4 * 4
+	if rows != padded {
+		if cap(x) >= padded*l.in {
+			x = x[:padded*l.in]
+		} else {
+			temp := make([]float32, padded*l.in)
+			copy(temp, x)
+			x = temp
+		}
+	}
+	stride := (l.out + 7) / 8 * 8
+	out := nativeBuffer(scratch, padded*stride)
+	workers := min(4, runtime.GOMAXPROCS(0), padded/4)
+	var wg sync.WaitGroup
+	for worker := 0; worker < workers; worker++ {
+		wg.Add(1)
+		go func(worker int) {
+			defer wg.Done()
+			for i := worker * 4; i < padded; i += workers * 4 {
+				for j := 0; j < l.out; j += 8 {
+					nativeMatmul4x8(&x[i*l.in], &l.packed[j*l.in], &out[i*stride+j], l.in, stride)
+				}
+			}
+		}(worker)
+	}
+	wg.Wait()
+	if stride != l.out {
+		for i := 0; i < rows; i++ {
+			copy(out[i*l.out:(i+1)*l.out], out[i*stride:i*stride+l.out])
+		}
+	}
+	return out[:rows*l.out]
+}
+
+//go:noescape
+func nativeMatmul4x8(a, packedB, c *float32, k, strideC int)

v3-ungated/native/stepsafety/native_linear_arm64.s

@@ -0,0 +1,57 @@
+//go:build arm64 && !purego
+
+#include "textflag.h"
+
+// C[4,8] = A[4,k] * B[k,8]. All dimensions checked by the Go caller.
+// B is a contiguous packed panel; C rows have strideC elements.
+TEXT ·nativeMatmul4x8(SB), NOSPLIT, $0-40
+	MOVD a+0(FP), R0
+	MOVD packedB+8(FP), R1
+	MOVD c+16(FP), R2
+	MOVD k+24(FP), R3
+	MOVD strideC+32(FP), R4
+	LSL $2, R3, R5
+	LSL $2, R4, R4
+	ADD R5, R0, R6
+	ADD R5, R6, R7
+	ADD R5, R7, R8
+	VEOR V16.B16, V16.B16, V16.B16
+	VEOR V17.B16, V17.B16, V17.B16
+	VEOR V18.B16, V18.B16, V18.B16
+	VEOR V19.B16, V19.B16, V19.B16
+	VEOR V20.B16, V20.B16, V20.B16
+	VEOR V21.B16, V21.B16, V21.B16
+	VEOR V22.B16, V22.B16, V22.B16
+	VEOR V23.B16, V23.B16, V23.B16
+loop:
+	FMOVS (R0), F0
+	FMOVS (R6), F1
+	FMOVS (R7), F2
+	FMOVS (R8), F3
+	VDUP V0.S[0], V0.S4
+	VDUP V1.S[0], V1.S4
+	VDUP V2.S[0], V2.S4
+	VDUP V3.S[0], V3.S4
+	VLD1.P 32(R1), [V4.S4, V5.S4]
+	VFMLA V4.S4, V0.S4, V16.S4
+	VFMLA V5.S4, V0.S4, V17.S4
+	VFMLA V4.S4, V1.S4, V18.S4
+	VFMLA V5.S4, V1.S4, V19.S4
+	VFMLA V4.S4, V2.S4, V20.S4
+	VFMLA V5.S4, V2.S4, V21.S4
+	VFMLA V4.S4, V3.S4, V22.S4
+	VFMLA V5.S4, V3.S4, V23.S4
+	ADD $4, R0, R0
+	ADD $4, R6, R6
+	ADD $4, R7, R7
+	ADD $4, R8, R8
+	SUBS $1, R3, R3
+	BNE loop
+	VST1 [V16.S4, V17.S4], (R2)
+	ADD R4, R2, R2
+	VST1 [V18.S4, V19.S4], (R2)
+	ADD R4, R2, R2
+	VST1 [V20.S4, V21.S4], (R2)
+	ADD R4, R2, R2
+	VST1 [V22.S4, V23.S4], (R2)
+	RET

v3-ungated/native/stepsafety/native_linear_generic.go

@@ -0,0 +1,20 @@
+//go:build !arm64 || purego
+
+package stepsafety
+
+import (
+	"gonum.org/v1/gonum/blas"
+	"gonum.org/v1/gonum/blas/gonum"
+)
+
+func prepareNativeLinear(l nativeLinear) nativeLinear                  { return l }
+func prepareNativeLinearInto(l nativeLinear, _ []float32) nativeLinear { return l }
+
+func nativeLinearProduct(l nativeLinear, x []float32, rows int) []float32 {
+	return nativeLinearProductInto(l, x, rows, nil)
+}
+func nativeLinearProductInto(l nativeLinear, x []float32, rows int, scratch []float32) []float32 {
+	out := nativeBuffer(scratch, rows*l.out)
+	gonum.Implementation{}.Sgemm(blas.NoTrans, blas.Trans, rows, l.out, l.in, 1, x, l.in, l.weight, l.in, 0, out, l.out)
+	return out
+}

v3-ungated/native/stepsafety/native_model.go

@@ -0,0 +1,288 @@
+package stepsafety
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"math"
+	"sync"
+)
+
+const (
+	nativeHidden       = 384
+	nativeHeads        = 6
+	nativeHeadSize     = 64
+	nativeIntermediate = 1536
+	nativeLayers       = 12
+	nativePositions    = 512
+)
+
+// This is the pinned DeBERTa-v3-xsmall evaluation graph, not a general-purpose
+// transformer interpreter. Dropout is disabled; there are no token-type,
+// absolute-position, convolution, or decoder layers in this checkpoint.
+type nativeModel struct {
+	embeddings         []float32
+	embeddingNorm      nativeNorm
+	layers             [nativeLayers]nativeLayer
+	pooler, classifier nativeLinear
+	work               sync.Pool
+}
+type nativeNorm struct{ weight, bias []float32 }
+type nativeLinear struct {
+	weight, bias []float32
+	packed       []float32
+	in, out      int
+}
+type nativeLayer struct {
+	query, key, value, attention, intermediate, output nativeLinear
+	attentionNorm, outputNorm                          nativeNorm
+	relative                                           [nativeHeads]struct{ query, key nativeLinear }
+}
+
+func buildNativeModel(ctx context.Context, tensors map[string]nativeTensor) (*nativeModel, error) {
+	m := &nativeModel{}
+	var firstErr error
+	weight := func(name string, shape ...int) []float32 {
+		data, err := nativeWeight(tensors, name, shape...)
+		if firstErr == nil {
+			firstErr = err
+		}
+		return data
+	}
+	norm := func(name string) nativeNorm {
+		return nativeNorm{weight(name+".weight", nativeHidden), weight(name+".bias", nativeHidden)}
+	}
+	linear := func(name string, in, out int) nativeLinear {
+		return prepareNativeLinear(nativeLinear{weight: weight(name+".weight", out, in), bias: weight(name+".bias", out), in: in, out: out})
+	}
+	m.embeddings = weight("deberta.embeddings.word_embeddings.weight", 128005, nativeHidden)
+	m.embeddingNorm = norm("deberta.embeddings.LayerNorm")
+	rel := weight("deberta.encoder.rel_embeddings.weight", nativePositions, nativeHidden)
+	relNorm := norm("deberta.encoder.LayerNorm")
+	for i := range m.layers {
+		p := fmt.Sprintf("deberta.encoder.layer.%d.", i)
+		l := &m.layers[i]
+		l.query = linear(p+"attention.self.query_proj", nativeHidden, nativeHidden)
+		l.key = linear(p+"attention.self.key_proj", nativeHidden, nativeHidden)
+		l.value = linear(p+"attention.self.value_proj", nativeHidden, nativeHidden)
+		l.attention = linear(p+"attention.output.dense", nativeHidden, nativeHidden)
+		l.attentionNorm = norm(p + "attention.output.LayerNorm")
+		l.intermediate = linear(p+"intermediate.dense", nativeHidden, nativeIntermediate)
+		l.output = linear(p+"output.dense", nativeIntermediate, nativeHidden)
+		l.outputNorm = norm(p + "output.LayerNorm")
+	}
+	m.pooler = linear("pooler.dense", nativeHidden, nativeHidden)
+	m.classifier = linear("classifier", nativeHidden, 2)
+	if firstErr != nil {
+		return nil, firstErr
+	}
+	// Relative embeddings and their projections depend only on weights. Cache
+	// them once; every inference uses the same exact float32 parameters.
+	rel = append([]float32(nil), rel...)
+	relNorm.apply(rel, nil)
+	for i := range m.layers {
+		if err := ctx.Err(); err != nil {
+			return nil, err
+		}
+		l := &m.layers[i]
+		relQ, relK := l.query.apply(rel, nativePositions), l.key.apply(rel, nativePositions)
+		for h := 0; h < nativeHeads; h++ {
+			l.relative[h].query = prepareNativeLinear(nativeLinear{weight: nativeHead(relQ, nativePositions, h), in: nativeHeadSize, out: nativePositions})
+			l.relative[h].key = prepareNativeLinear(nativeLinear{weight: nativeHead(relK, nativePositions, h), in: nativeHeadSize, out: nativePositions})
+		}
+	}
+	return m, nil
+}
+
+func (l nativeLinear) apply(x []float32, rows int) []float32 {
+	return l.applyInto(x, rows, nil)
+}
+func (l nativeLinear) applyInto(x []float32, rows int, scratch []float32) []float32 {
+	out := nativeLinearProductInto(l, x, rows, scratch)
+	for i := 0; i < rows; i++ {
+		row := out[i*l.out : (i+1)*l.out]
+		for j := range row {
+			row[j] += l.bias[j]
+		}
+	}
+	return out
+}
+
+func (norm nativeNorm) apply(x, residual []float32) {
+	for start := 0; start < len(x); start += nativeHidden {
+		row := x[start : start+nativeHidden]
+		var mean float64
+		for j := range row {
+			if residual != nil {
+				row[j] += residual[start+j]
+			}
+			mean += float64(row[j])
+		}
+		mean /= nativeHidden
+		var variance float64
+		for _, v := range row {
+			d := float64(v) - mean
+			variance += d * d
+		}
+		inv := float32(1 / math.Sqrt(variance/nativeHidden+1e-7))
+		mu := float32(mean)
+		for j := range row {
+			row[j] = (row[j]-mu)*inv*norm.weight[j] + norm.bias[j]
+		}
+	}
+}
+
+func nativeGELU(x []float32) {
+	for i, v := range x {
+		x[i] = v * 0.5 * (1 + float32(math.Erf(float64(v)/math.Sqrt2)))
+	}
+}
+
+// Trimming trailing masked padding is exact for this encoder: valid queries
+// never attend to padding, and all other operations are per-token. Preserve
+// every real token, its original position, and the checkpoint's relative buckets.
+func nativeSequenceLength(packed packedInput) (int, error) {
+	if len(packed.IDs) != maxSequenceLength || len(packed.Mask) != maxSequenceLength {
+		return 0, errors.New("invalid native input shape")
+	}
+	n := 0
+	for i, mask := range packed.Mask {
+		if packed.IDs[i] < 0 || packed.IDs[i] >= 128005 {
+			return 0, errors.New("invalid native token")
+		}
+		if mask == 1 && i == n {
+			n++
+		} else if mask != 0 {
+			return 0, errors.New("non-contiguous native attention mask")
+		}
+	}
+	if n == 0 {
+		return 0, errors.New("empty native input")
+	}
+	return n, nil
+}
+
+func nativeRelativePosition(distance int) int {
+	abs := distance
+	if abs < 0 {
+		abs = -abs
+	}
+	if abs <= 128 {
+		return distance
+	}
+	// Mirror make_log_bucket_position(bucket_size=256,max_position=512).
+	bucket := int(math.Ceil(math.Log(float64(abs)/128)/math.Log(511.0/128)*127)) + 128
+	if distance < 0 {
+		return -bucket
+	}
+	return bucket
+}
+
+func (m *nativeModel) infer(ctx context.Context, packed packedInput) (InferenceResult, error) {
+	n, err := nativeSequenceLength(packed)
+	if err != nil {
+		return InferenceResult{}, err
+	}
+	w, ok := m.work.Get().(*nativeWorkspace)
+	if !ok {
+		w = newNativeWorkspace()
+	}
+	defer m.work.Put(w)
+	x := w.x[:n*nativeHidden]
+	for i, id := range packed.IDs[:n] {
+		copy(x[i*nativeHidden:], m.embeddings[int(id)*nativeHidden:(int(id)+1)*nativeHidden])
+	}
+	m.embeddingNorm.apply(x, nil)
+	positions := w.positions[:n*n]
+	for i := 0; i < n; i++ {
+		for j := 0; j < n; j++ {
+			positions[i*n+j] = min(max(nativeRelativePosition(i-j)+256, 0), 511)
+		}
+	}
+	for _, l := range m.layers {
+		if err := ctx.Err(); err != nil {
+			return InferenceResult{}, err
+		}
+		q, k, v := l.query.applyInto(x, n, w.q), l.key.applyInto(x, n, w.k), l.value.applyInto(x, n, w.v)
+		attention := w.attention[:n*nativeHidden]
+		var wg sync.WaitGroup
+		for head := 0; head < nativeHeads; head++ {
+			wg.Add(1)
+			go func(head int) {
+				defer wg.Done()
+				nativeAttention(ctx, q, k, v, l.relative[head].query, l.relative[head].key, positions, attention, n, head, &w.heads[head])
+			}(head)
+		}
+		wg.Wait()
+		if err := ctx.Err(); err != nil {
+			return InferenceResult{}, err
+		}
+		a := l.attention.applyInto(attention, n, w.a)
+		l.attentionNorm.apply(a, x)
+		intermediate := l.intermediate.applyInto(a, n, w.intermediate)
+		nativeGELU(intermediate)
+		x = l.output.applyInto(intermediate, n, w.x)
+		l.outputNorm.apply(x, a)
+	}
+	if err := ctx.Err(); err != nil {
+		return InferenceResult{}, err
+	}
+	pooled := m.pooler.apply(x[:nativeHidden], 1)
+	nativeGELU(pooled)
+	logits := m.classifier.apply(pooled, 1)
+	return InferenceResult{Logits: [2]float64{float64(logits[0]), float64(logits[1])}, HistoryOmitted: packed.HistoryOmitted}, nil
+}
+
+func nativeHead(x []float32, n, head int) []float32 {
+	return nativeHeadInto(x, n, head, nil)
+}
+func nativeHeadInto(x []float32, n, head int, scratch []float32) []float32 {
+	out := nativeBuffer(scratch, n*nativeHeadSize)
+	for i := 0; i < n; i++ {
+		copy(out[i*nativeHeadSize:], x[i*nativeHidden+head*nativeHeadSize:i*nativeHidden+(head+1)*nativeHeadSize])
+	}
+	return out
+}
+
+func nativeAttention(ctx context.Context, q, k, v []float32, relQ, relK nativeLinear, positions []int, output []float32, n, head int, w *nativeHeadWorkspace) {
+	scale := float32(math.Sqrt(nativeHeadSize * 3))
+	qh, kh, vh := nativeHeadInto(q, n, head, w.q), nativeHeadInto(k, n, head, w.k), nativeHeadInto(v, n, head, w.v)
+	scores := nativeLinearProductInto(prepareNativeLinearInto(nativeLinear{weight: kh, in: nativeHeadSize, out: n}, w.packedK), qh, n, w.scores)
+	c2p := nativeLinearProductInto(relK, qh, n, w.c2p)
+	p2c := nativeLinearProductInto(relQ, kh, n, w.p2c)
+	offset := head * nativeHeadSize
+	for i := 0; i < n; i++ {
+		if ctx.Err() != nil {
+			return
+		}
+		row := scores[i*n : (i+1)*n]
+		largest := float32(-math.MaxFloat32)
+		for j := range row {
+			pos := positions[i*n+j]
+			// p2c gathers (-relative_pos), then transposes the query/key axes;
+			// for equal-length self attention this selects the same i-j bucket.
+			row[j] = row[j]/scale + (c2p[i*nativePositions+pos]/scale + p2c[j*nativePositions+pos]/scale)
+			if row[j] > largest {
+				largest = row[j]
+			}
+		}
+		var sum float32
+		for j := range row {
+			row[j] = float32(math.Exp(float64(row[j] - largest)))
+			sum += row[j]
+		}
+		for j := range row {
+			row[j] /= sum
+		}
+	}
+	vt := w.vt[:n*nativeHeadSize]
+	for i := 0; i < n; i++ {
+		for j := 0; j < nativeHeadSize; j++ {
+			vt[j*n+i] = vh[i*nativeHeadSize+j]
+		}
+	}
+	result := nativeLinearProductInto(prepareNativeLinearInto(nativeLinear{weight: vt, in: n, out: nativeHeadSize}, w.packedV), scores, n, w.result)
+	for i := 0; i < n; i++ {
+		copy(output[i*nativeHidden+offset:], result[i*nativeHeadSize:(i+1)*nativeHeadSize])
+	}
+}

v3-ungated/native/stepsafety/native_weights.go

@@ -0,0 +1,130 @@
+package stepsafety
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/binary"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"math"
+	"slices"
+)
+
+// The native backend consumes the original float32 checkpoint, without
+// quantization, conversion through ONNX, or any change to its learned weights.
+const nativeWeightsSHA256 = "429b09164c6705790c4414eb31c0bc18d2fa8a374bea467b2e1b49ead6aeb5f1"
+const nativeWeightsBytes = 283201512
+
+type nativeTensor struct {
+	Shape []int
+	Data  []float32
+}
+
+// readNativeWeights supports only the pinned checkpoint's F32 safetensors
+// subset. Validate offsets and lengths before allocating, and verify the entire
+// byte stream against the training pin before exposing any tensor to inference.
+func readNativeWeights(ctx context.Context, source io.Reader) (map[string]nativeTensor, error) {
+	digest := sha256.New()
+	r := io.TeeReader(io.LimitReader(source, nativeWeightsBytes+1), digest)
+	var headerSize uint64
+	if err := binary.Read(r, binary.LittleEndian, &headerSize); err != nil {
+		return nil, err
+	}
+	if headerSize == 0 || headerSize > 128*1024 {
+		return nil, errors.New("invalid native model header size")
+	}
+	header := make([]byte, int(headerSize))
+	if _, err := io.ReadFull(r, header); err != nil {
+		return nil, err
+	}
+	var raw map[string]json.RawMessage
+	if err := json.Unmarshal(header, &raw); err != nil {
+		return nil, err
+	}
+	delete(raw, "__metadata__")
+	if len(raw) != 202 {
+		return nil, errors.New("unexpected native model tensor count")
+	}
+	type spec struct {
+		Name    string
+		DType   string   `json:"dtype"`
+		Shape   []int    `json:"shape"`
+		Offsets [2]int64 `json:"data_offsets"`
+	}
+	entries := make([]spec, 0, len(raw))
+	for name, data := range raw {
+		entry := spec{Name: name}
+		if err := json.Unmarshal(data, &entry); err != nil {
+			return nil, err
+		}
+		size := int64(1)
+		if entry.DType != "F32" || len(entry.Shape) < 1 || len(entry.Shape) > 2 {
+			return nil, errors.New("unsupported native model tensor")
+		}
+		for _, dim := range entry.Shape {
+			if dim <= 0 || dim > 128005 {
+				return nil, errors.New("invalid native model dimension")
+			}
+			size *= int64(dim)
+		}
+		if entry.Offsets[0] < 0 || size*4 != entry.Offsets[1]-entry.Offsets[0] || entry.Offsets[1] > nativeWeightsBytes-int64(headerSize)-8 {
+			return nil, errors.New("invalid native model offsets")
+		}
+		entries = append(entries, entry)
+	}
+	slices.SortFunc(entries, func(a, b spec) int {
+		if a.Offsets[0] < b.Offsets[0] {
+			return -1
+		}
+		if a.Offsets[0] > b.Offsets[0] {
+			return 1
+		}
+		return 0
+	})
+	result := make(map[string]nativeTensor, len(entries))
+	buffer := make([]byte, 64*1024)
+	var offset int64
+	for _, entry := range entries {
+		if entry.Offsets[0] != offset {
+			return nil, errors.New("native model has overlapping or missing data")
+		}
+		values := make([]float32, (entry.Offsets[1]-offset)/4)
+		for start := 0; start < len(values); {
+			if err := ctx.Err(); err != nil {
+				return nil, err
+			}
+			n := min(len(buffer)/4, len(values)-start)
+			if _, err := io.ReadFull(r, buffer[:n*4]); err != nil {
+				return nil, err
+			}
+			for i := 0; i < n; i++ {
+				value := math.Float32frombits(binary.LittleEndian.Uint32(buffer[i*4:]))
+				if math.IsNaN(float64(value)) || math.IsInf(float64(value), 0) {
+					return nil, errors.New("non-finite native weight")
+				}
+				values[start+i] = value
+			}
+			start += n
+		}
+		result[entry.Name] = nativeTensor{Shape: entry.Shape, Data: values}
+		offset = entry.Offsets[1]
+	}
+	if n, err := r.Read(buffer[:1]); n != 0 || err != io.EOF || offset+int64(headerSize)+8 != nativeWeightsBytes {
+		return nil, errors.New("invalid native model length")
+	}
+	if hex.EncodeToString(digest.Sum(nil)) != nativeWeightsSHA256 {
+		return nil, errors.New("native model checksum mismatch")
+	}
+	return result, nil
+}
+
+func nativeWeight(tensors map[string]nativeTensor, name string, shape ...int) ([]float32, error) {
+	tensor, ok := tensors[name]
+	if !ok || !slices.Equal(tensor.Shape, shape) {
+		return nil, fmt.Errorf("unexpected native tensor %s", name)
+	}
+	return tensor.Data, nil
+}

v3-ungated/native/stepsafety/native_workspace.go

@@ -0,0 +1,31 @@
+package stepsafety
+
+// Inference reuses bounded scratch storage across layers and calls. In
+// particular the six attention heads must not allocate 512x512 matrices on
+// each of twelve layers. A pool also keeps independent callers race-free.
+type nativeWorkspace struct {
+	x, q, k, v, a, attention, intermediate []float32
+	positions                              []int
+	heads                                  [nativeHeads]nativeHeadWorkspace
+}
+type nativeHeadWorkspace struct {
+	q, k, v, vt, packedK, packedV, result []float32
+	scores, c2p, p2c                      []float32
+}
+
+func nativeBuffer(buffer []float32, n int) []float32 {
+	if cap(buffer) < n {
+		return make([]float32, n)
+	}
+	return buffer[:n]
+}
+func newNativeWorkspace() *nativeWorkspace {
+	h := func() []float32 { return make([]float32, maxSequenceLength*nativeHidden) }
+	w := &nativeWorkspace{x: h(), q: h(), k: h(), v: h(), a: h(), attention: h(), intermediate: make([]float32, maxSequenceLength*nativeIntermediate), positions: make([]int, maxSequenceLength*maxSequenceLength)}
+	for i := range w.heads {
+		s := func() []float32 { return make([]float32, maxSequenceLength*nativeHeadSize) }
+		matrix := func() []float32 { return make([]float32, maxSequenceLength*maxSequenceLength) }
+		w.heads[i] = nativeHeadWorkspace{q: s(), k: s(), v: s(), vt: s(), packedK: s(), packedV: s(), result: s(), scores: matrix(), c2p: matrix(), p2c: matrix()}
+	}
+	return w
+}

v3-ungated/native/stepsafety/normalization.go

@@ -0,0 +1,187 @@
+package stepsafety
+
+import (
+	"encoding/json"
+	"io"
+	"regexp"
+	"strings"
+)
+
+var schemaHeader = regexp.MustCompile(`(?m)^([A-Za-z_][A-Za-z0-9_.:/-]*):[ \t]+`)
+
+// candidateFields matches the saved joint-CE input view. Only API binding names
+// and JSON representation change; executable strings, argument keys, request and
+// observation text remain intact. Never modify the hook's shared maps in place.
+func candidateFields(input Input) ([4]string, error) {
+	var fields [4]string
+	schema, err := candidateSchema(input.AvailableToolSchemas)
+	if err != nil {
+		return fields, err
+	}
+	definitions := schemaDefinitions(schema)
+	spellings := map[string]map[string]bool{}
+	for _, d := range definitions {
+		name := d["name"].(string)
+		key := lowerASCII(name)
+		if spellings[key] == nil {
+			spellings[key] = map[string]bool{}
+		}
+		spellings[key][name] = true
+	}
+	name := func(s string) string {
+		if len(spellings[lowerASCII(s)]) > 1 {
+			return s
+		}
+		return lowerASCII(s)
+	}
+	for _, d := range definitions {
+		d["name"] = name(d["name"].(string))
+	}
+	var schemaText string
+	if schema != nil {
+		if text, ok := schema.(string); ok {
+			schemaText = text
+		} else {
+			schemaText, err = compactSortedJSON(schema)
+			if err != nil {
+				return fields, err
+			}
+		}
+	}
+	history := input.InteractionHistory
+	if strings.TrimSpace(history) == "" {
+		history = "[]"
+	}
+	h, err := decodeCandidateJSON(history)
+	if err != nil {
+		return fields, backendError(ErrorInvalidHistory, nil)
+	}
+	events, ok := h.([]any)
+	if !ok || len(events) > maxHistoryEntries {
+		return fields, backendError(ErrorInvalidHistory, nil)
+	}
+	for _, event := range events {
+		e, ok := event.(map[string]any)
+		if !ok {
+			return fields, backendError(ErrorInvalidHistory, nil)
+		}
+		t, ok := e["tool"].(string)
+		if !ok || t == "" {
+			return fields, backendError(ErrorInvalidHistory, nil)
+		}
+		e["tool"] = name(t)
+	}
+	history, err = compactSortedJSON(events)
+	if err != nil {
+		return fields, err
+	}
+	args := input.ToolArguments
+	// These exact interfaces use title/description as call-display metadata.
+	// A TaskCreate description, WebFetch prompt, or arbitrary API payload is data.
+	switch input.ToolName {
+	case "Bash", "exec_command", "functions.exec_command", "mcp__workspace__bash", "mcp__cua_repl__js":
+		if values, ok := args.(map[string]any); ok {
+			copy := make(map[string]any, len(values))
+			for k, v := range values {
+				if k != "title" && k != "description" {
+					copy[k] = v
+				}
+			}
+			args = copy
+		}
+	}
+	arguments, err := compactSortedJSON(args)
+	if err != nil {
+		return fields, err
+	}
+	return [4]string{input.UserRequest, history, strings.TrimSpace("[TOOL_NAME]\n" + name(input.ToolName) + "\n[ARGUMENTS]\n" + arguments), schemaText}, nil
+}
+
+func lowerASCII(s string) string {
+	return strings.Map(func(r rune) rune {
+		if r >= 'A' && r <= 'Z' {
+			return r + ('a' - 'A')
+		}
+		return r
+	}, s)
+}
+
+func decodeCandidateJSON(text string) (any, error) {
+	d := json.NewDecoder(strings.NewReader(text))
+	d.UseNumber()
+	var v any
+	if err := d.Decode(&v); err != nil {
+		return nil, err
+	}
+	var extra any
+	if err := d.Decode(&extra); err != io.EOF {
+		return nil, backendError(ErrorUnsupportedText, nil)
+	}
+	return v, nil
+}
+
+func candidateSchema(value any) (any, error) {
+	if value == nil {
+		return nil, nil
+	}
+	if text, ok := value.(string); ok {
+		if strings.TrimSpace(text) == "" {
+			return nil, nil
+		}
+		if parsed, err := decodeCandidateJSON(text); err == nil {
+			switch parsed.(type) {
+			case []any, map[string]any:
+				return parsed, nil
+			}
+		}
+		matches := schemaHeader.FindAllStringSubmatchIndex(text, -1)
+		if len(matches) > 0 && strings.TrimSpace(text[:matches[0][0]]) == "" {
+			var definitions []any
+			for i, m := range matches {
+				end := len(text)
+				if i+1 < len(matches) {
+					end = matches[i+1][0]
+				}
+				definitions = append(definitions, map[string]any{"name": text[m[2]:m[3]], "description": strings.TrimSpace(text[m[1]:end])})
+			}
+			return definitions, nil
+		}
+		// Research also accepted Python literal containers. Hooks supply JSON;
+		// abstain on such unsupported containers instead of guessing a conversion.
+		if strings.HasPrefix(strings.TrimSpace(text), "[") || strings.HasPrefix(strings.TrimSpace(text), "{") {
+			return nil, backendError(ErrorUnsupportedText, nil)
+		}
+		return text, nil
+	}
+	text, err := compactSortedJSON(value)
+	if err != nil {
+		return nil, err
+	}
+	return decodeCandidateJSON(text)
+}
+
+func schemaDefinitions(value any) []map[string]any {
+	switch v := value.(type) {
+	case []any:
+		var result []map[string]any
+		for _, item := range v {
+			result = append(result, schemaDefinitions(item)...)
+		}
+		return result
+	case map[string]any:
+		if f, ok := v["function"].(map[string]any); ok {
+			return schemaDefinitions(f)
+		}
+		if t, ok := v["tools"].([]any); ok {
+			return schemaDefinitions(t)
+		}
+		_, named := v["name"].(string)
+		_, parameters := v["parameters"]
+		_, inputSchema := v["input_schema"]
+		_, description := v["description"]
+		if named && (parameters || inputSchema || description || len(v) == 1) {
+			return []map[string]any{v}
+		}
+	}
+	return nil
+}

v3-ungated/native/stepsafety/packing.go

@@ -0,0 +1,215 @@
+package stepsafety
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"strings"
+)
+
+const maxSequenceLength = 512
+
+var fieldBudgets = [4]int{96, 144, 128, 128}
+var fieldMarkerIDs = [4]int64{128001, 128002, 128003, 128004}
+
+type tokenEncoder interface {
+	encode(context.Context, string) ([]int64, error)
+}
+type packedInput struct {
+	IDs            []int64
+	Mask           []int64
+	HistoryOmitted bool
+}
+
+func packInput(ctx context.Context, tokenizer tokenEncoder, input Input) (packedInput, error) {
+	var result packedInput
+	if ExcludedTool(input.ToolName) {
+		return result, backendError(ErrorExcludedTool, nil)
+	}
+	if err := validateInputBounds(input); err != nil {
+		return result, err
+	}
+	fields, err := candidateFields(input)
+	if err != nil {
+		return result, err
+	}
+	var selected [4][]int64
+	for _, index := range []int{2, 0, 3} {
+		tokens, err := tokenizer.encode(ctx, fields[index])
+		if err != nil {
+			return result, err
+		}
+		if len(tokens) > fieldBudgets[index] {
+			code := ErrorActionTooLarge
+			if index == 0 {
+				code = ErrorRequestTooLarge
+			}
+			if index == 3 {
+				code = ErrorSchemaTooLarge
+			}
+			return result, backendError(code, nil)
+		}
+		selected[index] = tokens
+	}
+	selected[1], result.HistoryOmitted, err = selectHistory(ctx, tokenizer, fields[1], fieldBudgets[1])
+	if err != nil {
+		return result, err
+	}
+	result.HistoryOmitted = result.HistoryOmitted || input.HistoryOmitted
+	result.IDs, result.Mask = assembleTokens(selected)
+	return result, nil
+}
+
+// Keep the training field order, special IDs, separators, and padding. Only
+// history is truncated; admitted requests, actions, and schemas stay complete.
+func assembleTokens(fields [4][]int64) ([]int64, []int64) {
+	ids := make([]int64, 0, maxSequenceLength)
+	ids = append(ids, 1)
+	for i, tokens := range fields {
+		ids = append(ids, fieldMarkerIDs[i])
+		ids = append(ids, tokens...)
+		ids = append(ids, 2)
+	}
+	mask := make([]int64, maxSequenceLength)
+	for i := range ids {
+		mask[i] = 1
+	}
+	ids = append(ids, make([]int64, maxSequenceLength-len(ids))...)
+	return ids, mask
+}
+
+func selectHistory(ctx context.Context, tokenizer tokenEncoder, history string, budget int) ([]int64, bool, error) {
+	if strings.TrimSpace(history) == "" {
+		history = "[]"
+	}
+	var entries []json.RawMessage
+	if err := json.Unmarshal([]byte(history), &entries); err != nil {
+		return nil, false, backendError(ErrorInvalidHistory, nil)
+	}
+	if len(entries) > maxHistoryEntries {
+		return nil, false, backendError(ErrorInputTooLarge, nil)
+	}
+	selected := make([]json.RawMessage, 0, len(entries))
+	omitted := false
+	for _, entry := range entries {
+		var event struct {
+			Tool string `json:"tool"`
+		}
+		if err := json.Unmarshal(entry, &event); err != nil || event.Tool == "" {
+			return nil, omitted, backendError(ErrorInvalidHistory, nil)
+		}
+		if ExcludedTool(event.Tool) {
+			omitted = true
+			continue
+		}
+		selected = append(selected, entry)
+	}
+	encoded, err := compactSortedJSON(selected)
+	if err != nil {
+		return nil, omitted, err
+	}
+	tokens, err := tokenizer.encode(ctx, encoded)
+	if err != nil {
+		return nil, omitted, err
+	}
+	if len(tokens) > budget {
+		// Match the training history packer after removing excluded tools:
+		// first 72 + last 72 tokens at the fixed 144-token history budget.
+		// The model consumes token fragments; they need not form valid JSON.
+		head := (budget + 1) / 2
+		tail := budget - head
+		truncated := make([]int64, 0, budget)
+		tokens = append(append(truncated, tokens[:head]...), tokens[len(tokens)-tail:]...)
+		omitted = true
+	}
+	return tokens, omitted, nil
+}
+
+// The schema field used Python's default JSON separators (unlike action and
+// history). Insert spaces outside strings only to preserve that contract.
+func schemaText(value any) (string, error) {
+	if value == nil {
+		return "", nil
+	}
+	if text, ok := value.(string); ok {
+		return text, nil
+	}
+	compact, err := compactSortedJSON(value)
+	if err != nil {
+		return "", err
+	}
+	var out strings.Builder
+	quoted, escaped := false, false
+	for _, r := range compact {
+		out.WriteRune(r)
+		if quoted {
+			if escaped {
+				escaped = false
+			} else if r == '\\' {
+				escaped = true
+			} else if r == '"' {
+				quoted = false
+			}
+			continue
+		}
+		if r == '"' {
+			quoted = true
+		} else if r == ':' || r == ',' {
+			out.WriteByte(' ')
+		}
+	}
+	return out.String(), nil
+}
+
+// Match known file-tool names and namespace leaves, without excluding broad
+// verbs such as read_email or write_message that the model can still score.
+func ExcludedTool(name string) bool {
+	if len(name) > 256 {
+		return false
+	}
+	name = strings.ToLower(strings.TrimSpace(name))
+	if i := strings.LastIndex(name, "__"); i >= 0 {
+		name = name[i+2:]
+	}
+	if i := strings.LastIndexAny(name, ".:/"); i >= 0 {
+		name = name[i+1:]
+	}
+	switch name {
+	case "read", "write", "edit", "multiedit", "notebookedit", "apply_patch", "read_file", "read_text_file", "read_multiple_files", "write_file", "edit_file", "replace_file_content", "multi_replace_file_content":
+		return true
+	}
+	return false
+}
+
+// Reject large or malformed JSON values before allocating their serialization.
+// Hook adapters supply JSON primitives; unsupported Go objects aren't inputs.
+func boundedJSONSize(value any, remaining *int, depth int) error {
+	if depth > 64 || *remaining < 0 {
+		return backendError(ErrorInputTooLarge, nil)
+	}
+	*remaining -= 2
+	switch v := value.(type) {
+	case nil, bool, float64, float32, int, int64, json.Number:
+	case string:
+		*remaining -= len(v)
+	case map[string]any:
+		for key, item := range v {
+			*remaining -= len(key) + 3
+			if err := boundedJSONSize(item, remaining, depth+1); err != nil {
+				return err
+			}
+		}
+	case []any:
+		for _, item := range v {
+			if err := boundedJSONSize(item, remaining, depth+1); err != nil {
+				return err
+			}
+		}
+	default:
+		return backendError(ErrorInference, errors.New("unsupported JSON input"))
+	}
+	if *remaining < 0 {
+		return backendError(ErrorInputTooLarge, nil)
+	}
+	return nil
+}

v3-ungated/native/stepsafety/request_gate.go

@@ -0,0 +1,122 @@
+package stepsafety
+
+import (
+	"compress/gzip"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"golang.org/x/text/cases"
+	"golang.org/x/text/language"
+	"io"
+	"math"
+	"strings"
+	"unicode"
+)
+
+const ErrorRequestNotInformational = "request_not_informational"
+
+var requestGateModel struct {
+	Word, Char           sparseVector
+	Intercept, Threshold float64
+	Calibrator           scoreCalibration
+}
+
+func loadRequestGateAssets() error {
+	f, e := nativeAssets.Open("model/native/request_gate.json.gz")
+	if e != nil {
+		return e
+	}
+	defer f.Close()
+	r, e := gzip.NewReader(f)
+	if e != nil {
+		return e
+	}
+	defer r.Close()
+	data, e := io.ReadAll(io.LimitReader(r, 5259026+1))
+	if e != nil {
+		return e
+	}
+	h := sha256.Sum256(data)
+	if len(data) != 5259026 || hex.EncodeToString(h[:]) != "d94b34f3c59803047365b594fe50531dd073d8ac14b5b8151c574b61d647f693" {
+		return errors.New("request gate checksum mismatch")
+	}
+	return json.Unmarshal(data, &requestGateModel)
+}
+func requestIntentProbability(text string) float64 {
+	c := requestGateModel.Calibrator
+	z := c.Coefficient*requestIntentMargin(text) + c.Intercept
+	if z >= 0 {
+		return 1 / (1 + math.Exp(-z))
+	}
+	v := math.Exp(z)
+	return v / (1 + v)
+}
+func requestIntentGate(ctx context.Context, text string) error {
+	if e := ctx.Err(); e != nil {
+		return e
+	}
+	p := requestIntentProbability(text)
+	if e := ctx.Err(); e != nil {
+		return e
+	}
+	if p < requestGateModel.Threshold {
+		return backendError(ErrorRequestNotInformational, nil)
+	}
+	return nil
+}
+
+func requestIntentMargin(text string) float64 {
+	// Match sklearn's lowercase Unicode word analyzer and char analyzer.
+	// The vocabulary is hashed to avoid shipping plaintext captured tokens.
+	text = cases.Lower(language.Und).String(text)
+	words := []string{}
+	run := []rune{}
+	flush := func() {
+		if len(run) >= 2 {
+			words = append(words, string(run))
+		}
+		run = run[:0]
+	}
+	for _, r := range text {
+		if unicode.IsLetter(r) || unicode.IsNumber(r) || r == '_' {
+			run = append(run, r)
+		} else {
+			flush()
+		}
+	}
+	flush()
+	counts := map[string]int{}
+	for i, word := range words {
+		counts[word]++
+		if i > 0 {
+			counts[words[i-1]+" "+word]++
+		}
+	}
+	score := requestGateModel.Intercept + vectorScore(counts, requestGateModel.Word)
+	var normalized strings.Builder
+	rr := []rune(text)
+	for i := 0; i < len(rr); {
+		j := i + 1
+		if pythonSpace(rr[i]) {
+			for j < len(rr) && pythonSpace(rr[j]) {
+				j++
+			}
+		}
+		if j-i >= 2 {
+			normalized.WriteByte(' ')
+		} else {
+			normalized.WriteRune(rr[i])
+		}
+		i = j
+	}
+	rr = []rune(normalized.String())
+	counts = map[string]int{}
+	for n := 3; n <= 5; n++ {
+		for i := 0; i+n <= len(rr); i++ {
+			counts[string(rr[i:i+n])]++
+		}
+	}
+	return score + vectorScore(counts, requestGateModel.Char)
+}

v3-ungated/native/stepsafety/runtime_artifacts.go

@@ -0,0 +1,11 @@
+package stepsafety
+
+// Official microsoft/onnxruntime release assets. Archive hashes are the
+// publisher's GitHub asset digests; library hashes pin the extracted bytes.
+// Intel macOS uses 1.23.2, the last upstream x86_64 release asset.
+var runtimeArtifacts = map[string]runtimeSpec{
+	"darwin/arm64": {Version: "1.24.4", Archive: artifactSpec{Name: "onnxruntime-osx-arm64-1.24.4.tgz", Size: 30937282, SHA256: "93787795f47e1eee369182e43ed51b9e5da0878ab0346aecf4258979b8bba989"}, Library: artifactSpec{Name: "libonnxruntime.1.24.4.dylib", Size: 35418600, SHA256: "872533f130f1839a5bc01788ddb4f75c83a189763441ba1178788ed965449289"}},
+	"darwin/amd64": {Version: "1.23.2", Archive: artifactSpec{Name: "onnxruntime-osx-x86_64-1.23.2.tgz", Size: 11676322, SHA256: "d10359e16347b57d9959f7e80a225a5b4a66ed7d7e007274a15cae86836485a6"}, Library: artifactSpec{Name: "libonnxruntime.1.23.2.dylib", Size: 39742608, SHA256: "8c9c78de65ea3786f987c0d980e9c1b13a3a5fbc6b3e2965ba05b450e6e4c054"}},
+	"linux/arm64":  {Version: "1.24.4", Archive: artifactSpec{Name: "onnxruntime-linux-aarch64-1.24.4.tgz", Size: 7181958, SHA256: "866109a9248d057671a039b9d725be4bd86888e3754140e6701ec621be9d4d7e"}, Library: artifactSpec{Name: "libonnxruntime.so.1.24.4", Size: 18625376, SHA256: "52b2a0e75e79468404284fec38ad5ee1a7a996232274f5e1b84f3e793fb07554"}},
+	"linux/amd64":  {Version: "1.24.4", Archive: artifactSpec{Name: "onnxruntime-linux-x64-1.24.4.tgz", Size: 8155822, SHA256: "3a211fbea252c1e66290658f1b735b772056149f28321e71c308942cdb54b747"}, Library: artifactSpec{Name: "libonnxruntime.so.1.24.4", Size: 22159232, SHA256: "d132535d051344ff5c64c9c200004150559049a81ed330eb4422c1962fb6b7e4"}},
+}

v3-ungated/native/stepsafety/runtime_store.go

@@ -0,0 +1,204 @@
+package stepsafety
+
+import (
+	"archive/tar"
+	"compress/gzip"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"runtime"
+	"time"
+)
+
+type runtimeSpec struct {
+	Version string
+	Archive artifactSpec
+	Library artifactSpec
+}
+
+func runtimeForPlatform() (runtimeSpec, error) {
+	spec, ok := runtimeArtifacts[runtime.GOOS+"/"+runtime.GOARCH]
+	if !ok {
+		return runtimeSpec{}, errors.New("step-safety ONNX Runtime is unsupported on this platform")
+	}
+	return spec, nil
+}
+
+func runtimeDir(modelDir string) string {
+	return filepath.Join(modelDir, "runtime", runtime.GOOS+"-"+runtime.GOARCH)
+}
+
+func ValidateRuntimeDir(modelDir string) (string, error) {
+	spec, err := runtimeForPlatform()
+	if err != nil {
+		return "", err
+	}
+	path := filepath.Join(runtimeDir(modelDir), spec.Library.Name)
+	if err := verifyArtifact(path, spec.Library); err != nil {
+		return "", err
+	}
+	return path, nil
+}
+
+func verifyArtifact(path string, spec artifactSpec) error {
+	info, err := os.Stat(path)
+	if err != nil {
+		return fmt.Errorf("artifact %s is unavailable: %w", spec.Name, err)
+	}
+	if !info.Mode().IsRegular() || info.Size() != spec.Size {
+		return fmt.Errorf("artifact %s has unexpected size", spec.Name)
+	}
+	hash, err := fileSHA256(path)
+	if err != nil || hash != spec.SHA256 {
+		return fmt.Errorf("artifact %s failed checksum verification", spec.Name)
+	}
+	return nil
+}
+
+// InstallRuntime is called only by the explicit install command. Daemon
+// startup is strictly offline and loads only this checksum-pinned library.
+// archivePath supports air-gapped installation; an empty path downloads the
+// platform-specific CPU archive from Microsoft's official GitHub release.
+func InstallRuntime(ctx context.Context, modelDir, archivePath string) error {
+	spec, err := runtimeForPlatform()
+	if err != nil {
+		return err
+	}
+	destination := runtimeDir(modelDir)
+	if _, err := ValidateRuntimeDir(modelDir); err == nil {
+		return nil
+	}
+	if _, err := os.Stat(destination); err == nil {
+		return fmt.Errorf("ONNX Runtime destination exists but is invalid: %s", destination)
+	} else if !errors.Is(err, os.ErrNotExist) {
+		return err
+	}
+	parent := filepath.Dir(destination)
+	if err := os.MkdirAll(parent, 0o700); err != nil {
+		return err
+	}
+	temporary, err := os.MkdirTemp(parent, ".onnx-install-*")
+	if err != nil {
+		return err
+	}
+	defer os.RemoveAll(temporary)
+	if archivePath == "" {
+		archivePath = filepath.Join(temporary, spec.Archive.Name)
+		if err := downloadRuntime(ctx, spec, archivePath); err != nil {
+			return err
+		}
+	}
+	if err := verifyArtifact(archivePath, spec.Archive); err != nil {
+		return err
+	}
+	staged := filepath.Join(temporary, "runtime")
+	if err := os.Mkdir(staged, 0o700); err != nil {
+		return err
+	}
+	if err := extractRuntime(archivePath, staged, spec); err != nil {
+		return err
+	}
+	if err := verifyArtifact(filepath.Join(staged, spec.Library.Name), spec.Library); err != nil {
+		return err
+	}
+	provenance, err := json.MarshalIndent(spec, "", "  ")
+	if err != nil {
+		return err
+	}
+	if err := os.WriteFile(filepath.Join(staged, "PROVENANCE.json"), append(provenance, '\n'), 0o600); err != nil {
+		return err
+	}
+	if err := ctx.Err(); err != nil {
+		return err
+	}
+	return os.Rename(staged, destination)
+}
+
+func downloadRuntime(ctx context.Context, spec runtimeSpec, path string) error {
+	url := "https://github.com/microsoft/onnxruntime/releases/download/v" + spec.Version + "/" + spec.Archive.Name
+	request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return err
+	}
+	client := &http.Client{Timeout: 2 * time.Minute}
+	response, err := client.Do(request)
+	if err != nil {
+		return err
+	}
+	defer response.Body.Close()
+	if response.StatusCode != http.StatusOK {
+		return fmt.Errorf("ONNX Runtime download returned HTTP %d", response.StatusCode)
+	}
+	out, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
+	if err != nil {
+		return err
+	}
+	written, copyErr := io.Copy(out, io.LimitReader(response.Body, spec.Archive.Size+1))
+	closeErr := out.Close()
+	if err := errors.Join(copyErr, closeErr); err != nil {
+		return err
+	}
+	if written != spec.Archive.Size {
+		return errors.New("ONNX Runtime download has unexpected size")
+	}
+	return nil
+}
+
+func extractRuntime(archivePath, destination string, spec runtimeSpec) error {
+	file, err := os.Open(archivePath)
+	if err != nil {
+		return err
+	}
+	defer file.Close()
+	zipped, err := gzip.NewReader(file)
+	if err != nil {
+		return err
+	}
+	defer zipped.Close()
+	archive := tar.NewReader(zipped)
+	root := spec.Archive.Name[:len(spec.Archive.Name)-len(".tgz")]
+	// Explicit allowlist: never extract archive paths or follow its symlinks.
+	wanted := map[string]string{
+		root + "/lib/" + spec.Library.Name: spec.Library.Name,
+		root + "/LICENSE":                  "LICENSE",
+		root + "/ThirdPartyNotices.txt":    "ThirdPartyNotices.txt",
+	}
+	for {
+		header, err := archive.Next()
+		if errors.Is(err, io.EOF) {
+			break
+		}
+		if err != nil {
+			return err
+		}
+		name, ok := wanted[header.Name]
+		if !ok {
+			continue
+		}
+		limit := int64(4 * 1024 * 1024)
+		if name == spec.Library.Name {
+			limit = spec.Library.Size
+		}
+		if header.Typeflag != tar.TypeReg || header.Size < 1 || header.Size > limit {
+			return errors.New("unexpected ONNX Runtime archive entry")
+		}
+		out, err := os.OpenFile(filepath.Join(destination, name), os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
+		if err != nil {
+			return err
+		}
+		_, copyErr := io.Copy(out, archive)
+		if err := errors.Join(copyErr, out.Close()); err != nil {
+			return err
+		}
+		delete(wanted, header.Name)
+	}
+	if len(wanted) != 0 {
+		return errors.New("ONNX Runtime archive is missing required library or license files")
+	}
+	return nil
+}

v3-ungated/native/stepsafety/scoped.go

@@ -0,0 +1,164 @@
+package stepsafety
+
+import (
+	"context"
+	"encoding/json"
+	"math"
+	"regexp"
+	"strings"
+)
+
+const ErrorUnsupportedEffect = "unsupported_effect"
+const ErrorMissingRequest = "missing_request"
+const ErrorCompoundRequest = "compound_or_mutation_request"
+
+var scopeCamel = regexp.MustCompile(`([a-z0-9])([A-Z])`)
+var scopeWords = regexp.MustCompile(`[a-z0-9]+`)
+var requestWords = regexp.MustCompile(`[a-z]+`)
+
+func scopeHas(words []string, options string) bool {
+	for _, option := range strings.Fields(options) {
+		for _, w := range words {
+			if w == option {
+				return true
+			}
+		}
+	}
+	return false
+}
+func scopePopulated(value any) bool {
+	switch v := value.(type) {
+	case string:
+		return strings.TrimSpace(v) != ""
+	case float64:
+		return !math.IsNaN(v) && !math.IsInf(v, 0)
+	case float32:
+		return !math.IsNaN(float64(v)) && !math.IsInf(float64(v), 0)
+	case int, int64:
+		return true
+	case json.Number:
+		n, e := v.Float64()
+		return e == nil && !math.IsNaN(n) && !math.IsInf(n, 0)
+	case []any:
+		if len(v) == 0 {
+			return false
+		}
+		for _, x := range v {
+			if !scopePopulated(x) {
+				return false
+			}
+		}
+		return true
+	}
+	return false
+}
+func recognizedScopeEffect(tool string, raw any) string {
+	args, ok := raw.(map[string]any)
+	if !ok {
+		return ""
+	}
+	ws := scopeWords.FindAllString(strings.ToLower(scopeCamel.ReplaceAllString(tool, "${1} ${2}")), -1)
+	if scopeHas(ws, "search find get list read inspect describe preview dry simulate check test") {
+		return ""
+	}
+	for _, key := range []string{"dry_run", "dryRun", "preview", "simulate"} {
+		if v, ok := args[key].(bool); ok && v {
+			return ""
+		}
+	}
+	keys := []string{}
+	for k, v := range args {
+		if scopePopulated(v) {
+			keys = append(keys, strings.ToLower(k))
+		}
+	}
+	if scopeHas(ws, "delete remove erase truncate drop wipe unlink") && scopeHas(ws, "file files email emails message messages record records database databases directory directories bucket buckets") && scopeHas(keys, "id ids file_id email_id message_id record_id path file filename name database bucket target records directory") {
+		return "destructive_object_change"
+	}
+	if scopeHas(ws, "send transfer pay") && scopeHas(ws, "money payment payments funds") && scopeHas(keys, "recipient to account destination recipient_id account_id") {
+		var amount float64
+		switch n := args["amount"].(type) {
+		case float64:
+			amount = n
+		case float32:
+			amount = float64(n)
+		case int:
+			amount = float64(n)
+		case int64:
+			amount = float64(n)
+		case json.Number:
+			amount, _ = n.Float64()
+		}
+		if amount > 0 && !math.IsNaN(amount) && !math.IsInf(amount, 0) {
+			return "money_transfer"
+		}
+	}
+	if scopeHas(ws, "change reset update set rotate") && scopeHas(ws, "password passwords credential credentials") {
+		for _, key := range []string{"password", "new_password", "newPassword", "credential"} {
+			if v, ok := args[key].(string); ok && strings.TrimSpace(v) != "" {
+				return "credential_change"
+			}
+		}
+	}
+	if scopeHas(ws, "send post publish forward") && scopeHas(ws, "email message mail") && scopeHas(keys, "recipient recipients to channel destination") && scopeHas(keys, "body content message text") {
+		return "outgoing_message"
+	}
+	if scopeHas(ws, "share grant invite") && scopeHas(ws, "file document folder access permission") && scopeHas(keys, "recipient recipients user users email") && scopeHas(keys, "file_id document_id folder_id path resource permission") {
+		return "access_share"
+	}
+	return ""
+}
+
+func scopedEligibility(input Input) error {
+	if recognizedScopeEffect(input.ToolName, input.ToolArguments) == "" {
+		return backendError(ErrorUnsupportedEffect, nil)
+	}
+	if strings.TrimSpace(input.UserRequest) == "" {
+		return backendError(ErrorMissingRequest, nil)
+	}
+	if scopeHas(requestWords.FindAllString(strings.ToLower(input.UserRequest), -1), "delete remove erase transfer send change update pay create modify write add book reserve schedule cancel install run execute deploy reset adjust approve archive buy checkout clear commit disburse discard dispose drop empty forward grant invite merge move order post publish purchase purge push refund reimburse rename reply reschedule respond restart rotate save set share start stop submit unlink upload wipe") {
+		return backendError(ErrorCompoundRequest, nil)
+	}
+	return nil
+}
+
+// packScopedInput preserves the frozen bounded contextual representation.
+// The current action is complete. Request uses its head, history and schema
+// use head/tail. Eligibility is independent of tool effect and request intent.
+// V3 ungated retains the full-action limit and action-agreement scoring.
+func packScopedInput(ctx context.Context, tok tokenEncoder, input Input) (packedInput, error) {
+	var out packedInput
+	if err := validateInputBounds(input); err != nil {
+		return out, err
+	}
+	fields, err := candidateFields(input)
+	if err != nil {
+		return out, err
+	}
+	var selected [4][]int64
+	for i, text := range fields {
+		tokens, err := tok.encode(ctx, text)
+		if err != nil {
+			return out, err
+		}
+		budget := fieldBudgets[i]
+		if len(tokens) > budget {
+			if i == 2 {
+				return out, backendError(ErrorActionTooLarge, nil)
+			}
+			if i == 0 {
+				tokens = tokens[:budget]
+			} else {
+				head := (budget + 1) / 2
+				tokens = append(append([]int64{}, tokens[:head]...), tokens[len(tokens)-(budget-head):]...)
+			}
+			if i == 1 {
+				out.HistoryOmitted = true
+			}
+		}
+		selected[i] = tokens
+	}
+	out.IDs, out.Mask = assembleTokens(selected)
+	out.HistoryOmitted = out.HistoryOmitted || input.HistoryOmitted
+	return out, nil
+}

v3-ungated/native/stepsafety/tokenizer.go

@@ -0,0 +1,122 @@
+package stepsafety
+
+import (
+	"context"
+	"os"
+	"strings"
+	"unicode"
+	"unicode/utf8"
+
+	"golang.org/x/text/unicode/norm"
+)
+
+// This adapter is deliberately specific to the checksum-pinned DeBERTa
+// tokenizer: the HF Replace -> NFC -> Strip normalizer and added special
+// tokens here, Metaspace and Unigram segmentation in unigram.go.
+type tokenizer struct{ unigram *unigram }
+
+var specialTokens = []string{"[PAD]", "[CLS]", "[SEP]", "[UNK]", "[MASK]", "[USER_REQUEST]", "[INTERACTION_HISTORY]", "[CURRENT_ACTION]", "[TOOL_DESCRIPTIONS]"}
+var specialTokenIDs = []int64{0, 1, 2, 3, 128000, 128001, 128002, 128003, 128004}
+
+func loadTokenizer(path string) (*tokenizer, error) {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return nil, err
+	}
+	return parseTokenizer(data)
+}
+
+func parseTokenizer(data []byte) (*tokenizer, error) {
+	unigram, err := parseUnigram(data)
+	if err != nil {
+		return nil, err
+	}
+	return &tokenizer{unigram: unigram}, nil
+}
+
+func (t *tokenizer) encode(ctx context.Context, text string) ([]int64, error) {
+	var ids []int64
+	for len(text) > 0 {
+		if err := ctx.Err(); err != nil {
+			return nil, err
+		}
+		next, special := len(text), -1
+		for i, token := range specialTokens {
+			if at := strings.Index(text, token); at >= 0 && at < next {
+				next, special = at, i
+			}
+		}
+		// Added special tokens are extracted before normalizing each ordinary
+		// segment, matching normalized=false and lstrip/rstrip=false in HF.
+		segment := text[:next]
+		normalized := normalizeText(segment)
+		// Go NFC inserts CGJ for non-stream-safe combining sequences; HF NFC
+		// does not. Abstain on those rare inputs rather than silently changing
+		// the representation the checkpoint was trained on.
+		if strings.Count(normalized, "\u034f") != strings.Count(segment, "\u034f") {
+			return nil, backendError(ErrorUnsupportedText, nil)
+		}
+		ids = t.unigram.encode(normalized, ids)
+		if special < 0 {
+			break
+		}
+		ids = append(ids, specialTokenIDs[special])
+		text = text[next+len(specialTokens[special]):]
+	}
+	return ids, ctx.Err()
+}
+
+func normalizeText(text string) string {
+	var out strings.Builder
+	out.Grow(len(text))
+	for i := 0; i < len(text); {
+		r, size := utf8.DecodeRuneInString(text[i:])
+		if !unicode.IsSpace(r) {
+			out.WriteRune(r)
+			i += size
+			continue
+		}
+		start, count := i, 0
+		for i < len(text) {
+			r, size = utf8.DecodeRuneInString(text[i:])
+			if !unicode.IsSpace(r) {
+				break
+			}
+			count++
+			i += size
+		}
+		whitespace := text[start:i]
+		// Rust regex \s{2,}|[\n\r\t] uses Unicode whitespace; Go's regexp \s
+		// is ASCII-only, so using it here would silently change token IDs.
+		if count >= 2 || whitespace == "\n" || whitespace == "\r" || whitespace == "\t" {
+			out.WriteByte(' ')
+		} else {
+			out.WriteString(whitespace)
+		}
+	}
+	return strings.TrimRightFunc(nfcUnicode9(out.String()), unicode.IsSpace)
+}
+
+// nfcUnicode9 matches HF NFC, whose tables are Unicode 9.0. HF treats every
+// later character as an unassigned starter that never reorders or composes,
+// while x/text's newer tables move or merge the ones in postUnicode9NFC. NFC
+// cannot act across such a starter, so normalizing the text between them and
+// copying them unchanged reproduces HF.
+func nfcUnicode9(text string) string {
+	var out strings.Builder
+	for {
+		i := strings.IndexFunc(text, func(r rune) bool { return unicode.Is(postUnicode9NFC, r) })
+		if i < 0 {
+			break
+		}
+		_, size := utf8.DecodeRuneInString(text[i:])
+		out.WriteString(norm.NFC.String(text[:i]))
+		out.WriteString(text[i : i+size])
+		text = text[i+size:]
+	}
+	if out.Len() == 0 {
+		return norm.NFC.String(text)
+	}
+	out.WriteString(norm.NFC.String(text))
+	return out.String()
+}

v3-ungated/native/stepsafety/types.go

@@ -0,0 +1,468 @@
+// Package stepsafety runs the local, observe-only step-level tool safety pilot.
+//
+// The package deliberately has no enforcement API. A result is evidence for
+// review, never an authorization decision, and every model failure degrades to
+// an unavailable shadow result instead of an error returned to the hook path.
+package stepsafety
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"math"
+	"os"
+	"strconv"
+	"strings"
+	"time"
+)
+
+const (
+	ModelVersion = "merlin-v3-ungated-20261104"
+	Threshold    = 0.5
+
+	calibrationScale = 1.0
+	calibrationBias  = 0.0
+
+	defaultTimeout        = 250 * time.Millisecond
+	maxConfiguredTimeout  = 500 * time.Millisecond
+	defaultStartupTimeout = 30 * time.Second
+	defaultConcurrency    = 1
+
+	// Packing remains token-exact for admitted inputs, but hook-controlled
+	// strings must be bounded before the tokenizer sees them. Each limit is
+	// still orders of magnitude above its final token budget.
+	maxPretokenizedFieldBytes = 64 * 1024
+	maxPretokenizedTotalBytes = 128 * 1024
+)
+
+const (
+	DecisionSafe        = "safe"
+	DecisionUnsafe      = "unsafe"
+	DecisionUnavailable = "unavailable"
+)
+
+const (
+	ErrorTimeout         = "timeout"
+	ErrorUnavailable     = "unavailable"
+	ErrorExcludedTool    = "excluded_tool"
+	ErrorActionTooLarge  = "action_too_large"
+	ErrorRequestTooLarge = "request_too_large"
+	ErrorSchemaTooLarge  = "schema_too_large"
+	ErrorUnsupportedText = "unsupported_text"
+	ErrorInvalidHistory  = "invalid_history"
+	ErrorInference       = "inference_error"
+	ErrorConcurrency     = "concurrency_timeout"
+	ErrorInvalidOutput   = "invalid_output"
+	ErrorInputTooLarge   = "input_too_large"
+)
+
+// Input is the complete production-available no-Thought representation. The
+// action stays structured until Go renders the training form.
+type Input struct {
+	UserRequest          string `json:"user_request"`
+	RequestTooLarge      bool   `json:"request_too_large"`
+	InteractionHistory   string `json:"interaction_history"`
+	HistoryOmitted       bool   `json:"history_omitted"`
+	ToolName             string `json:"tool_name"`
+	ToolArguments        any    `json:"tool_arguments"`
+	AvailableToolSchemas any    `json:"available_tool_schemas"`
+}
+
+// Evaluation is returned for every enabled PreToolUse call. UnsafeProbability
+// retains its wire name for compatibility; V3 supplies an agreement score, not
+// a calibrated risk probability. It is absent on abstention or failure; ShadowDecision then becomes unavailable and the real
+// tool authorization remains untouched (fail open for this pilot).
+type Evaluation struct {
+	UnsafeProbability  *float64 `json:"unsafe_probability,omitempty"`
+	ShadowDecision     string   `json:"shadow_decision"`
+	Threshold          float64  `json:"threshold"`
+	ModelVersion       string   `json:"model_version"`
+	LatencyMS          float64  `json:"latency_ms"`
+	ErrorCode          string   `json:"error_code,omitempty"`
+	Enforced           bool     `json:"enforced"`
+	UserRequestPresent bool     `json:"user_request_present"`
+	HistoryPresent     bool     `json:"history_present"`
+	HistoryOmitted     bool     `json:"history_omitted"`
+	ToolSchemasPresent bool     `json:"tool_schemas_present"`
+}
+
+type Health struct {
+	Enabled      bool   `json:"enabled"`
+	Status       string `json:"status"`
+	ModelVersion string `json:"model_version,omitempty"`
+	Device       string `json:"device,omitempty"`
+	ErrorCode    string `json:"error_code,omitempty"`
+}
+
+type Config struct {
+	Enabled bool
+	// ModelDir is retained for compatibility with historical development callers.
+	// Production inference always uses the embedded checkpoint.
+	ModelDir       string
+	Timeout        time.Duration
+	StartupTimeout time.Duration
+	MaxConcurrency int
+	ModelVersion   string
+}
+
+// Backend is the narrow boundary between safety policy plumbing and local
+// inference. Production uses the embedded Go model; tests can use deterministic fakes.
+type Backend interface {
+	Infer(context.Context, Input) (InferenceResult, error)
+	Health(context.Context) (Health, error)
+	Close() error
+}
+
+type InferenceResult struct {
+	Logits         [2]float64
+	HistoryOmitted bool
+}
+
+type codedError struct {
+	code string
+	err  error
+}
+
+func (e codedError) Error() string { return e.err.Error() }
+func (e codedError) Unwrap() error { return e.err }
+
+func backendError(code string, err error) error {
+	if err == nil {
+		err = errors.New(code)
+	}
+	return codedError{code: code, err: err}
+}
+
+type Evaluator struct {
+	backend      Backend
+	timeout      time.Duration
+	slots        chan struct{}
+	modelVersion string
+	unavailable  string
+}
+
+func ConfigFromEnv(_ string) (Config, error) {
+	// Advisory assessment is enabled unless the daemon explicitly opts out.
+	enabled, err := envBool("KONTEXT_STEP_SAFETY_SHADOW", true)
+	if err != nil {
+		return Config{}, err
+	}
+	if !enabled {
+		return Config{Enabled: false}, nil
+	}
+	timeout, err := envDuration("KONTEXT_STEP_SAFETY_TIMEOUT", defaultTimeout)
+	if err != nil {
+		return Config{}, err
+	}
+	if timeout > maxConfiguredTimeout {
+		return Config{}, fmt.Errorf("KONTEXT_STEP_SAFETY_TIMEOUT must not exceed %s", maxConfiguredTimeout)
+	}
+	startupTimeout, err := envDuration("KONTEXT_STEP_SAFETY_STARTUP_TIMEOUT", defaultStartupTimeout)
+	if err != nil {
+		return Config{}, err
+	}
+	concurrency, err := envInt("KONTEXT_STEP_SAFETY_MAX_CONCURRENCY", defaultConcurrency)
+	if err != nil {
+		return Config{}, err
+	}
+	if concurrency != 1 {
+		return Config{}, errors.New("KONTEXT_STEP_SAFETY_MAX_CONCURRENCY must be 1 for the singleton pilot session")
+	}
+	// One evaluation runs at a time. Admission stays within the same deadline.
+	return Config{
+		Enabled:        enabled,
+		Timeout:        timeout,
+		StartupTimeout: startupTimeout,
+		MaxConcurrency: concurrency,
+		ModelVersion:   ModelVersion,
+	}, nil
+}
+
+// New loads the embedded native Go checkpoint when enabled. Initialization
+// failures produce an unavailable evaluator rather than failing daemon startup.
+func New(ctx context.Context, cfg Config) *Evaluator {
+	if !cfg.Enabled {
+		return nil
+	}
+	cfg = normalizeConfig(cfg)
+	backend, err := newNativeBackend(ctx, cfg)
+	if err != nil {
+		return &Evaluator{
+			timeout:      cfg.Timeout,
+			slots:        make(chan struct{}, cfg.MaxConcurrency),
+			modelVersion: cfg.ModelVersion,
+			unavailable:  errorCode(err),
+		}
+	}
+	return NewWithBackend(backend, cfg.Timeout, cfg.MaxConcurrency, cfg.ModelVersion)
+}
+
+func NewWithBackend(backend Backend, timeout time.Duration, maxConcurrency int, modelVersion string) *Evaluator {
+	if timeout <= 0 {
+		timeout = defaultTimeout
+	}
+	if maxConcurrency <= 0 {
+		maxConcurrency = defaultConcurrency
+	}
+	if strings.TrimSpace(modelVersion) == "" {
+		modelVersion = ModelVersion
+	}
+	return &Evaluator{
+		backend:      backend,
+		timeout:      timeout,
+		slots:        make(chan struct{}, maxConcurrency),
+		modelVersion: modelVersion,
+	}
+}
+
+func normalizeConfig(cfg Config) Config {
+	if cfg.Timeout <= 0 {
+		cfg.Timeout = defaultTimeout
+	}
+	if cfg.StartupTimeout <= 0 {
+		cfg.StartupTimeout = defaultStartupTimeout
+	}
+	if cfg.MaxConcurrency <= 0 {
+		cfg.MaxConcurrency = defaultConcurrency
+	}
+	if strings.TrimSpace(cfg.ModelVersion) == "" {
+		cfg.ModelVersion = ModelVersion
+	}
+	return cfg
+}
+
+func (e *Evaluator) Evaluate(ctx context.Context, input Input) Evaluation {
+	started := time.Now()
+	result := Evaluation{
+		ShadowDecision:     DecisionUnavailable,
+		Threshold:          Threshold,
+		ModelVersion:       ModelVersion,
+		Enforced:           false,
+		UserRequestPresent: input.RequestTooLarge || strings.TrimSpace(input.UserRequest) != "",
+		HistoryPresent:     historyIsPresent(input.InteractionHistory),
+		ToolSchemasPresent: input.AvailableToolSchemas != nil,
+	}
+	if e == nil {
+		return result
+	}
+	result.ModelVersion = e.modelVersion
+	result.HistoryOmitted = input.HistoryOmitted
+	if ExcludedTool(input.ToolName) {
+		result.ErrorCode = ErrorExcludedTool
+		result.LatencyMS = elapsedMilliseconds(started)
+		return result
+	}
+	if err := validateInputBounds(input); err != nil {
+		result.ErrorCode = errorCode(err)
+		result.LatencyMS = elapsedMilliseconds(started)
+		return result
+	}
+	if e.unavailable != "" || e.backend == nil {
+		result.ErrorCode = firstNonEmpty(e.unavailable, ErrorUnavailable)
+		result.LatencyMS = elapsedMilliseconds(started)
+		return result
+	}
+
+	callCtx, cancel := context.WithTimeout(ctx, e.timeout)
+	defer cancel()
+	select {
+	case e.slots <- struct{}{}:
+	case <-callCtx.Done():
+		result.ErrorCode = ErrorConcurrency
+		result.LatencyMS = elapsedMilliseconds(started)
+		return result
+	}
+
+	// Keep the slot until native execution has actually stopped, even when
+	// the hook deadline wins. This bounds outstanding work and prevents late
+	// results from being attributed to a later request. The native binding does
+	// not interrupt Run, so its completion must not extend the hook deadline.
+	type outcome struct {
+		value InferenceResult
+		err   error
+	}
+	done := make(chan outcome, 1)
+	go func() {
+		defer func() { <-e.slots }()
+		value, err := e.backend.Infer(callCtx, input)
+		done <- outcome{value, err}
+	}()
+	var output outcome
+	select {
+	case output = <-done:
+	case <-callCtx.Done():
+		output.err = callCtx.Err()
+	}
+	result.LatencyMS = elapsedMilliseconds(started)
+	if callCtx.Err() != nil {
+		output.err = callCtx.Err()
+	}
+	if output.err != nil {
+		result.ErrorCode = errorCode(output.err)
+		return result
+	}
+	result.HistoryOmitted = result.HistoryOmitted || output.value.HistoryOmitted
+	logits := output.value.Logits
+	if math.IsNaN(logits[0]) || math.IsNaN(logits[1]) || math.IsInf(logits[0], 0) || math.IsInf(logits[1], 0) {
+		result.ErrorCode = ErrorInvalidOutput
+		return result
+	}
+	probability := CalibratedProbability(logits[1] - logits[0])
+	result.UnsafeProbability = &probability
+	result.ShadowDecision = DecisionSafe
+	if probability >= Threshold {
+		result.ShadowDecision = DecisionUnsafe
+	}
+	return result
+}
+
+func historyIsPresent(history string) bool {
+	normalized := strings.TrimSpace(history)
+	return normalized != "" && normalized != "[]"
+}
+
+func validateInputBounds(input Input) error {
+	if input.RequestTooLarge || len(input.UserRequest) > maxPretokenizedFieldBytes || len(input.InteractionHistory) > maxPretokenizedFieldBytes || len(input.ToolName) > 256 {
+		return backendError(ErrorInputTooLarge, nil)
+	}
+	for _, value := range []any{input.ToolArguments, input.AvailableToolSchemas} {
+		remaining := maxPretokenizedFieldBytes
+		if err := boundedJSONSize(value, &remaining, 0); err != nil {
+			return err
+		}
+	}
+
+	arguments, err := json.Marshal(input.ToolArguments)
+	if err != nil {
+		return backendError(ErrorInference, errors.New("step-safety arguments are not JSON encodable"))
+	}
+	schemas := []byte(nil)
+	if input.AvailableToolSchemas != nil {
+		schemas, err = json.Marshal(input.AvailableToolSchemas)
+		if err != nil {
+			return backendError(ErrorInference, errors.New("step-safety schemas are not JSON encodable"))
+		}
+	}
+	fields := []int{
+		len(input.UserRequest),
+		len(input.InteractionHistory),
+		len(input.ToolName) + len(arguments) + len("[TOOL_NAME]\n\n[ARGUMENTS]\n"),
+		len(schemas),
+	}
+	total := 0
+	for _, size := range fields {
+		if size > maxPretokenizedFieldBytes {
+			return backendError(ErrorInputTooLarge, errors.New("step-safety input field exceeds preprocessing bound"))
+		}
+		total += size
+	}
+	if total > maxPretokenizedTotalBytes {
+		return backendError(ErrorInputTooLarge, errors.New("step-safety input exceeds preprocessing bound"))
+	}
+	return nil
+}
+
+func CalibratedProbability(margin float64) float64 {
+	z := calibrationScale*margin + calibrationBias
+	if z >= 0 {
+		return 1 / (1 + math.Exp(-z))
+	}
+	exp := math.Exp(z)
+	return exp / (1 + exp)
+}
+
+func (e *Evaluator) Health(ctx context.Context) Health {
+	if e == nil {
+		return Health{Enabled: false, Status: "disabled"}
+	}
+	if e.unavailable != "" || e.backend == nil {
+		return Health{Enabled: true, Status: "unavailable", ModelVersion: e.modelVersion, ErrorCode: firstNonEmpty(e.unavailable, ErrorUnavailable)}
+	}
+	health, err := e.backend.Health(ctx)
+	if err != nil {
+		return Health{Enabled: true, Status: "unavailable", ModelVersion: e.modelVersion, ErrorCode: errorCode(err)}
+	}
+	health.Enabled = true
+	if health.ModelVersion == "" {
+		health.ModelVersion = e.modelVersion
+	}
+	return health
+}
+
+func (e *Evaluator) Close() error {
+	if e == nil || e.backend == nil {
+		return nil
+	}
+	return e.backend.Close()
+}
+
+func errorCode(err error) string {
+	if err == nil {
+		return ""
+	}
+	if errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
+		return ErrorTimeout
+	}
+	var coded codedError
+	if errors.As(err, &coded) && coded.code != "" {
+		return coded.code
+	}
+	return ErrorInference
+}
+
+func elapsedMilliseconds(started time.Time) float64 {
+	return float64(time.Since(started).Microseconds()) / 1000
+}
+
+func firstNonEmpty(values ...string) string {
+	for _, value := range values {
+		if value != "" {
+			return value
+		}
+	}
+	return ""
+}
+
+func envString(key, fallback string) string {
+	if value := strings.TrimSpace(os.Getenv(key)); value != "" {
+		return value
+	}
+	return fallback
+}
+
+func envBool(key string, fallback bool) (bool, error) {
+	value := strings.TrimSpace(os.Getenv(key))
+	if value == "" {
+		return fallback, nil
+	}
+	parsed, err := strconv.ParseBool(value)
+	if err != nil {
+		return false, fmt.Errorf("%s must be a boolean: %w", key, err)
+	}
+	return parsed, nil
+}
+
+func envDuration(key string, fallback time.Duration) (time.Duration, error) {
+	value := strings.TrimSpace(os.Getenv(key))
+	if value == "" {
+		return fallback, nil
+	}
+	parsed, err := time.ParseDuration(value)
+	if err != nil || parsed <= 0 {
+		return 0, fmt.Errorf("%s must be a positive duration", key)
+	}
+	return parsed, nil
+}
+
+func envInt(key string, fallback int) (int, error) {
+	value := strings.TrimSpace(os.Getenv(key))
+	if value == "" {
+		return fallback, nil
+	}
+	parsed, err := strconv.Atoi(value)
+	if err != nil {
+		return 0, fmt.Errorf("%s must be an integer: %w", key, err)
+	}
+	return parsed, nil
+}

v3-ungated/native/stepsafety/unicode9.go

@@ -0,0 +1,79 @@
+// Code generated by UPDATE_GOLDEN=1 go test -run TestPostUnicode9TableCoversXText; DO NOT EDIT.
+
+package stepsafety
+
+import "unicode"
+
+// postUnicode9NFC holds the runes Unicode 9.0 leaves unassigned that the
+// Unicode 17.0.0 normalization tables reorder, compose or decompose.
+var postUnicode9NFC = &unicode.RangeTable{
+	R16: []unicode.Range16{
+		{Lo: 0x07fd, Hi: 0x07fd, Stride: 1},
+		{Lo: 0x0897, Hi: 0x089f, Stride: 1},
+		{Lo: 0x08ca, Hi: 0x08d3, Stride: 1},
+		{Lo: 0x09fe, Hi: 0x09fe, Stride: 1},
+		{Lo: 0x0c3c, Hi: 0x0c3c, Stride: 1},
+		{Lo: 0x0d3b, Hi: 0x0d3c, Stride: 1},
+		{Lo: 0x0eba, Hi: 0x0eba, Stride: 1},
+		{Lo: 0x1715, Hi: 0x1715, Stride: 1},
+		{Lo: 0x1abf, Hi: 0x1add, Stride: 1},
+		{Lo: 0x1ae0, Hi: 0x1aeb, Stride: 1},
+		{Lo: 0x1df6, Hi: 0x1dfa, Stride: 1},
+		{Lo: 0xa82c, Hi: 0xa82c, Stride: 1},
+	},
+	R32: []unicode.Range32{
+		{Lo: 0x105c9, Hi: 0x105c9, Stride: 1},
+		{Lo: 0x105d2, Hi: 0x105d2, Stride: 1},
+		{Lo: 0x105da, Hi: 0x105da, Stride: 1},
+		{Lo: 0x105e4, Hi: 0x105e4, Stride: 1},
+		{Lo: 0x10d24, Hi: 0x10d27, Stride: 1},
+		{Lo: 0x10d69, Hi: 0x10d6d, Stride: 1},
+		{Lo: 0x10eab, Hi: 0x10eac, Stride: 1},
+		{Lo: 0x10efa, Hi: 0x10efb, Stride: 1},
+		{Lo: 0x10efd, Hi: 0x10eff, Stride: 1},
+		{Lo: 0x10f46, Hi: 0x10f50, Stride: 1},
+		{Lo: 0x10f82, Hi: 0x10f85, Stride: 1},
+		{Lo: 0x11070, Hi: 0x11070, Stride: 1},
+		{Lo: 0x1133b, Hi: 0x1133b, Stride: 1},
+		{Lo: 0x11382, Hi: 0x11385, Stride: 1},
+		{Lo: 0x1138b, Hi: 0x1138b, Stride: 1},
+		{Lo: 0x1138e, Hi: 0x1138e, Stride: 1},
+		{Lo: 0x11390, Hi: 0x11391, Stride: 1},
+		{Lo: 0x113b8, Hi: 0x113b8, Stride: 1},
+		{Lo: 0x113bb, Hi: 0x113bb, Stride: 1},
+		{Lo: 0x113c2, Hi: 0x113c2, Stride: 1},
+		{Lo: 0x113c5, Hi: 0x113c5, Stride: 1},
+		{Lo: 0x113c7, Hi: 0x113c9, Stride: 1},
+		{Lo: 0x113ce, Hi: 0x113d0, Stride: 1},
+		{Lo: 0x1145e, Hi: 0x1145e, Stride: 1},
+		{Lo: 0x11839, Hi: 0x1183a, Stride: 1},
+		{Lo: 0x11930, Hi: 0x11930, Stride: 1},
+		{Lo: 0x11935, Hi: 0x11935, Stride: 1},
+		{Lo: 0x11938, Hi: 0x11938, Stride: 1},
+		{Lo: 0x1193d, Hi: 0x1193e, Stride: 1},
+		{Lo: 0x11943, Hi: 0x11943, Stride: 1},
+		{Lo: 0x119e0, Hi: 0x119e0, Stride: 1},
+		{Lo: 0x11a34, Hi: 0x11a34, Stride: 1},
+		{Lo: 0x11a47, Hi: 0x11a47, Stride: 1},
+		{Lo: 0x11a99, Hi: 0x11a99, Stride: 1},
+		{Lo: 0x11d42, Hi: 0x11d42, Stride: 1},
+		{Lo: 0x11d44, Hi: 0x11d45, Stride: 1},
+		{Lo: 0x11d97, Hi: 0x11d97, Stride: 1},
+		{Lo: 0x11f41, Hi: 0x11f42, Stride: 1},
+		{Lo: 0x1611e, Hi: 0x16129, Stride: 1},
+		{Lo: 0x1612f, Hi: 0x1612f, Stride: 1},
+		{Lo: 0x16d63, Hi: 0x16d63, Stride: 1},
+		{Lo: 0x16d67, Hi: 0x16d6a, Stride: 1},
+		{Lo: 0x16ff0, Hi: 0x16ff1, Stride: 1},
+		{Lo: 0x1e08f, Hi: 0x1e08f, Stride: 1},
+		{Lo: 0x1e130, Hi: 0x1e136, Stride: 1},
+		{Lo: 0x1e2ae, Hi: 0x1e2ae, Stride: 1},
+		{Lo: 0x1e2ec, Hi: 0x1e2ef, Stride: 1},
+		{Lo: 0x1e4ec, Hi: 0x1e4ef, Stride: 1},
+		{Lo: 0x1e5ee, Hi: 0x1e5ef, Stride: 1},
+		{Lo: 0x1e6e3, Hi: 0x1e6e3, Stride: 1},
+		{Lo: 0x1e6e6, Hi: 0x1e6e6, Stride: 1},
+		{Lo: 0x1e6ee, Hi: 0x1e6ef, Stride: 1},
+		{Lo: 0x1e6f5, Hi: 0x1e6f5, Stride: 1},
+	},
+}

v3-ungated/native/stepsafety/unigram.go

@@ -0,0 +1,167 @@
+package stepsafety
+
+import (
+	"encoding/json"
+	"errors"
+	"math"
+	"strings"
+	"unicode/utf8"
+)
+
+const metaspace = "▁"
+
+// unigram is the Metaspace pre-tokenizer and Unigram model of the pinned
+// tokenizer.json, matching HF tokenizers 0.22.2 (Unigram::encode_optimized)
+// exactly: every vocab entry is matchable, including the literal "<0xNN>"
+// and bracketed special pieces; path scores accumulate in float64; the first
+// candidate to reach a position wins ties; an unknown rune scores the lowest
+// piece score minus 10, and consecutive unknowns fuse into one [UNK].
+//
+// The vocabulary is a map whose keys share one backing string, about 8 MiB.
+// A byte trie with a 256-pointer array per node held the same vocabulary in
+// more than 600 MiB for the lifetime of the daemon.
+type unigram struct {
+	ids      map[string]int32
+	scores   []float32 // the artifact's scores are exact float32 values
+	maxLen   int       // longest piece in bytes
+	unkID    int32
+	unkScore float64
+}
+
+func parseUnigram(data []byte) (*unigram, error) {
+	var config struct {
+		Model struct {
+			Type         string            `json:"type"`
+			UnkID        *int              `json:"unk_id"`
+			ByteFallback bool              `json:"byte_fallback"`
+			Vocab        []json.RawMessage `json:"vocab"`
+		} `json:"model"`
+	}
+	if err := json.Unmarshal(data, &config); err != nil {
+		return nil, err
+	}
+	model := config.Model
+	if model.Type != "Unigram" || model.UnkID == nil || *model.UnkID < 0 || *model.UnkID >= len(model.Vocab) || model.ByteFallback {
+		return nil, errors.New("unsupported tokenizer model")
+	}
+	pieces := make([]string, len(model.Vocab))
+	scores := make([]float32, len(model.Vocab))
+	size, minScore := 0, math.Inf(1)
+	for i, raw := range model.Vocab {
+		var entry [2]json.RawMessage
+		var score float64
+		if err := json.Unmarshal(raw, &entry); err != nil {
+			return nil, err
+		}
+		if err := json.Unmarshal(entry[0], &pieces[i]); err != nil {
+			return nil, err
+		}
+		if err := json.Unmarshal(entry[1], &score); err != nil {
+			return nil, err
+		}
+		if float64(float32(score)) != score || pieces[i] == "" || !utf8.ValidString(pieces[i]) {
+			return nil, errors.New("unsupported tokenizer vocabulary entry")
+		}
+		scores[i] = float32(score)
+		minScore = math.Min(minScore, score)
+		size += len(pieces[i])
+	}
+	var backing strings.Builder
+	backing.Grow(size)
+	for _, piece := range pieces {
+		backing.WriteString(piece)
+	}
+	all := backing.String()
+	u := &unigram{ids: make(map[string]int32, len(pieces)), scores: scores, unkID: int32(*model.UnkID), unkScore: minScore - 10}
+	offset := 0
+	for i, piece := range pieces {
+		// HF resolves duplicate pieces to the later id.
+		u.ids[all[offset:offset+len(piece)]] = int32(i)
+		offset += len(piece)
+		u.maxLen = max(u.maxLen, len(piece))
+	}
+	return u, nil
+}
+
+// encode pre-tokenizes one normalized segment that holds no special tokens:
+// spaces become ▁, a ▁ is prepended unless present, and every word starts at
+// a ▁. Each word is segmented independently.
+func (u *unigram) encode(normalized string, ids []int64) []int64 {
+	if normalized == "" {
+		return ids
+	}
+	text := strings.ReplaceAll(normalized, " ", metaspace)
+	if !strings.HasPrefix(text, metaspace) {
+		text = metaspace + text
+	}
+	for len(text) > 0 {
+		word := text
+		if next := strings.Index(text[len(metaspace):], metaspace); next >= 0 {
+			word = text[:len(metaspace)+next]
+		}
+		ids = u.segment(word, ids)
+		text = text[len(word):]
+	}
+	return ids
+}
+
+type unigramNode struct {
+	score float64
+	start int // -1 until reached
+	id    int32
+}
+
+// segment appends the best-scoring segmentation of word.
+func (u *unigram) segment(word string, ids []int64) []int64 {
+	n := len(word)
+	nodes := make([]unigramNode, n+1)
+	for i := range nodes {
+		nodes[i].start = -1
+	}
+	relax := func(end, start int, id int32, score float64) {
+		if node := &nodes[end]; node.start < 0 || score > node.score {
+			*node = unigramNode{score: score, start: start, id: id}
+		}
+	}
+	for start := 0; start < n; {
+		here := nodes[start].score
+		_, runeLen := utf8.DecodeRuneInString(word[start:])
+		single := false
+		for end := start + 1; end <= min(n, start+u.maxLen); end++ {
+			// Pieces are valid UTF-8, so they end on rune boundaries.
+			if end < n && !utf8.RuneStart(word[end]) {
+				continue
+			}
+			id, ok := u.ids[word[start:end]]
+			if !ok {
+				continue
+			}
+			relax(end, start, id, here+float64(u.scores[id]))
+			single = single || end-start == runeLen
+		}
+		if !single {
+			relax(start+runeLen, start, u.unkID, here+u.unkScore)
+		}
+		start += runeLen
+	}
+	first := len(ids)
+	for end := n; end > 0; {
+		node := nodes[end]
+		start, id := node.start, node.id
+		if id == u.unkID {
+			for start > 0 && nodes[start].id == u.unkID {
+				start = nodes[start].start
+			}
+			// HF maps the fused surface string back through the vocabulary.
+			if known, ok := u.ids[word[start:end]]; ok {
+				id = known
+			}
+		}
+		ids = append(ids, int64(id))
+		end = start
+	}
+	for i, j := first, len(ids)-1; i < j; i, j = i+1, j-1 {
+		ids[i], ids[j] = ids[j], ids[i]
+	}
+	return ids
+}

v3-ungated/release_manifest.json

@@ -0,0 +1,180 @@
+{
+  "contains_private_examples": false,
+  "files": {
+    "LICENSE": {
+      "sha256": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
+      "size": 11357
+    },
+    "NOTICE": {
+      "sha256": "5f307f4fc93430c9abe22bec00c00f3844291724bef614ac0a527a163ca5eab1",
+      "size": 271
+    },
+    "README.md": {
+      "sha256": "5d4285e2dd08af2d9a449889780ab2cf55c8ba13c1d8154aae04c734b2b04128",
+      "size": 4424
+    },
+    "evaluation.json": {
+      "sha256": "186ec26dd500b5f8791283a619200450c0f10e523ce3faca92a296f6acec1562",
+      "size": 1517
+    },
+    "native/cmd/merlin/main.go": {
+      "sha256": "0a9bbc83520ef46b1b3fc031f16fb2315f7a94d73e8dfef0d78374e64e466376",
+      "size": 1414
+    },
+    "native/go.mod": {
+      "sha256": "97ffbb6542f80e9cbc14bf46354388a7ba0ce18a1ad4822b64c750b7eda7abe4",
+      "size": 129
+    },
+    "native/go.sum": {
+      "sha256": "a121c8ef8f666600e1b0dde83144361a6a67d2850f36515c00d2860adfb53b23",
+      "size": 312
+    },
+    "native/stepsafety/agreement.go": {
+      "sha256": "a19ce6cba08771b455741ba7d2622334636c50b7ffde44fc98341c767f1d2363",
+      "size": 3798
+    },
+    "native/stepsafety/context.go": {
+      "sha256": "616f619ec4ff787d810085e1566836d13e5106ec6ee42a086411999329dd1563",
+      "size": 7264
+    },
+    "native/stepsafety/model/ATTRIBUTION.md": {
+      "sha256": "35596b6f09742aa91e8e11fea77170605db0e5d62f2c818f5c90a7b6d16b520e",
+      "size": 616
+    },
+    "native/stepsafety/model/LICENSE": {
+      "sha256": "646f8936b8ddcd14e13e578ff6857e368780b0d1a4f6066bee89211923a373e2",
+      "size": 1073
+    },
+    "native/stepsafety/model/PROVENANCE.json": {
+      "sha256": "dec7f4875979af4bab488e1aca9c6fcad97b8d56ea361a40dd1da8588ae1a3ab",
+      "size": 3748
+    },
+    "native/stepsafety/model/native/PROVENANCE.json": {
+      "sha256": "2432e56272cca72e32504a0c3605c2611f957ff58e70d23095bbbad7f3844739",
+      "size": 2622
+    },
+    "native/stepsafety/model/native/agreement.json.gz": {
+      "sha256": "6ffbd35e56a031df9f2ca08f7520486cc8ef65592c8d03cfe966fbd42523bbdf",
+      "size": 2290715
+    },
+    "native/stepsafety/model/native/candidate.json": {
+      "sha256": "776a05f6f1b3854527919a24a98815d17df8a7cd2928f0d05b4cfe286217c119",
+      "size": 3675
+    },
+    "native/stepsafety/model/native/manifest.json": {
+      "sha256": "ec4ca2647df5326ac2fc919e3eb71ad82fc5597e0e077db759b430ba941e5ee5",
+      "size": 1591
+    },
+    "native/stepsafety/model/native/request_gate.json.gz": {
+      "sha256": "acfb7025661b5a424d482aae1fd351f31f714bf9c22b7c18dcabb623ca2e72c3",
+      "size": 2492102
+    },
+    "native/stepsafety/model/native/tokenizer.json.gz": {
+      "sha256": "956061a0f96d88a118848b4f32e4e5fcdc8c363f4cbc223fdc175232668aa41c",
+      "size": 1736709
+    },
+    "native/stepsafety/model/native/weights.00.gz": {
+      "sha256": "2c69b3cccc79582da64dd47ab44835604b5b4f5c676b1102eacef17dc52cacad",
+      "size": 46014808
+    },
+    "native/stepsafety/model/native/weights.01.gz": {
+      "sha256": "129de5fde020ece20ce7cf2529564c779e486e29b0c4725f7ed09af086f6db04",
+      "size": 41825526
+    },
+    "native/stepsafety/model/native/weights.02.gz": {
+      "sha256": "7cb825c60a553c37d70ee6859d597a49b0afa0e34a2a243d62903da5d297f47e",
+      "size": 42596160
+    },
+    "native/stepsafety/model/native/weights.03.gz": {
+      "sha256": "d5a9ce4740491f638c497b8e88d6d4236a1f26b3debfa9f7566a22c960828b6c",
+      "size": 62319867
+    },
+    "native/stepsafety/model/native/weights.04.gz": {
+      "sha256": "9316ce90d589cba6e3ce02f5e643fa244ff7e7bb2b5117b8508f176f01ed3000",
+      "size": 13716366
+    },
+    "native/stepsafety/modelstore.go": {
+      "sha256": "d1dfee52f8984b8aafae9f6e519bd28e5cefc2dce274fee46cd3e4ba18b0fd92",
+      "size": 12453
+    },
+    "native/stepsafety/native_assets.go": {
+      "sha256": "73781cc21e22742983297f3e216ec4ede722e9c2281eaa8bfaee4515eb02a72b",
+      "size": 403
+    },
+    "native/stepsafety/native_assets_nomodel.go": {
+      "sha256": "d1c5d28591ae19d1f9003ffb29f56a7e2a0491c03a16ac2ab6c6400ec32e4dc6",
+      "size": 454
+    },
+    "native/stepsafety/native_backend.go": {
+      "sha256": "e94731e7b57cd0bdac1b045dd76f10fcf8690a5bec67ef2b5f1b97d7141f97b1",
+      "size": 3611
+    },
+    "native/stepsafety/native_linear_arm64.go": {
+      "sha256": "2a510eb9fc849bdf59d4e13fc9288dc14bca41034fb307f76f84ab30d49cb53b",
+      "size": 2108
+    },
+    "native/stepsafety/native_linear_arm64.s": {
+      "sha256": "f7d41e764cb85dc8b855d6a7f94834681c9aff236e67f3d1b04d2efa138de4b7",
+      "size": 1340
+    },
+    "native/stepsafety/native_linear_generic.go": {
+      "sha256": "c54f032dcb5a9b574cd539b838eacd83b71376b898edbb771e06ee5a191d947b",
+      "size": 690
+    },
+    "native/stepsafety/native_model.go": {
+      "sha256": "02fc641a5439fd359bd5f8dd5d6f26f4396ee2bbf0f3b06b6dac2c029989addc",
+      "size": 9618
+    },
+    "native/stepsafety/native_weights.go": {
+      "sha256": "104a1ea3851d97d920e1ae7b60d2b21be3ad58470a089fc5538eb2ba511af511",
+      "size": 4065
+    },
+    "native/stepsafety/native_workspace.go": {
+      "sha256": "6aa87fec23a4a88783990463dca83015512c93d9fcca1829e559cab729b64211",
+      "size": 1420
+    },
+    "native/stepsafety/normalization.go": {
+      "sha256": "b67a2422a587b6d59719dfb4027058b6f32495dec29ddb75d8524d1c0e1ba434",
+      "size": 5107
+    },
+    "native/stepsafety/packing.go": {
+      "sha256": "694c59c3d9e4724b912d34a7101d07fcf6fb5d2449883216eba11379e1c4f9d5",
+      "size": 5942
+    },
+    "native/stepsafety/request_gate.go": {
+      "sha256": "cb94dd7a32ab72e7a3f75017d297ee74a17ca0baa56283194f6f8febac2b7d83",
+      "size": 2758
+    },
+    "native/stepsafety/runtime_artifacts.go": {
+      "sha256": "cee5b858b63b60227d32ac241a47e7428df5f9d98a66c3397cf64f1fa035a3f4",
+      "size": 1672
+    },
+    "native/stepsafety/runtime_store.go": {
+      "sha256": "cb4fc9d15c372d303a76bd878ef2a992bb72eea728636792169c1f727ea5b25f",
+      "size": 5702
+    },
+    "native/stepsafety/scoped.go": {
+      "sha256": "00ffb738e773127099f9b25bcd4e7772e7b6796af17ef3ae76cb63960d1e2927",
+      "size": 5510
+    },
+    "native/stepsafety/tokenizer.go": {
+      "sha256": "3530fd17e4d5898636f19ff9246e6305b7e322373905d1bd83508042274f5b5c",
+      "size": 3663
+    },
+    "native/stepsafety/types.go": {
+      "sha256": "faa85a9c0438c6bffed0184d18d1531481d42500d0b7c9679f77d1a6a110c54a",
+      "size": 13934
+    },
+    "native/stepsafety/unicode9.go": {
+      "sha256": "3b110e09e6901a4d79cb085bbfaaeda4186e03c75a3ce196de29a3b7ee669c3d",
+      "size": 3024
+    },
+    "native/stepsafety/unigram.go": {
+      "sha256": "0a79d9c5c75306be59647997d8d29d2d2a917455f72a7fdf1d8a40bc62a09df0",
+      "size": 5041
+    }
+  },
+  "runtime": "merlin-v3-ungated-20261104",
+  "schema_version": 1,
+  "weights_candidate": "merlin-v3-precision-20261007"
+}