Add selected BF16 model weights, tokenizer and offline inference runner

CITATION.cff

@@ -9,4 +9,4 @@ date-released: 2026-10-14
 url: "https://github.com/williamzebrowskI/OpenSML-JSONLite-175M"
 repository-code: "https://github.com/williamzebrowskI/OpenSML-JSONLite-175M"
 license: Apache-2.0
-abstract: "Source, tokenizer, training provenance and observed evaluation records for a 175.6M-parameter field extractor with JSON assembly. Source license only; weights are not included."
+abstract: "Source, tokenizer, training provenance and observed evaluation records for a 175.6M-parameter field extractor with JSON assembly. The Hugging Face package includes the selected weights and native JSON adapter; the source license does not assign a model-weight license."

PACKAGING_VERIFICATION.json

@@ -0,0 +1,33 @@
+{
+  "model": "OpenSML-JSONLite-175M-Instruct",
+  "date": "2026-10-14",
+  "weight_file": "model.safetensors",
+  "weight_bytes": 351242342,
+  "weight_sha256": "d552c1380c71c7c0569e9d692ec9c46c5c94bb87a7b024e56df0f0e36df4d798",
+  "runtime": "stock MLX 0.32.3, Apple Silicon GPU, Python 3.11",
+  "root_runner_verification": {
+    "command": "python inference.py --example 29",
+    "default_weights": true,
+    "source_bundle_integrity": true,
+    "frozen_tokenizer_integrity": true,
+    "schema_valid": true,
+    "natural_stop": true,
+    "exact": true,
+    "value": {
+      "track": [
+        "Blue Lantern"
+      ],
+      "artist": [
+        "Mira Sol"
+      ],
+      "album": []
+    }
+  },
+  "tokenizer_root_matches_bundle": true,
+  "prior_native_fixture_parity": {
+    "requests": 40,
+    "value_and_stop_matches": 40,
+    "correct_against_reference": 29
+  },
+  "scope": "Package entry-point validation; no new blind benchmark or full-test cross-runtime parity claim."
+}

README.md

@@ -25,7 +25,7 @@ datasets:
 
 **Author:** William Zebrowski · **Model:** OpenSML-JSONLite-175M-Instruct
 
-[Technical report](TECHNICAL_REPORT.md) · [Results](JSONLITE_RESULTS.json) · [Checkpoint provenance](checkpoint_provenance.json) · [Native inference verification](INFERENCE_VERIFICATION.json)
+[Run locally](RUN_GUIDE.md) · [Technical report](TECHNICAL_REPORT.md) · [Results](JSONLITE_RESULTS.json) · [Checkpoint provenance](checkpoint_provenance.json) · [Native inference verification](INFERENCE_VERIFICATION.json)
 
 **A 175.6M-parameter English field extractor with a schema-guided JSON adapter.**
 
@@ -35,27 +35,26 @@ The weights were trained from scratch through OpenSML Stage A pretraining, then
 
 ## Availability and inference
 
-This repository contains the model card, configuration, detailed technical report, tokenizer/inference source bundle, verified extraction results and checkpoint identity. **The selected weights have not yet been uploaded to this repository.** They remain preserved in the owner's research workspace. A weight artifact and model-weight license are separate release items.
+This repository includes the selected **BF16 `model.safetensors`**, frozen tokenizer, configuration, native inference runner, source bundle, examples and verified extraction results. Download the complete repository to run the model locally. The verified platform is Apple Silicon macOS with Python 3.11 and stock MLX 0.32.3.
 
-This repository is not deployed by a hosted inference provider. It is not a Transformers AutoModel or mlx-lm auto-loading export.
-
-The included `native_bundle.zip` contains the standalone inference entry point, frozen tokenizer, required model code, field adapter, examples, evaluation script, provenance and source notices. It contains no weight or optimizer binaries. Supply the matching local BF16 weight file to run it:
+This is a custom native MLX architecture. Use the supplied runner; it is not a Transformers AutoModel or mlx-lm auto-loading export. The source bundle contains the exact model implementation and schema-guided extraction/JSON adapter. Optimizer state is unnecessary for inference and is not included.
 
 ```bash
-# Authenticate if access requires it.
-hf auth login
-hf download wzebrowski/OpenSML-JSONLite-175M --local-dir ./OpenSML-JSONLite-175M
-cd OpenSML-JSONLite-175M
-python -m zipfile -e native_bundle.zip native_bundle
-cd native_bundle
 python3 -m venv .venv
 source .venv/bin/activate
+python -m pip install huggingface_hub
+# If access requires authentication: hf auth login
+hf download wzebrowski/OpenSML-JSONLite-175M --local-dir ./OpenSML-JSONLite-175M
+cd OpenSML-JSONLite-175M
 python -m pip install -r requirements.txt
-python scripts/verify_snapshot.py
-python inference.py --weights /path/to/model.safetensors --example 29
+python inference.py --example 29
+# Your own source and schema:
+python inference.py --text-file request.txt --schema-file schema.json
 ```
 
-There is no downloadable weight file in this repository yet. The exact weight SHA-256 required by the runner is:
+The root runner defaults to `model.safetensors`, verifies and temporarily extracts `native_bundle.zip`, then runs the frozen native implementation. After downloading the artifacts and installing dependencies, inference runs offline. The root `tokenizer.json` and `tokenizer_config.json` mirror the frozen tokenizer in the bundle, which includes its additional integrity metadata. See [RUN_GUIDE.md](RUN_GUIDE.md) for complete instructions and an example schema.
+
+The uploaded weight SHA-256 is:
 
 `d552c1380c71c7c0569e9d692ec9c46c5c94bb87a7b024e56df0f0e36df4d798`
 

RUN_GUIDE.md

@@ -0,0 +1,96 @@
+# Run OpenSML-JSONLite-175M-Instruct locally
+
+The repository includes the selected BF16 `model.safetensors`, frozen tokenizer,
+configuration, native MLX implementation and schema-to-extraction/JSON adapter.
+No training workspace, optimizer state or external inference service is needed.
+
+The verified runtime is **Apple Silicon macOS with Python 3.11 and MLX 0.32.3**.
+This is a custom native architecture: use the supplied runner, rather than
+`transformers.AutoModel` or `mlx_lm.load`. CUDA, CPU-only and quantized exports
+have not been verified.
+
+## Download and run
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install huggingface_hub
+# If access requires authentication: hf auth login
+hf download wzebrowski/OpenSML-JSONLite-175M --local-dir OpenSML-JSONLite-175M
+cd OpenSML-JSONLite-175M
+python -m pip install -r requirements.txt
+python inference.py --example 29
+```
+
+Once downloaded and dependencies installed, generation runs offline. The runner
+defaults to `model.safetensors` beside it, verifies its SHA-256, verifies and
+unpacks the native source bundle into a temporary directory, and loads the
+frozen tokenizer. It cleans up the temporary code after execution.
+
+Example 29 source: `Play the track Blue Lantern by Mira Sol.` Its result is:
+
+```json
+{"track": ["Blue Lantern"], "artist": ["Mira Sol"], "album": []}
+```
+
+The command prints a result record including `value`, `schema_valid`,
+`success`, intermediate extractions and stop reasons. Only example mode adds
+`exact` by comparing the generated value to a caller-side reference. Reference
+answers are never given to the model.
+
+## Your own text and schema
+
+```bash
+python inference.py --text-file request.txt --schema-file schema.json
+```
+
+`request.txt` contains your source text. `schema.json` requests a closed object
+with one to eight required properties, each an array of strings. For the example:
+
+```json
+{
+  "type": "object",
+  "properties": {
+    "track": {"type": "array", "items": {"type": "string"}, "description": "track name"},
+    "artist": {"type": "array", "items": {"type": "string"}, "description": "performer or musical artist"},
+    "album": {"type": "array", "items": {"type": "string"}, "description": "album name"}
+  },
+  "required": ["track", "artist", "album"],
+  "additionalProperties": false
+}
+```
+
+Only the 39 documented field definitions are supported. Missing values are `[]`.
+The schema adapter calls the model once per field, parses labeled values,
+assembles JSON and validates it. This is not unrestricted arbitrary-schema
+JSON generation. See the model card for supported fields, limits and measured
+accuracy. Structural success does not establish content correctness.
+
+The included source archive contains 40 examples in `examples/cases.json` and
+`inference_native/adapter.py` contains the complete field glossary.
+
+## Integrity and lower-level use
+
+On macOS, verify all downloadable artifacts with:
+
+```bash
+shasum -a 256 -c SHA256SUMS
+```
+
+The chosen weight identity is
+`d552c1380c71c7c0569e9d692ec9c46c5c94bb87a7b024e56df0f0e36df4d798`.
+Root tokenizer files are byte-identical to the native bundle's frozen tokenizer.
+`tokenizer_manifest.json` records its native metadata; the bundle also contains
+all companion tokenizer files needed for its integrity checks.
+
+For direct access to the implementation and evaluator:
+
+```bash
+python -m zipfile -e native_bundle.zip native_bundle
+python native_bundle/scripts/verify_snapshot.py
+python native_bundle/inference.py --weights "$PWD/model.safetensors" --example 29
+```
+
+Generation uses greedy BF16 native MLX, repetition penalty 1, maximum 160 tokens
+per field and a 2,048-token context. Overlong input or unsupported schemas are
+rejected. No silent truncation, fuzzy correction or online API calls occur.

SHA256SUMS

@@ -1,12 +1,20 @@
-7814ab50b4bcdb1d0d9cf36134ac0d494360693131f4954ffe6b87266bfe0a03  CITATION.cff
+d0b49f29bcb68dbb1d9967dc026b972ef9fa4a22eb67c9fea88539a90a786d03  CITATION.cff
 82b9908c9fd21fd8c9d10ad809e7c48446485ee98a56bfd9d64bf9fb353761c2  INFERENCE_VERIFICATION.json
 d29cf5952a82db444a6df56446806b1f4e2ccbee1b4aa6b93cb7eeeabcab8536  JSONLITE_RESULTS.json
 c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4  LICENSE
-12ca825efe5784d85a45ed2e98b7d12e7deb9b914e8d095cbc341ce92e0a0692  README.md
-64519f25aded3e1fa74b0b9eafdf35df4490e7bb550027338a9bf1a25f9203bb  SOURCE_NOTICE.md
-6b7b7e931c7a4e119411ef93dd2535927ed2adb647273471ef5995110df144f8  TECHNICAL_REPORT.md
+8179f766c423a10e719ccd5efeb097b60e460323507bf2644b9626edc0a441c6  PACKAGING_VERIFICATION.json
+8ef3023f43332c3ae0a42270b9b7595d3a1126d9248077b57f4a20d71381227e  README.md
+994a1798d7b961e34b581bf5337dd5dd14f533e9ebfb4fe1be10cd7b17edf0ec  RUN_GUIDE.md
+5bc419e245a6d8f074186210d9cba59df3e2cc82dfd04452c625268108a6796d  SOURCE_NOTICE.md
+5edeb60b77046aebf671ebc2c670d0637658f1929c621f66b9de89b445a3c623  TECHNICAL_REPORT.md
 423f0d1e0e2690505c24e6d31308db728fba60657c5166e177bb269d6a7198d0  THIRD_PARTY_NOTICES.md
 6f8de0b591c8a07885c8adcab359b8ca1cdc81be5f90638cc0ce286f3899c6c5  checkpoint_provenance.json
 fcc46a0f69fc4afb8d6e25361a013e7656ac6758ae5a49cc6b629f95bf728c1a  complete-request-exact.svg
 2f918a15b7b06a792e48d476fa9884dfc7fb46d5ccd0f756bd3890a8ad259d96  config.json
+5f66ff6f09a6ce26603220defe3aa2bc0231f7144785bc33865683d0097a2c34  inference.py
+d552c1380c71c7c0569e9d692ec9c46c5c94bb87a7b024e56df0f0e36df4d798  model.safetensors
 459b495441931bf520ea14274401509bc6441cd701bbbf2a1d162fcae5eacbc4  native_bundle.zip
+c7b1a3c0efb09ef3b5416a24072e48b25e2cc1f122fb5ed638c0f95b72594af9  requirements.txt
+e095f0413a11fa91fbb58f37d00e4c6bcda59beb89b28f6f680db1436a7e8891  tokenizer.json
+619f500553367eb76f5be00a98625130ec857b9c367561334984096c029ba2a9  tokenizer_config.json
+1194292c6d906ac19e6f01c6ad2b42825143fad426a22aa8856d903776701ecb  tokenizer_manifest.json

SOURCE_NOTICE.md

@@ -2,7 +2,7 @@
 
 OpenSML-JSONLite-175M is extracted from William Zebrowski's OpenSML research workspace. Its inherited weights follow Stage A/132005 → SFT-065/469 → SFT-068/379 → recreated SFT-093/160. Supporting code dependencies and unused representations are not additional weight ancestors.
 
-Author-controlled source contributions use the root Apache-2.0 license. Upstream dependencies, datasets and their underlying content retain their own notices. No model-weight license is assigned here; no weights are committed to this repository.
+Author-controlled source contributions use the root Apache-2.0 license. Upstream dependencies, datasets and their underlying content retain their own notices. The Hugging Face package includes the selected BF16 weights. This source notice does not assign a model-weight license.
 
 Pretraining uses pinned FineWeb-Edu, DCLM-Edu, English FineWiki, Cosmopedia v2 and the Python-Edu text mirror. Grounded QA uses original selected SQuAD 2.0/Dolly answers plus programmatically checked fictional examples. Extraction uses original SNIPS chunk labels. See the technical report for exact revisions, sampling, exclusions and selected exposure.
 

TECHNICAL_REPORT.md

@@ -1,13 +1,13 @@
 # OpenSML-JSONLite-175M: SFT-093/update 160
 
-Hugging Face companion edition. Archive-relative links in this report resolve inside `native_bundle.zip`.
+Hugging Face companion edition. Archive-relative links in this report resolve inside `native_bundle.zip`. The Hugging Face repository additionally includes the selected `model.safetensors`, root tokenizer files and a runnable entry point; see `RUN_GUIDE.md`.
 
 ## Technical report, training provenance, evaluation and public release specification
 
 **Author/project:** William Zebrowski / OpenSML
 **Report date:** October 14, 2026 (America/New_York)
 **Artifact:** recreated SFT-093/update 160, retained at the owner's request
-**Status:** public source/documentation repository for a preserved experimental candidate; weights are not yet published and no public inference endpoint is deployed.
+The Hugging Face package includes the selected BF16 weights and offline native runner. No hosted inference endpoint is deployed.
 
 This report describes one exact model and its accompanying inference pipeline. The model is a 175,611,648-parameter decoder pretrained from random initialization, then adapted through grounded QA, requested-field extraction and focused single-field extraction. Its learned output is a labeled list. Software accepts a supported JSON Schema, turns each requested property into an extraction question, strictly parses generated field lines, assembles JSON and validates its structure.
 
@@ -748,7 +748,7 @@ The initial base report described an Apache-2.0 code license, but a root `LICENS
 
 Completed work includes preservation of the checkpoint, verification of its lineage and hashes, recording of development, complete observed-test and playground metrics, documentation of the field/schema adapter, and verification of the local API and UI. The original and recreated model identities are recorded separately.
 
-The public GitHub source repository is now OpenSML-JSONLite-175M. Its source code uses the included Apache-2.0 license; dependency/source notices, a native inference runner, pinned public runtime dependencies, a 40-case clean-environment parity record and a repository integrity manifest are included. Remaining weight-release items are the owner-assigned weight license and public weight artifact. Broader external evaluation and full-suite parity still need separate verification. Evaluation beyond the reused SNIPS panel and 12 authored cases is still needed to establish transfer to intended production inputs. An experimental release can report these results and limitations; preserving this checkpoint does not override its failed retention flags or prove reliable general extraction.
+The frozen source package is OpenSML-JSONLite-175M. Its source code uses the included Apache-2.0 license; dependency/source notices, a native inference runner, pinned public runtime dependencies, a 40-case clean-environment parity record and a repository integrity manifest are included. The Hugging Face companion package additionally includes the selected weight artifact and tokenizer; the model-weight license remains unassigned. Broader external evaluation and full-suite parity still need separate verification. Evaluation beyond the reused SNIPS panel and 12 authored cases is still needed to establish transfer to intended production inputs. An experimental release can report these results and limitations; preserving this checkpoint does not override its failed retention flags or prove reliable general extraction.
 
 ## 19. Artifact and evidence index
 

inference.py

@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+"""Run JSONLite offline with the bundled native MLX implementation.
+
+Defaults to model.safetensors beside this script. The verified source bundle
+includes the exact model architecture, tokenizer, field adapter and examples.
+"""
+from pathlib import Path
+import hashlib
+import runpy
+import sys
+import tempfile
+import zipfile
+
+ROOT = Path(__file__).resolve().parent
+BUNDLE_SHA256 = "459b495441931bf520ea14274401509bc6441cd701bbbf2a1d162fcae5eacbc4"
+
+def main():
+    archive = ROOT / "native_bundle.zip"
+    if not archive.is_file():
+        raise SystemExit("Missing native_bundle.zip; download the complete repository.")
+    if hashlib.sha256(archive.read_bytes()).hexdigest() != BUNDLE_SHA256:
+        raise SystemExit("Native source bundle failed its integrity check.")
+    if not any(arg == "--weights" or arg.startswith("--weights=") for arg in sys.argv[1:]):
+        sys.argv.extend(["--weights", str(ROOT / "model.safetensors")])
+    with tempfile.TemporaryDirectory(prefix="jsonlite-runtime-") as directory:
+        runtime = Path(directory)
+        with zipfile.ZipFile(archive) as bundle:
+            for member in bundle.infolist():
+                path = Path(member.filename)
+                if path.is_absolute() or ".." in path.parts or (member.external_attr >> 16) & 0o170000 == 0o120000:
+                    raise SystemExit("Unsafe source-bundle entry.")
+            bundle.extractall(runtime)
+        runpy.run_path(str(runtime / "inference.py"), run_name="__main__")
+
+if __name__ == "__main__":
+    main()

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d552c1380c71c7c0569e9d692ec9c46c5c94bb87a7b024e56df0f0e36df4d798
+size 351242342

requirements.txt

@@ -0,0 +1,4 @@
+mlx==0.32.3
+numpy==2.4.2
+tokenizers==0.22.2
+jsonschema==4.26.0

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|doc_end|>",
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}

tokenizer_manifest.json

@@ -0,0 +1,31 @@
+{
+  "boundary_policy": "raw-text-then-one-document-EOS-v1",
+  "corpus_fingerprint": "a89f69d6f63cfa4385da233693a445404234a18194170fe1d2b4a74bf501f5d6",
+  "format": "sml-v2-tokenizer-v1",
+  "recipe": {
+    "corpus": "a89f69d6f63cfa4385da233693a445404234a18194170fe1d2b4a74bf501f5d6",
+    "min_frequency": 2,
+    "sample_manifest": "30ead6f502f11e99b479e3711a81ce7198787408d0700153615fc3255b244f46",
+    "specials": [
+      "<|pad|>",
+      "<|doc_end|>",
+      "<|turn_start|>",
+      "<|turn_end|>"
+    ],
+    "tokenizers": "0.22.2",
+    "version": 1,
+    "vocab_size": 32000
+  },
+  "sha256": {
+    "corpus.json": "e4c47b7b6aba08f948de7c2c7cb94c58dbd09dde2ded6ac185c60856ee686d27",
+    "tokenizer.json": "e095f0413a11fa91fbb58f37d00e4c6bcda59beb89b28f6f680db1436a7e8891",
+    "tokenizer_config.json": "619f500553367eb76f5be00a98625130ec857b9c367561334984096c029ba2a9"
+  },
+  "special_ids": {
+    "eos": 1,
+    "pad": 0,
+    "turn_end": 3,
+    "turn_start": 2
+  },
+  "vocab_size": 32000
+}