Add ExecuTorch WebGPU program for smolvlm-500m-instruct

.gitattributes

@@ -36,3 +36,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 vlm_decoder.pte filter=lfs diff=lfs merge=lfs -text
 vlm_prefill.pte filter=lfs diff=lfs merge=lfs -text
 vlm_vision_q8.pte filter=lfs diff=lfs merge=lfs -text
+smolvlm_decoder_q8.pte filter=lfs diff=lfs merge=lfs -text
+smolvlm_vision_q8.pte filter=lfs diff=lfs merge=lfs -text

README.md

@@ -16,12 +16,12 @@ tags:
 
 This repository contains an [ExecuTorch](https://github.com/pytorch/executorch) conversion of
 [HuggingFaceTB/SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct) for the ExecuTorch **WebGPU** backend, so the model runs entirely in a web browser on
-the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)).
+the visitor's GPU. It powers the corresponding demo in the [ExecuTorch WebGPU Playground](https://executorch.ai/playground/).
 
 - **Base model:** [HuggingFaceTB/SmolVLM-500M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct)
 - **Format:** ExecuTorch `.pte` programs delegated to the WebGPU backend
-- **Quantization:** int8 vision encoder and token embeddings; decoder as exported
-- **Total size:** 893.9 MB
+- **Quantization:** 8-bit linears in the vision encoder and decoder (int8 static activations in two passes, int8 per-channel GPTQ weights); int8 token embeddings
+- **Total size:** 497.7 MB
 
 ## About the original model
 
@@ -43,15 +43,16 @@ Not for high-stakes decisions about people (for example identity, employment or
 
 ## What changed in this conversion
 
-- Vision encoder and token embeddings quantized to int8
-- Vision encoder, prefill and decoder exported as separate programs
-- Context capped at 512 tokens
+- All linear layers of the vision encoder and decoder use 8-bit weights (per-channel, GPTQ) and 8-bit activations with one static scale per layer, applied in two passes (the second pass quantizes the first pass's rounding error)
+- Token embeddings quantized to int8
+- The key/value of the first prompt token (`<|im_start|>`) are built into the decoder, so prompts must start with it
+- Context capped at 512 tokens; quantization changes outputs slightly compared with the original model
 
 ## Inputs and outputs
 
-- `vlm_vision_q8.pte`: one image to visual tokens
-- `vlm_prefill.pte` and `vlm_decoder.pte`: prompt prefill, then token-by-token decoding; embeddings come from `embed_tokens.i8.bin` with `embed_tokens.scale.f32.bin`
-- Tokenize with the files in `tokenizer/`; `vlm_meta.json` lists the special tokens and sizes
+- `smolvlm_vision_q8.pte`: `pixels float32 [1, 3, 512, 512]` (scaled as `(x/255 - 0.5) / 0.5`) -> `image_embeds float32 [1, 64, 960]`
+- `smolvlm_decoder_q8.pte`: `forward(embeds float32 [1, 4, 960], input_pos int64 [1]) -> logits float32 [1, 49280]` for the last of the 4 rows; feed windows of 4 consecutive positions, the KV cache lives inside the program
+- Text embeddings come from `embed_tokens.i8.bin` with `embed_tokens.scale.f32.bin`; tokenize with the files in `tokenizer/`; `vlm_meta.json` lists the special tokens and sizes
 
 ## Files
 
@@ -59,30 +60,30 @@ Not for high-stakes decisions about people (for example identity, employment or
 |---|---|---|
 | `embed_tokens.i8.bin` | 45.1 MB | `d3ea47247be0b070…` |
 | `embed_tokens.scale.f32.bin` | 192.5 KB | `adefdadaae81d572…` |
+| `smolvlm_decoder_q8.pte` | 349.9 MB | `a54d81fbafd289c3…` |
+| `smolvlm_vision_q8.pte` | 99.0 MB | `a48d6919aa2067be…` |
 | `tokenizer/special_tokens_map.json` | 1.0 KB | `aa0ff906077086df…` |
 | `tokenizer/tokenizer.json` | 3.4 MB | `5ece781dc8d2b2f3…` |
 | `tokenizer/tokenizer_config.json` | 27.6 KB | `36c6fd44d07d10fd…` |
-| `vlm_decoder.pte` | 370.7 MB | `d92697a8c32eeaa1…` |
 | `vlm_meta.json` | 201 B | `b36fcf7ef7da8c2a…` |
-| `vlm_prefill.pte` | 370.7 MB | `1e8671eb77ae2f60…` |
-| `vlm_vision_q8.pte` | 103.8 MB | `5f09d6028d9bb945…` |
 
 Full checksums are in `manifest.json`. The playground verifies them after download.
 
 ## Running it
 
 These programs were exported with the ExecuTorch WebGPU backend
-([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu)) and validated with the
-WebAssembly runtime build used by the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)). That runtime build is not published yet; until it is,
-the playground is the supported way to run them. It downloads these files once, caches them in the browser,
+([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu)) and run in a browser (headless Chromium, WebGPU on SwiftShader) with the WebAssembly runtime build used by the [ExecuTorch WebGPU Playground](https://executorch.ai/playground/), where every generated token (78 of 78 over 4 turns) matched the same programs run on CPU. That runtime build is published in
+[executorch-community/ExecuTorch-WebGPU-Runtime](https://huggingface.co/executorch-community/ExecuTorch-WebGPU-Runtime); the playground is the easiest way to run them. It downloads these files once, caches them in the browser,
 and runs everything locally.
 
 Requirements: a browser with WebGPU enabled.
 
 ## Provenance
 
-- Converted from the upstream checkpoint by the ExecuTorch team with the WebGPU export path.
-- A reproducible export recipe will be published alongside the open-source ExecuTorch WebGPU runtime.
+- **Upstream:** `HuggingFaceTB/SmolVLM-500M-Instruct` at revision `a7da5b986cb59b408707209984f360a5f4ad7e47` (official)
+- **Exported with:** executorch 1.5.1+cpu, torch 2.14.1+cpu, torchao 0.18.0+cpu, transformers 5.19.0 (executorch git `3b6068392324`)
+- **Reproduce:** `python recipes/smolvlm-500m-instruct-q8.py --out out/smolvlm-500m-instruct-q8` (recipe in `export/`)
+- Re-running the recipe produces byte-identical files.
 
 ## Citation
 

export/_common.py

@@ -0,0 +1,330 @@
+"""Shared helpers for the ExecuTorch WebGPU export recipes in this directory.
+
+Each recipe builds an eager model from a pinned upstream revision and calls
+`export_vision` (or its own LLM path), which writes:
+
+  <out>/<name>_webgpu.pte    program lowered with WebGPUPartitioner (delegate id "VulkanBackend")
+  <out>/<name>_portable.pte  same exported graph without a delegate, for CPU validation only
+  <out>/recipe.json          upstream id + revision, versions, shapes, dtype, command line
+  <out>/validation.json      eager vs exported-program vs portable .pte (and goldens, if given)
+"""
+
+import hashlib
+import io
+import json
+import platform
+import shlex
+import sys
+import urllib.request
+import zipfile
+from pathlib import Path
+
+import numpy as np
+import torch
+
+PROXY_HINT = (
+    "set https_proxy if the machine needs a proxy for huggingface.co / pypi.org"
+)
+
+
+def versions():
+    import executorch
+    import torchao
+    from executorch.version import git_version
+
+    out = {
+        "python": platform.python_version(),
+        "torch": torch.__version__,
+        "torchao": torchao.__version__,
+        "executorch": getattr(executorch, "__version__", None)
+        or __import__("executorch.version").version.__version__,
+        "executorch_git": git_version,
+    }
+    try:
+        import transformers
+
+        out["transformers"] = transformers.__version__
+    except ImportError:
+        pass
+    try:
+        import huggingface_hub
+
+        out["huggingface_hub"] = huggingface_hub.__version__
+    except ImportError:
+        pass
+    return out
+
+
+def sha256_file(path):
+    h = hashlib.sha256()
+    with open(path, "rb") as fh:
+        for chunk in iter(lambda: fh.read(1 << 20), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+def hf_file(repo_id, filename, revision, expected_sha256=None, cache_dir=None):
+    from huggingface_hub import hf_hub_download
+
+    path = hf_hub_download(
+        repo_id=repo_id, filename=filename, revision=revision, cache_dir=cache_dir
+    )
+    digest = sha256_file(path)
+    if expected_sha256 and digest != expected_sha256:
+        sys.exit(
+            f"{repo_id}@{revision}:{filename} sha256 {digest} != {expected_sha256}"
+        )
+    return Path(path), digest
+
+
+def pypi_wheel_member(url, expected_sha256, member):
+    """Read one file out of a PyPI wheel, verifying the wheel's sha256."""
+    with urllib.request.urlopen(url, timeout=120) as resp:
+        blob = resp.read()
+    digest = hashlib.sha256(blob).hexdigest()
+    if digest != expected_sha256:
+        sys.exit(f"{url}: sha256 {digest} != {expected_sha256} ({PROXY_HINT})")
+    with zipfile.ZipFile(io.BytesIO(blob)) as zf:
+        return zf.read(member).decode()
+
+
+def scrub(text, out_dir):
+    """Replace machine-specific paths so recipe.json can ship with the model."""
+    text = text.replace(str(Path(out_dir).resolve()), "<out>")
+    text = text.replace(str(Path(__file__).resolve().parent), "recipes")
+    return text.replace(str(Path.home()), "~")
+
+
+def command_line(out_dir):
+    argv = [f"recipes/{Path(sys.argv[0]).name}", *sys.argv[1:]]
+    args = []
+    for a in argv:
+        if a.startswith("/") or a.startswith("."):
+            a = scrub(str(Path(a).resolve()), out_dir)
+        args.append(a)
+    return scrub(" ".join(["python", *(shlex.quote(a) for a in args)]), out_dir)
+
+
+def lower_webgpu(ep):
+    from executorch.backends.webgpu.partitioner import WebGPUPartitioner
+    from executorch.exir import to_edge_transform_and_lower
+
+    return to_edge_transform_and_lower(
+        ep, partitioner=[WebGPUPartitioner()]
+    ).to_executorch()
+
+
+def lower_portable(ep):
+    from executorch.exir import to_edge_transform_and_lower
+
+    return to_edge_transform_and_lower(ep).to_executorch()
+
+
+def delegate_summary(pte_bytes):
+    """Non-delegated kernels and per-delegate op histograms of a serialized program."""
+    import collections
+
+    from executorch.backends.vulkan.serialization.vulkan_graph_serialize import (
+        extract_vk_flatbuffer,
+        flatbuffer_to_vk_graph,
+    )
+    from executorch.exir._serialize._program import deserialize_pte_binary
+
+    pte = deserialize_pte_binary(pte_bytes)
+    prog = pte.program if hasattr(pte, "program") else pte
+    seg_base = int.from_bytes(pte_bytes[24:32], "little")
+    plan = prog.execution_plan[0]
+    ops = [f"{o.name}.{o.overload}" for o in plan.operators]
+    kernels = collections.Counter()
+    for chain in plan.chains:
+        for ins in chain.instructions:
+            if type(ins.instr_args).__name__ == "KernelCall":
+                kernels[ops[ins.instr_args.op_index]] += 1
+    delegates = []
+    for d in plan.delegates:
+        proc = d.processed
+        if proc.location.name == "INLINE":
+            blob = prog.backend_delegate_data[proc.index].data
+        else:
+            seg = prog.segments[proc.index]
+            blob = pte_bytes[seg_base + seg.offset : seg_base + seg.offset + seg.size]
+        g = flatbuffer_to_vk_graph(extract_vk_flatbuffer(blob))
+        delegates.append(
+            {
+                "id": d.id,
+                "ops": dict(collections.Counter(c.name for c in g.chain)),
+                "inputs": len(g.input_ids),
+                "outputs": len(g.output_ids),
+            }
+        )
+    return {"non_delegated_kernels": dict(kernels), "delegates": delegates}
+
+
+def compare(a, b):
+    a = np.asarray(a, dtype=np.float64).ravel()
+    b = np.asarray(b, dtype=np.float64).ravel()
+    n = min(a.size, b.size)
+    a, b = a[:n], b[:n]
+    cos = float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-30))
+    return {
+        "max_abs": float(np.max(np.abs(a - b))),
+        "mean_abs": float(np.mean(np.abs(a - b))),
+        "cosine": cos,
+    }
+
+
+def run_pte(pte_bytes, inputs):
+    from executorch.extension.pybindings.portable_lib import (
+        _load_for_executorch_from_buffer,
+    )
+
+    m = _load_for_executorch_from_buffer(pte_bytes)
+    return [
+        o.detach().numpy() if isinstance(o, torch.Tensor) else o
+        for o in m.forward(inputs)
+    ]
+
+
+def try_run_webgpu_pte_on_cpu(pte_bytes, inputs):
+    try:
+        return run_pte(pte_bytes, inputs), None
+    except Exception as e:  # noqa: BLE001 - the error text is the result
+        return None, f"{type(e).__name__}: {str(e).splitlines()[0][:300]}"
+
+
+def export_vision(
+    *,
+    name,
+    model,
+    example_input,
+    out_dir,
+    upstream,
+    extra_recipe,
+    golden=None,
+    output_index=None,
+):
+    """Export, lower, write both .pte files, recipe.json, and validation.json.
+
+    `golden` is an optional (input ndarray, output ndarray) pair; when its input
+    shape matches `example_input`, all paths are also compared against it.
+    `output_index` picks one tensor when the eager model returns a tuple.
+    """
+    out_dir = Path(out_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+    torch.manual_seed(0)
+    model = model.eval()
+
+    class Wrapped(torch.nn.Module):
+        def __init__(self, m):
+            super().__init__()
+            self.m = m
+
+        def forward(self, x):
+            y = self.m(x)
+            return y[output_index] if output_index is not None else y
+
+    wrapped = Wrapped(model).eval()
+    with torch.no_grad():
+        ep = torch.export.export(wrapped, (example_input,), strict=False)
+        webgpu = lower_webgpu(ep)
+        portable = lower_portable(ep)
+
+    webgpu_path = out_dir / f"{name}_webgpu.pte"
+    portable_path = out_dir / f"{name}_portable.pte"
+    webgpu_path.write_bytes(webgpu.buffer)
+    portable_path.write_bytes(portable.buffer)
+
+    summary = delegate_summary(webgpu.buffer)
+    out_shape = list(ep.module()(example_input).shape)
+    recipe = {
+        "model": name,
+        "upstream": upstream,
+        "versions": versions(),
+        "export": {
+            "api": "torch.export.export(strict=False) -> to_edge_transform_and_lower(partitioner=[WebGPUPartitioner()]) -> to_executorch()",
+            "partitioner": "executorch.backends.webgpu.partitioner.WebGPUPartitioner() (default compile options; wraps VulkanPartitioner)",
+            "inputs": [
+                {
+                    "shape": list(example_input.shape),
+                    "dtype": "float32",
+                    "layout": "NCHW",
+                }
+            ],
+            "outputs": [{"shape": out_shape, "dtype": "float32"}],
+            "dtype": "float32",
+            "quantization": "none (fp32)",
+            **extra_recipe,
+        },
+        "files": {
+            webgpu_path.name: {
+                "bytes": webgpu_path.stat().st_size,
+                "sha256": sha256_file(webgpu_path),
+            },
+            portable_path.name: {
+                "bytes": portable_path.stat().st_size,
+                "sha256": sha256_file(portable_path),
+                "note": "CPU-only validation artifact (portable kernels, no delegate); not for release",
+            },
+        },
+        "graph": summary,
+        "command": command_line(out_dir),
+    }
+    (out_dir / "recipe.json").write_text(json.dumps(recipe, indent=2) + "\n")
+
+    # Validation on CPU.
+    gen = torch.Generator().manual_seed(1234)
+    rand_input = torch.rand(example_input.shape, generator=gen)
+    cases = {"random": rand_input}
+    if golden is not None and tuple(golden[0].shape) == tuple(example_input.shape):
+        cases["golden"] = torch.from_numpy(golden[0].copy())
+    report = {
+        "cpu_runtime": "executorch pybindings (portable_lib) from the pip wheel",
+        "cases": {},
+    }
+    webgpu_cpu_error = None
+    for label, x in cases.items():
+        with torch.no_grad():
+            eager = wrapped(x).numpy()
+            exported = ep.module()(x).numpy()
+        port = run_pte(portable.buffer, (x,))[0]
+        res = {
+            "exported_vs_eager": compare(exported, eager),
+            "portable_pte_vs_eager": compare(port, eager),
+        }
+        wg, err = try_run_webgpu_pte_on_cpu(webgpu.buffer, (x,))
+        if wg is not None:
+            res["webgpu_pte_vs_eager"] = compare(wg[0], eager)
+        else:
+            webgpu_cpu_error = err
+        if label == "golden":
+            ref = golden[1]
+            res["internal_golden_vs_eager"] = compare(eager, ref)
+            res["internal_golden_vs_portable_pte"] = compare(port, ref)
+            res["golden_note"] = (
+                "reference output from the internal goldens.manifest.json"
+            )
+        report["cases"][label] = res
+    report["webgpu_pte_on_cpu"] = (
+        "ran"
+        if webgpu_cpu_error is None
+        else f"not runnable on CPU pybindings: {webgpu_cpu_error}"
+    )
+    (out_dir / "validation.json").write_text(json.dumps(report, indent=2) + "\n")
+    print(
+        json.dumps(
+            {"recipe": recipe["files"], "graph": summary, "validation": report},
+            indent=2,
+        )
+    )
+    return ep, webgpu, portable
+
+
+def load_golden(manifest_path, key):
+    man = json.loads(Path(manifest_path).read_text())[key]["tensors"]
+    base = Path(manifest_path).parent
+
+    def ld(t):
+        return np.fromfile(base / t["file"], dtype=np.float32).reshape(t["shape"])
+
+    inp = man.get("input")
+    return ld(inp), ld(man["output"])

export/_smolvlm_q8.py

@@ -0,0 +1,545 @@
+"""Building blocks for the 8-bit SmolVLM WebGPU recipe (smolvlm-500m-instruct-q8.py).
+
+The one 8-bit linear that public ExecuTorch lowers for WebGPU and the public WebGPU runtime runs is
+et_vk.linear_q8ta_q8csw: int8 activations with one static per-tensor scale, times int8 weights with
+one scale per output channel. It is emitted for linear(dq(q(x, s, z)), dq_per_channel(W8, ws, 0)),
+which Q8Linear below writes out directly with quantized_decomposed ops (no PT2E prepare/convert:
+the scales come from torchao MinMaxObservers run over calibration data on the fp32 model).
+
+Two things make it accurate enough for this model (see VLM_FOLLOWUP.md for the measurements):
+
+- Two passes. y = W8.dq(q(x, s1)) + W8.dq(q(r, s2)), r = x - dq(q(x, s1)): the second pass quantizes
+  the first pass's rounding error with its own static scale (s2 ~ s1/255) and reuses the same int8
+  weight, so the activation error drops from s1/2 to about s1/510. The residual is written with
+  plain ops (round/clamp/mul): the lowering would fold an explicit dq(q(x)) back to x for a
+  non-quantized consumer, which makes the residual zero.
+- A built-in attention sink. The text decoder's position 0 is always <|im_start|>, whose hidden
+  state carries massive activations (|x| ~ 3000 at the input of layer 3's down projection, vs ~4 at
+  every other position). Its K/V are the same in every conversation, so they are computed once in
+  fp32 and written into the cache by the program itself whenever a call starts at position 0
+  (SinkKVCache); the calibration excludes that row, so it never sets a scale.
+"""
+
+import math
+
+import torch
+import torch.nn.functional as F
+
+qd = torch.ops.quantized_decomposed
+QMIN, QMAX = -128, 127
+
+
+# ----------------------------------------------------------------------------- quantization
+
+
+def weight_int8(w):
+    """VulkanQuantizer's weight spec: int8 [-128, 127], per-output-channel symmetric, scale from
+    torchao PerChannelMinMaxObserver (amax / 127.5). Returns (int8 [N, K], fp32 scales [N])."""
+    from torchao.quantization.pt2e import PerChannelMinMaxObserver
+
+    obs = PerChannelMinMaxObserver(
+        ch_axis=0,
+        dtype=torch.int8,
+        quant_min=QMIN,
+        quant_max=QMAX,
+        qscheme=torch.per_channel_symmetric,
+    )
+    obs(w)
+    scale, zp = obs.calculate_qparams()
+    assert int(zp.abs().max()) == 0
+    scale = scale.float()
+    q = torch.clamp(torch.round(w / scale[:, None]), QMIN, QMAX).to(torch.int8)
+    return q, scale
+
+
+def act_observer():
+    """Static per-tensor affine int8 activation observer (min/max of everything it saw)."""
+    from torchao.quantization.pt2e import MinMaxObserver
+
+    return MinMaxObserver(
+        dtype=torch.int8,
+        quant_min=QMIN,
+        quant_max=QMAX,
+        qscheme=torch.per_tensor_affine,
+    )
+
+
+def qparams(obs):
+    s, z = obs.calculate_qparams()
+    return float(s), int(z)
+
+
+def fake_quant(x, s, z):
+    """dq(q(x)) exactly as quantized_decomposed computes it (round(x * (1/s)) + z, clamp)."""
+    return (torch.clamp(torch.round(x * (1.0 / s)) + z, QMIN, QMAX) - z) * s
+
+
+class Q8Linear(torch.nn.Module):
+    """et_vk.linear_q8ta_q8csw, one or two passes (see the module docstring)."""
+
+    def __init__(self, lin, act, second=None):
+        super().__init__()
+        q, scale = weight_int8(lin.weight.detach().float())
+        self.register_buffer("weight_int8", q)
+        self.register_buffer("weight_scale", scale)
+        self.register_buffer("weight_zero", torch.zeros(q.shape[0], dtype=torch.int64))
+        self.bias = (
+            None
+            if lin.bias is None
+            else torch.nn.Parameter(lin.bias.detach().float(), requires_grad=False)
+        )
+        self.act = (float(act[0]), int(act[1]))
+        self.second = None if second is None else (float(second[0]), int(second[1]))
+
+    def forward(self, x):
+        s1, z1 = self.act
+        w = qd.dequantize_per_channel(
+            self.weight_int8,
+            self.weight_scale,
+            self.weight_zero,
+            0,
+            QMIN,
+            QMAX,
+            torch.int8,
+        )
+        x1 = qd.dequantize_per_tensor(
+            qd.quantize_per_tensor(x, s1, z1, QMIN, QMAX, torch.int8),
+            s1,
+            z1,
+            QMIN,
+            QMAX,
+            torch.int8,
+        )
+        y = F.linear(x1, w, self.bias)
+        if self.second is not None:
+            s2, z2 = self.second
+            r = x - fake_quant(x, s1, z1)
+            r1 = qd.dequantize_per_tensor(
+                qd.quantize_per_tensor(r, s2, z2, QMIN, QMAX, torch.int8),
+                s2,
+                z2,
+                QMIN,
+                QMAX,
+                torch.int8,
+            )
+            y = y + F.linear(r1, w)
+        return y
+
+
+def linear_names(module):
+    return [n for n, m in module.named_modules() if isinstance(m, torch.nn.Linear)]
+
+
+def set_module(root, name, new):
+    parent, _, leaf = name.rpartition(".")
+    setattr(root.get_submodule(parent) if parent else root, leaf, new)
+
+
+class Recorder:
+    """Forward pre-hooks feeding one observer per linear. `skip_first_row` (set by the feeder while
+    a call starts at position 0) drops row 0, the attention-sink position, from what is observed."""
+
+    def __init__(self, module, names, transform=None):
+        self.obs = {n: act_observer() for n in names}
+        self.skip_first_row = False
+        self.hooks = []
+        for n in names:
+
+            def hook(m, args, n=n):
+                x = args[0].detach().float()
+                if self.skip_first_row and x.dim() == 3:
+                    x = x[:, 1:]
+                x = x.reshape(-1, x.shape[-1])
+                if x.numel():
+                    self.obs[n](transform(n, x) if transform else x)
+
+            self.hooks.append(module.get_submodule(n).register_forward_pre_hook(hook))
+
+    def close(self):
+        for h in self.hooks:
+            h.remove()
+        return {n: qparams(o) for n, o in self.obs.items()}
+
+
+def calibrate(module, names, feed, two_pass):
+    """Static activation qparams for every linear in `names`: {name: (first, second or None)}.
+    feed(recorder) runs the fp32 module over the calibration data (twice for two passes: the
+    second run observes each linear's first-pass rounding error)."""
+    rec = Recorder(module, names)
+    with torch.no_grad():
+        feed(rec)
+    first = rec.close()
+    second = {}
+    if two_pass:
+        rec = Recorder(
+            module, names, transform=lambda n, x: x - fake_quant(x, *first[n])
+        )
+        with torch.no_grad():
+            feed(rec)
+        second = rec.close()
+    return {n: (first[n], second.get(n)) for n in names}
+
+
+def quantize(module, act):
+    """Replace every linear named in `act` by Q8Linear (in place)."""
+    for n, (first, second) in act.items():
+        set_module(module, n, Q8Linear(module.get_submodule(n), first, second))
+    return module
+
+
+_FAKE_FIXED = []
+
+
+def fix_q8ta_fake():
+    """executorch 1.5.1's reference kernel for et_vk.linear_q8ta_q8csw computes linear(x_int8, w_fp32),
+    so under FakeTensorMode (the Vulkan FusePatternsPass re-trace) its output comes out int8. Every
+    consumer then inherits int8 metadata: llama::custom_sdpa's meta check fails on the query, and
+    the serialized delegate would declare fp32 activations as int8 (the WebGPU kernel rejects a
+    non-fp32 output). A fake kernel with the real output dtype (fp32, [..., N]) fixes the metadata;
+    the serialized ops and constants are otherwise unchanged."""
+    if _FAKE_FIXED:
+        return
+    import executorch.backends.vulkan.custom_ops_lib  # noqa: F401 (defines et_vk ops)
+
+    @torch.library.register_fake("et_vk::linear_q8ta_q8csw")
+    def _fake(x, input_scale, input_zero_point, weights, weight_sums, weight_scales, bias=None):
+        return x.new_empty((*x.shape[:-1], weights.shape[0]), dtype=torch.float32)
+
+    _FAKE_FIXED.append(True)
+
+
+# ----------------------------------------------------------------------------- attention sink
+
+
+class SinkKVCache(torch.nn.Module):
+    """executorch's CustomKVCache (llama::update_cache into a [1, C, H, D] cache) with the K/V of
+    position 0 built in: when a call starts at position 0, its row 0 is replaced by the constants
+    before the cache update. sink = relu(1 - input_pos) is 1 at position 0 and 0 everywhere else,
+    so later calls are untouched (x + 0 * (c - x) = x)."""
+
+    def __init__(self, base, k0, v0, rows):
+        super().__init__()
+        self.max_context_length = base.max_context_length
+        self.max_batch_size, self.n_heads, self.head_dim = (
+            base.max_batch_size,
+            base.n_heads,
+            base.head_dim,
+        )
+        self.register_buffer("k_cache", base.k_cache)
+        self.register_buffer("v_cache", base.v_cache)
+        # [B, S, H, D], the cache layout (update() receives [B, H, S, D] and transposes back first,
+        # so the arithmetic below sees one dim order: the portable kernels require that).
+        self.register_buffer("k0", k0.detach().float().contiguous())
+        self.register_buffer("v0", v0.detach().float().contiguous())
+        first_row = torch.zeros(1, rows, 1, 1)
+        first_row[0, 0, 0, 0] = 1.0
+        self.register_buffer("first_row", first_row)
+
+    def update(self, input_pos, k_val, v_val, indices=None):
+        assert indices is None
+        k_val = k_val.transpose(1, 2)
+        v_val = v_val.transpose(1, 2)
+        sink = torch.relu(1.0 - input_pos[:1].to(torch.float32)).reshape(1, 1, 1, 1) * self.first_row
+        k_val = k_val + sink * (self.k0 - k_val)
+        v_val = v_val + sink * (self.v0 - v_val)
+        start_pos = input_pos[0].item()
+        _ = torch.ops.llama.update_cache(k_val, self.k_cache, start_pos)
+        _ = torch.ops.llama.update_cache(v_val, self.v_cache, start_pos)
+        return self.k_cache.transpose(1, 2), self.v_cache.transpose(1, 2)
+
+
+def sink_kv(transformer, row0):
+    """Per-layer K/V of position 0 for the fp32 transformer, given its input row [hidden]."""
+    for name, buf in transformer.named_buffers():
+        if name.endswith("k_cache") or name.endswith("v_cache"):
+            buf.zero_()
+    with torch.no_grad():
+        transformer(None, {"input_pos": torch.tensor([0])}, row0[None, None])
+    out = []
+    for layer in transformer.layers:
+        kv = layer.attention.kv_cache
+        out.append((kv.k_cache[:, :1].clone(), kv.v_cache[:, :1].clone()))
+        kv.k_cache.zero_()
+        kv.v_cache.zero_()
+    return out
+
+
+def install_sink(transformer, kvs, rows):
+    for layer, (k0, v0) in zip(transformer.layers, kvs):
+        layer.attention.kv_cache = SinkKVCache(layer.attention.kv_cache, k0, v0, rows)
+    return transformer
+
+
+class RefSDPA(torch.nn.Module):
+    """Causal attention over the KV cache in plain PyTorch, used only while collecting calibration
+    statistics. llama::custom_sdpa's CPU kernel divides its work by the threads it gets, so its float
+    sums (and with them a few activation ranges, in the last bits) vary from run to run; this keeps
+    the calibration, and the exported files, byte-reproducible. Same inputs/outputs as SDPACustom."""
+
+    def __init__(self, dim, n_rep):
+        super().__init__()
+        self.dim, self.n_rep = dim, n_rep
+
+    def forward(self, input_pos, q, k, v, bsz, seqlen, mask):
+        # q [B, Hq, S, D]; k, v [B, Hkv, L, D] (the whole cache)
+        end = int(input_pos[0]) + seqlen
+        k = k[:, :, :end].repeat_interleave(self.n_rep, dim=1)
+        v = v[:, :, :end].repeat_interleave(self.n_rep, dim=1)
+        scores = (q @ k.transpose(-1, -2)) / math.sqrt(q.shape[-1])
+        rows = torch.arange(end - seqlen, end)[:, None]
+        scores = scores.masked_fill(torch.arange(end)[None, :] > rows, float("-inf"))
+        out = torch.softmax(scores, dim=-1) @ v
+        return out.transpose(1, 2).reshape(bsz, seqlen, self.dim)
+
+
+class reference_attention:
+    """Context manager: every layer's SDPA module replaced by RefSDPA, restored on exit."""
+
+    def __init__(self, transformer):
+        self.t = transformer
+
+    def __enter__(self):
+        self.saved = []
+        for layer in self.t.layers:
+            a = layer.attention
+            self.saved.append(a.SDPA)
+            a.SDPA = RefSDPA(a.SDPA.dim, a.n_heads // a.n_kv_heads)
+        return self.t
+
+    def __exit__(self, *exc):
+        for layer, sdpa in zip(self.t.layers, self.saved):
+            layer.attention.SDPA = sdpa
+
+
+# ----------------------------------------------------------------------------- windows
+
+
+def window_starts(start, end, rows):
+    """First positions of the `rows`-row windows that feed positions [start, end): each window ends
+    at most at `end`, the last exactly there (the same plan as windowStarts() in vlm.js)."""
+    if start >= end:
+        return []
+    assert end >= rows, (start, end, rows)
+    out, stop = [], min(start + rows, end)
+    while True:
+        out.append(stop - rows)
+        if stop == end:
+            return out
+        stop = min(stop + rows, end)
+
+
+# ----------------------------------------------------------------------------- inspection
+
+
+def q8_ops_vs_eager(pte_bytes, module):
+    """Every et_vk.linear_q8ta_q8csw in the delegates: its int8 weight and weight scales must be
+    bit-identical to one Q8Linear's buffers, and its activation scale / zero point one of that
+    linear's static qparams (scale to 1e-7 relative: the flatc JSON dump prints about 8 significant
+    digits). Returns counts."""
+    import hashlib
+
+    import _smolvlm as S
+    import numpy as np
+    from executorch.backends.vulkan.serialization.vulkan_graph_serialize import (
+        VulkanDelegateHeader,
+    )
+    from executorch.exir._serialize._program import deserialize_pte_binary
+
+    eager = {}
+    for m in module.modules():
+        if isinstance(m, Q8Linear):
+            key = hashlib.sha256(m.weight_int8.numpy().tobytes()).hexdigest()
+            eager[key] = (
+                m.weight_scale.numpy(),
+                {m.act} | ({m.second} if m.second else set()),
+            )
+    pte = deserialize_pte_binary(pte_bytes)
+    prog = pte.program if hasattr(pte, "program") else pte
+    nd = getattr(pte, "named_data", None)
+    seg_base = int.from_bytes(pte_bytes[24:32], "little")
+    total = weights_ok = scales_ok = act_ok = 0
+    for plan in prog.execution_plan:
+        for d in plan.delegates:
+            proc = d.processed
+            if proc.location.name == "INLINE":
+                blob = bytes(prog.backend_delegate_data[proc.index].data)
+            else:
+                seg = prog.segments[proc.index]
+                blob = pte_bytes[
+                    seg_base + seg.offset : seg_base + seg.offset + seg.size
+                ]
+            h = VulkanDelegateHeader.from_bytes(
+                blob[: VulkanDelegateHeader.EXPECTED_LENGTH]
+            )
+            raw = blob[h.bytes_offset : h.bytes_offset + h.bytes_size]
+            g = S._vk_graph(blob)
+
+            def const(vid):
+                t = g.values[vid].value
+                c = g.constants[t.constant_id]
+                if c.named_key:
+                    return bytes(
+                        nd.buffers[nd.pte_data[c.named_key].buffer_index]
+                    ), list(t.dims)
+                return raw[c.offset : c.offset + c.length], list(t.dims)
+
+            for op in g.chain:
+                if op.name != "et_vk.linear_q8ta_q8csw.default":
+                    continue
+                total += 1
+                wb, (n, k) = const(op.args[3])
+                hit = eager.get(hashlib.sha256(wb[: n * k]).hexdigest())
+                if hit is None:
+                    continue
+                weights_ok += 1
+                sb, _ = const(op.args[5])
+                scales = np.frombuffer(sb, dtype=np.float32)[:n]
+                scales_ok += int(np.array_equal(scales, hit[0]))
+                s = g.values[op.args[1]].value.double_val
+                z = g.values[op.args[2]].value.int_val
+                act_ok += int(
+                    any(abs(s - a) <= 1e-7 * a and z == b for a, b in hit[1])
+                )
+    return {
+        "linear_q8ta_q8csw_ops": total,
+        "weight_int8_bit_identical": weights_ok,
+        "weight_scales_bit_identical": scales_ok,
+        "activation_qparams_match": act_ok,
+        "eager_q8_linears": len(eager),
+    }
+
+
+# ----------------------------------------------------------------------------- GPTQ (optional)
+
+
+def gptq_int8(w, H, damp=0.01, block=128):
+    """GPTQ (Frantar et al. 2022) for the same int8 grid as weight_int8 (per-output-channel symmetric,
+    scale = amax / 127.5 of the original weight): columns are rounded in order and each rounding
+    error is pushed onto the not-yet-rounded columns through the inverse Hessian H = sum x x^T of the
+    layer's calibration inputs. Returns (int8 [N, K], fp32 scales [N]); float64 throughout."""
+    _, scale = weight_int8(w)
+    s = scale.double()[:, None]
+    w = w.detach().double().clone()
+    H = H.detach().double().clone()
+    K = w.shape[1]
+    dead = torch.diag(H) == 0
+    H[dead, dead] = 1
+    w[:, dead] = 0
+    H += damp * torch.mean(torch.diag(H)) * torch.eye(K, dtype=H.dtype)
+    hinv = torch.linalg.cholesky(torch.cholesky_inverse(torch.linalg.cholesky(H)), upper=True)
+    q = torch.zeros_like(w)
+    for i1 in range(0, K, block):
+        i2 = min(i1 + block, K)
+        w1 = w[:, i1:i2].clone()
+        err1 = torch.zeros_like(w1)
+        h1 = hinv[i1:i2, i1:i2]
+        for i in range(i2 - i1):
+            col = w1[:, i : i + 1]
+            qc = torch.clamp(torch.round(col / s), QMIN, QMAX)
+            q[:, i1 + i : i1 + i + 1] = qc
+            e = (col - qc * s) / h1[i, i]
+            w1[:, i:] -= e * h1[i : i + 1, i:]
+            err1[:, i : i + 1] = e
+        w[:, i2:] -= err1 @ hinv[i1:i2, i2:]
+    return q.to(torch.int8), scale
+
+
+def hessians(module, names, feed):
+    """sum x x^T (float64) of every named linear's input over feed(recorder-like ctx)."""
+    hs, hooks = {}, []
+
+    class Ctx:
+        skip_first_row = False
+
+    ctx = Ctx()
+    for n in names:
+
+        def hook(m, args, n=n):
+            x = args[0].detach().double()
+            if ctx.skip_first_row and x.dim() == 3:
+                x = x[:, 1:]
+            x = x.reshape(-1, x.shape[-1])
+            hs[n] = hs[n] + x.T @ x if n in hs else x.T @ x
+            rows[n] = rows.get(n, 0) + x.shape[0]
+
+        hooks.append(module.get_submodule(n).register_forward_pre_hook(hook))
+    rows = {}
+    with torch.no_grad():
+        feed(ctx)
+    for h in hooks:
+        h.remove()
+    hessians.rows = rows
+    return hs
+
+
+# ----------------------------------------------------------------------------- model builders
+
+
+def quantize_with(module, act, weights=None):
+    """Like quantize(), with optional precomputed (int8, scale) weights per linear (GPTQ)."""
+    for n, (first, second) in act.items():
+        lin = module.get_submodule(n)
+        q = Q8Linear(lin, first, second)
+        if weights and n in weights:
+            q.weight_int8.copy_(weights[n][0])
+            q.weight_scale.copy_(weights[n][1])
+        set_module(module, n, q)
+    return module
+
+
+def build_vision(vision_fp32, images, *, two_pass, gptq):
+    """Quantize a (deep-copied) fp32 VisionEncoder: every linear becomes Q8Linear, scales from the
+    calibration images (pixel tensors [1,3,S,S])."""
+    import copy
+
+    v = copy.deepcopy(vision_fp32).eval()
+    names = linear_names(v)
+
+    def feed(rec):
+        for px in images:
+            v(px)
+
+    act = calibrate(v, names, feed, two_pass)
+    weights = None
+    if gptq:
+        # GPTQ only where the calibration rows outnumber the input features: the connector
+        # (12288 inputs, 64 rows per image) keeps round-to-nearest.
+        hs = hessians(v, names, feed)
+        weights = {n: gptq_int8(v.get_submodule(n).weight, hs[n]) for n in names if hessians.rows[n] >= hs[n].shape[0]}
+        del hs
+    return quantize_with(v, act, weights), act
+
+
+def build_text(transformer, calib_rows, *, rows, two_pass, gptq, sink_row=None):
+    """Quantize the fp32 executorch llama Transformer in place.
+
+    calib_rows: list of [n, hidden] embedding sequences (prompt + continuation), fed in the same
+    `rows`-row windows the browser uses; row 0 (the sink) is excluded from every statistic.
+    sink_row: the input row of position 0 (the <|im_start|> embedding as the host feeds it), or None
+    for no built-in sink. Returns (transformer, activation qparams, sink K/V)."""
+    import _smolvlm as S
+
+    names = linear_names(transformer)
+    sink = sink_row is not None
+    with reference_attention(transformer):
+        kvs = sink_kv(transformer, sink_row) if sink else None
+
+    def feed(rec):
+        for seq in calib_rows:
+            S.reset_kv(transformer)
+            for start in window_starts(0, seq.shape[0], rows):
+                rec.skip_first_row = sink and start == 0
+                transformer(None, {"input_pos": torch.tensor([start])}, seq[None, start : start + rows])
+            rec.skip_first_row = False
+        S.reset_kv(transformer)
+
+    with reference_attention(transformer):
+        act = calibrate(transformer, names, feed, two_pass)
+        hs = hessians(transformer, names, feed) if gptq else None
+    weights = None
+    if gptq:
+        weights = {n: gptq_int8(transformer.get_submodule(n).weight, hs.pop(n)) for n in names}
+    quantize_with(transformer, act, weights)
+    if sink:
+        install_sink(transformer, kvs, rows)
+    return transformer, act, kvs

export/recipe.json

@@ -0,0 +1,3510 @@
+{
+  "model": "smolvlm-500m-instruct-q8",
+  "variant": "q8",
+  "upstream": {
+    "model": "HuggingFaceTB/SmolVLM-500M-Instruct",
+    "revision": "a7da5b986cb59b408707209984f360a5f4ad7e47",
+    "official": true,
+    "sha256": {
+      "model.safetensors": "d05b567eeaf534e83d375551f068ed57b5f52d37c657197f644af5ef9db091a2",
+      "config.json": "daacbbca6af3c34e50466c72aed2df4553a08084f6a0551d4ae420a241cb66c6",
+      "generation_config.json": "067a2a54e5f87162ecac6e0e911cc4665fc8f7f3324794ecbac0f76badb56636",
+      "preprocessor_config.json": "6cb6e36d6fcb88ca1502c4a26750715dc3e7dedddc9a8f17b27d8d167d1457e7",
+      "tokenizer.json": "5ece781dc8d2b2f3e2f289ca0ae50b17cfc27dd27bfe7971bb8241e0b964331a",
+      "tokenizer_config.json": "36c6fd44d07d10fd8180ee6b46dcccf69fb7c06753968ff0d7e17b8bfe17b777",
+      "special_tokens_map.json": "aa0ff906077086dfa9734a7f97f68c825877a48f9468807be65504495cdeef09"
+    }
+  },
+  "versions": {
+    "python": "3.12.15+meta",
+    "torch": "2.14.1+cpu",
+    "torchao": "0.18.0+cpu",
+    "executorch": "1.5.1+cpu",
+    "executorch_git": "3b60683923245cf472b7323426920e15623ba361",
+    "transformers": "5.19.0",
+    "huggingface_hub": "2.2.0"
+  },
+  "export": {
+    "partitioner": "executorch.backends.webgpu.partitioner.WebGPUPartitioner, default options (static shapes)",
+    "linear": "et_vk.linear_q8ta_q8csw: int8 activations, one static per-tensor affine scale per pass (torchao MinMaxObserver over the calibration data, [-128, 127]); int8 weights, per-output-channel symmetric (VulkanQuantizer weight spec, scale amax/127.5)",
+    "export_workaround": "_smolvlm_q8.fix_q8ta_fake(): a fake kernel giving et_vk.linear_q8ta_q8csw its real fp32 output dtype during lowering (executorch 1.5.1's reference kernel reports int8, which breaks llama::custom_sdpa's meta check and would mis-declare tensors). Metadata only",
+    "programs": {
+      "smolvlm_vision_q8.pte": {
+        "api": "torch.export.export(strict=False) -> to_edge_transform_and_lower(partitioner=[WebGPUPartitioner()]) -> to_executorch()",
+        "module": "transformers Idefics3 vision tower (eager attention) + post_layernorm + connector; identity position ids for a full 512x512 image",
+        "inputs": [
+          {
+            "name": "pixel_values",
+            "shape": [
+              1,
+              3,
+              512,
+              512
+            ],
+            "dtype": "float32",
+            "range": "(x/255 - 0.5)/0.5"
+          }
+        ],
+        "outputs": [
+          {
+            "name": "image_embeds",
+            "shape": [
+              1,
+              64,
+              960
+            ],
+            "dtype": "float32"
+          }
+        ],
+        "quantization": "all 73 linears 8-bit, two passes (second pass on the first pass's rounding error, same int8 weight), weights GPTQ (damp 0.01, block 128, float64) on the calibration inputs, same grid (the connector, with fewer calibration rows than inputs, round to nearest); patch embedding conv, layer norms, position table fp32"
+      },
+      "smolvlm_decoder_q8.pte": {
+        "api": "LLMEdgeManager: export(strict) -> run_canonical_optimizations -> export_to_edge -> to_backend([WebGPUPartitioner()]) -> to_executorch; static shapes",
+        "module": "executorch examples/models/llama Transformer (apply_embedding=False, use_hf_rope, enable_dynamic_shape for the start position) with weights from executorch.examples.models.smolvlm.convert_weights; source transforms from export_llama_lib._get_source_transforms(use_sdpa_with_kv_cache, use_kv_cache)",
+        "params": {
+          "dim": 960,
+          "hidden_dim": 2560,
+          "n_heads": 15,
+          "n_kv_heads": 5,
+          "head_dim": 64,
+          "n_layers": 32,
+          "norm_eps": 1e-05,
+          "rope_theta": 100000.0,
+          "use_scaled_rope": false,
+          "use_hf_rope": true,
+          "attention_qkv_bias": false,
+          "vocab_size": 49280,
+          "apply_embedding": false
+        },
+        "inputs": [
+          {
+            "name": "embeds",
+            "shape": [
+              1,
+              4,
+              960
+            ],
+            "dtype": "float32",
+            "note": "rows of positions input_pos .. input_pos + rows - 1"
+          },
+          {
+            "name": "input_pos",
+            "shape": [
+              1
+            ],
+            "dtype": "int64",
+            "note": "position of the first row"
+          }
+        ],
+        "outputs": [
+          {
+            "name": "logits",
+            "shape": [
+              1,
+              49280
+            ],
+            "dtype": "float32",
+            "note": "last row only"
+          }
+        ],
+        "kv_cache": "in the program: 32 layers x k/v [1,512,5,64] fp32; every call writes its rows' positions",
+        "attention_sink": "position 0 must be <|im_start|> (id 1), as the chat template always starts: its per-layer K/V (fp32 model, int8-table embedding row) are constants the program writes over row 0 when input_pos is 0; that row is excluded from calibration",
+        "host_contract": "feed positions in windows of `rows` rows ending at the newest position (window_starts in _smolvlm_q8.py / windowStarts in vlm.js); input_pos + rows <= max_context",
+        "max_context_length": 512,
+        "quantization": "all 225 linears incl. lm_head 8-bit, two passes (second pass on the first pass's rounding error, same int8 weight), weights GPTQ (damp 0.01, block 128, float64) on the calibration inputs, same grid; RMSNorm weights fp32"
+      }
+    },
+    "calibration": {
+      "pairs": [
+        {
+          "image": "yolo_cat.jpg",
+          "image_sha256": "88b9eb3e6ec026c5ab96aab516f4a4a2773522a474abc52e934a52d86bba21e5",
+          "question": "What animal is this?",
+          "rows": 85,
+          "continuation": " This is a cat."
+        },
+        {
+          "image": "yolo_people.jpg",
+          "image_sha256": "16d73869e3267a7d4ed00de8e860833bd1657c1b252e94c0c348277adc7b6edb",
+          "question": "Describe this image",
+          "rows": 110,
+          "continuation": " The image depicts two men standing on a field, likely during a sports event or a public gathering. The men are dressed in formal attire, suggesting that they might"
+        },
+        {
+          "image": "yolo_street.jpg",
+          "image_sha256": "c02019c4979c191eb739ddd944445ef408dad5679acab6fd520ef9d434bfbc63",
+          "question": "What vehicles can you see?",
+          "rows": 88,
+          "continuation": " There are buses in the image."
+        },
+        {
+          "image": "yolo_table.jpg",
+          "image_sha256": "0ef0c3f7caa3ad4ec92dc8968f9ac0e81ede5b4805aae89648a4c7c4236768df",
+          "question": "What objects are on the table?",
+          "rows": 110,
+          "continuation": " There are cups, saucers, a cup with coffee, a bowl with pastries, and a plate with a napkin on the table."
+        },
+        {
+          "image": "sample_shapes.jpg",
+          "image_sha256": "f7911a973ef5a6dc962b92349f65a34f33b15e4c078052f8166018fe8338e074",
+          "question": "What shapes are in this picture?",
+          "rows": 114,
+          "continuation": " The image contains four distinct geometric shapes. Here is a detailed description of each shape:\n\n1. **Red Triangle**: Located on the left side of the image"
+        },
+        {
+          "image": "chihuahuas.jpg",
+          "image_sha256": "6f18287a5728ce32e3714b28a971f5c62d57e2cf0f689786611d438d5d7b7522",
+          "question": "How many dogs are there?",
+          "rows": 89,
+          "continuation": " There are two dogs in the image."
+        },
+        {
+          "image": "kitchen.jpg",
+          "image_sha256": "dbecca629f4ebe4c6d311c07f55de921bd77d32d6c85ab32e7a8033175556580",
+          "question": "Describe this image",
+          "rows": 110,
+          "continuation": " A kitchen with a large island in the center, which has a granite countertop and a sink with a double faucet. There are two large bowls on the island"
+        },
+        {
+          "image": "giraffes.jpg",
+          "image_sha256": "e10d0f49a1e3b0d01bb6186bf59ca25a2c310a8d7d8ca7781eec76313ccc4ea1",
+          "question": "What is happening here?",
+          "rows": 88,
+          "continuation": " There are giraffes walking in the field."
+        }
+      ],
+      "continuation_tokens": 32,
+      "decoder_windows": "4 rows, the browser's window plan",
+      "vision": "the 8 calibration images"
+    },
+    "activation_qparams": {
+      "vision": {
+        "layers.0.self_attn.k_proj": [
+          [
+            0.11057502031326294,
+            0
+          ],
+          [
+            0.00043362739961594343,
+            -1
+          ]
+        ],
+        "layers.0.self_attn.v_proj": [
+          [
+            0.11057502031326294,
+            0
+          ],
+          [
+            0.00043362739961594343,
+            -1
+          ]
+        ],
+        "layers.0.self_attn.q_proj": [
+          [
+            0.11057502031326294,
+            0
+          ],
+          [
+            0.00043362739961594343,
+            -1
+          ]
+        ],
+        "layers.0.self_attn.out_proj": [
+          [
+            0.03311925381422043,
+            9
+          ],
+          [
+            0.00012987936497665942,
+            -1
+          ]
+        ],
+        "layers.0.mlp.fc1": [
+          [
+            0.29603707790374756,
+            3
+          ],
+          [
+            0.0011609295615926385,
+            -1
+          ]
+        ],
+        "layers.0.mlp.fc2": [
+          [
+            0.04607229679822922,
+            -124
+          ],
+          [
+            0.00018067570636048913,
+            0
+          ]
+        ],
+        "layers.1.self_attn.k_proj": [
+          [
+            0.11797387152910233,
+            -7
+          ],
+          [
+            0.00046264249249361455,
+            0
+          ]
+        ],
+        "layers.1.self_attn.v_proj": [
+          [
+            0.11797387152910233,
+            -7
+          ],
+          [
+            0.00046264249249361455,
+            0
+          ]
+        ],
+        "layers.1.self_attn.q_proj": [
+          [
+            0.11797387152910233,
+            -7
+          ],
+          [
+            0.00046264249249361455,
+            0
+          ]
+        ],
+        "layers.1.self_attn.out_proj": [
+          [
+            0.025301799178123474,
+            5
+          ],
+          [
+            9.922267781803384e-05,
+            0
+          ]
+        ],
+        "layers.1.mlp.fc1": [
+          [
+            0.15761414170265198,
+            -11
+          ],
+          [
+            0.0006180948112159967,
+            -1
+          ]
+        ],
+        "layers.1.mlp.fc2": [
+          [
+            0.021325966343283653,
+            -120
+          ],
+          [
+            8.363127562915906e-05,
+            0
+          ]
+        ],
+        "layers.2.self_attn.k_proj": [
+          [
+            0.18797777593135834,
+            -25
+          ],
+          [
+            0.0007371677202172577,
+            -1
+          ]
+        ],
+        "layers.2.self_attn.v_proj": [
+          [
+            0.18797777593135834,
+            -25
+          ],
+          [
+            0.0007371677202172577,
+            -1
+          ]
+        ],
+        "layers.2.self_attn.q_proj": [
+          [
+            0.18797777593135834,
+            -25
+          ],
+          [
+            0.0007371677202172577,
+            -1
+          ]
+        ],
+        "layers.2.self_attn.out_proj": [
+          [
+            0.02796168625354767,
+            -9
+          ],
+          [
+            0.0001096536434488371,
+            -1
+          ]
+        ],
+        "layers.2.mlp.fc1": [
+          [
+            0.15474054217338562,
+            -36
+          ],
+          [
+            0.0006068257498554885,
+            0
+          ]
+        ],
+        "layers.2.mlp.fc2": [
+          [
+            0.019050095230340958,
+            -119
+          ],
+          [
+            7.470636046491563e-05,
+            -1
+          ]
+        ],
+        "layers.3.self_attn.k_proj": [
+          [
+            0.18854397535324097,
+            -16
+          ],
+          [
+            0.0007393882842734456,
+            -1
+          ]
+        ],
+        "layers.3.self_attn.v_proj": [
+          [
+            0.18854397535324097,
+            -16
+          ],
+          [
+            0.0007393882842734456,
+            -1
+          ]
+        ],
+        "layers.3.self_attn.q_proj": [
+          [
+            0.18854397535324097,
+            -16
+          ],
+          [
+            0.0007393882842734456,
+            -1
+          ]
+        ],
+        "layers.3.self_attn.out_proj": [
+          [
+            0.028024116531014442,
+            26
+          ],
+          [
+            0.00010989848669851199,
+            -1
+          ]
+        ],
+        "layers.3.mlp.fc1": [
+          [
+            0.1748019903898239,
+            -28
+          ],
+          [
+            0.0006854981184005737,
+            -1
+          ]
+        ],
+        "layers.3.mlp.fc2": [
+          [
+            0.021401548758149147,
+            -120
+          ],
+          [
+            8.392766176257282e-05,
+            0
+          ]
+        ],
+        "layers.4.self_attn.k_proj": [
+          [
+            0.2204122543334961,
+            -24
+          ],
+          [
+            0.0008643616456538439,
+            0
+          ]
+        ],
+        "layers.4.self_attn.v_proj": [
+          [
+            0.2204122543334961,
+            -24
+          ],
+          [
+            0.0008643616456538439,
+            0
+          ]
+        ],
+        "layers.4.self_attn.q_proj": [
+          [
+            0.2204122543334961,
+            -24
+          ],
+          [
+            0.0008643616456538439,
+            0
+          ]
+        ],
+        "layers.4.self_attn.out_proj": [
+          [
+            0.026799356564879417,
+            5
+          ],
+          [
+            0.00010509543790249154,
+            0
+          ]
+        ],
+        "layers.4.mlp.fc1": [
+          [
+            0.21318072080612183,
+            -15
+          ],
+          [
+            0.000836003280710429,
+            0
+          ]
+        ],
+        "layers.4.mlp.fc2": [
+          [
+            0.03725965693593025,
+            -123
+          ],
+          [
+            0.00014611640654038638,
+            0
+          ]
+        ],
+        "layers.5.self_attn.k_proj": [
+          [
+            0.1860852837562561,
+            -7
+          ],
+          [
+            0.0007297461852431297,
+            -1
+          ]
+        ],
+        "layers.5.self_attn.v_proj": [
+          [
+            0.1860852837562561,
+            -7
+          ],
+          [
+            0.0007297461852431297,
+            -1
+          ]
+        ],
+        "layers.5.self_attn.q_proj": [
+          [
+            0.1860852837562561,
+            -7
+          ],
+          [
+            0.0007297461852431297,
+            -1
+          ]
+        ],
+        "layers.5.self_attn.out_proj": [
+          [
+            0.03250356763601303,
+            21
+          ],
+          [
+            0.00012746502761729062,
+            0
+          ]
+        ],
+        "layers.5.mlp.fc1": [
+          [
+            0.22218218445777893,
+            -6
+          ],
+          [
+            0.0008713023853488266,
+            -1
+          ]
+        ],
+        "layers.5.mlp.fc2": [
+          [
+            0.12603624165058136,
+            -127
+          ],
+          [
+            0.0004942595842294395,
+            0
+          ]
+        ],
+        "layers.6.self_attn.k_proj": [
+          [
+            0.20868107676506042,
+            -1
+          ],
+          [
+            0.000818357162643224,
+            0
+          ]
+        ],
+        "layers.6.self_attn.v_proj": [
+          [
+            0.20868107676506042,
+            -1
+          ],
+          [
+            0.000818357162643224,
+            0
+          ]
+        ],
+        "layers.6.self_attn.q_proj": [
+          [
+            0.20868107676506042,
+            -1
+          ],
+          [
+            0.000818357162643224,
+            0
+          ]
+        ],
+        "layers.6.self_attn.out_proj": [
+          [
+            0.028652627021074295,
+            6
+          ],
+          [
+            0.00011236334830755368,
+            -1
+          ]
+        ],
+        "layers.6.mlp.fc1": [
+          [
+            0.18634845316410065,
+            8
+          ],
+          [
+            0.0007307780906558037,
+            0
+          ]
+        ],
+        "layers.6.mlp.fc2": [
+          [
+            0.04141681641340256,
+            -124
+          ],
+          [
+            0.00016241887351498008,
+            0
+          ]
+        ],
+        "layers.7.self_attn.k_proj": [
+          [
+            0.19424130022525787,
+            7
+          ],
+          [
+            0.00076173065463081,
+            -1
+          ]
+        ],
+        "layers.7.self_attn.v_proj": [
+          [
+            0.19424130022525787,
+            7
+          ],
+          [
+            0.00076173065463081,
+            -1
+          ]
+        ],
+        "layers.7.self_attn.q_proj": [
+          [
+            0.19424130022525787,
+            7
+          ],
+          [
+            0.00076173065463081,
+            -1
+          ]
+        ],
+        "layers.7.self_attn.out_proj": [
+          [
+            0.025726741179823875,
+            -21
+          ],
+          [
+            0.00010088913404615596,
+            0
+          ]
+        ],
+        "layers.7.mlp.fc1": [
+          [
+            0.18025848269462585,
+            27
+          ],
+          [
+            0.0007068958948366344,
+            0
+          ]
+        ],
+        "layers.7.mlp.fc2": [
+          [
+            0.0317450687289238,
+            -123
+          ],
+          [
+            0.0001244904997292906,
+            -1
+          ]
+        ],
+        "layers.8.self_attn.k_proj": [
+          [
+            0.2215735763311386,
+            19
+          ],
+          [
+            0.0008689158130437136,
+            -1
+          ]
+        ],
+        "layers.8.self_attn.v_proj": [
+          [
+            0.2215735763311386,
+            19
+          ],
+          [
+            0.0008689158130437136,
+            -1
+          ]
+        ],
+        "layers.8.self_attn.q_proj": [
+          [
+            0.2215735763311386,
+            19
+          ],
+          [
+            0.0008689158130437136,
+            -1
+          ]
+        ],
+        "layers.8.self_attn.out_proj": [
+          [
+            0.03458048775792122,
+            20
+          ],
+          [
+            0.00013560972001869231,
+            0
+          ]
+        ],
+        "layers.8.mlp.fc1": [
+          [
+            0.18420721590518951,
+            17
+          ],
+          [
+            0.000722382334060967,
+            -1
+          ]
+        ],
+        "layers.8.mlp.fc2": [
+          [
+            0.028086207807064056,
+            -122
+          ],
+          [
+            0.0001101423695217818,
+            -1
+          ]
+        ],
+        "layers.9.self_attn.k_proj": [
+          [
+            0.24290066957473755,
+            31
+          ],
+          [
+            0.000952551665250212,
+            0
+          ]
+        ],
+        "layers.9.self_attn.v_proj": [
+          [
+            0.24290066957473755,
+            31
+          ],
+          [
+            0.000952551665250212,
+            0
+          ]
+        ],
+        "layers.9.self_attn.q_proj": [
+          [
+            0.24290066957473755,
+            31
+          ],
+          [
+            0.000952551665250212,
+            0
+          ]
+        ],
+        "layers.9.self_attn.out_proj": [
+          [
+            0.03084946796298027,
+            10
+          ],
+          [
+            0.00012097826402168721,
+            0
+          ]
+        ],
+        "layers.9.mlp.fc1": [
+          [
+            0.3155311346054077,
+            -4
+          ],
+          [
+            0.0012373793870210648,
+            -1
+          ]
+        ],
+        "layers.9.mlp.fc2": [
+          [
+            0.06736534833908081,
+            -125
+          ],
+          [
+            0.0002641777682583779,
+            0
+          ]
+        ],
+        "layers.10.self_attn.k_proj": [
+          [
+            0.2827392816543579,
+            33
+          ],
+          [
+            0.0011087822495028377,
+            0
+          ]
+        ],
+        "layers.10.self_attn.v_proj": [
+          [
+            0.2827392816543579,
+            33
+          ],
+          [
+            0.0011087822495028377,
+            0
+          ]
+        ],
+        "layers.10.self_attn.q_proj": [
+          [
+            0.2827392816543579,
+            33
+          ],
+          [
+            0.0011087822495028377,
+            0
+          ]
+        ],
+        "layers.10.self_attn.out_proj": [
+          [
+            0.02676890231668949,
+            10
+          ],
+          [
+            0.00010497611219761893,
+            -1
+          ]
+        ],
+        "layers.10.mlp.fc1": [
+          [
+            0.6608939170837402,
+            -3
+          ],
+          [
+            0.002591740805655718,
+            0
+          ]
+        ],
+        "layers.10.mlp.fc2": [
+          [
+            0.11313328146934509,
+            -126
+          ],
+          [
+            0.0004436599847394973,
+            0
+          ]
+        ],
+        "layers.11.self_attn.k_proj": [
+          [
+            0.29342105984687805,
+            23
+          ],
+          [
+            0.0011506739538162947,
+            -1
+          ]
+        ],
+        "layers.11.self_attn.v_proj": [
+          [
+            0.29342105984687805,
+            23
+          ],
+          [
+            0.0011506739538162947,
+            -1
+          ]
+        ],
+        "layers.11.self_attn.q_proj": [
+          [
+            0.29342105984687805,
+            23
+          ],
+          [
+            0.0011506739538162947,
+            -1
+          ]
+        ],
+        "layers.11.self_attn.out_proj": [
+          [
+            0.03917478024959564,
+            -11
+          ],
+          [
+            0.00015362711565103382,
+            0
+          ]
+        ],
+        "layers.11.mlp.fc1": [
+          [
+            0.5828185081481934,
+            -9
+          ],
+          [
+            0.002285564551129937,
+            0
+          ]
+        ],
+        "layers.11.mlp.fc2": [
+          [
+            0.42501285672187805,
+            -128
+          ],
+          [
+            0.0016667165327817202,
+            0
+          ]
+        ],
+        "connector.modality_projection.proj": [
+          [
+            0.18216344714164734,
+            19
+          ],
+          [
+            0.000714366149622947,
+            -1
+          ]
+        ]
+      },
+      "decoder": {
+        "layers.0.attention.wq": [
+          [
+            0.017823828384280205,
+            40
+          ],
+          [
+            6.989705434534699e-05,
+            0
+          ]
+        ],
+        "layers.0.attention.wk": [
+          [
+            0.017823828384280205,
+            40
+          ],
+          [
+            6.989705434534699e-05,
+            0
+          ]
+        ],
+        "layers.0.attention.wv": [
+          [
+            0.017823828384280205,
+            40
+          ],
+          [
+            6.989705434534699e-05,
+            0
+          ]
+        ],
+        "layers.0.attention.wo": [
+          [
+            0.007807944901287556,
+            -40
+          ],
+          [
+            3.0619306926382706e-05,
+            0
+          ]
+        ],
+        "layers.0.feed_forward.w1": [
+          [
+            0.08885412663221359,
+            -112
+          ],
+          [
+            0.000348444766132161,
+            0
+          ]
+        ],
+        "layers.0.feed_forward.w2": [
+          [
+            0.04989893361926079,
+            8
+          ],
+          [
+            0.0001956819323822856,
+            -1
+          ]
+        ],
+        "layers.0.feed_forward.w3": [
+          [
+            0.08885412663221359,
+            -112
+          ],
+          [
+            0.000348444766132161,
+            0
+          ]
+        ],
+        "layers.1.attention.wq": [
+          [
+            0.035501036792993546,
+            53
+          ],
+          [
+            0.000139219468110241,
+            -1
+          ]
+        ],
+        "layers.1.attention.wk": [
+          [
+            0.035501036792993546,
+            53
+          ],
+          [
+            0.000139219468110241,
+            -1
+          ]
+        ],
+        "layers.1.attention.wv": [
+          [
+            0.035501036792993546,
+            53
+          ],
+          [
+            0.000139219468110241,
+            -1
+          ]
+        ],
+        "layers.1.attention.wo": [
+          [
+            0.019016792997717857,
+            -27
+          ],
+          [
+            7.457551691913977e-05,
+            -1
+          ]
+        ],
+        "layers.1.feed_forward.w1": [
+          [
+            0.008405232802033424,
+            6
+          ],
+          [
+            3.296163777122274e-05,
+            -1
+          ]
+        ],
+        "layers.1.feed_forward.w2": [
+          [
+            0.040298622101545334,
+            29
+          ],
+          [
+            0.00015803368296474218,
+            -1
+          ]
+        ],
+        "layers.1.feed_forward.w3": [
+          [
+            0.008405232802033424,
+            6
+          ],
+          [
+            3.296163777122274e-05,
+            -1
+          ]
+        ],
+        "layers.2.attention.wq": [
+          [
+            0.03276512399315834,
+            8
+          ],
+          [
+            0.00012849035556428134,
+            -1
+          ]
+        ],
+        "layers.2.attention.wk": [
+          [
+            0.03276512399315834,
+            8
+          ],
+          [
+            0.00012849035556428134,
+            -1
+          ]
+        ],
+        "layers.2.attention.wv": [
+          [
+            0.03276512399315834,
+            8
+          ],
+          [
+            0.00012849035556428134,
+            -1
+          ]
+        ],
+        "layers.2.attention.wo": [
+          [
+            0.011804603971540928,
+            -4
+          ],
+          [
+            4.6292545448523015e-05,
+            -1
+          ]
+        ],
+        "layers.2.feed_forward.w1": [
+          [
+            0.010259619913995266,
+            -1
+          ],
+          [
+            4.0233793697552755e-05,
+            0
+          ]
+        ],
+        "layers.2.feed_forward.w2": [
+          [
+            0.0317905955016613,
+            -17
+          ],
+          [
+            0.00012466894986573607,
+            0
+          ]
+        ],
+        "layers.2.feed_forward.w3": [
+          [
+            0.010259619913995266,
+            -1
+          ],
+          [
+            4.0233793697552755e-05,
+            0
+          ]
+        ],
+        "layers.3.attention.wq": [
+          [
+            0.03699791058897972,
+            -2
+          ],
+          [
+            0.00014508973981719464,
+            0
+          ]
+        ],
+        "layers.3.attention.wk": [
+          [
+            0.03699791058897972,
+            -2
+          ],
+          [
+            0.00014508973981719464,
+            0
+          ]
+        ],
+        "layers.3.attention.wv": [
+          [
+            0.03699791058897972,
+            -2
+          ],
+          [
+            0.00014508973981719464,
+            0
+          ]
+        ],
+        "layers.3.attention.wo": [
+          [
+            0.021720269694924355,
+            32
+          ],
+          [
+            8.517743117408827e-05,
+            0
+          ]
+        ],
+        "layers.3.feed_forward.w1": [
+          [
+            0.014318770729005337,
+            -17
+          ],
+          [
+            5.615192276309244e-05,
+            0
+          ]
+        ],
+        "layers.3.feed_forward.w2": [
+          [
+            0.33642786741256714,
+            -104
+          ],
+          [
+            0.001319324248470366,
+            0
+          ]
+        ],
+        "layers.3.feed_forward.w3": [
+          [
+            0.014318770729005337,
+            -17
+          ],
+          [
+            5.615192276309244e-05,
+            0
+          ]
+        ],
+        "layers.4.attention.wq": [
+          [
+            0.0462617389857769,
+            25
+          ],
+          [
+            0.00018141795590054244,
+            0
+          ]
+        ],
+        "layers.4.attention.wk": [
+          [
+            0.0462617389857769,
+            25
+          ],
+          [
+            0.00018141795590054244,
+            0
+          ]
+        ],
+        "layers.4.attention.wv": [
+          [
+            0.0462617389857769,
+            25
+          ],
+          [
+            0.00018141795590054244,
+            0
+          ]
+        ],
+        "layers.4.attention.wo": [
+          [
+            0.01729913242161274,
+            2
+          ],
+          [
+            6.783962453482673e-05,
+            0
+          ]
+        ],
+        "layers.4.feed_forward.w1": [
+          [
+            0.015150951221585274,
+            -27
+          ],
+          [
+            5.9415444411570206e-05,
+            0
+          ]
+        ],
+        "layers.4.feed_forward.w2": [
+          [
+            0.04258466139435768,
+            15
+          ],
+          [
+            0.00016699809930287302,
+            -1
+          ]
+        ],
+        "layers.4.feed_forward.w3": [
+          [
+            0.015150951221585274,
+            -27
+          ],
+          [
+            5.9415444411570206e-05,
+            0
+          ]
+        ],
+        "layers.5.attention.wq": [
+          [
+            0.04702253267168999,
+            17
+          ],
+          [
+            0.00018440181156620383,
+            0
+          ]
+        ],
+        "layers.5.attention.wk": [
+          [
+            0.04702253267168999,
+            17
+          ],
+          [
+            0.00018440181156620383,
+            0
+          ]
+        ],
+        "layers.5.attention.wv": [
+          [
+            0.04702253267168999,
+            17
+          ],
+          [
+            0.00018440181156620383,
+            0
+          ]
+        ],
+        "layers.5.attention.wo": [
+          [
+            0.015384010970592499,
+            -16
+          ],
+          [
+            6.032934470567852e-05,
+            0
+          ]
+        ],
+        "layers.5.feed_forward.w1": [
+          [
+            0.012542051263153553,
+            -31
+          ],
+          [
+            4.9184363888343796e-05,
+            0
+          ]
+        ],
+        "layers.5.feed_forward.w2": [
+          [
+            0.0316687673330307,
+            -13
+          ],
+          [
+            0.00012419100676197559,
+            -1
+          ]
+        ],
+        "layers.5.feed_forward.w3": [
+          [
+            0.012542051263153553,
+            -31
+          ],
+          [
+            4.9184363888343796e-05,
+            0
+          ]
+        ],
+        "layers.6.attention.wq": [
+          [
+            0.053652409464120865,
+            20
+          ],
+          [
+            0.00021040139836259186,
+            0
+          ]
+        ],
+        "layers.6.attention.wk": [
+          [
+            0.053652409464120865,
+            20
+          ],
+          [
+            0.00021040139836259186,
+            0
+          ]
+        ],
+        "layers.6.attention.wv": [
+          [
+            0.053652409464120865,
+            20
+          ],
+          [
+            0.00021040139836259186,
+            0
+          ]
+        ],
+        "layers.6.attention.wo": [
+          [
+            0.016384968534111977,
+            -14
+          ],
+          [
+            6.425465835491195e-05,
+            -1
+          ]
+        ],
+        "layers.6.feed_forward.w1": [
+          [
+            0.013905930332839489,
+            -22
+          ],
+          [
+            5.453297490021214e-05,
+            -1
+          ]
+        ],
+        "layers.6.feed_forward.w2": [
+          [
+            0.06569673120975494,
+            -7
+          ],
+          [
+            0.000257634004810825,
+            0
+          ]
+        ],
+        "layers.6.feed_forward.w3": [
+          [
+            0.013905930332839489,
+            -22
+          ],
+          [
+            5.453297490021214e-05,
+            -1
+          ]
+        ],
+        "layers.7.attention.wq": [
+          [
+            0.048247963190078735,
+            20
+          ],
+          [
+            0.00018920755246654153,
+            0
+          ]
+        ],
+        "layers.7.attention.wk": [
+          [
+            0.048247963190078735,
+            20
+          ],
+          [
+            0.00018920755246654153,
+            0
+          ]
+        ],
+        "layers.7.attention.wv": [
+          [
+            0.048247963190078735,
+            20
+          ],
+          [
+            0.00018920755246654153,
+            0
+          ]
+        ],
+        "layers.7.attention.wo": [
+          [
+            0.01625002734363079,
+            -19
+          ],
+          [
+            6.372541247401386e-05,
+            0
+          ]
+        ],
+        "layers.7.feed_forward.w1": [
+          [
+            0.013832789845764637,
+            -6
+          ],
+          [
+            5.424611299531534e-05,
+            -1
+          ]
+        ],
+        "layers.7.feed_forward.w2": [
+          [
+            0.055258866399526596,
+            -3
+          ],
+          [
+            0.00021670118439942598,
+            -1
+          ]
+        ],
+        "layers.7.feed_forward.w3": [
+          [
+            0.013832789845764637,
+            -6
+          ],
+          [
+            5.424611299531534e-05,
+            -1
+          ]
+        ],
+        "layers.8.attention.wq": [
+          [
+            0.046826478093862534,
+            25
+          ],
+          [
+            0.0001836329756770283,
+            -1
+          ]
+        ],
+        "layers.8.attention.wk": [
+          [
+            0.046826478093862534,
+            25
+          ],
+          [
+            0.0001836329756770283,
+            -1
+          ]
+        ],
+        "layers.8.attention.wv": [
+          [
+            0.046826478093862534,
+            25
+          ],
+          [
+            0.0001836329756770283,
+            -1
+          ]
+        ],
+        "layers.8.attention.wo": [
+          [
+            0.020178530365228653,
+            6
+          ],
+          [
+            7.91311904322356e-05,
+            0
+          ]
+        ],
+        "layers.8.feed_forward.w1": [
+          [
+            0.015370944514870644,
+            -26
+          ],
+          [
+            6.0277943703113124e-05,
+            0
+          ]
+        ],
+        "layers.8.feed_forward.w2": [
+          [
+            0.06391426175832748,
+            34
+          ],
+          [
+            0.0002506438468117267,
+            0
+          ]
+        ],
+        "layers.8.feed_forward.w3": [
+          [
+            0.015370944514870644,
+            -26
+          ],
+          [
+            6.0277943703113124e-05,
+            0
+          ]
+        ],
+        "layers.9.attention.wq": [
+          [
+            0.05844179913401604,
+            14
+          ],
+          [
+            0.00022918317699804902,
+            -1
+          ]
+        ],
+        "layers.9.attention.wk": [
+          [
+            0.05844179913401604,
+            14
+          ],
+          [
+            0.00022918317699804902,
+            -1
+          ]
+        ],
+        "layers.9.attention.wv": [
+          [
+            0.05844179913401604,
+            14
+          ],
+          [
+            0.00022918317699804902,
+            -1
+          ]
+        ],
+        "layers.9.attention.wo": [
+          [
+            0.020972328260540962,
+            -10
+          ],
+          [
+            8.224398334277794e-05,
+            -1
+          ]
+        ],
+        "layers.9.feed_forward.w1": [
+          [
+            0.01582447625696659,
+            -30
+          ],
+          [
+            6.205654062796384e-05,
+            -1
+          ]
+        ],
+        "layers.9.feed_forward.w2": [
+          [
+            0.061518240720033646,
+            -41
+          ],
+          [
+            0.00024124795163515955,
+            -1
+          ]
+        ],
+        "layers.9.feed_forward.w3": [
+          [
+            0.01582447625696659,
+            -30
+          ],
+          [
+            6.205654062796384e-05,
+            -1
+          ]
+        ],
+        "layers.10.attention.wq": [
+          [
+            0.057099029421806335,
+            13
+          ],
+          [
+            0.00022391745005734265,
+            0
+          ]
+        ],
+        "layers.10.attention.wk": [
+          [
+            0.057099029421806335,
+            13
+          ],
+          [
+            0.00022391745005734265,
+            0
+          ]
+        ],
+        "layers.10.attention.wv": [
+          [
+            0.057099029421806335,
+            13
+          ],
+          [
+            0.00022391745005734265,
+            0
+          ]
+        ],
+        "layers.10.attention.wo": [
+          [
+            0.02361183613538742,
+            25
+          ],
+          [
+            9.259540820494294e-05,
+            -1
+          ]
+        ],
+        "layers.10.feed_forward.w1": [
+          [
+            0.015977581962943077,
+            -31
+          ],
+          [
+            6.265722186071798e-05,
+            0
+          ]
+        ],
+        "layers.10.feed_forward.w2": [
+          [
+            0.05527188628911972,
+            -12
+          ],
+          [
+            0.000216752290725708,
+            0
+          ]
+        ],
+        "layers.10.feed_forward.w3": [
+          [
+            0.015977581962943077,
+            -31
+          ],
+          [
+            6.265722186071798e-05,
+            0
+          ]
+        ],
+        "layers.11.attention.wq": [
+          [
+            0.05523655563592911,
+            18
+          ],
+          [
+            0.0002166134217986837,
+            0
+          ]
+        ],
+        "layers.11.attention.wk": [
+          [
+            0.05523655563592911,
+            18
+          ],
+          [
+            0.0002166134217986837,
+            0
+          ]
+        ],
+        "layers.11.attention.wv": [
+          [
+            0.05523655563592911,
+            18
+          ],
+          [
+            0.0002166134217986837,
+            0
+          ]
+        ],
+        "layers.11.attention.wo": [
+          [
+            0.02736421301960945,
+            -27
+          ],
+          [
+            0.000107310559542384,
+            0
+          ]
+        ],
+        "layers.11.feed_forward.w1": [
+          [
+            0.022182175889611244,
+            -12
+          ],
+          [
+            8.698877354618162e-05,
+            -1
+          ]
+        ],
+        "layers.11.feed_forward.w2": [
+          [
+            0.048375293612480164,
+            -32
+          ],
+          [
+            0.00018970677047036588,
+            -1
+          ]
+        ],
+        "layers.11.feed_forward.w3": [
+          [
+            0.022182175889611244,
+            -12
+          ],
+          [
+            8.698877354618162e-05,
+            -1
+          ]
+        ],
+        "layers.12.attention.wq": [
+          [
+            0.060529887676239014,
+            17
+          ],
+          [
+            0.00023737164156045765,
+            -1
+          ]
+        ],
+        "layers.12.attention.wk": [
+          [
+            0.060529887676239014,
+            17
+          ],
+          [
+            0.00023737164156045765,
+            -1
+          ]
+        ],
+        "layers.12.attention.wv": [
+          [
+            0.060529887676239014,
+            17
+          ],
+          [
+            0.00023737164156045765,
+            -1
+          ]
+        ],
+        "layers.12.attention.wo": [
+          [
+            0.024945423007011414,
+            -17
+          ],
+          [
+            9.782514098333195e-05,
+            -1
+          ]
+        ],
+        "layers.12.feed_forward.w1": [
+          [
+            0.02477002516388893,
+            -32
+          ],
+          [
+            9.713695908430964e-05,
+            0
+          ]
+        ],
+        "layers.12.feed_forward.w2": [
+          [
+            0.03289053589105606,
+            -7
+          ],
+          [
+            0.00012898237037006766,
+            -1
+          ]
+        ],
+        "layers.12.feed_forward.w3": [
+          [
+            0.02477002516388893,
+            -32
+          ],
+          [
+            9.713695908430964e-05,
+            0
+          ]
+        ],
+        "layers.13.attention.wq": [
+          [
+            0.06709799915552139,
+            18
+          ],
+          [
+            0.00026312837144359946,
+            0
+          ]
+        ],
+        "layers.13.attention.wk": [
+          [
+            0.06709799915552139,
+            18
+          ],
+          [
+            0.00026312837144359946,
+            0
+          ]
+        ],
+        "layers.13.attention.wv": [
+          [
+            0.06709799915552139,
+            18
+          ],
+          [
+            0.00026312837144359946,
+            0
+          ]
+        ],
+        "layers.13.attention.wo": [
+          [
+            0.02714546211063862,
+            -8
+          ],
+          [
+            0.00010645274596754462,
+            -1
+          ]
+        ],
+        "layers.13.feed_forward.w1": [
+          [
+            0.0362023301422596,
+            -55
+          ],
+          [
+            0.00014196964912116528,
+            -1
+          ]
+        ],
+        "layers.13.feed_forward.w2": [
+          [
+            0.29110389947891235,
+            -97
+          ],
+          [
+            0.0011415830813348293,
+            -1
+          ]
+        ],
+        "layers.13.feed_forward.w3": [
+          [
+            0.0362023301422596,
+            -55
+          ],
+          [
+            0.00014196964912116528,
+            -1
+          ]
+        ],
+        "layers.14.attention.wq": [
+          [
+            0.06070156395435333,
+            20
+          ],
+          [
+            0.0002380445512244478,
+            -1
+          ]
+        ],
+        "layers.14.attention.wk": [
+          [
+            0.06070156395435333,
+            20
+          ],
+          [
+            0.0002380445512244478,
+            -1
+          ]
+        ],
+        "layers.14.attention.wv": [
+          [
+            0.06070156395435333,
+            20
+          ],
+          [
+            0.0002380445512244478,
+            -1
+          ]
+        ],
+        "layers.14.attention.wo": [
+          [
+            0.02148558758199215,
+            -7
+          ],
+          [
+            8.425699343206361e-05,
+            0
+          ]
+        ],
+        "layers.14.feed_forward.w1": [
+          [
+            0.016782715916633606,
+            15
+          ],
+          [
+            6.581440538866445e-05,
+            -1
+          ]
+        ],
+        "layers.14.feed_forward.w2": [
+          [
+            0.036045271903276443,
+            -1
+          ],
+          [
+            0.00014135372475720942,
+            0
+          ]
+        ],
+        "layers.14.feed_forward.w3": [
+          [
+            0.016782715916633606,
+            15
+          ],
+          [
+            6.581440538866445e-05,
+            -1
+          ]
+        ],
+        "layers.15.attention.wq": [
+          [
+            0.052386704832315445,
+            26
+          ],
+          [
+            0.00020543771097436547,
+            -1
+          ]
+        ],
+        "layers.15.attention.wk": [
+          [
+            0.052386704832315445,
+            26
+          ],
+          [
+            0.00020543771097436547,
+            -1
+          ]
+        ],
+        "layers.15.attention.wv": [
+          [
+            0.052386704832315445,
+            26
+          ],
+          [
+            0.00020543771097436547,
+            -1
+          ]
+        ],
+        "layers.15.attention.wo": [
+          [
+            0.02741842530667782,
+            10
+          ],
+          [
+            0.00010752306843642145,
+            0
+          ]
+        ],
+        "layers.15.feed_forward.w1": [
+          [
+            0.016052227467298508,
+            13
+          ],
+          [
+            6.294966442510486e-05,
+            -1
+          ]
+        ],
+        "layers.15.feed_forward.w2": [
+          [
+            0.036300696432590485,
+            -10
+          ],
+          [
+            0.0001423554349457845,
+            0
+          ]
+        ],
+        "layers.15.feed_forward.w3": [
+          [
+            0.016052227467298508,
+            13
+          ],
+          [
+            6.294966442510486e-05,
+            -1
+          ]
+        ],
+        "layers.16.attention.wq": [
+          [
+            0.057359110563993454,
+            24
+          ],
+          [
+            0.00022493749565910548,
+            -1
+          ]
+        ],
+        "layers.16.attention.wk": [
+          [
+            0.057359110563993454,
+            24
+          ],
+          [
+            0.00022493749565910548,
+            -1
+          ]
+        ],
+        "layers.16.attention.wv": [
+          [
+            0.057359110563993454,
+            24
+          ],
+          [
+            0.00022493749565910548,
+            -1
+          ]
+        ],
+        "layers.16.attention.wo": [
+          [
+            0.022664036601781845,
+            -4
+          ],
+          [
+            8.887828153092414e-05,
+            -1
+          ]
+        ],
+        "layers.16.feed_forward.w1": [
+          [
+            0.015661122277379036,
+            21
+          ],
+          [
+            6.141592893982306e-05,
+            0
+          ]
+        ],
+        "layers.16.feed_forward.w2": [
+          [
+            0.029099423438310623,
+            9
+          ],
+          [
+            0.0001141151660704054,
+            -1
+          ]
+        ],
+        "layers.16.feed_forward.w3": [
+          [
+            0.015661122277379036,
+            21
+          ],
+          [
+            6.141592893982306e-05,
+            0
+          ]
+        ],
+        "layers.17.attention.wq": [
+          [
+            0.06801969558000565,
+            22
+          ],
+          [
+            0.00026674376567825675,
+            -1
+          ]
+        ],
+        "layers.17.attention.wk": [
+          [
+            0.06801969558000565,
+            22
+          ],
+          [
+            0.00026674376567825675,
+            -1
+          ]
+        ],
+        "layers.17.attention.wv": [
+          [
+            0.06801969558000565,
+            22
+          ],
+          [
+            0.00026674376567825675,
+            -1
+          ]
+        ],
+        "layers.17.attention.wo": [
+          [
+            0.03552840277552605,
+            -11
+          ],
+          [
+            0.00013932664296589792,
+            0
+          ]
+        ],
+        "layers.17.feed_forward.w1": [
+          [
+            0.016431303694844246,
+            16
+          ],
+          [
+            6.443633901653811e-05,
+            -1
+          ]
+        ],
+        "layers.17.feed_forward.w2": [
+          [
+            0.03217285871505737,
+            6
+          ],
+          [
+            0.00012616798630915582,
+            0
+          ]
+        ],
+        "layers.17.feed_forward.w3": [
+          [
+            0.016431303694844246,
+            16
+          ],
+          [
+            6.443633901653811e-05,
+            -1
+          ]
+        ],
+        "layers.18.attention.wq": [
+          [
+            0.059997912496328354,
+            22
+          ],
+          [
+            0.00023528531892225146,
+            0
+          ]
+        ],
+        "layers.18.attention.wk": [
+          [
+            0.059997912496328354,
+            22
+          ],
+          [
+            0.00023528531892225146,
+            0
+          ]
+        ],
+        "layers.18.attention.wv": [
+          [
+            0.059997912496328354,
+            22
+          ],
+          [
+            0.00023528531892225146,
+            0
+          ]
+        ],
+        "layers.18.attention.wo": [
+          [
+            0.03692711889743805,
+            22
+          ],
+          [
+            0.00014481216203421354,
+            -1
+          ]
+        ],
+        "layers.18.feed_forward.w1": [
+          [
+            0.018235012888908386,
+            16
+          ],
+          [
+            7.15096975909546e-05,
+            -1
+          ]
+        ],
+        "layers.18.feed_forward.w2": [
+          [
+            0.03604672849178314,
+            4
+          ],
+          [
+            0.0001413596619386226,
+            -1
+          ]
+        ],
+        "layers.18.feed_forward.w3": [
+          [
+            0.018235012888908386,
+            16
+          ],
+          [
+            7.15096975909546e-05,
+            -1
+          ]
+        ],
+        "layers.19.attention.wq": [
+          [
+            0.05455257371068001,
+            32
+          ],
+          [
+            0.00021393051429186016,
+            0
+          ]
+        ],
+        "layers.19.attention.wk": [
+          [
+            0.05455257371068001,
+            32
+          ],
+          [
+            0.00021393051429186016,
+            0
+          ]
+        ],
+        "layers.19.attention.wv": [
+          [
+            0.05455257371068001,
+            32
+          ],
+          [
+            0.00021393051429186016,
+            0
+          ]
+        ],
+        "layers.19.attention.wo": [
+          [
+            0.024298356845974922,
+            -11
+          ],
+          [
+            9.528754162602127e-05,
+            -1
+          ]
+        ],
+        "layers.19.feed_forward.w1": [
+          [
+            0.016636382788419724,
+            14
+          ],
+          [
+            6.52406524750404e-05,
+            -1
+          ]
+        ],
+        "layers.19.feed_forward.w2": [
+          [
+            0.037912607192993164,
+            16
+          ],
+          [
+            0.00014867674326524138,
+            0
+          ]
+        ],
+        "layers.19.feed_forward.w3": [
+          [
+            0.016636382788419724,
+            14
+          ],
+          [
+            6.52406524750404e-05,
+            -1
+          ]
+        ],
+        "layers.20.attention.wq": [
+          [
+            0.05900881439447403,
+            26
+          ],
+          [
+            0.00023140695702750236,
+            -1
+          ]
+        ],
+        "layers.20.attention.wk": [
+          [
+            0.05900881439447403,
+            26
+          ],
+          [
+            0.00023140695702750236,
+            -1
+          ]
+        ],
+        "layers.20.attention.wv": [
+          [
+            0.05900881439447403,
+            26
+          ],
+          [
+            0.00023140695702750236,
+            -1
+          ]
+        ],
+        "layers.20.attention.wo": [
+          [
+            0.029481958597898483,
+            -2
+          ],
+          [
+            0.00011561506835278124,
+            0
+          ]
+        ],
+        "layers.20.feed_forward.w1": [
+          [
+            0.01973850466310978,
+            13
+          ],
+          [
+            7.740584260318428e-05,
+            0
+          ]
+        ],
+        "layers.20.feed_forward.w2": [
+          [
+            0.05077813193202019,
+            15
+          ],
+          [
+            0.00019912967400159687,
+            -1
+          ]
+        ],
+        "layers.20.feed_forward.w3": [
+          [
+            0.01973850466310978,
+            13
+          ],
+          [
+            7.740584260318428e-05,
+            0
+          ]
+        ],
+        "layers.21.attention.wq": [
+          [
+            0.0625663548707962,
+            29
+          ],
+          [
+            0.00024535812553949654,
+            0
+          ]
+        ],
+        "layers.21.attention.wk": [
+          [
+            0.0625663548707962,
+            29
+          ],
+          [
+            0.00024535812553949654,
+            0
+          ]
+        ],
+        "layers.21.attention.wv": [
+          [
+            0.0625663548707962,
+            29
+          ],
+          [
+            0.00024535812553949654,
+            0
+          ]
+        ],
+        "layers.21.attention.wo": [
+          [
+            0.02409578487277031,
+            -14
+          ],
+          [
+            9.449264325667173e-05,
+            -1
+          ]
+        ],
+        "layers.21.feed_forward.w1": [
+          [
+            0.02278539538383484,
+            17
+          ],
+          [
+            8.935417281463742e-05,
+            0
+          ]
+        ],
+        "layers.21.feed_forward.w2": [
+          [
+            0.07497410476207733,
+            1
+          ],
+          [
+            0.0002940157719422132,
+            0
+          ]
+        ],
+        "layers.21.feed_forward.w3": [
+          [
+            0.02278539538383484,
+            17
+          ],
+          [
+            8.935417281463742e-05,
+            0
+          ]
+        ],
+        "layers.22.attention.wq": [
+          [
+            0.05468488484621048,
+            23
+          ],
+          [
+            0.0002144504978787154,
+            -1
+          ]
+        ],
+        "layers.22.attention.wk": [
+          [
+            0.05468488484621048,
+            23
+          ],
+          [
+            0.0002144504978787154,
+            -1
+          ]
+        ],
+        "layers.22.attention.wv": [
+          [
+            0.05468488484621048,
+            23
+          ],
+          [
+            0.0002144504978787154,
+            -1
+          ]
+        ],
+        "layers.22.attention.wo": [
+          [
+            0.025124160572886467,
+            6
+          ],
+          [
+            9.852605580817908e-05,
+            0
+          ]
+        ],
+        "layers.22.feed_forward.w1": [
+          [
+            0.0270016361027956,
+            10
+          ],
+          [
+            0.00010588859004201367,
+            -1
+          ]
+        ],
+        "layers.22.feed_forward.w2": [
+          [
+            0.10119497030973434,
+            6
+          ],
+          [
+            0.00039684283547103405,
+            -1
+          ]
+        ],
+        "layers.22.feed_forward.w3": [
+          [
+            0.0270016361027956,
+            10
+          ],
+          [
+            0.00010588859004201367,
+            -1
+          ]
+        ],
+        "layers.23.attention.wq": [
+          [
+            0.050478190183639526,
+            27
+          ],
+          [
+            0.00019795302068814635,
+            0
+          ]
+        ],
+        "layers.23.attention.wk": [
+          [
+            0.050478190183639526,
+            27
+          ],
+          [
+            0.00019795302068814635,
+            0
+          ]
+        ],
+        "layers.23.attention.wv": [
+          [
+            0.050478190183639526,
+            27
+          ],
+          [
+            0.00019795302068814635,
+            0
+          ]
+        ],
+        "layers.23.attention.wo": [
+          [
+            0.0362025685608387,
+            -4
+          ],
+          [
+            0.00014197066775523126,
+            -1
+          ]
+        ],
+        "layers.23.feed_forward.w1": [
+          [
+            0.0261356383562088,
+            11
+          ],
+          [
+            0.00010249243496218696,
+            0
+          ]
+        ],
+        "layers.23.feed_forward.w2": [
+          [
+            0.16038192808628082,
+            -29
+          ],
+          [
+            0.0006289482116699219,
+            -1
+          ]
+        ],
+        "layers.23.feed_forward.w3": [
+          [
+            0.0261356383562088,
+            11
+          ],
+          [
+            0.00010249243496218696,
+            0
+          ]
+        ],
+        "layers.24.attention.wq": [
+          [
+            0.05497172102332115,
+            5
+          ],
+          [
+            0.0002155762631446123,
+            0
+          ]
+        ],
+        "layers.24.attention.wk": [
+          [
+            0.05497172102332115,
+            5
+          ],
+          [
+            0.0002155762631446123,
+            0
+          ]
+        ],
+        "layers.24.attention.wv": [
+          [
+            0.05497172102332115,
+            5
+          ],
+          [
+            0.0002155762631446123,
+            0
+          ]
+        ],
+        "layers.24.attention.wo": [
+          [
+            0.03732874244451523,
+            -30
+          ],
+          [
+            0.000146387072163634,
+            -1
+          ]
+        ],
+        "layers.24.feed_forward.w1": [
+          [
+            0.025813216343522072,
+            8
+          ],
+          [
+            0.00010122814273927361,
+            0
+          ]
+        ],
+        "layers.24.feed_forward.w2": [
+          [
+            0.12522609531879425,
+            -27
+          ],
+          [
+            0.000491082202643156,
+            -1
+          ]
+        ],
+        "layers.24.feed_forward.w3": [
+          [
+            0.025813216343522072,
+            8
+          ],
+          [
+            0.00010122814273927361,
+            0
+          ]
+        ],
+        "layers.25.attention.wq": [
+          [
+            0.047167424112558365,
+            10
+          ],
+          [
+            0.0001849701948231086,
+            -1
+          ]
+        ],
+        "layers.25.attention.wk": [
+          [
+            0.047167424112558365,
+            10
+          ],
+          [
+            0.0001849701948231086,
+            -1
+          ]
+        ],
+        "layers.25.attention.wv": [
+          [
+            0.047167424112558365,
+            10
+          ],
+          [
+            0.0001849701948231086,
+            -1
+          ]
+        ],
+        "layers.25.attention.wo": [
+          [
+            0.03641931712627411,
+            21
+          ],
+          [
+            0.00014282084885053337,
+            0
+          ]
+        ],
+        "layers.25.feed_forward.w1": [
+          [
+            0.024308975785970688,
+            7
+          ],
+          [
+            9.532903641229495e-05,
+            0
+          ]
+        ],
+        "layers.25.feed_forward.w2": [
+          [
+            0.22050899267196655,
+            -18
+          ],
+          [
+            0.0008647405775263906,
+            0
+          ]
+        ],
+        "layers.25.feed_forward.w3": [
+          [
+            0.024308975785970688,
+            7
+          ],
+          [
+            9.532903641229495e-05,
+            0
+          ]
+        ],
+        "layers.26.attention.wq": [
+          [
+            0.05102696269750595,
+            16
+          ],
+          [
+            0.00020010532170999795,
+            -1
+          ]
+        ],
+        "layers.26.attention.wk": [
+          [
+            0.05102696269750595,
+            16
+          ],
+          [
+            0.00020010532170999795,
+            -1
+          ]
+        ],
+        "layers.26.attention.wv": [
+          [
+            0.05102696269750595,
+            16
+          ],
+          [
+            0.00020010532170999795,
+            -1
+          ]
+        ],
+        "layers.26.attention.wo": [
+          [
+            0.04784587025642395,
+            -11
+          ],
+          [
+            0.0001876299938885495,
+            -1
+          ]
+        ],
+        "layers.26.feed_forward.w1": [
+          [
+            0.024701740592718124,
+            9
+          ],
+          [
+            9.686937846709043e-05,
+            -1
+          ]
+        ],
+        "layers.26.feed_forward.w2": [
+          [
+            0.20375220477581024,
+            24
+          ],
+          [
+            0.0007990262238308787,
+            -1
+          ]
+        ],
+        "layers.26.feed_forward.w3": [
+          [
+            0.024701740592718124,
+            9
+          ],
+          [
+            9.686937846709043e-05,
+            -1
+          ]
+        ],
+        "layers.27.attention.wq": [
+          [
+            0.05648134648799896,
+            19
+          ],
+          [
+            0.00022149378492031246,
+            -1
+          ]
+        ],
+        "layers.27.attention.wk": [
+          [
+            0.05648134648799896,
+            19
+          ],
+          [
+            0.00022149378492031246,
+            -1
+          ]
+        ],
+        "layers.27.attention.wv": [
+          [
+            0.05648134648799896,
+            19
+          ],
+          [
+            0.00022149378492031246,
+            -1
+          ]
+        ],
+        "layers.27.attention.wo": [
+          [
+            0.05714663490653038,
+            16
+          ],
+          [
+            0.00022410399105865508,
+            0
+          ]
+        ],
+        "layers.27.feed_forward.w1": [
+          [
+            0.024300716817378998,
+            16
+          ],
+          [
+            9.529681119602174e-05,
+            -1
+          ]
+        ],
+        "layers.27.feed_forward.w2": [
+          [
+            0.23974163830280304,
+            -23
+          ],
+          [
+            0.0009401627467013896,
+            -1
+          ]
+        ],
+        "layers.27.feed_forward.w3": [
+          [
+            0.024300716817378998,
+            16
+          ],
+          [
+            9.529681119602174e-05,
+            -1
+          ]
+        ],
+        "layers.28.attention.wq": [
+          [
+            0.05538808926939964,
+            18
+          ],
+          [
+            0.00021720785298384726,
+            -1
+          ]
+        ],
+        "layers.28.attention.wk": [
+          [
+            0.05538808926939964,
+            18
+          ],
+          [
+            0.00021720785298384726,
+            -1
+          ]
+        ],
+        "layers.28.attention.wv": [
+          [
+            0.05538808926939964,
+            18
+          ],
+          [
+            0.00021720785298384726,
+            -1
+          ]
+        ],
+        "layers.28.attention.wo": [
+          [
+            0.04831133037805557,
+            1
+          ],
+          [
+            0.00018945617193821818,
+            0
+          ]
+        ],
+        "layers.28.feed_forward.w1": [
+          [
+            0.02387886494398117,
+            10
+          ],
+          [
+            9.364249854115769e-05,
+            -1
+          ]
+        ],
+        "layers.28.feed_forward.w2": [
+          [
+            0.21306206285953522,
+            16
+          ],
+          [
+            0.0008355369791388512,
+            -1
+          ]
+        ],
+        "layers.28.feed_forward.w3": [
+          [
+            0.02387886494398117,
+            10
+          ],
+          [
+            9.364249854115769e-05,
+            -1
+          ]
+        ],
+        "layers.29.attention.wq": [
+          [
+            0.05176341533660889,
+            23
+          ],
+          [
+            0.00020299319294281304,
+            -1
+          ]
+        ],
+        "layers.29.attention.wk": [
+          [
+            0.05176341533660889,
+            23
+          ],
+          [
+            0.00020299319294281304,
+            -1
+          ]
+        ],
+        "layers.29.attention.wv": [
+          [
+            0.05176341533660889,
+            23
+          ],
+          [
+            0.00020299319294281304,
+            -1
+          ]
+        ],
+        "layers.29.attention.wo": [
+          [
+            0.06051347032189369,
+            -5
+          ],
+          [
+            0.0002373075985815376,
+            0
+          ]
+        ],
+        "layers.29.feed_forward.w1": [
+          [
+            0.02441590465605259,
+            13
+          ],
+          [
+            9.574845898896456e-05,
+            0
+          ]
+        ],
+        "layers.29.feed_forward.w2": [
+          [
+            0.2622620761394501,
+            -9
+          ],
+          [
+            0.001028476282954216,
+            -1
+          ]
+        ],
+        "layers.29.feed_forward.w3": [
+          [
+            0.02441590465605259,
+            13
+          ],
+          [
+            9.574845898896456e-05,
+            0
+          ]
+        ],
+        "layers.30.attention.wq": [
+          [
+            0.03940032795071602,
+            21
+          ],
+          [
+            0.00015451051876880229,
+            -1
+          ]
+        ],
+        "layers.30.attention.wk": [
+          [
+            0.03940032795071602,
+            21
+          ],
+          [
+            0.00015451051876880229,
+            -1
+          ]
+        ],
+        "layers.30.attention.wv": [
+          [
+            0.03940032795071602,
+            21
+          ],
+          [
+            0.00015451051876880229,
+            -1
+          ]
+        ],
+        "layers.30.attention.wo": [
+          [
+            0.06193621829152107,
+            5
+          ],
+          [
+            0.0002428869338473305,
+            -1
+          ]
+        ],
+        "layers.30.feed_forward.w1": [
+          [
+            0.029521800577640533,
+            -9
+          ],
+          [
+            0.000115771617856808,
+            0
+          ]
+        ],
+        "layers.30.feed_forward.w2": [
+          [
+            0.2706244885921478,
+            -7
+          ],
+          [
+            0.0010612712940201163,
+            -1
+          ]
+        ],
+        "layers.30.feed_forward.w3": [
+          [
+            0.029521800577640533,
+            -9
+          ],
+          [
+            0.000115771617856808,
+            0
+          ]
+        ],
+        "layers.31.attention.wq": [
+          [
+            0.03465617075562477,
+            23
+          ],
+          [
+            0.00013590618618763983,
+            0
+          ]
+        ],
+        "layers.31.attention.wk": [
+          [
+            0.03465617075562477,
+            23
+          ],
+          [
+            0.00013590618618763983,
+            0
+          ]
+        ],
+        "layers.31.attention.wv": [
+          [
+            0.03465617075562477,
+            23
+          ],
+          [
+            0.00013590618618763983,
+            0
+          ]
+        ],
+        "layers.31.attention.wo": [
+          [
+            0.04215332493185997,
+            18
+          ],
+          [
+            0.00016530673019587994,
+            0
+          ]
+        ],
+        "layers.31.feed_forward.w1": [
+          [
+            0.04697251319885254,
+            -11
+          ],
+          [
+            0.00018420587002765387,
+            0
+          ]
+        ],
+        "layers.31.feed_forward.w2": [
+          [
+            0.5093206763267517,
+            19
+          ],
+          [
+            0.001997330691665411,
+            0
+          ]
+        ],
+        "layers.31.feed_forward.w3": [
+          [
+            0.04697251319885254,
+            -11
+          ],
+          [
+            0.00018420587002765387,
+            0
+          ]
+        ],
+        "output": [
+          [
+            0.23516124486923218,
+            -15
+          ],
+          [
+            0.0009221913642250001,
+            0
+          ]
+        ]
+      }
+    },
+    "host_data": "embed_tokens / vlm_meta.json / tokenizer exactly as in smolvlm-500m-instruct.py"
+  },
+  "files": {
+    "embed_tokens.i8.bin": {
+      "bytes": 47308800,
+      "sha256": "d3ea47247be0b0703ca1246335cb3bca5f68a4068e177475388e287d9eb5f385"
+    },
+    "embed_tokens.scale.f32.bin": {
+      "bytes": 197120,
+      "sha256": "adefdadaae81d572c6094b6826e850f263939d716b74da03484dc20c0769562a"
+    },
+    "vlm_meta.json": {
+      "bytes": 201,
+      "sha256": "b36fcf7ef7da8c2ace5271aa5da5ffaa5f10a779db2095cb69f5eadfc9972ae4"
+    },
+    "tokenizer/tokenizer.json": {
+      "sha256": "5ece781dc8d2b2f3e2f289ca0ae50b17cfc27dd27bfe7971bb8241e0b964331a"
+    },
+    "tokenizer/tokenizer_config.json": {
+      "sha256": "36c6fd44d07d10fd8180ee6b46dcccf69fb7c06753968ff0d7e17b8bfe17b777"
+    },
+    "tokenizer/special_tokens_map.json": {
+      "sha256": "aa0ff906077086dfa9734a7f97f68c825877a48f9468807be65504495cdeef09"
+    },
+    "smolvlm_vision_q8.pte": {
+      "bytes": 103839108,
+      "sha256": "a48d6919aa2067be9a8d615e46f1cac5c8e0ac94a6cea7af9e4798290f5a9747"
+    },
+    "smolvlm_decoder_q8.pte": {
+      "bytes": 366912132,
+      "sha256": "a54d81fbafd289c3b68891b150aa7ba9b069ffc5ecabe4114cc437fa726bc08a"
+    }
+  },
+  "graph": {
+    "smolvlm_vision_q8.pte": {
+      "forward": {
+        "inputs": [
+          {
+            "shape": [
+              1,
+              3,
+              512,
+              512
+            ],
+            "dtype": "FLOAT"
+          }
+        ],
+        "outputs": [
+          {
+            "shape": [
+              1,
+              64,
+              960
+            ],
+            "dtype": "FLOAT"
+          }
+        ],
+        "non_delegated_kernels": {},
+        "delegates": [
+          {
+            "id": "VulkanBackend",
+            "ops": {
+              "aten._softmax.default": 12,
+              "aten.add.Tensor": 171,
+              "aten.bmm.default": 24,
+              "aten.clamp.default": 73,
+              "aten.clone.default": 268,
+              "aten.convolution.default": 1,
+              "aten.gelu.default": 12,
+              "aten.mul.Tensor": 158,
+              "aten.native_layer_norm.default": 25,
+              "aten.permute_copy.default": 63,
+              "aten.round.default": 73,
+              "aten.squeeze_copy.dims": 48,
+              "aten.sub.Tensor": 146,
+              "aten.view_copy.default": 76,
+              "dim_order_ops._to_dim_order_copy.default": 146,
+              "et_vk.linear_q8ta_q8csw.default": 146,
+              "et_vk.prepack.default": 305,
+              "quantized_decomposed.quantize_per_tensor.default": 146
+            },
+            "op_arity": {
+              "aten._softmax.default": [
+                4
+              ],
+              "aten.add.Tensor": [
+                4
+              ],
+              "aten.bmm.default": [
+                3
+              ],
+              "aten.clamp.default": [
+                4
+              ],
+              "aten.clone.default": [
+                3
+              ],
+              "aten.convolution.default": [
+                10
+              ],
+              "aten.gelu.default": [
+                3
+              ],
+              "aten.mul.Tensor": [
+                3
+              ],
+              "aten.native_layer_norm.default": [
+                6
+              ],
+              "aten.permute_copy.default": [
+                3
+              ],
+              "aten.round.default": [
+                2
+              ],
+              "aten.squeeze_copy.dims": [
+                3
+              ],
+              "aten.sub.Tensor": [
+                4
+              ],
+              "aten.view_copy.default": [
+                3
+              ],
+              "dim_order_ops._to_dim_order_copy.default": [
+                8
+              ],
+              "et_vk.linear_q8ta_q8csw.default": [
+                8
+              ],
+              "et_vk.prepack.default": [
+                2
+              ],
+              "quantized_decomposed.quantize_per_tensor.default": [
+                7
+              ]
+            },
+            "inputs": 1,
+            "outputs": 1
+          }
+        ]
+      }
+    },
+    "smolvlm_decoder_q8.pte": {
+      "forward": {
+        "inputs": [
+          {
+            "shape": [
+              1,
+              4,
+              960
+            ],
+            "dtype": "FLOAT"
+          },
+          {
+            "shape": [
+              1
+            ],
+            "dtype": "LONG"
+          }
+        ],
+        "outputs": [
+          {
+            "shape": [
+              1,
+              49280
+            ],
+            "dtype": "FLOAT"
+          }
+        ],
+        "non_delegated_kernels": {},
+        "delegates": [
+          {
+            "id": "VulkanBackend",
+            "ops": {
+              "aten._to_copy.default": 482,
+              "aten.add.Tensor": 578,
+              "aten.clamp.default": 225,
+              "aten.clone.default": 450,
+              "aten.mul.Tensor": 610,
+              "aten.relu.default": 32,
+              "aten.round.default": 225,
+              "aten.select_copy.int": 1,
+              "aten.sigmoid.default": 32,
+              "aten.sub.Tensor": 546,
+              "aten.view_copy.default": 160,
+              "et_vk.apply_rotary_emb_hf.default": 32,
+              "et_vk.linear_q8ta_q8csw.default": 450,
+              "et_vk.prepack.default": 1030,
+              "et_vk.rms_norm.default": 65,
+              "et_vk.select_as_symint.default": 2,
+              "quantized_decomposed.quantize_per_tensor.default": 450,
+              "sdpa_with_kv_cache.default": 32
+            },
+            "op_arity": {
+              "aten._to_copy.default": [
+                8
+              ],
+              "aten.add.Tensor": [
+                4
+              ],
+              "aten.clamp.default": [
+                4
+              ],
+              "aten.clone.default": [
+                3
+              ],
+              "aten.mul.Tensor": [
+                3
+              ],
+              "aten.relu.default": [
+                2
+              ],
+              "aten.round.default": [
+                2
+              ],
+              "aten.select_copy.int": [
+                4
+              ],
+              "aten.sigmoid.default": [
+                2
+              ],
+              "aten.sub.Tensor": [
+                4
+              ],
+              "aten.view_copy.default": [
+                3
+              ],
+              "et_vk.apply_rotary_emb_hf.default": [
+                6
+              ],
+              "et_vk.linear_q8ta_q8csw.default": [
+                8
+              ],
+              "et_vk.prepack.default": [
+                2
+              ],
+              "et_vk.rms_norm.default": [
+                4
+              ],
+              "et_vk.select_as_symint.default": [
+                4
+              ],
+              "quantized_decomposed.quantize_per_tensor.default": [
+                7
+              ],
+              "sdpa_with_kv_cache.default": [
+                12
+              ]
+            },
+            "inputs": 2,
+            "outputs": 1
+          }
+        ]
+      },
+      "get_max_context_len": {
+        "inputs": [],
+        "outputs": [
+          {
+            "value": 512
+          }
+        ],
+        "non_delegated_kernels": {},
+        "delegates": []
+      },
+      "get_n_layers": {
+        "inputs": [],
+        "outputs": [
+          {
+            "value": 32
+          }
+        ],
+        "non_delegated_kernels": {},
+        "delegates": []
+      },
+      "get_vocab_size": {
+        "inputs": [],
+        "outputs": [
+          {
+            "value": 49280
+          }
+        ],
+        "non_delegated_kernels": {},
+        "delegates": []
+      },
+      "use_kv_cache": {
+        "inputs": [],
+        "outputs": [
+          {
+            "value": true
+          }
+        ],
+        "non_delegated_kernels": {},
+        "delegates": []
+      },
+      "use_sdpa_with_kv_cache": {
+        "inputs": [],
+        "outputs": [
+          {
+            "value": true
+          }
+        ],
+        "non_delegated_kernels": {},
+        "delegates": []
+      },
+      "enable_dynamic_shape": {
+        "inputs": [],
+        "outputs": [
+          {
+            "value": false
+          }
+        ],
+        "non_delegated_kernels": {},
+        "delegates": []
+      },
+      "get_max_seq_len": {
+        "inputs": [],
+        "outputs": [
+          {
+            "value": 5
+          }
+        ],
+        "non_delegated_kernels": {},
+        "delegates": []
+      }
+    }
+  },
+  "delegate_weight_check": {
+    "smolvlm_decoder_q8.pte": {
+      "linear_q8ta_q8csw_ops": 450,
+      "weight_int8_bit_identical": 450,
+      "weight_scales_bit_identical": 450,
+      "activation_qparams_match": 450,
+      "eager_q8_linears": 225
+    },
+    "smolvlm_vision_q8.pte": {
+      "linear_q8ta_q8csw_ops": 146,
+      "weight_int8_bit_identical": 146,
+      "weight_scales_bit_identical": 146,
+      "activation_qparams_match": 146,
+      "eager_q8_linears": 73
+    }
+  },
+  "command": "python recipes/smolvlm-500m-instruct-q8.py --out out/smolvlm-500m-instruct-q8"
+}

export/requirements.txt

@@ -0,0 +1,15 @@
+# Environment the re-exports in REEXPORT.md were produced with (Python 3.12, Linux x86_64, CPU only).
+#   python -m venv venv
+#   venv/bin/pip install -r recipes/requirements.txt --extra-index-url https://download.pytorch.org/whl/cpu
+# executorch 1.5.1 = pytorch/executorch git 3b60683923245cf472b7323426920e15623ba361; its wheel ships
+# backends/webgpu (WebGPUPartitioner) and examples/models/llama (export_llm).
+executorch==1.5.1
+torch==2.14.1+cpu
+torchvision==0.29.1+cpu
+torchao==0.18.0+cpu
+transformers==5.19.0
+huggingface_hub==2.2.0
+safetensors==0.8.0
+tokenizers==0.23.3
+numpy==2.5.4
+pillow==12.3.0

export/smolvlm-500m-instruct-q8.py

@@ -0,0 +1,754 @@
+#!/usr/bin/env python3
+"""SmolVLM 500M Instruct -> ExecuTorch WebGPU with 8-bit linears (public ExecuTorch only).
+
+The 4-bit public export (smolvlm-500m-instruct.py) loses too much answer quality. This variant uses
+the one 8-bit linear that public ExecuTorch lowers for WebGPU and the public WebGPU runtime runs,
+et_vk.linear_q8ta_q8csw (static per-tensor int8 activations x per-channel int8 weights), with the
+additions described in _smolvlm_q8.py (two-pass activations, built-in attention sink, optional GPTQ).
+
+Outputs in --out:
+  smolvlm_vision_q8.pte          vision tower + connector: pixels[1,3,512,512] -> image embeds[1,64,960]
+  smolvlm_decoder_q8.pte         text decoder, one static window of R rows per call:
+                                 embeds[1,R,960], input_pos[1] -> logits[1,49280] of the last row;
+                                 KV cache [1,512,5,64] x 32 layers in the program
+  embed_tokens.i8.bin, embed_tokens.scale.f32.bin, vlm_meta.json, tokenizer/
+                                 identical to the 4-bit export (and to the internal files)
+  recipe.json, validation.json
+
+The decoder has static shapes because the Vulkan partitioner does not delegate
+quantized_decomposed.quantize_per_tensor with a dynamic dimension ("no dynamic shape support"), and
+the public WebGPU quantize / q8ta kernels have no resize hooks. One program means one KV cache:
+prompts are fed in windows of R rows, the last aligned to end on the prompt's last position, and
+each decode step feeds the new token with the R-1 rows before it (vlm.js, window_starts()).
+"""
+
+import argparse
+import copy
+import hashlib
+import json
+import shutil
+import sys
+import time
+import types
+from pathlib import Path
+
+import numpy as np
+import torch
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import _common  # noqa: E402
+import _smolvlm as S  # noqa: E402
+import _smolvlm_q8 as Q  # noqa: E402
+
+SPEC = {
+    "name": "smolvlm-500m-instruct-q8",
+    "upstream_repo": "HuggingFaceTB/SmolVLM-500M-Instruct",
+    "revision": "a7da5b986cb59b408707209984f360a5f4ad7e47",
+    "sha256": {
+        "model.safetensors": "d05b567eeaf534e83d375551f068ed57b5f52d37c657197f644af5ef9db091a2",
+        "config.json": "daacbbca6af3c34e50466c72aed2df4553a08084f6a0551d4ae420a241cb66c6",
+        "generation_config.json": "067a2a54e5f87162ecac6e0e911cc4665fc8f7f3324794ecbac0f76badb56636",
+        "preprocessor_config.json": "6cb6e36d6fcb88ca1502c4a26750715dc3e7dedddc9a8f17b27d8d167d1457e7",
+        "tokenizer.json": "5ece781dc8d2b2f3e2f289ca0ae50b17cfc27dd27bfe7971bb8241e0b964331a",
+        "tokenizer_config.json": "36c6fd44d07d10fd8180ee6b46dcccf69fb7c06753968ff0d7e17b8bfe17b777",
+        "special_tokens_map.json": "aa0ff906077086dfa9734a7f97f68c825877a48f9468807be65504495cdeef09",
+    },
+}
+TOKENIZER_FILES = ["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json"]
+
+# End-to-end check: the 3 fixed pairs of the 4-bit export (images from the playground samples).
+PAIRS = [
+    ("astronaut.jpg", "Describe this image"),
+    ("receipt.jpg", "What text can you read?"),
+    ("city_ny.jpg", "How many people are there?"),
+]
+# Calibration: held-out image + question pairs. None of these images is in the 3-pair check or in
+# the 12-pair quality evaluation (VLM_FOLLOWUP.md); the continuation is the fp32 model's own
+# greedy answer (CALIB_NEW tokens).
+CALIB = [
+    ("yolo_cat.jpg", "What animal is this?"),
+    ("yolo_people.jpg", "Describe this image"),
+    ("yolo_street.jpg", "What vehicles can you see?"),
+    ("yolo_table.jpg", "What objects are on the table?"),
+    ("sample_shapes.jpg", "What shapes are in this picture?"),
+    ("chihuahuas.jpg", "How many dogs are there?"),
+    ("kitchen.jpg", "Describe this image"),
+    ("giraffes.jpg", "What is happening here?"),
+]
+CALIB_NEW = 32
+
+VARIANTS = {
+    # vision: q8x2 = two-pass 8-bit, q8 = one pass; two_pass: decoder; gptq: weights of both programs
+    "q8": {"vision": "q8x2", "two_pass": True, "gptq": True},
+    "q8-rtn": {"vision": "q8x2", "two_pass": True, "gptq": False},
+    "q8-1pass": {"vision": "q8", "two_pass": False, "gptq": True},
+}
+
+
+# ----------------------------------------------------------------------------- inputs
+
+
+def find_image(name, dirs):
+    for d in dirs:
+        if (Path(d) / name).exists():
+            return Path(d) / name
+    sys.exit(f"image {name} not found in {[str(d) for d in dirs]}")
+
+
+class Prompts:
+    """Playground prompt construction (vlm.js firstTurnIds) and preprocessing (normalizeRGBA)."""
+
+    def __init__(self, tok, meta):
+        self.tok, self.meta = tok, meta
+        self.eos = set(meta["eos_token_ids"])
+
+    def ids(self, question):
+        conv = self.tok.convert_tokens_to_ids
+        user = self.tok.encode("<|im_start|>User:", add_special_tokens=False)
+        asst = self.tok.encode(
+            "<end_of_utterance>\nAssistant:", add_special_tokens=False
+        )
+        image = (
+            [conv(S.FAKE_IMAGE), conv(S.GLOBAL_IMAGE)]
+            + [self.meta["image_token_id"]] * self.meta["n_image_tokens"]
+            + [conv(S.FAKE_IMAGE)]
+        )
+        return user + image + self.tok.encode(
+            question, add_special_tokens=False
+        ) + asst, len(user) + 2
+
+    def pixels(self, path):
+        """Stretch to 512x512 (PIL bilinear stands in for canvas drawImage), rescale, normalize, CHW."""
+        from PIL import Image
+
+        size = self.meta["image_size"]
+        im = Image.open(path).convert("RGB").resize((size, size), Image.BILINEAR)
+        a = np.asarray(im, dtype=np.float32).transpose(2, 0, 1) * np.float32(
+            self.meta["rescale"]
+        )
+        a = (a - np.float32(self.meta["mean"])) / np.float32(self.meta["std"])
+        return torch.from_numpy(np.ascontiguousarray(a))[None]
+
+
+# ----------------------------------------------------------------------------- export
+
+
+def lower_static(module, example, name):
+    from executorch.backends.webgpu.partitioner import WebGPUPartitioner
+    from executorch.exir import to_edge_transform_and_lower
+
+    with torch.no_grad():
+        ep = torch.export.export(module, example, strict=False)
+        webgpu = (
+            to_edge_transform_and_lower(ep, partitioner=[WebGPUPartitioner()])
+            .to_executorch()
+            .buffer
+        )
+        portable = (
+            to_edge_transform_and_lower(
+                torch.export.export(module, example, strict=False)
+            )
+            .to_executorch()
+            .buffer
+        )
+    return ep, webgpu, portable
+
+
+def export_decoder(dec, params, args):
+    from executorch.backends.webgpu.partitioner import WebGPUPartitioner
+    from executorch.extension.llm.export.builder import DType, LLMEdgeManager
+
+    def manager():
+        return LLMEdgeManager(
+            model=dec,
+            modelname="smolvlm_decoder_q8",
+            max_seq_len=args.rows + 1,
+            use_kv_cache=True,
+            example_inputs=(
+                torch.zeros(1, args.rows, params["dim"]),
+                torch.tensor([0], dtype=torch.long),
+            ),
+            dtype=DType.fp32,
+            enable_dynamic_shape=False,
+            metadata={
+                "get_max_context_len": args.max_context,
+                "get_n_layers": params["n_layers"],
+                "get_vocab_size": params["vocab_size"],
+                "use_kv_cache": True,
+                "use_sdpa_with_kv_cache": True,
+                "enable_dynamic_shape": False,
+            },
+            dynamic_shapes=None,
+        )
+
+    mgr = manager()
+    mgr.export()
+    mgr.run_canonical_optimizations()
+    mgr.export_to_edge()
+    graph = copy.deepcopy(mgr.pre_autograd_graph_module)
+    mgr.to_backend([WebGPUPartitioner()])
+    mgr.to_executorch(passes=[])
+    webgpu = mgr.export_program.buffer
+    del mgr
+    pmgr = manager()
+    pmgr.export()
+    pmgr.run_canonical_optimizations()
+    pmgr.export_to_edge()
+    pmgr.to_executorch(passes=[])
+    portable = pmgr.export_program.buffer
+    del pmgr
+    return graph, webgpu, portable
+
+
+# ----------------------------------------------------------------------------- validation
+
+
+class Pipeline:
+    """Host logic of vlm.js around a vision callable and a windowed decoder callable
+    decoder(rows [1, R, hidden], start) -> logits [vocab] of the last row."""
+
+    def __init__(self, prompts, vision, decoder, embed, rows):
+        self.p, self.vision, self.decoder, self.embed, self.rows = (
+            prompts,
+            vision,
+            decoder,
+            embed,
+            rows,
+        )
+
+    def feed(self, seq, start, end):
+        logits = None
+        for s in Q.window_starts(start, end, self.rows):
+            logits = self.decoder(seq[None, s : s + self.rows].contiguous(), s)
+        return logits
+
+    def prompt_rows(self, image, question):
+        ids, start = self.p.ids(question)
+        rows = self.embed[torch.tensor(ids)].clone()
+        rows[start : start + image.shape[0]] = image
+        return ids, rows
+
+    def greedy(self, image, question, steps):
+        ids, rows = self.prompt_rows(image, question)
+        logits = self.feed(rows, 0, len(ids))
+        out = []
+        for _ in range(steps):
+            t = int(logits.argmax())
+            if t in self.p.eos:
+                break
+            out.append(t)
+            rows = torch.cat([rows, self.embed[t][None]])
+            logits = self.feed(rows, rows.shape[0] - 1, rows.shape[0])
+        return out
+
+    def teacher_forced(self, image, question, ref_tokens, ref_logits):
+        ids, rows = self.prompt_rows(image, question)
+        logits = self.feed(rows, 0, len(ids))
+        hits, cos = [], []
+        for i in range(len(ref_tokens) + 1):
+            hits.append(int(logits.argmax()) == int(ref_logits[i].argmax()))
+            cos.append(_common.compare(logits.numpy(), ref_logits[i].numpy())["cosine"])
+            if i == len(ref_tokens):
+                break
+            rows = torch.cat([rows, self.embed[ref_tokens[i]][None]])
+            logits = self.feed(rows, rows.shape[0] - 1, rows.shape[0])
+        return {
+            "prefill_top1_match": hits[0],
+            "prefill_logits_cosine": cos[0],
+            "teacher_forced_top1_agreement": sum(hits) / len(hits),
+            "teacher_forced_mean_logits_cosine": float(np.mean(cos)),
+        }
+
+
+def agreement(a, b):
+    prefix = 0
+    for x, y in zip(a, b):
+        if x != y:
+            break
+        prefix += 1
+    n = max(len(a), len(b))
+    return {
+        "tokens": len(a),
+        "reference_tokens": len(b),
+        "prefix_match": prefix,
+        "positional_agreement": (sum(x == y for x, y in zip(a, b)) / n) if n else 1.0,
+    }
+
+
+def validate(hf, prompts, args, vis_q, vis_ep, dec, dec_graph, programs, emb_i8):
+    import executorch.extension.llm.custom_ops.custom_ops  # noqa: F401 (llama:: CPU kernels)
+    import executorch.kernels.quantized  # noqa: F401 (quantized_decomposed CPU kernels)
+    from executorch.extension.pybindings.portable_lib import (
+        _load_for_executorch_from_buffer,
+    )
+
+    report = {
+        "cpu_runtime": "executorch pybindings (portable_lib) from the pip wheel, with executorch.extension.llm.custom_ops (llama::update_cache, custom_sdpa) and executorch.kernels.quantized (quantized_decomposed::quantize/dequantize_per_tensor, dequantize_per_channel)",
+        "portable_programs": "the same exported graphs without a delegate: int8 weights, static activation scales and the quantize/dequantize ops run as stored",
+        "note": "the WebGPU programs cannot run on CPU; see VLM_FOLLOWUP.md for the browser run",
+    }
+    vprog = _load_for_executorch_from_buffer(programs["vision_portable"])
+    dprog = _load_for_executorch_from_buffer(programs["decoder_portable"])
+    for name in ("vision", "decoder"):
+        try:
+            probe = (
+                (torch.zeros(1, 3, 512, 512),)
+                if name == "vision"
+                else (torch.zeros(1, args.rows, 960), torch.tensor([0]))
+            )
+            _load_for_executorch_from_buffer(programs[name]).forward(probe)
+            report[f"{name}_webgpu_pte_on_cpu"] = "ran"
+        except Exception as e:  # noqa: BLE001 - the error text is the result
+            report[f"{name}_webgpu_pte_on_cpu"] = (
+                f"not runnable on CPU pybindings (expected: VulkanBackend is not linked): {type(e).__name__}: {str(e).splitlines()[0][:200]}"
+            )
+
+    gen = torch.Generator().manual_seed(1234)
+    x = torch.rand(1, 3, 512, 512, generator=gen) * 2 - 1
+    with torch.no_grad():
+        ve = vis_q(x)
+        report["vision_random_input"] = {
+            "exported_vs_eager": _common.compare(
+                vis_ep.module()(x).numpy(), ve.numpy()
+            ),
+            "portable_pte_vs_eager": _common.compare(
+                np.asarray(vprog.forward((x.contiguous(),))[0]), ve.numpy()
+            ),
+        }
+    # Decoder: a real prompt's rows through a sequence of windows (overlapping re-feeds included).
+    img = find_image(PAIRS[0][0], args.images)
+    with torch.no_grad():
+        image = vis_q(prompts.pixels(img))[0]
+    ids, start = prompts.ids(PAIRS[0][1])
+    rows = emb_i8[torch.tensor(ids)].clone()
+    rows[start : start + 64] = image
+    R = args.rows
+    starts = [0, R, R // 2, R + 1, R + 2]
+    out = {}
+    for label, fn in (
+        ("eager", lambda e, p: dec(e, torch.tensor([p]))),
+        ("exported", lambda e, p: dec_graph(e, torch.tensor([p]))),
+        (
+            "portable_pte",
+            lambda e, p: torch.from_numpy(
+                np.asarray(dprog.forward((e.contiguous(), torch.tensor([p])))[0])
+            ),
+        ),
+    ):
+        S.reset_kv(dec)
+        S.reset_kv(dec_graph)
+        with torch.no_grad():
+            out[label] = [
+                fn(rows[None, s : s + R].contiguous(), s).clone() for s in starts
+            ]
+    report["decoder_windows"] = {
+        "starts": starts,
+        **{
+            f"{k}_vs_eager": [
+                _common.compare(a.numpy(), b.numpy())
+                for a, b in zip(out[k], out["eager"])
+            ]
+            for k in ("exported", "portable_pte")
+        },
+    }
+    del dprog
+
+    # End to end on the 3 fixed pairs.
+    dprog = _load_for_executorch_from_buffer(programs["decoder_portable"])
+    pipelines = {
+        "eager_quantized": Pipeline(
+            prompts,
+            lambda px: vis_q(px)[0].detach(),
+            lambda e, p: dec(e, torch.tensor([p]))[0].detach(),
+            emb_i8,
+            R,
+        ),
+        "export_portable_pte": Pipeline(
+            prompts,
+            lambda px: torch.from_numpy(
+                np.asarray(vprog.forward((px.contiguous(),))[0])
+            )[0],
+            lambda e, p: torch.from_numpy(
+                np.asarray(dprog.forward((e.contiguous(), torch.tensor([p])))[0])
+            )[0],
+            emb_i8,
+            R,
+        ),
+    }
+    vis_fp32 = S.VisionEncoder(hf).eval()
+    pairs = {}
+    for name, question in PAIRS:
+        px = prompts.pixels(find_image(name, args.images))
+        ids, _ = prompts.ids(question)
+        with torch.no_grad():
+            g = hf.generate(
+                input_ids=torch.tensor([ids]),
+                pixel_values=px[None],
+                max_new_tokens=args.new_tokens,
+                do_sample=False,
+                num_beams=1,
+            )
+            ref = [t for t in g[0, len(ids) :].tolist() if t not in prompts.eos]
+            ref_logits = hf(
+                input_ids=torch.tensor([ids + ref]), pixel_values=px[None]
+            ).logits[0][len(ids) - 1 :]
+            ref_image = vis_fp32(px)[0]
+        entry = {
+            "question": question,
+            "image": name,
+            "prompt_tokens": len(ids),
+            "transformers_fp32": {
+                "text": prompts.tok.decode(ref, skip_special_tokens=True),
+                "token_ids": ref,
+            },
+        }
+        for label, pipe in pipelines.items():
+            S.reset_kv(dec)
+            t0 = time.time()
+            with torch.no_grad():
+                image = pipe.vision(px)
+                toks = pipe.greedy(image, question, args.new_tokens)
+                S.reset_kv(dec)
+                tf = pipe.teacher_forced(image, question, ref, ref_logits)
+            entry[label] = {
+                "text": prompts.tok.decode(toks, skip_special_tokens=True),
+                "token_ids": toks,
+                **agreement(toks, ref),
+                **tf,
+                "image_embeds_vs_fp32": _common.compare(
+                    image.numpy(), ref_image.numpy()
+                ),
+                "seconds": round(time.time() - t0, 1),
+            }
+        pairs[name] = entry
+        print(json.dumps(entry, default=str)[:1500], flush=True)
+    summary = {}
+    for label in pipelines:
+        rows_ = [pairs[p][label] for p in pairs]
+        summary[label] = {
+            "exact_answers": sum(
+                r["token_ids"] == pairs[p]["transformers_fp32"]["token_ids"]
+                for p, r in zip(pairs, rows_)
+            ),
+            "positional_agreement_mean": float(
+                np.mean([r["positional_agreement"] for r in rows_])
+            ),
+            "teacher_forced_top1_agreement_mean": float(
+                np.mean([r["teacher_forced_top1_agreement"] for r in rows_])
+            ),
+            "prefill_top1_matches": sum(r["prefill_top1_match"] for r in rows_),
+        }
+    report["end_to_end"] = {
+        "decoding": f"greedy, at most {args.new_tokens} new tokens, stop at eos {sorted(prompts.eos)}; prompt fed in {R}-row windows, then one window per token (vlm.js)",
+        "preprocessing": "PIL bilinear resize to 512x512 (stand-in for canvas drawImage), (x/255 - 0.5)/0.5",
+        "reference": "transformers Idefics3ForConditionalGeneration fp32, eager attention, same input_ids and pixel_values",
+        "pipelines": {
+            "eager_quantized": "the quantized eager modules (the numerics of the exported graphs) + int8 host embeddings",
+            "export_portable_pte": "the exported programs as portable .pte via pybindings + int8 host embeddings: the shipped pipeline minus the WebGPU kernels",
+        },
+        "pairs": pairs,
+        "summary": summary,
+    }
+    return report
+
+
+# ----------------------------------------------------------------------------- main
+
+
+def build(args, var, out):
+    """Host data (written to `out`), calibration data and the quantized eager vision tower and
+    decoder. Shared with the quality evaluation, so it measures exactly what is exported."""
+    snap = S.snapshot(SPEC["upstream_repo"], SPEC["revision"], SPEC["sha256"])
+    hf = S.load_hf(snap)
+    from transformers import AutoTokenizer
+
+    tok = AutoTokenizer.from_pretrained(snap)
+    files = {}
+    q, s = S.embedding_table(hf)
+    files["embed_tokens.i8.bin"] = S.write_bytes(
+        out / "embed_tokens.i8.bin", q.tobytes()
+    )
+    files["embed_tokens.scale.f32.bin"] = S.write_bytes(
+        out / "embed_tokens.scale.f32.bin", s.tobytes()
+    )
+    meta = S.vlm_meta(snap, hf.config)
+    files["vlm_meta.json"] = S.write_bytes(
+        out / "vlm_meta.json", json.dumps(meta, indent=1).encode()
+    )
+    (out / "tokenizer").mkdir(exist_ok=True)
+    for f in TOKENIZER_FILES:
+        shutil.copyfile(snap / f, out / "tokenizer" / f)
+        files[f"tokenizer/{f}"] = {"sha256": _common.sha256_file(out / "tokenizer" / f)}
+    emb_i8 = torch.from_numpy(q.astype(np.float32) * s[:, None])
+    prompts = Prompts(tok, meta)
+
+    # Calibration data: images for the vision tower; prompt + fp32 greedy continuation rows for the
+    # decoder (the image rows come from the quantized vision tower, as at run time).
+    calib_imgs = [find_image(n, args.images) for n, _ in CALIB]
+    calib_px = [prompts.pixels(p) for p in calib_imgs]
+    calib_info = []
+    t0 = time.time()
+    vis_fp32 = S.VisionEncoder(hf).eval()
+    if var["vision"] == "q4gsw":
+        vis_q = S.quantize_4w(copy.deepcopy(vis_fp32), 32)
+        vis_act = None
+    else:
+        vis_q, vis_act = Q.build_vision(
+            vis_fp32, calib_px, two_pass=var["vision"] == "q8x2", gptq=var["gptq"]
+        )
+    calib_rows = []
+    for (name, question), path, px in zip(CALIB, calib_imgs, calib_px):
+        ids, start = prompts.ids(question)
+        with torch.no_grad():
+            g = hf.generate(
+                input_ids=torch.tensor([ids]),
+                pixel_values=px[None],
+                max_new_tokens=CALIB_NEW,
+                do_sample=False,
+                num_beams=1,
+            )
+            cont = [t for t in g[0, len(ids) :].tolist() if t not in prompts.eos]
+            image = vis_q(px)[0]
+        rows = emb_i8[torch.tensor(ids + cont)].clone()
+        rows[start : start + 64] = image
+        calib_rows.append(rows)
+        calib_info.append(
+            {
+                "image": name,
+                "image_sha256": _common.sha256_file(path),
+                "question": question,
+                "rows": int(rows.shape[0]),
+                "continuation": tok.decode(cont, skip_special_tokens=True),
+            }
+        )
+    params = S.text_params(hf.config)
+    transformer = S.build_text(
+        snap,
+        params,
+        max_context=args.max_context,
+        prefill_max=args.rows,
+        group_size=None,
+    )
+    sink_row = emb_i8[tok.encode("<|im_start|>", add_special_tokens=False)[0]]
+    transformer, dec_act, _ = Q.build_text(
+        transformer,
+        calib_rows,
+        rows=args.rows,
+        two_pass=var["two_pass"],
+        gptq=var["gptq"],
+        sink_row=sink_row,
+    )
+    dec = S.TextDecoder(transformer).eval()
+    print(f"calibration + quantization {time.time() - t0:.0f}s", flush=True)
+    return types.SimpleNamespace(**locals())
+
+
+def main():
+    ap = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--variant", choices=sorted(VARIANTS), default="q8")
+    ap.add_argument(
+        "--rows", type=int, default=4, help="decoder rows per call (static)"
+    )
+    ap.add_argument("--max-context", type=int, default=512)
+    ap.add_argument("--new-tokens", type=int, default=48)
+    ap.add_argument("--no-validate", action="store_true")
+    ap.add_argument("--threads", type=int, default=16, help="torch CPU threads (fixed for reproducible calibration)")
+    ap.add_argument("--save-portable", type=Path, help="also write the portable (CPU) programs here (validation aid, not a release file)")
+    ap.add_argument(
+        "--images",
+        type=Path,
+        nargs="+",
+        default=[
+            Path.home() / "etai-playground/_internal/assets/dashboard/web",
+            Path.home() / "etai-playground/site/playground/webgpu/assets/samples",
+        ],
+        help="directories searched for the evaluation and calibration images",
+    )
+    args = ap.parse_args()
+    var = VARIANTS[args.variant]
+    out = Path(args.out).resolve()
+    out.mkdir(parents=True, exist_ok=True)
+    torch.manual_seed(0)
+    # The activation scales come from fp32 forward passes, whose summation order depends on the
+    # thread count; a fixed count keeps re-runs byte-identical.
+    torch.set_num_threads(args.threads)
+    import executorch.kernels.quantized  # noqa: F401 (out variants of quantized_decomposed for to_executorch)
+
+    Q.fix_q8ta_fake()
+
+    m = build(args, var, out)
+    hf, tok, meta, prompts, emb_i8, files, calib_info = m.hf, m.tok, m.meta, m.prompts, m.emb_i8, m.files, m.calib_info
+    vis_q, vis_act, params, transformer, dec_act, dec = m.vis_q, m.vis_act, m.params, m.transformer, m.dec_act, m.dec
+
+    vision_ep, vision_pte, vision_portable = lower_static(
+        vis_q, (torch.zeros(1, 3, 512, 512),), "vision"
+    )
+    vname = (
+        "smolvlm_vision_q8.pte"
+        if var["vision"] != "q4gsw"
+        else "smolvlm_vision_q4gsw.pte"
+    )
+    files[vname] = S.write_bytes(out / vname, vision_pte)
+    dec_graph, decoder_pte, decoder_portable = export_decoder(dec, params, args)
+    files["smolvlm_decoder_q8.pte"] = S.write_bytes(
+        out / "smolvlm_decoder_q8.pte", decoder_pte
+    )
+
+    graph = {
+        vname: S.program_summary(vision_pte),
+        "smolvlm_decoder_q8.pte": S.program_summary(decoder_pte),
+    }
+    for name, g in graph.items():
+        fwd = g["forward"]
+        assert not fwd["non_delegated_kernels"], (name, fwd["non_delegated_kernels"])
+        assert len(fwd["delegates"]) == 1, (name, len(fwd["delegates"]))
+    weight_check = {
+        "smolvlm_decoder_q8.pte": Q.q8_ops_vs_eager(decoder_pte, transformer)
+    }
+    if var["vision"] != "q4gsw":
+        weight_check[vname] = Q.q8_ops_vs_eager(vision_pte, vis_q)
+    for name, c in weight_check.items():
+        n = c["linear_q8ta_q8csw_ops"]
+        assert (
+            n
+            == c["weight_int8_bit_identical"]
+            == c["weight_scales_bit_identical"]
+            == c["activation_qparams_match"]
+        ), (name, c)
+
+    two = {
+        True: "two passes (second pass on the first pass's rounding error, same int8 weight)",
+        False: "one pass",
+    }
+    wq = (
+        "GPTQ (damp 0.01, block 128, float64) on the calibration inputs, same grid"
+        if var["gptq"]
+        else "round to nearest"
+    )
+    recipe = {
+        "model": SPEC["name"],
+        "variant": args.variant,
+        "upstream": {
+            "model": SPEC["upstream_repo"],
+            "revision": SPEC["revision"],
+            "official": True,
+            "sha256": SPEC["sha256"],
+        },
+        "versions": _common.versions(),
+        "export": {
+            "partitioner": "executorch.backends.webgpu.partitioner.WebGPUPartitioner, default options (static shapes)",
+            "linear": "et_vk.linear_q8ta_q8csw: int8 activations, one static per-tensor affine scale per pass (torchao MinMaxObserver over the calibration data, [-128, 127]); int8 weights, per-output-channel symmetric (VulkanQuantizer weight spec, scale amax/127.5)",
+            "export_workaround": "_smolvlm_q8.fix_q8ta_fake(): a fake kernel giving et_vk.linear_q8ta_q8csw its real fp32 output dtype during lowering (executorch 1.5.1's reference kernel reports int8, which breaks llama::custom_sdpa's meta check and would mis-declare tensors). Metadata only",
+            "programs": {
+                vname: {
+                    "api": "torch.export.export(strict=False) -> to_edge_transform_and_lower(partitioner=[WebGPUPartitioner()]) -> to_executorch()",
+                    "module": "transformers Idefics3 vision tower (eager attention) + post_layernorm + connector; identity position ids for a full 512x512 image",
+                    "inputs": [
+                        {
+                            "name": "pixel_values",
+                            "shape": [1, 3, 512, 512],
+                            "dtype": "float32",
+                            "range": "(x/255 - 0.5)/0.5",
+                        }
+                    ],
+                    "outputs": [
+                        {
+                            "name": "image_embeds",
+                            "shape": [1, meta["n_image_tokens"], meta["hidden"]],
+                            "dtype": "float32",
+                        }
+                    ],
+                    "quantization": "4-bit groupwise (as the 4-bit export)"
+                    if var["vision"] == "q4gsw"
+                    else f"all 73 linears 8-bit, {two[var['vision'] == 'q8x2']}, weights {wq}{' (the connector, with fewer calibration rows than inputs, round to nearest)' if var['gptq'] else ''}; patch embedding conv, layer norms, position table fp32",
+                },
+                "smolvlm_decoder_q8.pte": {
+                    "api": "LLMEdgeManager: export(strict) -> run_canonical_optimizations -> export_to_edge -> to_backend([WebGPUPartitioner()]) -> to_executorch; static shapes",
+                    "module": "executorch examples/models/llama Transformer (apply_embedding=False, use_hf_rope, enable_dynamic_shape for the start position) with weights from executorch.examples.models.smolvlm.convert_weights; source transforms from export_llama_lib._get_source_transforms(use_sdpa_with_kv_cache, use_kv_cache)",
+                    "params": params,
+                    "inputs": [
+                        {
+                            "name": "embeds",
+                            "shape": [1, args.rows, meta["hidden"]],
+                            "dtype": "float32",
+                            "note": "rows of positions input_pos .. input_pos + rows - 1",
+                        },
+                        {
+                            "name": "input_pos",
+                            "shape": [1],
+                            "dtype": "int64",
+                            "note": "position of the first row",
+                        },
+                    ],
+                    "outputs": [
+                        {
+                            "name": "logits",
+                            "shape": [1, meta["vocab"]],
+                            "dtype": "float32",
+                            "note": "last row only",
+                        }
+                    ],
+                    "kv_cache": f"in the program: 32 layers x k/v [1,{args.max_context},5,64] fp32; every call writes its rows' positions",
+                    "attention_sink": "position 0 must be <|im_start|> (id 1), as the chat template always starts: its per-layer K/V (fp32 model, int8-table embedding row) are constants the program writes over row 0 when input_pos is 0; that row is excluded from calibration",
+                    "host_contract": "feed positions in windows of `rows` rows ending at the newest position (window_starts in _smolvlm_q8.py / windowStarts in vlm.js); input_pos + rows <= max_context",
+                    "max_context_length": args.max_context,
+                    "quantization": f"all 225 linears incl. lm_head 8-bit, {two[var['two_pass']]}, weights {wq}; RMSNorm weights fp32",
+                },
+            },
+            "calibration": {
+                "pairs": calib_info,
+                "continuation_tokens": CALIB_NEW,
+                "decoder_windows": f"{args.rows} rows, the browser's window plan",
+                "vision": "the 8 calibration images",
+            },
+            "activation_qparams": {
+                "vision": vis_act and {k: list(v) for k, v in vis_act.items()},
+                "decoder": {k: list(v) for k, v in dec_act.items()},
+            },
+            "host_data": "embed_tokens / vlm_meta.json / tokenizer exactly as in smolvlm-500m-instruct.py",
+        },
+        "files": files,
+        "graph": graph,
+        "delegate_weight_check": weight_check,
+        "command": _common.command_line(out),
+    }
+    (out / "recipe.json").write_text(
+        json.dumps(
+            recipe,
+            indent=2,
+            default=lambda o: list(o) if isinstance(o, tuple) else str(o),
+        )
+        + "\n"
+    )
+    print(json.dumps({"files": files}, indent=2), flush=True)
+
+    if args.save_portable:
+        args.save_portable.mkdir(parents=True, exist_ok=True)
+        (args.save_portable / "vision_portable.pte").write_bytes(vision_portable)
+        (args.save_portable / "decoder_portable.pte").write_bytes(decoder_portable)
+    if not args.no_validate:
+        programs = {
+            "vision": vision_pte,
+            "decoder": decoder_pte,
+            "vision_portable": vision_portable,
+            "decoder_portable": decoder_portable,
+        }
+        report = validate(
+            hf, prompts, args, vis_q, vision_ep, dec, dec_graph, programs, emb_i8
+        )
+        report["portable_program_files"] = {
+            k: {
+                "bytes": len(programs[k]),
+                "sha256": hashlib.sha256(programs[k]).hexdigest(),
+            }
+            for k in ("vision_portable", "decoder_portable")
+        }
+        (out / "validation.json").write_text(json.dumps(report, indent=2) + "\n")
+        print(json.dumps(report["end_to_end"]["summary"], indent=2))
+
+
+if __name__ == "__main__":
+    main()

manifest.json

@@ -12,6 +12,16 @@
       "bytes": 197120,
       "sha256": "adefdadaae81d572c6094b6826e850f263939d716b74da03484dc20c0769562a"
     },
+    {
+      "path": "smolvlm_decoder_q8.pte",
+      "bytes": 366912132,
+      "sha256": "a54d81fbafd289c3b68891b150aa7ba9b069ffc5ecabe4114cc437fa726bc08a"
+    },
+    {
+      "path": "smolvlm_vision_q8.pte",
+      "bytes": 103839108,
+      "sha256": "a48d6919aa2067be9a8d615e46f1cac5c8e0ac94a6cea7af9e4798290f5a9747"
+    },
     {
       "path": "tokenizer/special_tokens_map.json",
       "bytes": 1069,
@@ -27,25 +37,10 @@
       "bytes": 28249,
       "sha256": "36c6fd44d07d10fd8180ee6b46dcccf69fb7c06753968ff0d7e17b8bfe17b777"
     },
-    {
-      "path": "vlm_decoder.pte",
-      "bytes": 388664832,
-      "sha256": "d92697a8c32eeaa1d73603cf608d658cf297fdf41ed81384ec29b7025e78e996"
-    },
     {
       "path": "vlm_meta.json",
       "bytes": 201,
       "sha256": "b36fcf7ef7da8c2ace5271aa5da5ffaa5f10a779db2095cb69f5eadfc9972ae4"
-    },
-    {
-      "path": "vlm_prefill.pte",
-      "bytes": 388664320,
-      "sha256": "1e8671eb77ae2f60c1eb80073454230bfaeb1eb22cee36dd9b028d8e291a3e3f"
-    },
-    {
-      "path": "vlm_vision_q8.pte",
-      "bytes": 108885888,
-      "sha256": "5f09d6028d9bb945ce135d4be9ad499c1081f13d46655d2d1288c207f536ee4e"
     }
   ]
 }

vlm_decoder.pte → smolvlm_decoder_q8.pte

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d92697a8c32eeaa1d73603cf608d658cf297fdf41ed81384ec29b7025e78e996
-size 388664832
+oid sha256:a54d81fbafd289c3b68891b150aa7ba9b069ffc5ecabe4114cc437fa726bc08a
+size 366912132

vlm_prefill.pte → smolvlm_vision_q8.pte

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1e8671eb77ae2f60c1eb80073454230bfaeb1eb22cee36dd9b028d8e291a3e3f
-size 388664320
+oid sha256:a48d6919aa2067be9a8d615e46f1cac5c8e0ac94a6cea7af9e4798290f5a9747
+size 103839108

vlm_vision_q8.pte

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5f09d6028d9bb945ce135d4be9ad499c1081f13d46655d2d1288c207f536ee4e
-size 108885888