Add files using upload-large-folder tool

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Viewer-HX
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

MODEL_CARD.md

@@ -0,0 +1,110 @@
+---
+license: mit
+library_name: binprov
+tags:
+- binary-analysis
+- compilation-provenance
+- optimization-level-classification
+- pytorch
+- roberta
+metrics:
+- accuracy
+---
+
+# BinProv Opt4
+
+`binprov` classifies the compiler optimization level of x86-64 ELF
+binaries from raw `.text` bytes: O0, O1, O2, O3.
+It follows the BinProv method and does not disassemble the input.
+
+This is a BinProv-native PyTorch checkpoint. Its byte encoder is RoBERTa-shaped,
+but its pooling and classification head are custom, so it is not compatible
+with `transformers.AutoModel`. Use the bundled `inference.py`; no remote code or
+`trust_remote_code` is involved.
+
+## Model details
+
+- Architecture: RoBERTa-style byte encoder, seq 2048 bytes, 12 layers x 768 hidden; 261-token byte vocabulary
+- Pooling: `border` with 4 heads
+- Input: 2048-byte windows at a 512-byte evaluation stride
+- Output: four probabilities corresponding to O0, O1, O2, and O3
+- Framework: PyTorch with Hugging Face Transformers components
+- License: MIT
+
+The model follows [BinProv: Binary Code Provenance Identification without
+Disassembly](https://doi.org/10.1145/3545948.3545956), RAID 2022. This checkpoint
+comes from a clean reproduction run of the repository's best single-model
+recipe.
+Evaluation (this export): sequence accuracy **84.2255%**. Binary-level accuracy **93.8830%**. A fresh training run's numbers vary by ~1-2 points with seed.
+
+## Direct use
+
+```bash
+python -m venv .venv
+source .venv/bin/activate                    # Windows: .venv\Scripts\activate
+pip install -r requirements.txt
+
+python inference.py --model model --elf /path/to/binary.elf
+python inference.py --model model --text-bytes raw_text.bin --json
+```
+
+The script extracts `.text` from an ELF or accepts already extracted bytes. It
+evaluates overlapping 2048-byte inputs every 512 bytes, then averages the
+per-window probabilities for one binary. It automatically selects CUDA, Apple
+MPS, or CPU. Use `--batch-size` to trade memory for speed.
+
+## Training data
+
+The model was trained and evaluated on the x86-64 subset of BinKit Normal,
+covering GCC and Clang binaries at optimization levels O0 through O3. Splits are
+grouped by source program so the same program does not cross train and test.
+Exact training arguments and environment versions are in `metadata.json` and
+`evidence/args.json`.
+
+## Evaluation
+
+| metric | value |
+|---|---:|
+| sequence accuracy | **84.2255%** |
+| balanced sequence accuracy | **83.0951%** |
+| binary soft-vote accuracy | **93.8830%** |
+| test windows | 116,321 |
+| test binaries | 376 |
+
+The sequence metric uses 512-byte target positions with 2048-byte overlapping
+model inputs. Binary accuracy averages the saved per-window probabilities. The
+recomputed evaluation in `evidence/evaluation.json` matches the recorded test
+accuracy. Results can vary with retraining and should be compared on the same
+program-grouped split.
+
+## Intended use and limitations
+
+This checkpoint is intended for research and analysis of compiler optimization
+provenance in ordinary x86-64 ELF code produced by toolchains represented in
+BinKit. It has not been validated as a general malware, authorship, attribution,
+or security-decision system. Packed, obfuscated, self-modifying, non-ELF, other
+architecture, and out-of-distribution toolchain inputs may be unreliable. The
+softmax scores are not calibrated confidence estimates.
+
+## Files
+
+- `model/` — `model.pt`, `binprov_config.json`, `head.json` (BinProv checkpoint)
+- `binprov/` — minimal source package needed by `inference.py`
+- `requirements.txt` — runtime dependencies
+- `labels.json` — class id -> label
+- `metadata.json` — exact training args + eval results + environment
+- `environment.json` — runtime versions
+- `evidence/` — args, result, evaluation report, and saved probabilities
+- `LICENSE` — repository license
+
+## Citation
+
+```bibtex
+@inproceedings{he2022binprov,
+  title={BinProv: Binary Code Provenance Identification without Disassembly},
+  author={He, Xu and Wang, Shu and Xing, Yunlong and Feng, Pengbin and Wang, Haining and Li, Qi and Chen, Songqing and Sun, Kun},
+  booktitle={RAID},
+  year={2022},
+  doi={10.1145/3545948.3545956}
+}
+```

README.md

@@ -0,0 +1,110 @@
+---
+license: mit
+library_name: binprov
+tags:
+- binary-analysis
+- compilation-provenance
+- optimization-level-classification
+- pytorch
+- roberta
+metrics:
+- accuracy
+---
+
+# BinProv Opt4
+
+`binprov` classifies the compiler optimization level of x86-64 ELF
+binaries from raw `.text` bytes: O0, O1, O2, O3.
+It follows the BinProv method and does not disassemble the input.
+
+This is a BinProv-native PyTorch checkpoint. Its byte encoder is RoBERTa-shaped,
+but its pooling and classification head are custom, so it is not compatible
+with `transformers.AutoModel`. Use the bundled `inference.py`; no remote code or
+`trust_remote_code` is involved.
+
+## Model details
+
+- Architecture: RoBERTa-style byte encoder, seq 2048 bytes, 12 layers x 768 hidden; 261-token byte vocabulary
+- Pooling: `border` with 4 heads
+- Input: 2048-byte windows at a 512-byte evaluation stride
+- Output: four probabilities corresponding to O0, O1, O2, and O3
+- Framework: PyTorch with Hugging Face Transformers components
+- License: MIT
+
+The model follows [BinProv: Binary Code Provenance Identification without
+Disassembly](https://doi.org/10.1145/3545948.3545956), RAID 2022. This checkpoint
+comes from a clean reproduction run of the repository's best single-model
+recipe.
+Evaluation (this export): sequence accuracy **84.2255%**. Binary-level accuracy **93.8830%**. A fresh training run's numbers vary by ~1-2 points with seed.
+
+## Direct use
+
+```bash
+python -m venv .venv
+source .venv/bin/activate                    # Windows: .venv\Scripts\activate
+pip install -r requirements.txt
+
+python inference.py --model model --elf /path/to/binary.elf
+python inference.py --model model --text-bytes raw_text.bin --json
+```
+
+The script extracts `.text` from an ELF or accepts already extracted bytes. It
+evaluates overlapping 2048-byte inputs every 512 bytes, then averages the
+per-window probabilities for one binary. It automatically selects CUDA, Apple
+MPS, or CPU. Use `--batch-size` to trade memory for speed.
+
+## Training data
+
+The model was trained and evaluated on the x86-64 subset of BinKit Normal,
+covering GCC and Clang binaries at optimization levels O0 through O3. Splits are
+grouped by source program so the same program does not cross train and test.
+Exact training arguments and environment versions are in `metadata.json` and
+`evidence/args.json`.
+
+## Evaluation
+
+| metric | value |
+|---|---:|
+| sequence accuracy | **84.2255%** |
+| balanced sequence accuracy | **83.0951%** |
+| binary soft-vote accuracy | **93.8830%** |
+| test windows | 116,321 |
+| test binaries | 376 |
+
+The sequence metric uses 512-byte target positions with 2048-byte overlapping
+model inputs. Binary accuracy averages the saved per-window probabilities. The
+recomputed evaluation in `evidence/evaluation.json` matches the recorded test
+accuracy. Results can vary with retraining and should be compared on the same
+program-grouped split.
+
+## Intended use and limitations
+
+This checkpoint is intended for research and analysis of compiler optimization
+provenance in ordinary x86-64 ELF code produced by toolchains represented in
+BinKit. It has not been validated as a general malware, authorship, attribution,
+or security-decision system. Packed, obfuscated, self-modifying, non-ELF, other
+architecture, and out-of-distribution toolchain inputs may be unreliable. The
+softmax scores are not calibrated confidence estimates.
+
+## Files
+
+- `model/` — `model.pt`, `binprov_config.json`, `head.json` (BinProv checkpoint)
+- `binprov/` — minimal source package needed by `inference.py`
+- `requirements.txt` — runtime dependencies
+- `labels.json` — class id -> label
+- `metadata.json` — exact training args + eval results + environment
+- `environment.json` — runtime versions
+- `evidence/` — args, result, evaluation report, and saved probabilities
+- `LICENSE` — repository license
+
+## Citation
+
+```bibtex
+@inproceedings{he2022binprov,
+  title={BinProv: Binary Code Provenance Identification without Disassembly},
+  author={He, Xu and Wang, Shu and Xing, Yunlong and Feng, Pengbin and Wang, Haining and Li, Qi and Chen, Songqing and Sun, Kun},
+  booktitle={RAID},
+  year={2022},
+  doi={10.1145/3545948.3545956}
+}
+```

binprov/__init__.py

@@ -0,0 +1,37 @@
+"""BinProv — compilation provenance identification straight from binary code.
+
+A re-implementation of the experiment pipeline from:
+
+    Xu He, Shu Wang, Yunlong Xing, Pengbin Feng, Haining Wang, Qi Li,
+    Songqing Chen, Kun Sun. "BinProv: Binary Code Provenance Identification
+    without Disassembly." RAID 2022. https://doi.org/10.1145/3545948.3545956
+
+The pipeline, mirroring the paper's four stages (§3):
+
+    1. pre-process  .text section -> fixed-length byte sequences
+    2. embed        byte-level RoBERTa/BERT encoder, MLM pre-trained
+    3. classify     2-layer FC head, fine-tuned per provenance task
+    4. joint infer  majority vote over sequences of a function / a binary
+
+The original code was lost; this rebuild follows the paper and marks every place
+where the paper leaves a detail unspecified — see docs/REPRODUCTION.md.
+"""
+
+from __future__ import annotations
+
+__version__ = "0.1.0"
+
+# Only the torch-free modules are imported eagerly, so that corpus building and
+# inspection work in an environment without torch/transformers installed.
+# `binprov.model` and `binprov.data` are imported directly where needed.
+from . import corpus, discover, elf, metrics, provenance, vocab, vote
+
+__all__ = [
+    "corpus",
+    "discover",
+    "elf",
+    "metrics",
+    "provenance",
+    "vocab",
+    "vote",
+]

binprov/corpus.py

@@ -0,0 +1,445 @@
+"""The packed corpus: the only thing BinProv needs to keep on disk.
+
+Motivation is disk space. A BinKit binary averages ~200 KB, but BinProv reads
+nothing except the ``.text`` bytes and (for function-level voting) the function
+boundaries. So corpus building concatenates every ``.text`` section into one
+flat ``uint8`` file and records offsets alongside — after which **the binaries
+themselves can be deleted**. In practice that is a 3-5x reduction versus keeping
+the ELF files, and it makes training I/O a single memory-mapped sequential read.
+
+On-disk layout of a corpus directory::
+
+    text.u8            all .text sections concatenated, no padding
+    binaries.jsonl.gz  one JSON record per binary (offset, length, labels, funcs)
+    meta.json          corpus-level counts, label tallies, build parameters
+    splits/<name>.json binary-id lists for train/test/pretrain
+
+Sequences (the paper's fixed-length 512-byte units, §3.1) are *not* stored.
+They are cut on the fly by :meth:`Corpus.sequences`, so changing the sequence
+length or stride costs nothing and needs no rebuild.
+"""
+
+from __future__ import annotations
+
+import gzip
+import json
+import os
+import random
+from dataclasses import asdict, dataclass, field
+from pathlib import Path
+
+import numpy as np
+
+TEXT_FILE = "text.u8"
+INDEX_FILE = "binaries.jsonl.gz"
+META_FILE = "meta.json"
+SPLIT_DIR = "splits"
+
+
+@dataclass
+class BinaryRecord:
+    """One binary's slice of the packed corpus."""
+
+    bid: int
+    path: str  # original path, for traceability only
+    package: str  # e.g. "coreutils-8.24"
+    program: str  # e.g. "ls" — the grouping key for leak-free splits
+    labels: dict  # see provenance.Labels.as_dict()
+    text_off: int  # offset into text.u8
+    text_len: int
+    text_vaddr: int  # virtual address of .text[0], for symbol mapping
+    arch_elf: str  # architecture as read from the ELF header
+    stripped: bool
+    functions: list = field(default_factory=list)  # [[off, size], ...] within .text
+    func_names: list = field(default_factory=list)  # parallel to functions, may be empty
+
+    @property
+    def group(self) -> str:
+        """Split-grouping key: all compiled variants of one program share it."""
+        return f"{self.package}/{self.program}"
+
+
+# ---------------------------------------------------------------------------
+# writing
+# ---------------------------------------------------------------------------
+
+
+class CorpusWriter:
+    """Streams ``.text`` sections into a packed corpus.
+
+    Use as a context manager. ``text.u8`` is appended to as we go, so peak
+    memory stays at one binary regardless of corpus size.
+    """
+
+    def __init__(self, out_dir: str | os.PathLike, *, build_params: dict | None = None):
+        self.dir = Path(out_dir)
+        self.dir.mkdir(parents=True, exist_ok=True)
+        self._text = open(self.dir / TEXT_FILE, "wb")
+        self._index = gzip.open(self.dir / INDEX_FILE, "wt", compresslevel=6)
+        self._offset = 0
+        self._n = 0
+        self._build_params = dict(build_params or {})
+        self._label_counts: dict[str, dict[str, int]] = {}
+
+    def add(
+        self,
+        text: bytes,
+        *,
+        path: str,
+        package: str,
+        program: str,
+        labels: dict,
+        text_vaddr: int,
+        arch_elf: str,
+        stripped: bool,
+        functions: list | None = None,
+        func_names: list | None = None,
+    ) -> BinaryRecord:
+        rec = BinaryRecord(
+            bid=self._n,
+            path=path,
+            package=package,
+            program=program,
+            labels=labels,
+            text_off=self._offset,
+            text_len=len(text),
+            text_vaddr=text_vaddr,
+            arch_elf=arch_elf,
+            stripped=stripped,
+            functions=functions or [],
+            func_names=func_names or [],
+        )
+        self._text.write(text)
+        self._index.write(json.dumps(asdict(rec), separators=(",", ":")) + "\n")
+        self._offset += len(text)
+        self._n += 1
+        for key in ("arch", "compiler", "opt", "extra", "obfuscation"):
+            val = labels.get(key)
+            if val is not None:
+                self._label_counts.setdefault(key, {})
+                self._label_counts[key][val] = self._label_counts[key].get(val, 0) + 1
+        return rec
+
+    @property
+    def num_binaries(self) -> int:
+        return self._n
+
+    @property
+    def num_bytes(self) -> int:
+        return self._offset
+
+    def close(self) -> None:
+        self._text.close()
+        self._index.close()
+        meta = {
+            "num_binaries": self._n,
+            "num_text_bytes": self._offset,
+            "label_counts": self._label_counts,
+            "build_params": self._build_params,
+        }
+        (self.dir / META_FILE).write_text(json.dumps(meta, indent=2) + "\n")
+
+    def __enter__(self) -> CorpusWriter:
+        return self
+
+    def __exit__(self, *exc) -> None:
+        self.close()
+
+
+# ---------------------------------------------------------------------------
+# reading
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class SequenceIndex:
+    """A set of fixed-length byte sequences cut out of a corpus.
+
+    Arrays are parallel and all the same length:
+
+        bid    which binary each sequence came from
+        start  absolute offset into ``text.u8``
+        length valid bytes (< seq_len only for the tail of a unit)
+        unit   index into ``record.functions``, or -1 for a binary-level cut
+
+    ``unit`` is what makes function-level majority voting (§3.4) possible: it
+    identifies the entity a sequence votes for.
+    """
+
+    bid: np.ndarray
+    start: np.ndarray
+    length: np.ndarray
+    unit: np.ndarray
+    seq_len: int
+    level: str  # "binary" or "function"
+
+    def __len__(self) -> int:
+        return int(self.bid.shape[0])
+
+    def group_ids(self) -> np.ndarray:
+        """Integer id of the entity each sequence votes for.
+
+        Binary level: one group per binary. Function level: one group per
+        (binary, function) pair.
+        """
+        if self.level == "binary":
+            keys = self.bid.astype(np.int64)
+        else:
+            keys = self.bid.astype(np.int64) * (2**24) + (self.unit.astype(np.int64) + 1)
+        # dense re-indexing keeps downstream bincount cheap
+        _, inverse = np.unique(keys, return_inverse=True)
+        return inverse.astype(np.int64)
+
+    def subset(self, mask: np.ndarray) -> SequenceIndex:
+        return SequenceIndex(
+            bid=self.bid[mask],
+            start=self.start[mask],
+            length=self.length[mask],
+            unit=self.unit[mask],
+            seq_len=self.seq_len,
+            level=self.level,
+        )
+
+
+class Corpus:
+    """Read-only view over a packed corpus directory."""
+
+    def __init__(self, path: str | os.PathLike):
+        self.dir = Path(path)
+        if not (self.dir / TEXT_FILE).exists():
+            raise FileNotFoundError(f"{self.dir} is not a corpus (no {TEXT_FILE})")
+        self.meta = json.loads((self.dir / META_FILE).read_text())
+        self.records: list[BinaryRecord] = []
+        with gzip.open(self.dir / INDEX_FILE, "rt") as fh:
+            for line in fh:
+                self.records.append(BinaryRecord(**json.loads(line)))
+        # np.memmap keeps the whole thing out of RAM; the OS page cache does the
+        # right thing for the sequential-ish access pattern of training.
+        self.text = np.memmap(self.dir / TEXT_FILE, dtype=np.uint8, mode="r")
+
+    def __len__(self) -> int:
+        return len(self.records)
+
+    def __repr__(self) -> str:
+        mb = self.text.shape[0] / 1e6
+        return f"<Corpus {self.dir.name}: {len(self.records)} binaries, {mb:.1f} MB .text>"
+
+    # -- selection ---------------------------------------------------------
+
+    def filter_bids(self, **constraints) -> list[int]:
+        """Binary ids whose labels match every constraint.
+
+        A constraint value may be a scalar or a collection; ``None`` means "no
+        constraint". Example::
+
+            corpus.filter_bids(arch="x86_64", extra="normal", opt=("O2", "O3"))
+        """
+        out = []
+        for rec in self.records:
+            ok = True
+            for key, want in constraints.items():
+                if want is None:
+                    continue
+                got = rec.labels.get(key)
+                if isinstance(want, (list, tuple, set, frozenset)):
+                    if got not in want:
+                        ok = False
+                        break
+                elif got != want:
+                    ok = False
+                    break
+            if ok:
+                out.append(rec.bid)
+        return out
+
+    # -- sequence cutting --------------------------------------------------
+
+    def sequences(
+        self,
+        *,
+        seq_len: int = 512,
+        stride: int | None = None,
+        level: str = "binary",
+        bids: list[int] | None = None,
+        min_bytes: int = 16,
+        max_seqs_per_binary: int | None = None,
+        function_context: bool = False,
+        rng: random.Random | None = None,
+    ) -> SequenceIndex:
+        """Cut the corpus into fixed-length byte sequences.
+
+        Args:
+            seq_len: bytes per sequence (paper uses 512, a multiple of 8 so that
+                fixed-width ARM/MIPS instructions stay aligned).
+            stride: step between sequence starts. Defaults to ``seq_len``
+                (non-overlapping, as in §3.1). A smaller stride is used for
+                function-level evaluation, where most functions are shorter than
+                one sequence and non-overlapping cuts would give nothing to vote
+                over — see docs/REPRODUCTION.md.
+            level: ``"binary"`` cuts the whole ``.text``; ``"function"`` cuts
+                each function separately so no sequence straddles a boundary.
+            min_bytes: drop trailing fragments shorter than this.
+            max_seqs_per_binary: uniformly subsample when a binary is huge.
+                Keeps a corpus-wide cap on training cost without biasing toward
+                long binaries.
+            function_context: at ``level="function"``, give a function shorter
+                than ``seq_len`` a full-length window centred on it rather than
+                its own bytes padded. See the comment at the use site: most
+                functions are far shorter than one sequence, so without this the
+                function level measures a harder input distribution than the
+                sequence level rather than a voted one.
+        """
+        if seq_len <= 0:
+            raise ValueError("seq_len must be positive")
+        step = stride or seq_len
+        if step <= 0:
+            raise ValueError("stride must be positive")
+        if level not in ("binary", "function"):
+            raise ValueError(f"level must be 'binary' or 'function', got {level!r}")
+
+        wanted = self.records if bids is None else [self.records[b] for b in bids]
+        rng = rng or random.Random(0)
+        b_out: list[int] = []
+        s_out: list[int] = []
+        l_out: list[int] = []
+        u_out: list[int] = []
+
+        for rec in wanted:
+            if level == "binary":
+                spans = [(-1, 0, rec.text_len)]
+            else:
+                spans = [(i, off, size) for i, (off, size) in enumerate(rec.functions)]
+
+            local: list[tuple[int, int, int]] = []  # (unit, start_abs, length)
+            for unit, span_off, span_len in spans:
+                if function_context and unit >= 0 and span_len < seq_len:
+                    # A short function padded to seq_len sees far less context
+                    # than an ordinary sequence, which makes it *harder*, not
+                    # easier -- and in BinKit 81.5% of functions are shorter than
+                    # 512 bytes (median 127), so this is the common case. Centre a
+                    # full-length window on the function instead, borrowing
+                    # neighbouring bytes and clamping to the section. The label is
+                    # still the function's, and provenance is a property of the
+                    # whole binary, so the borrowed bytes carry the same label.
+                    centre = span_off + span_len // 2
+                    start = max(0, min(centre - seq_len // 2, rec.text_len - seq_len))
+                    n = min(seq_len, rec.text_len)
+                    local.append((unit, rec.text_off + max(0, start), n))
+                    continue
+                pos = 0
+                while pos < span_len:
+                    n = min(seq_len, span_len - pos)
+                    # keep a short unit whole (a 40-byte helper function is a
+                    # legitimate sample), but drop a mere tail fragment
+                    if n < min_bytes and pos > 0:
+                        break
+                    local.append((unit, rec.text_off + span_off + pos, n))
+                    if n < seq_len:
+                        break
+                    pos += step
+            if max_seqs_per_binary is not None and len(local) > max_seqs_per_binary:
+                local = rng.sample(local, max_seqs_per_binary)
+                local.sort(key=lambda t: t[1])
+            for unit, start, n in local:
+                b_out.append(rec.bid)
+                s_out.append(start)
+                l_out.append(n)
+                u_out.append(unit)
+
+        return SequenceIndex(
+            bid=np.asarray(b_out, dtype=np.int32),
+            start=np.asarray(s_out, dtype=np.int64),
+            length=np.asarray(l_out, dtype=np.int32),
+            unit=np.asarray(u_out, dtype=np.int32),
+            seq_len=seq_len,
+            level=level,
+        )
+
+    # -- splits ------------------------------------------------------------
+
+    def make_splits(
+        self,
+        *,
+        train_ratio: float = 0.8,
+        seed: int = 1234,
+        group_by: str = "program",
+        bids: list[int] | None = None,
+    ) -> dict[str, list[int]]:
+        """Split binary ids into train/test.
+
+        The paper splits 8:2 and states that train and test must not overlap so
+        that generalization is measured honestly (§4.1). ``group_by="program"``
+        goes one step further: every compiled variant of a program (all
+        compilers, all optimization levels) lands on the same side, so the model
+        cannot recognise a test program it has already seen at another
+        optimization level. Use ``group_by="binary"`` for the looser
+        binary-level split.
+        """
+        pool = self.records if bids is None else [self.records[b] for b in bids]
+        rng = random.Random(seed)
+        if group_by == "binary":
+            keys = [str(r.bid) for r in pool]
+        elif group_by == "program":
+            keys = [r.group for r in pool]
+        elif group_by == "package":
+            keys = [r.package for r in pool]
+        else:
+            raise ValueError(f"unknown group_by {group_by!r}")
+
+        uniq = sorted(set(keys))
+        rng.shuffle(uniq)
+        n_train = int(round(len(uniq) * train_ratio))
+        train_keys = set(uniq[:n_train])
+        train, test = [], []
+        for rec, key in zip(pool, keys):
+            (train if key in train_keys else test).append(rec.bid)
+        return {"train": train, "test": test}
+
+    def pretrain_bids(
+        self,
+        *,
+        per_package: int = 1,
+        restrict_to: list[int] | None = None,
+        seed: int = 1234,
+    ) -> list[int]:
+        """Select the MLM pre-training set (paper §4.1).
+
+        "We first construct the pre-training set by selecting at least one
+        binary (2 x 4 variants) from each software project." So: pick
+        ``per_package`` programs per package and take *all* their compiled
+        variants, giving broad coverage of projects at modest size.
+
+        ``restrict_to`` should normally be the training split — pre-training on
+        binaries that later appear in the test set would leak, even though the
+        MLM task has no labels.
+        """
+        allowed = None if restrict_to is None else set(restrict_to)
+        by_pkg: dict[str, dict[str, list[int]]] = {}
+        for rec in self.records:
+            if allowed is not None and rec.bid not in allowed:
+                continue
+            by_pkg.setdefault(rec.package, {}).setdefault(rec.program, []).append(rec.bid)
+
+        rng = random.Random(seed)
+        out: list[int] = []
+        for pkg in sorted(by_pkg):
+            programs = sorted(by_pkg[pkg])
+            rng.shuffle(programs)
+            for prog in programs[:per_package]:
+                out.extend(by_pkg[pkg][prog])
+        return sorted(out)
+
+    def save_splits(self, name: str, splits: dict[str, list[int]]) -> Path:
+        d = self.dir / SPLIT_DIR
+        d.mkdir(exist_ok=True)
+        path = d / f"{name}.json"
+        path.write_text(json.dumps({k: sorted(v) for k, v in splits.items()}, indent=1))
+        return path
+
+    def load_splits(self, name: str) -> dict[str, list[int]]:
+        path = self.dir / SPLIT_DIR / f"{name}.json"
+        if not path.exists():
+            raise FileNotFoundError(
+                f"no split {name!r} in {self.dir}; create one with scripts/build_corpus.py"
+            )
+        return json.loads(path.read_text())

binprov/data.py

@@ -0,0 +1,367 @@
+"""Torch datasets and collators over a packed corpus.
+
+The dataset returns raw byte slices; tokenization and MLM masking happen in the
+collator, vectorised over the batch. That keeps DataLoader workers doing almost
+nothing but memory-mapped reads, and it makes the masking *dynamic*: a fresh
+random mask every time a sequence is drawn, which is what the paper means by
+"these masked bytes are different at each epoch" (§4.1).
+"""
+
+from __future__ import annotations
+
+import numpy as np
+import torch
+from torch.utils.data import Dataset
+
+from .corpus import Corpus, SequenceIndex
+from .provenance import Task
+from .vocab import BYTE_OFFSET, EOS_ID, BOS_ID, MASK_ID, PAD_ID, VOCAB_SIZE
+
+IGNORE_INDEX = -100
+
+
+def sequence_labels(corpus: Corpus, index: SequenceIndex, task: Task) -> np.ndarray:
+    """Per-sequence class index, ``-1`` where the binary is out of task scope."""
+    per_bid = np.full(len(corpus.records), -1, dtype=np.int64)
+    for rec in corpus.records:
+        lab = task.label_of(rec.labels)
+        if lab is not None:
+            per_bid[rec.bid] = lab
+    return per_bid[index.bid]
+
+
+def drop_unlabelled(index: SequenceIndex, labels: np.ndarray) -> tuple[SequenceIndex, np.ndarray]:
+    """Filter out sequences whose binary the task does not cover.
+
+    Needed for the O0/O1 and O2/O3 tasks of Table 4, which each use half the
+    optimization levels.
+    """
+    keep = labels >= 0
+    return index.subset(keep), labels[keep]
+
+
+class ByteSequenceDataset(Dataset):
+    """Fixed-length byte sequences cut from a corpus.
+
+    Yields ``(bytes, label, position)`` where ``position`` is the sequence's
+    index in the :class:`SequenceIndex` — evaluation needs it to map predictions
+    back to the voting group.
+    """
+
+    def __init__(self, corpus: Corpus, index: SequenceIndex, labels: np.ndarray | None = None):
+        if labels is not None and len(labels) != len(index):
+            raise ValueError(f"labels ({len(labels)}) and index ({len(index)}) disagree")
+        self.text = corpus.text
+        self.index = index
+        self.labels = labels
+
+    def __len__(self) -> int:
+        return len(self.index)
+
+    def __getitem__(self, i: int):
+        start = int(self.index.start[i])
+        n = int(self.index.length[i])
+        chunk = np.asarray(self.text[start : start + n], dtype=np.uint8)
+        label = -1 if self.labels is None else int(self.labels[i])
+        return chunk, label, i
+
+
+
+class JitteredByteSequenceDataset(ByteSequenceDataset):
+    """Training sequences whose window start is re-drawn every time they are read.
+
+    Motivation: a non-overlapping cut (§3.1) gives a *fixed* set of windows, so
+    an N-epoch fine-tune shows the model the identical byte strings N times. The
+    corpus holds 211 MB of ``.text`` but only 315,710 distinct 512-byte windows,
+    and where a window happens to begin is an artefact of the cut, not a
+    property of the code. Re-drawing the start turns those 315,710 samples into
+    a continuum, and it also stops the model keying on "byte 0 of a window is
+    usually an instruction boundary" -- which is true of the training cut and
+    only accidentally true at deployment.
+
+    ``jitter`` is the maximum shift in bytes. ``jitter < 0`` means "start
+    anywhere in this binary's ``.text``", the strongest form. The window is
+    always clamped to stay inside the binary it came from, so a jittered sample
+    never mixes two binaries and its label stays correct.
+    """
+
+    def __init__(
+        self,
+        corpus: Corpus,
+        index: SequenceIndex,
+        labels: np.ndarray | None = None,
+        *,
+        jitter: int = 256,
+        seed: int = 0,
+    ):
+        super().__init__(corpus, index, labels)
+        self.jitter = jitter
+        self._rng = np.random.default_rng(seed)
+        # Per-sequence bounds on the window start, so the clamp needs no lookup
+        # into corpus.records at __getitem__ time (which runs in a worker).
+        off = np.asarray([r.text_off for r in corpus.records], dtype=np.int64)
+        ln = np.asarray([r.text_len for r in corpus.records], dtype=np.int64)
+        bid = index.bid.astype(np.int64)
+        self._lo = off[bid]
+        self._hi = np.maximum(off[bid], off[bid] + ln[bid] - index.length.astype(np.int64))
+
+    def __getitem__(self, i: int):
+        start = int(self.index.start[i])
+        n = int(self.index.length[i])
+        lo, hi = int(self._lo[i]), int(self._hi[i])
+        if self.jitter < 0:
+            start = int(self._rng.integers(lo, hi + 1)) if hi > lo else lo
+        elif self.jitter > 0:
+            shift = int(self._rng.integers(-self.jitter, self.jitter + 1))
+            start = min(max(start + shift, lo), hi)
+        chunk = np.asarray(self.text[start : start + n], dtype=np.uint8)
+        label = -1 if self.labels is None else int(self.labels[i])
+        return chunk, label, i
+
+
+
+class ContextWindowDataset(ByteSequenceDataset):
+    """The paper's window set, each window widened to ``length`` centred bytes.
+
+    This exists to make a longer-input model *comparable*. Training on 2048-byte
+    sequences changes the unit being classified, so per-sequence accuracy at 2048
+    bytes and at 512 bytes are different metrics and must not be put in the same
+    column. Keeping the 512-byte index and only widening what the model reads
+    around each window fixes that: the row count, the labels and the voting
+    groups are identical to the baseline's, and the only thing that changed is
+    how much context the prediction had.
+
+    It is the same manoeuvre as ``--function-context``, and it carries the same
+    caveat: the widened window contains bytes the 512-byte window does not, so
+    this is not the paper's sequence level. Provenance is a property of the whole
+    binary, so those bytes share the label and this is not label leakage — but it
+    is a different, clearly-labelled operating point.
+    """
+
+    def __init__(self, corpus: Corpus, index: SequenceIndex, labels=None, *, length: int = 2048):
+        super().__init__(corpus, index, labels)
+        self.length = length
+        off = np.asarray([r.text_off for r in corpus.records], dtype=np.int64)
+        ln = np.asarray([r.text_len for r in corpus.records], dtype=np.int64)
+        bid = index.bid.astype(np.int64)
+        self._lo = off[bid]
+        self._hi = off[bid] + ln[bid]
+
+    def __getitem__(self, i: int):
+        start = int(self.index.start[i])
+        n = int(self.index.length[i])
+        lo, hi = int(self._lo[i]), int(self._hi[i])
+        want = min(self.length, hi - lo)
+        centre = start + n // 2
+        s = min(max(centre - want // 2, lo), max(lo, hi - want))
+        chunk = np.asarray(self.text[s : s + want], dtype=np.uint8)
+        label = -1 if self.labels is None else int(self.labels[i])
+        return chunk, label, i
+
+class MultiViewByteSequenceDataset(ByteSequenceDataset):
+    """One sequence, several views of it — the evaluation side of jitter.
+
+    Item ``i`` of this dataset is view ``i % n_views`` of sequence
+    ``i // n_views``, so a plain non-shuffled pass yields every view of every
+    sequence and the caller averages the ``n_views`` probability rows back down.
+    The ``positions`` field still carries ``i``, so :func:`engine.predict`
+    scatters correctly without knowing anything about views.
+
+    Two modes, kept separate because they differ in what information the
+    prediction is allowed to use:
+
+    ``crop``
+        Views are sub-windows *inside* the sequence's own bytes (length
+        ``seq_len - k*shift`` for view ``k``). Nothing outside the 512 bytes is
+        read, so an averaged prediction is still a sequence-level prediction and
+        is comparable to the paper's number.
+
+    ``shift``
+        Views slide the full-length window off the sequence's start by up to
+        ``span`` bytes, clamped to the binary. This reads bytes the sequence does
+        not contain, so it is *not* the paper's sequence level -- report it as
+        its own row, the way ``--function-context`` is.
+    """
+
+    def __init__(
+        self,
+        corpus: Corpus,
+        index: SequenceIndex,
+        labels: np.ndarray | None = None,
+        *,
+        mode: str = "crop",
+        n_views: int = 4,
+        span: int = 128,
+    ):
+        super().__init__(corpus, index, labels)
+        if mode not in ("crop", "shift"):
+            raise ValueError(f"mode must be 'crop' or 'shift', got {mode!r}")
+        if n_views < 1:
+            raise ValueError("n_views must be >= 1")
+        self.mode = mode
+        self.n_views = n_views
+        self.span = span
+        off = np.asarray([r.text_off for r in corpus.records], dtype=np.int64)
+        ln = np.asarray([r.text_len for r in corpus.records], dtype=np.int64)
+        bid = index.bid.astype(np.int64)
+        self._lo = off[bid]
+        self._hi = np.maximum(off[bid], off[bid] + ln[bid] - index.length.astype(np.int64))
+
+    def __len__(self) -> int:
+        return len(self.index) * self.n_views
+
+    def _offsets(self) -> list[int]:
+        """Signed shifts for the views, symmetric around 0 and including 0."""
+        if self.n_views == 1:
+            return [0]
+        half = self.n_views // 2
+        step = max(1, self.span // max(1, half))
+        out = [0]
+        for k in range(1, half + 1):
+            out.append(-k * step)
+            if len(out) < self.n_views:
+                out.append(k * step)
+        return out[: self.n_views]
+
+    def __getitem__(self, j: int):
+        i, view = divmod(j, self.n_views)
+        start = int(self.index.start[i])
+        n = int(self.index.length[i])
+        delta = self._offsets()[view]
+        if self.mode == "crop":
+            # keep the window inside its own bytes: trim from one end
+            trim = abs(delta)
+            if delta >= 0:
+                start, n = start + trim, max(16, n - trim)
+            else:
+                n = max(16, n - trim)
+        else:
+            start = min(max(start + delta, int(self._lo[i])), int(self._hi[i]))
+        chunk = np.asarray(self.text[start : start + n], dtype=np.uint8)
+        label = -1 if self.labels is None else int(self.labels[i])
+        return chunk, label, j
+
+class PairedByteSequenceDataset(ByteSequenceDataset):
+    """Optionally splices two half-sequences from different binaries.
+
+    The paper's segment embedding ``E_s`` exists to mark "which binary program
+    each byte belongs to, when a byte sequence contains multiple fragments from
+    different programs" (§3.2). With plain single-binary sequences that embedding
+    never varies and stays dead weight. Setting ``pair_prob > 0`` during
+    pre-training makes it meaningful. Off by default, since the paper does not
+    say it used spliced inputs.
+    """
+
+    def __init__(self, corpus: Corpus, index: SequenceIndex, *, pair_prob: float = 0.0, seed: int = 0):
+        super().__init__(corpus, index, None)
+        self.pair_prob = pair_prob
+        self._rng = np.random.default_rng(seed)
+
+    def __getitem__(self, i: int):
+        chunk, _, _ = super().__getitem__(i)
+        if self.pair_prob <= 0 or self._rng.random() >= self.pair_prob or len(chunk) < 16:
+            return chunk, -1, i
+        j = int(self._rng.integers(0, len(self.index)))
+        other, _, _ = super().__getitem__(j)
+        if self.index.bid[j] == self.index.bid[i] or len(other) < 16:
+            return chunk, -1, i
+        cut = len(chunk) // 2
+        spliced = np.concatenate([chunk[:cut], other[: len(chunk) - cut]])
+        # negative position marks "segment boundary at `cut`" for the collator
+        return spliced, -(cut + 1), i
+
+
+def _tokenize_batch(batch, seq_tokens: int):
+    """Build ``(input_ids, attention_mask, token_type_ids)`` for a batch.
+
+    Layout per row: ``<s> b0 b1 ... bn-1 </s> <pad> ...``
+    """
+    bsz = len(batch)
+    ids = np.full((bsz, seq_tokens), PAD_ID, dtype=np.int64)
+    attn = np.zeros((bsz, seq_tokens), dtype=np.int64)
+    types = np.zeros((bsz, seq_tokens), dtype=np.int64)
+
+    max_bytes = seq_tokens - 2
+    for row, (chunk, label, _pos) in enumerate(batch):
+        n = min(len(chunk), max_bytes)
+        ids[row, 0] = BOS_ID
+        ids[row, 1 : 1 + n] = chunk[:n].astype(np.int64) + BYTE_OFFSET
+        ids[row, 1 + n] = EOS_ID
+        attn[row, : n + 2] = 1
+        if label is not None and label < -1:  # spliced pair, see dataset above
+            cut = -label - 1
+            types[row, 1 + cut : 1 + n] = 1
+    return ids, attn, types
+
+
+class MLMCollator:
+    """Masked-language-model batches (paper §3.2 / §4.1).
+
+    Masking follows Pei et al. as the paper states: 20% of bytes are chosen; of
+    those, 50% become ``<mask>`` and 50% become a random byte value. Note there
+    is no BERT-style "keep original 10%" bucket here — that is the paper's
+    setting, not an omission.
+    """
+
+    def __init__(
+        self,
+        seq_tokens: int,
+        *,
+        mask_prob: float = 0.20,
+        mask_replace: float = 0.5,
+        random_replace: float = 0.5,
+        seed: int | None = None,
+    ):
+        if mask_replace + random_replace > 1.0 + 1e-9:
+            raise ValueError("mask_replace + random_replace must not exceed 1")
+        self.seq_tokens = seq_tokens
+        self.mask_prob = mask_prob
+        self.mask_replace = mask_replace
+        self.random_replace = random_replace
+        self.rng = np.random.default_rng(seed)
+
+    def __call__(self, batch):
+        ids, attn, types = _tokenize_batch(batch, self.seq_tokens)
+
+        # only real byte tokens are maskable: never <s>, </s> or padding
+        maskable = (ids >= BYTE_OFFSET) & (attn == 1)
+        selected = maskable & (self.rng.random(ids.shape) < self.mask_prob)
+
+        labels = np.full(ids.shape, IGNORE_INDEX, dtype=np.int64)
+        labels[selected] = ids[selected]
+
+        draw = self.rng.random(ids.shape)
+        to_mask = selected & (draw < self.mask_replace)
+        to_random = selected & (draw >= self.mask_replace) & (
+            draw < self.mask_replace + self.random_replace
+        )
+        ids[to_mask] = MASK_ID
+        n_rand = int(to_random.sum())
+        if n_rand:
+            ids[to_random] = self.rng.integers(BYTE_OFFSET, VOCAB_SIZE, size=n_rand)
+
+        return {
+            "input_ids": torch.from_numpy(ids),
+            "attention_mask": torch.from_numpy(attn),
+            "token_type_ids": torch.from_numpy(types),
+            "labels": torch.from_numpy(labels),
+        }
+
+
+class ClassificationCollator:
+    """Batches for fine-tuning and evaluation."""
+
+    def __init__(self, seq_tokens: int):
+        self.seq_tokens = seq_tokens
+
+    def __call__(self, batch):
+        ids, attn, types = _tokenize_batch(batch, self.seq_tokens)
+        labels = np.asarray([max(b[1], 0) for b in batch], dtype=np.int64)
+        positions = np.asarray([b[2] for b in batch], dtype=np.int64)
+        return {
+            "input_ids": torch.from_numpy(ids),
+            "attention_mask": torch.from_numpy(attn),
+            "token_type_ids": torch.from_numpy(types),
+            "labels": torch.from_numpy(labels),
+            "positions": torch.from_numpy(positions),
+        }

binprov/discover.py

@@ -0,0 +1,217 @@
+"""Find labelled binaries in a directory tree.
+
+Two layouts are handled, because the two data paths differ:
+
+**BinKit** puts the whole toolchain in the *filename*, two levels deep::
+
+    gnu_debug/a2ps/a2ps-4.14_clang-7.0_x86_64_O2_a2ps.elf
+              |    |         |         |       |  `-- program
+              |    |         |         |       `-- optimization level
+              |    |         |         `-- architecture
+              |    |         `-- compiler and version
+              |    `-- package and version
+              `-- package directory
+
+**scripts/build_local_dataset.py** puts it in a directory instead::
+
+    <root>/<package>/<arch>-<compiler>-<version>-<opt>/<program>
+
+Labels come from :func:`binprov.provenance.parse_labels`, which scans every path
+component for recognisable tokens and so copes with both. Verified against all
+67,680 paths of BinKit Normal: architecture, compiler, compiler version and
+optimization level are recovered correctly for every one.
+
+The *program name* needs more care than the labels do. Under the BinKit layout a
+naive ``path.name`` makes each compiled variant look like a different program,
+which would silently defeat the program-grouped train/test split — the same
+binary at O2 could train while its O3 twin is tested. :func:`program_name`
+therefore parses the filename grammar to recover the real program.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+from collections import Counter
+from dataclasses import dataclass
+from pathlib import Path
+
+from . import elf
+from .provenance import Labels, parse_labels, toolchain_key
+
+#: never worth parsing — build artefacts, sources, docs
+SKIP_SUFFIXES = frozenset(
+    """.c .h .cc .cpp .hpp .py .sh .m4 .am .in .ac .md .txt .log .json .yaml .yml
+    .gz .xz .bz2 .zip .tar .patch .diff .po .pot .1 .3 .html .css .js .png .jpg
+    .o .lo .la .a .cmake .pc .mk .d .gmo .info .texi""".split()
+)
+
+
+@dataclass
+class Candidate:
+    """A file on disk that looks like a labelled binary."""
+
+    path: Path
+    package: str
+    program: str
+    labels: Labels
+
+    @property
+    def toolchain(self) -> str:
+        return toolchain_key(self.labels.as_dict())
+
+
+def _looks_like_source_tree(name: str) -> bool:
+    return name in {".git", ".svn", "autom4te.cache", "__pycache__", "node_modules"}
+
+
+# BinKit's filename grammar:
+#   {package}-{version}_{compiler}-{cver}_{arch}_{width}_{opt}_{program}
+# The program is the trailing group, so names containing underscores survive
+# (gdbm_dump, gdbm_load are the two real cases in BinKit Normal).
+_BINKIT_STEM = re.compile(
+    r"^(?P<pkgver>.+?)"
+    r"_(?P<compiler>(?:gcc|clang|g\+\+|clang\+\+)-[0-9][0-9.]*)"
+    r"_(?P<arch>(?:x86|x64|i386|arm|aarch64|mips|mipseb|mipsel|ppc|riscv)_(?:32|64))"
+    r"_(?P<opt>O[0-3s]|Ofast)"
+    r"_(?P<program>.+)$"
+)
+
+
+def program_name(path: Path) -> str:
+    """The program a binary was built from, stripped of provenance tokens.
+
+    Returns the trailing program field for a BinKit-style filename, and the plain
+    filename otherwise (the directory-based layout, where the filename already
+    *is* the program). This is the grouping key for leak-free splits, so getting
+    it wrong does not fail loudly — it just quietly inflates accuracy.
+    """
+    stem = path.name
+    for suffix in (".elf", ".exe", ".so"):
+        if stem.endswith(suffix):
+            stem = stem[: -len(suffix)]
+            break
+    m = _BINKIT_STEM.match(stem)
+    return m.group("program") if m else path.name
+
+
+def walk(root: str | os.PathLike, *, follow_symlinks: bool = False):
+    """Yield paths of ELF files under ``root``, cheaply.
+
+    Filters by extension first and reads only 4 bytes to confirm the ELF magic,
+    so walking a build tree full of objects and sources stays fast.
+    """
+    root = Path(root)
+    for dirpath, dirnames, filenames in os.walk(root, followlinks=follow_symlinks):
+        dirnames[:] = [d for d in dirnames if not _looks_like_source_tree(d)]
+        for fn in filenames:
+            p = Path(dirpath) / fn
+            if p.suffix.lower() in SKIP_SUFFIXES:
+                continue
+            if p.is_symlink() and not follow_symlinks:
+                continue
+            try:
+                if p.stat().st_size < 128:
+                    continue
+            except OSError:
+                continue
+            if elf.is_elf(p):
+                yield p
+
+
+def discover(
+    root: str | os.PathLike,
+    *,
+    arch=None,
+    compiler=None,
+    compiler_version=None,
+    opt=None,
+    extra=None,
+    obfuscation=None,
+    require_complete: bool = True,
+    per_toolchain_limit: int | None = None,
+    packages=None,
+):
+    """Yield :class:`Candidate` objects for binaries matching the filters.
+
+    Filters accept a scalar or a collection; ``None`` means "no constraint".
+    They are applied *before* the file is read, which is the whole point — the
+    paper uses 4 architectures x 2 compilers x 4 opt levels, i.e. 32 of BinKit
+    Normal's 288 toolchains, so filtering here avoids touching ~89% of the
+    dataset.
+
+    Args:
+        per_toolchain_limit: keep at most this many binaries per compilation
+            configuration. The cheapest way to bound corpus size while staying
+            balanced across configurations.
+        packages: restrict to these package names.
+    """
+    root = Path(root)
+    want = {
+        "arch": arch,
+        "compiler": compiler,
+        "compiler_version": compiler_version,
+        "opt": opt,
+        "extra": extra,
+        "obfuscation": obfuscation,
+    }
+    pkg_filter = None if packages is None else set(packages)
+    seen_per_toolchain: Counter[str] = Counter()
+
+    for path in walk(root):
+        rel = path.relative_to(root)
+        parts = rel.parts
+        package = parts[0] if len(parts) > 1 else "_root"
+        if pkg_filter is not None and package not in pkg_filter:
+            continue
+
+        labels = parse_labels(str(rel))
+        if require_complete and not labels.is_complete():
+            continue
+
+        d = labels.as_dict()
+        if not _matches(d, want):
+            continue
+
+        if per_toolchain_limit is not None:
+            key = toolchain_key(d)
+            if seen_per_toolchain[key] >= per_toolchain_limit:
+                continue
+            seen_per_toolchain[key] += 1
+
+        yield Candidate(
+            path=path, package=package, program=program_name(path), labels=labels
+        )
+
+
+def _matches(got: dict, want: dict) -> bool:
+    for key, expected in want.items():
+        if expected is None:
+            continue
+        value = got.get(key)
+        if isinstance(expected, (list, tuple, set, frozenset)):
+            if value not in expected:
+                return False
+        elif value != expected:
+            return False
+    return True
+
+
+def summarize(candidates) -> str:
+    """A short report of what discovery found, for ``--dry-run``."""
+    cands = list(candidates)
+    if not cands:
+        return "no binaries matched"
+    by_tc = Counter(c.toolchain for c in cands)
+    by_pkg = Counter(c.package for c in cands)
+    lines = [
+        f"{len(cands)} binaries, {len(by_pkg)} packages, {len(by_tc)} toolchains",
+        "",
+        "toolchains (count):",
+    ]
+    for tc, n in sorted(by_tc.items()):
+        lines.append(f"  {n:7d}  {tc}")
+    lines += ["", "sample paths:"]
+    for c in cands[:8]:
+        lines.append(f"  {c.package} | {c.program} | {c.toolchain}")
+    return "\n".join(lines)

binprov/elf.py

@@ -0,0 +1,313 @@
+"""A small, dependency-free ELF reader.
+
+BinProv's whole input is the ``.text`` section (paper §3.1), plus — only for the
+function-level majority voting of §3.4 — the function boundaries recorded in the
+symbol table. That is all this module extracts. It deliberately avoids
+``pyelftools``/``lief`` so that corpus building has no native dependencies and
+so that the same code path handles every architecture in the paper
+(x86_32, x86_64, ARM_64, MIPS_64), including big-endian MIPS.
+
+Everything is parsed from an in-memory ``bytes``/``memoryview`` buffer.
+Malformed or truncated files raise :class:`ElfError` rather than crashing, so a
+corpus build can skip them and keep going.
+"""
+
+from __future__ import annotations
+
+import struct
+from dataclasses import dataclass, field
+
+ELF_MAGIC = b"\x7fELF"
+
+# section header types we care about
+SHT_SYMTAB = 2
+SHT_DYNSYM = 11
+SHT_NOBITS = 8
+
+STT_FUNC = 2
+
+# e_machine -> canonical architecture name. The width/endianness suffix is
+# resolved from EI_CLASS / EI_DATA, because e.g. EM_MIPS covers 4 variants.
+_MACHINE = {
+    3: "x86",
+    62: "x86",  # EM_X86_64 (width comes from EI_CLASS)
+    40: "arm",
+    183: "arm",  # EM_AARCH64
+    8: "mips",
+    10: "mips",  # EM_MIPS_RS3_LE
+    20: "ppc",
+    21: "ppc",
+    22: "s390",
+    243: "riscv",
+    258: "loongarch",
+}
+
+
+class ElfError(Exception):
+    """Raised for anything that is not a parseable ELF file."""
+
+
+@dataclass(frozen=True)
+class Function:
+    """A function boundary taken from the symbol table."""
+
+    name: str
+    offset: int  # byte offset within the .text section
+    size: int  # size in bytes
+
+
+@dataclass
+class TextSection:
+    """The extracted ``.text`` section and its metadata."""
+
+    data: bytes
+    vaddr: int  # virtual address of .text[0]
+    file_offset: int
+    arch: str  # e.g. "x86_64", "mips_64", "mipseb_64"
+    bits: int  # 32 or 64
+    little_endian: bool
+    functions: list[Function] = field(default_factory=list)
+    comment: str = ""  # contents of .comment, if present
+    stripped: bool = True  # no SHT_SYMTAB present
+
+    def __len__(self) -> int:
+        return len(self.data)
+
+
+def _u(buf: memoryview, fmt: str, off: int):
+    """Unpack ``fmt`` at ``off``, raising ElfError on truncation."""
+    size = struct.calcsize(fmt)
+    if off < 0 or off + size > len(buf):
+        raise ElfError(f"truncated file: want {size} bytes at {off}, have {len(buf)}")
+    return struct.unpack_from(fmt, buf, off)
+
+
+def _cstr(buf: memoryview, off: int) -> str:
+    """Read a NUL-terminated string; tolerates a missing terminator."""
+    if not 0 <= off < len(buf):
+        return ""
+    end = bytes(buf[off:]).find(b"\x00")
+    raw = bytes(buf[off:] if end < 0 else buf[off : off + end])
+    return raw.decode("utf-8", "replace")
+
+
+def _arch_name(machine: int, bits: int, little_endian: bool) -> str:
+    base = _MACHINE.get(machine, f"machine{machine}")
+    # BinKit names its big-endian MIPS targets "mipseb"; mirror that so labels
+    # from paths and from ELF headers agree.
+    if base == "mips" and not little_endian:
+        base = "mipseb"
+    return f"{base}_{bits}"
+
+
+def parse(buf: bytes, *, want_functions: bool = True) -> TextSection:
+    """Extract ``.text`` (and optionally function boundaries) from an ELF image.
+
+    Args:
+        buf: the whole file contents.
+        want_functions: parse the symbol table for function boundaries. Skip it
+            when only doing binary-level work — it saves a scan over the
+            symbol table, which in a debug build can be larger than ``.text``.
+
+    Raises:
+        ElfError: not an ELF, unsupported class, or no ``.text`` section.
+    """
+    mv = memoryview(buf)
+    if len(mv) < 64 or bytes(mv[:4]) != ELF_MAGIC:
+        raise ElfError("not an ELF file")
+
+    ei_class, ei_data = mv[4], mv[5]
+    if ei_class == 1:
+        bits = 32
+    elif ei_class == 2:
+        bits = 64
+    else:
+        raise ElfError(f"bad EI_CLASS {ei_class}")
+    if ei_data == 1:
+        little_endian = True
+    elif ei_data == 2:
+        little_endian = False
+    else:
+        raise ElfError(f"bad EI_DATA {ei_data}")
+
+    e = "<" if little_endian else ">"
+    (machine,) = _u(mv, e + "H", 18)
+
+    # --- ELF header: locate the section header table -----------------------
+    if bits == 64:
+        e_shoff, = _u(mv, e + "Q", 40)
+        e_shentsize, e_shnum, e_shstrndx = _u(mv, e + "HHH", 58)
+    else:
+        e_shoff, = _u(mv, e + "I", 32)
+        e_shentsize, e_shnum, e_shstrndx = _u(mv, e + "HHH", 46)
+
+    if e_shoff == 0 or e_shnum == 0:
+        raise ElfError("no section header table (stripped of sections?)")
+    if e_shentsize < (64 if bits == 64 else 40):
+        raise ElfError(f"implausible e_shentsize {e_shentsize}")
+
+    # --- section headers ---------------------------------------------------
+    # (name_off, type, addr, offset, size, link, entsize)
+    sections: list[tuple[int, int, int, int, int, int, int]] = []
+    for i in range(e_shnum):
+        base = e_shoff + i * e_shentsize
+        if bits == 64:
+            sh_name, sh_type = _u(mv, e + "II", base)
+            sh_addr, sh_offset, sh_size = _u(mv, e + "QQQ", base + 16)
+            sh_link, = _u(mv, e + "I", base + 40)
+            sh_entsize, = _u(mv, e + "Q", base + 56)
+        else:
+            sh_name, sh_type = _u(mv, e + "II", base)
+            sh_addr, sh_offset, sh_size = _u(mv, e + "III", base + 12)
+            sh_link, = _u(mv, e + "I", base + 24)
+            sh_entsize, = _u(mv, e + "I", base + 36)
+        sections.append((sh_name, sh_type, sh_addr, sh_offset, sh_size, sh_link, sh_entsize))
+
+    # section-name string table
+    shstr_off = 0
+    if 0 <= e_shstrndx < len(sections):
+        shstr_off = sections[e_shstrndx][3]
+
+    def sec_name(idx: int) -> str:
+        return _cstr(mv, shstr_off + sections[idx][0]) if shstr_off else ""
+
+    names = [sec_name(i) for i in range(len(sections))]
+
+    # --- .text -------------------------------------------------------------
+    try:
+        text_idx = names.index(".text")
+    except ValueError:
+        raise ElfError("no .text section") from None
+
+    _, t_type, t_addr, t_off, t_size, _, _ = sections[text_idx]
+    if t_type == SHT_NOBITS or t_size == 0:
+        raise ElfError(".text is empty")
+    if t_off + t_size > len(mv):
+        # Truncated download / corrupt file: take what is actually there.
+        t_size = max(0, len(mv) - t_off)
+        if t_size == 0:
+            raise ElfError(".text lies past end of file")
+    text = bytes(mv[t_off : t_off + t_size])
+
+    # --- .comment (compiler fingerprint, used to sanity-check labels) ------
+    comment = ""
+    if ".comment" in names:
+        _, _, _, c_off, c_size, _, _ = sections[names.index(".comment")]
+        if c_size and c_off + c_size <= len(mv):
+            comment = bytes(mv[c_off : c_off + c_size]).replace(b"\x00", b" ").decode(
+                "utf-8", "replace"
+            ).strip()
+
+    stripped = not any(s[1] == SHT_SYMTAB for s in sections)
+
+    result = TextSection(
+        data=text,
+        vaddr=t_addr,
+        file_offset=t_off,
+        arch=_arch_name(machine, bits, little_endian),
+        bits=bits,
+        little_endian=little_endian,
+        comment=comment,
+        stripped=stripped,
+    )
+    if want_functions:
+        result.functions = _read_functions(mv, e, bits, sections, text_idx, t_addr, t_size, machine)
+    return result
+
+
+def _read_functions(
+    mv: memoryview,
+    e: str,
+    bits: int,
+    sections,
+    text_idx: int,
+    text_vaddr: int,
+    text_size: int,
+    machine: int,
+) -> list[Function]:
+    """Collect STT_FUNC symbols that live inside ``.text``.
+
+    Prefers SHT_SYMTAB; falls back to SHT_DYNSYM for stripped binaries (which
+    yields far fewer functions, but is better than nothing).
+    """
+    symtabs = [s for s in sections if s[1] == SHT_SYMTAB]
+    if not symtabs:
+        symtabs = [s for s in sections if s[1] == SHT_DYNSYM]
+    if not symtabs:
+        return []
+
+    sym_size = 24 if bits == 64 else 16
+    out: dict[int, Function] = {}  # offset -> Function, de-duplicates aliases
+
+    for _, _, _, sh_offset, sh_size, sh_link, sh_entsize in symtabs:
+        entsize = sh_entsize or sym_size
+        if entsize < sym_size or sh_size == 0:
+            continue
+        # linked string table
+        strtab_off = sections[sh_link][3] if 0 <= sh_link < len(sections) else 0
+        n = sh_size // entsize
+        for i in range(n):
+            base = sh_offset + i * entsize
+            try:
+                if bits == 64:
+                    st_name, = _u(mv, e + "I", base)
+                    st_info = mv[base + 4]
+                    st_shndx, = _u(mv, e + "H", base + 6)
+                    st_value, st_sz = _u(mv, e + "QQ", base + 8)
+                else:
+                    st_name, st_value, st_sz = _u(mv, e + "III", base)
+                    st_info = mv[base + 12]
+                    st_shndx, = _u(mv, e + "H", base + 14)
+            except ElfError:
+                break  # truncated symbol table; keep what we have
+
+            if (st_info & 0xF) != STT_FUNC or st_sz == 0 or st_shndx != text_idx:
+                continue
+            # ARM/Thumb interworking sets bit 0 of st_value; MIPS16/microMIPS
+            # does the same. Clear it so the offset lands on a real byte.
+            if machine in (40, 8, 10):
+                st_value &= ~1
+            off = st_value - text_vaddr
+            if off < 0 or off >= text_size:
+                continue
+            size = min(st_sz, text_size - off)
+            if size <= 0 or off in out:
+                continue
+            out[off] = Function(name=_cstr(mv, strtab_off + st_name), offset=off, size=size)
+
+    return sorted(out.values(), key=lambda f: f.offset)
+
+
+def guess_compiler(comment: str) -> tuple[str | None, str | None]:
+    """Best-effort ``(family, version)`` from a ``.comment`` string.
+
+    Used to cross-check labels derived from file paths — a cheap guard against a
+    mislabeled corpus. Returns ``(None, None)`` when nothing is recognisable.
+    Note that a binary linked from both GCC- and Clang-compiled objects can
+    carry both fingerprints; the first match wins.
+    """
+    import re
+
+    if not comment:
+        return None, None
+    low = comment.lower()
+    m = re.search(r"clang version ([0-9]+(?:\.[0-9]+)*)", low)
+    if m:
+        return "clang", m.group(1)
+    m = re.search(r"gcc[:\s(]+.*?([0-9]+\.[0-9]+\.[0-9]+)", low)
+    if m:
+        return "gcc", m.group(1)
+    if "clang" in low:
+        return "clang", None
+    if "gcc" in low or "gnu" in low:
+        return "gcc", None
+    return None, None
+
+
+def is_elf(path) -> bool:
+    """Cheap check used to filter build trees before a full parse."""
+    try:
+        with open(path, "rb") as fh:
+            return fh.read(4) == ELF_MAGIC
+    except OSError:
+        return False

binprov/metrics.py

@@ -0,0 +1,71 @@
+"""Metrics and table formatting.
+
+The paper reports accuracy as its headline number, because the dataset is
+balanced across compilers and optimization levels (§4.2), and adds
+precision/recall/F1 per optimization level to expose *where* the O2/O3
+confusion sits (Table 5). Both are here, with no sklearn dependency so that
+evaluation runs in any environment that can run the model.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+
+def accuracy(pred: np.ndarray, true: np.ndarray) -> float:
+    if len(true) == 0:
+        return 0.0
+    return float((pred == true).mean())
+
+
+def confusion_matrix(pred: np.ndarray, true: np.ndarray, num_classes: int) -> np.ndarray:
+    """Rows are ground truth, columns are predictions."""
+    flat = true.astype(np.int64) * num_classes + pred.astype(np.int64)
+    return np.bincount(flat, minlength=num_classes**2).reshape(num_classes, num_classes)
+
+
+def per_class_prf(pred: np.ndarray, true: np.ndarray, num_classes: int) -> dict:
+    """Precision / recall / F1 per class, plus macro averages (Table 5)."""
+    cm = confusion_matrix(pred, true, num_classes)
+    tp = np.diag(cm).astype(np.float64)
+    predicted = cm.sum(axis=0).astype(np.float64)
+    actual = cm.sum(axis=1).astype(np.float64)
+    with np.errstate(divide="ignore", invalid="ignore"):
+        precision = np.where(predicted > 0, tp / predicted, 0.0)
+        recall = np.where(actual > 0, tp / actual, 0.0)
+        denom = precision + recall
+        f1 = np.where(denom > 0, 2 * precision * recall / denom, 0.0)
+    return {
+        "precision": precision,
+        "recall": recall,
+        "f1": f1,
+        "support": actual.astype(np.int64),
+        "macro_precision": float(precision.mean()),
+        "macro_recall": float(recall.mean()),
+        "macro_f1": float(f1.mean()),
+        "accuracy": float(tp.sum() / cm.sum()) if cm.sum() else 0.0,
+        "confusion": cm,
+    }
+
+
+def pct(x: float) -> str:
+    return f"{100 * x:.2f}%"
+
+
+def markdown_table(header: list[str], rows: list[list[str]]) -> str:
+    """Render a markdown table with columns padded to a readable width."""
+    widths = [len(h) for h in header]
+    for row in rows:
+        for i, cell in enumerate(row):
+            widths[i] = max(widths[i], len(str(cell)))
+    out = ["| " + " | ".join(h.ljust(widths[i]) for i, h in enumerate(header)) + " |"]
+    out.append("|" + "|".join("-" * (w + 2) for w in widths) + "|")
+    for row in rows:
+        out.append("| " + " | ".join(str(c).ljust(widths[i]) for i, c in enumerate(row)) + " |")
+    return "\n".join(out)
+
+
+def confusion_as_markdown(cm: np.ndarray, classes) -> str:
+    header = ["true \\ pred", *classes]
+    rows = [[classes[i], *[str(int(v)) for v in cm[i]]] for i in range(len(classes))]
+    return markdown_table(header, rows)

binprov/model.py

@@ -0,0 +1,453 @@
+"""BinProv's networks: a byte-level RoBERTa encoder and a provenance classifier.
+
+Two pieces, matching the paper's two training phases:
+
+* :func:`build_mlm_model` — the embedding model ``f_e`` with a softmax head,
+  pre-trained on masked language modeling (§3.2, Eq. 1). 12 transformer layers,
+  hidden size 768, which lands at ~85M trainable parameters — the paper's
+  "around 80 million".
+* :class:`BinProvForProvenance` — ``f_c(f_e(...))``, the encoder plus the
+  two-layer fully connected classifier of §3.3, fine-tuned end-to-end.
+
+The paper describes the classifier's first layer as one that "reshape[s] the
+input vectors and weaken[s] the border weights of the embeddings", to cope with
+x86's variable-length instructions being cut mid-instruction at a sequence
+boundary. It does not give the mechanism. :class:`BorderTaperedPool` implements
+it as a learned attention over token positions whose logits are *initialised*
+with a taper at both ends: borders start down-weighted, and training is free to
+change that. Ablate it with ``--pool mean`` to see what it buys.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+from dataclasses import dataclass, asdict
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import RobertaConfig, RobertaForMaskedLM, RobertaModel
+
+from .vocab import BOS_ID, EOS_ID, MASK_ID, PAD_ID, VOCAB_SIZE
+
+
+@dataclass
+class BinProvConfig:
+    """Architecture knobs. Stored next to every checkpoint."""
+
+    seq_bytes: int = 512  # bytes per sequence (paper §3.1)
+    hidden_size: int = 768
+    num_hidden_layers: int = 12  # "consists of 12 transformers" (§3.2)
+    num_attention_heads: int = 12
+    intermediate_size: int = 3072
+    max_segments: int = 2  # segment embedding E_s (§3.2)
+    hidden_dropout_prob: float = 0.1
+    attention_probs_dropout_prob: float = 0.1
+
+    @property
+    def seq_tokens(self) -> int:
+        """Sequence length in tokens: the bytes plus <s> and </s>."""
+        return self.seq_bytes + 2
+
+    def to_roberta(self) -> RobertaConfig:
+        return RobertaConfig(
+            vocab_size=VOCAB_SIZE,
+            hidden_size=self.hidden_size,
+            num_hidden_layers=self.num_hidden_layers,
+            num_attention_heads=self.num_attention_heads,
+            intermediate_size=self.intermediate_size,
+            # HF's RoBERTa derives position ids as padding_idx + 1 + i, so the
+            # table needs two slots of headroom beyond the token count.
+            max_position_embeddings=self.seq_tokens + 2,
+            type_vocab_size=self.max_segments,
+            pad_token_id=PAD_ID,
+            bos_token_id=BOS_ID,
+            eos_token_id=EOS_ID,
+            mask_token_id=MASK_ID,
+            hidden_dropout_prob=self.hidden_dropout_prob,
+            attention_probs_dropout_prob=self.attention_probs_dropout_prob,
+            position_embedding_type="absolute",
+        )
+
+    def save(self, path: str | Path) -> None:
+        Path(path).write_text(json.dumps(asdict(self), indent=2) + "\n")
+
+    @classmethod
+    def load(cls, path: str | Path) -> BinProvConfig:
+        return cls(**json.loads(Path(path).read_text()))
+
+
+def build_mlm_model(cfg: BinProvConfig) -> RobertaForMaskedLM:
+    """The pre-training model: encoder + softmax head over the 261-byte vocab."""
+    return RobertaForMaskedLM(cfg.to_roberta())
+
+
+class BorderTaperedPool(nn.Module):
+    """Pool token embeddings into one vector, down-weighting sequence borders.
+
+    This is the first of the classifier's two layers (§3.3). Weights are a
+    softmax over learned per-position logits, masked to the valid tokens, so
+    padding and the <s>/</s> markers contribute nothing. Logits start as a
+    linear ramp over the outermost ``taper`` byte positions, encoding the prior
+    that a byte near the cut is likely part of a broken instruction.
+    """
+
+    def __init__(self, n_positions: int, hidden_size: int, taper: int = 32, taper_floor: float = -2.0):
+        super().__init__()
+        logits = torch.zeros(n_positions)
+        taper = max(0, min(taper, n_positions // 2))
+        if taper:
+            ramp = torch.linspace(taper_floor, 0.0, taper + 1)[:-1]
+            logits[:taper] = ramp
+            logits[n_positions - taper :] = ramp.flip(0)
+        self.pos_logits = nn.Parameter(logits)
+        self.proj = nn.Linear(hidden_size, hidden_size)
+        self.norm = nn.LayerNorm(hidden_size)
+
+    def forward(self, hidden: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+        # hidden: (B, T, H); attention_mask: (B, T) with 1 on real tokens
+        n = hidden.size(1)
+        logits = self.pos_logits[:n].unsqueeze(0).expand(hidden.size(0), -1)
+        logits = logits.masked_fill(attention_mask == 0, torch.finfo(logits.dtype).min)
+        weights = logits.softmax(dim=-1).unsqueeze(-1)
+        pooled = (hidden * weights).sum(dim=1)
+        return self.norm(F.gelu(self.proj(pooled)))
+
+
+class MeanPool(nn.Module):
+    """Plain masked mean pooling — the ablation baseline for the head."""
+
+    def __init__(self, hidden_size: int):
+        super().__init__()
+        self.proj = nn.Linear(hidden_size, hidden_size)
+        self.norm = nn.LayerNorm(hidden_size)
+
+    def forward(self, hidden: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+        mask = attention_mask.unsqueeze(-1).to(hidden.dtype)
+        pooled = (hidden * mask).sum(1) / mask.sum(1).clamp(min=1.0)
+        return self.norm(F.gelu(self.proj(pooled)))
+
+
+
+class AttentionPool(nn.Module):
+    """Content-based attention pooling: weight a position by *what* is there.
+
+    :class:`BorderTaperedPool` weights a position by *where* it is, which makes
+    it a fixed weighted mean. That is the right prior for the border problem it
+    was written for, and the wrong one for O2/O3. The byte patterns that
+    separate the two levels -- a vectorised loop body, one copy of an unrolled
+    loop, a particular alignment pad -- occupy a handful of positions in a
+    512-byte window and are absent from the rest of it, so any weighted mean
+    dilutes the evidence by the fraction of the window that carries it.
+    Attention can concentrate on those positions instead.
+
+    ``heads > 1`` lets the head attend to several kinds of evidence at once
+    (say, a vector opcode in one head and a padding pattern in another).
+    """
+
+    def __init__(self, hidden_size: int, heads: int = 4):
+        super().__init__()
+        self.heads = heads
+        self.score = nn.Linear(hidden_size, heads)
+        self.proj = nn.Linear(hidden_size * heads, hidden_size)
+        self.norm = nn.LayerNorm(hidden_size)
+
+    def forward(self, hidden: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+        scores = self.score(hidden)  # (B, T, heads)
+        keep = attention_mask.unsqueeze(-1) == 0
+        scores = scores.masked_fill(keep, torch.finfo(scores.dtype).min)
+        weights = scores.softmax(dim=1)  # over positions
+        pooled = torch.einsum("bth,btd->bhd", weights, hidden).flatten(1)
+        return self.norm(F.gelu(self.proj(pooled)))
+
+
+class MaxPool(nn.Module):
+    """Per-dimension max over positions -- the hard version of AttentionPool.
+
+    Cheaper and with no parameters of its own in the aggregation step, at the
+    cost of committing to one position per feature dimension.
+    """
+
+    def __init__(self, hidden_size: int):
+        super().__init__()
+        self.proj = nn.Linear(hidden_size, hidden_size)
+        self.norm = nn.LayerNorm(hidden_size)
+
+    def forward(self, hidden: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+        keep = attention_mask.unsqueeze(-1) == 0
+        pooled = hidden.masked_fill(keep, torch.finfo(hidden.dtype).min).max(dim=1).values
+        return self.norm(F.gelu(self.proj(pooled)))
+
+
+class MILHead(nn.Module):
+    """Per-token classifier plus a smooth max over positions (returns logits).
+
+    The multiple-instance view of the task: a window is O3 if it *contains* an
+    O3 fingerprint, so the window score should be a max over per-token scores
+    rather than a score computed from an average of tokens. Pooling after the
+    classifier rather than before it is the whole point -- it keeps a single
+    strongly-O3 position from being averaged into 500 neutral ones.
+
+    ``logsumexp`` is the differentiable max: ``tau -> 0`` recovers a hard max,
+    large ``tau`` approaches the mean, so one module spans both ends and ``tau``
+    is a knob rather than a commitment. ``log n`` is subtracted so the result is
+    a log-*mean*-exp and therefore invariant to how many valid tokens a window
+    has -- without it a short window would score systematically lower on every
+    class, which is exactly the confound that makes padded short functions look
+    hard.
+    """
+
+    def __init__(self, hidden_size: int, num_labels: int, *, tau: float = 1.0, dropout: float = 0.1):
+        super().__init__()
+        self.proj = nn.Linear(hidden_size, hidden_size)
+        self.norm = nn.LayerNorm(hidden_size)
+        self.drop = nn.Dropout(dropout)
+        self.token_clf = nn.Linear(hidden_size, num_labels)
+        self.tau = tau
+        nn.init.normal_(self.token_clf.weight, std=0.02)
+        nn.init.zeros_(self.token_clf.bias)
+
+    def forward(self, hidden: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+        h = self.norm(F.gelu(self.proj(hidden)))
+        tok = self.token_clf(self.drop(h))  # (B, T, C)
+        keep = attention_mask.unsqueeze(-1) == 0
+        tok = tok.masked_fill(keep, torch.finfo(tok.dtype).min)
+        n = attention_mask.sum(dim=1, keepdim=True).clamp(min=1).to(tok.dtype)
+        return self.tau * (torch.logsumexp(tok / self.tau, dim=1) - torch.log(n))
+
+    def token_logits(self, hidden: torch.Tensor) -> torch.Tensor:
+        """Per-position class scores, for locating *where* the evidence is."""
+        return self.token_clf(self.norm(F.gelu(self.proj(hidden))))
+
+
+class BinProvForProvenance(nn.Module):
+    """Encoder + classifier, i.e. Eq. (2) of the paper.
+
+    ``y_hat = f_c(f_e(concat(E_b, E_p, E_s)))``
+    """
+
+    def __init__(
+        self,
+        cfg: BinProvConfig,
+        num_labels: int,
+        *,
+        pool: str = "border",
+        classifier_dropout: float = 0.1,
+        taper: int = 32,
+        pool_heads: int = 4,
+        mil_tau: float = 1.0,
+        label_smoothing: float = 0.0,
+    ):
+        super().__init__()
+        self.cfg = cfg
+        self.num_labels = num_labels
+        self.pool_kind = pool
+        self.pool_heads = pool_heads
+        self.mil_tau = mil_tau
+        self.label_smoothing = label_smoothing
+        self.encoder = RobertaModel(cfg.to_roberta(), add_pooling_layer=False)
+        # A "mil" head produces class logits itself, so there is no pooled
+        # vector for a separate classifier to consume; `pools_to_logits` records
+        # which of the two shapes we are in.
+        self.pools_to_logits = pool == "mil"
+        if pool == "border":
+            self.pool = BorderTaperedPool(cfg.seq_tokens, cfg.hidden_size, taper=taper)
+        elif pool == "mean":
+            self.pool = MeanPool(cfg.hidden_size)
+        elif pool == "attn":
+            self.pool = AttentionPool(cfg.hidden_size, heads=pool_heads)
+        elif pool == "max":
+            self.pool = MaxPool(cfg.hidden_size)
+        elif pool == "mil":
+            self.pool = MILHead(
+                cfg.hidden_size, num_labels, tau=mil_tau, dropout=classifier_dropout
+            )
+        elif pool == "cls":
+            self.pool = None
+        else:
+            raise ValueError(f"unknown pool {pool!r} (border|mean|attn|max|mil|cls)")
+        self.dropout = nn.Dropout(classifier_dropout)
+        if self.pools_to_logits:
+            self.classifier = nn.Identity()
+        else:
+            self.classifier = nn.Linear(cfg.hidden_size, num_labels)
+            nn.init.normal_(self.classifier.weight, std=0.02)
+            nn.init.zeros_(self.classifier.bias)
+
+    def forward(
+        self,
+        input_ids: torch.Tensor,
+        attention_mask: torch.Tensor,
+        token_type_ids: torch.Tensor | None = None,
+        labels: torch.Tensor | None = None,
+    ):
+        out = self.encoder(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+        )
+        hidden = out.last_hidden_state  # E_final, shape (B, T, H)
+        if self.pools_to_logits:
+            logits = self.pool(hidden, attention_mask)
+        else:
+            pooled = hidden[:, 0] if self.pool is None else self.pool(hidden, attention_mask)
+            logits = self.classifier(self.dropout(pooled))
+        loss = None
+        if labels is not None:
+            loss = F.cross_entropy(
+                logits.float(), labels, label_smoothing=self.label_smoothing
+            )
+        return {"loss": loss, "logits": logits}
+
+    # -- checkpointing -----------------------------------------------------
+
+    @staticmethod
+    def _resize_position_embeddings(state: dict, want: int, how: str = "tile") -> dict:
+        """Extend a pre-trained absolute position table to a longer sequence.
+
+        The MLM checkpoint learned 516 position embeddings because the paper's
+        sequence is 512 bytes. A longer-sequence encoder needs more, and the extra
+        rows have to come from somewhere. Two ways, and the difference is not
+        cosmetic — measured on the O2/O3 task, ``interp`` costs 5 points against
+        the 512-byte baseline while ``tile`` does not:
+
+        ``interp``
+            Linear interpolation, the move that extends a vision transformer to a
+            larger image. It preserves the table's global *shape* and destroys its
+            local structure: stretching 2x makes each adjacent pair of positions
+            nearly identical. For an image patch grid that is harmless. For a byte
+            encoder whose pre-training learned that neighbouring positions differ
+            — instruction boundaries, alignment padding — it is not, and it shows
+            up as an encoder that will not fit the training set any more.
+
+        ``tile``
+            Position ``k`` reuses the learned row ``k mod (have - 1)``. Local
+            structure is preserved exactly; what is lost is the ability to tell
+            absolute position 100 from position 614. For this task that is the
+            cheaper thing to give up, since a provenance cue means the same thing
+            wherever in the window it appears.
+
+        Rows are handled from index 1 up, because HF derives RoBERTa position ids
+        as ``padding_idx + 1 + i`` and so never uses row 0.
+        """
+        key = "embeddings.position_embeddings.weight"
+        if key not in state or state[key].shape[0] == want:
+            return state
+        have = state[key]
+        head, tail = have[:1], have[1:]
+        if how == "interp":
+            grown = F.interpolate(
+                tail.T.unsqueeze(0).float(), size=want - 1, mode="linear", align_corners=True
+            ).squeeze(0).T.to(have.dtype)
+        elif how == "tile":
+            reps = -(-(want - 1) // tail.shape[0])
+            grown = tail.repeat(reps, 1)[: want - 1]
+        else:
+            raise ValueError(f"unknown position extension {how!r} (tile|interp)")
+        state = dict(state)
+        state[key] = torch.cat([head, grown], dim=0)
+        # position_ids is a plain arange buffer; regenerate rather than stretch
+        for k in list(state):
+            if k.endswith("embeddings.position_ids") or k.endswith("embeddings.token_type_ids"):
+                state.pop(k)
+        return state
+
+    def load_encoder_from_mlm(
+        self, mlm_dir: str | Path, *, strict: bool = False, pos_extend: str = "tile"
+    ) -> list[str]:
+        """Warm-start the encoder from an MLM checkpoint (the transfer step).
+
+        Returns the list of encoder parameters that were *not* found, which
+        should be empty for a matching config — worth asserting in a run log,
+        since a silent mismatch here would quietly turn fine-tuning into
+        training from scratch.
+        """
+        mlm = RobertaForMaskedLM.from_pretrained(str(mlm_dir))
+        state = self._resize_position_embeddings(
+            mlm.roberta.state_dict(),
+            self.encoder.embeddings.position_embeddings.weight.shape[0],
+            pos_extend,
+        )
+        missing, unexpected = self.encoder.load_state_dict(state, strict=strict)
+        del mlm
+        if strict and (missing or unexpected):
+            raise RuntimeError(f"encoder mismatch: missing={missing} unexpected={unexpected}")
+        return list(missing)
+
+    def save(self, out_dir: str | Path, *, extra: dict | None = None) -> None:
+        out = Path(out_dir)
+        out.mkdir(parents=True, exist_ok=True)
+        torch.save(self.state_dict(), out / "model.pt")
+        self.cfg.save(out / "binprov_config.json")
+        (out / "head.json").write_text(
+            json.dumps(
+                {
+                    "num_labels": self.num_labels,
+                    "pool": self.pool_kind,
+                    "pool_heads": self.pool_heads,
+                    "mil_tau": self.mil_tau,
+                    **(extra or {}),
+                },
+                indent=2,
+            )
+            + "\n"
+        )
+
+    @classmethod
+    def load(cls, ckpt_dir: str | Path, *, map_location="cpu") -> tuple[BinProvForProvenance, dict]:
+        d = Path(ckpt_dir)
+        cfg = BinProvConfig.load(d / "binprov_config.json")
+        head = json.loads((d / "head.json").read_text())
+        model = cls(
+            cfg,
+            head["num_labels"],
+            pool=head.get("pool", "border"),
+            pool_heads=head.get("pool_heads", 4),
+            mil_tau=head.get("mil_tau", 1.0),
+        )
+        state = torch.load(d / "model.pt", map_location=map_location, weights_only=True)
+        model.load_state_dict(state)
+        return model, head
+
+
+def count_parameters(model: nn.Module) -> int:
+    return sum(p.numel() for p in model.parameters() if p.requires_grad)
+
+
+def describe(model: nn.Module) -> str:
+    n = count_parameters(model)
+    return f"{type(model).__name__}: {n:,} trainable parameters ({n / 1e6:.1f}M)"
+
+
+def param_groups(model: nn.Module, weight_decay: float = 0.01):
+    """AdamW groups that exempt biases and LayerNorm from weight decay."""
+    decay, no_decay = [], []
+    for name, p in model.named_parameters():
+        if not p.requires_grad:
+            continue
+        if p.ndim == 1 or name.endswith(".bias") or "LayerNorm" in name or "norm" in name:
+            no_decay.append(p)
+        else:
+            decay.append(p)
+    # Empty groups are dropped: a frozen encoder would otherwise hand AdamW a
+    # group with no parameters.
+    groups = [
+        {"params": decay, "weight_decay": weight_decay},
+        {"params": no_decay, "weight_decay": 0.0},
+    ]
+    return [g for g in groups if g["params"]]
+
+
+def cosine_schedule_with_warmup(optimizer, warmup_steps: int, total_steps: int, min_ratio: float = 0.02):
+    """Linear warmup then cosine decay — standard for MLM pre-training."""
+
+    def lr_lambda(step: int) -> float:
+        if warmup_steps > 0 and step < warmup_steps:
+            return step / max(1, warmup_steps)
+        progress = (step - warmup_steps) / max(1, total_steps - warmup_steps)
+        progress = min(1.0, max(0.0, progress))
+        return min_ratio + (1 - min_ratio) * 0.5 * (1 + math.cos(math.pi * progress))
+
+    return torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)

binprov/provenance.py

@@ -0,0 +1,277 @@
+"""Provenance labels and the classification tasks defined on them.
+
+The paper decomposes provenance identification into independent sub-tasks
+(§3.3) rather than predicting the 8-way compiler×optimization product, because
+"specialized classifiers have better performance". Each entry in :data:`TASKS`
+is one such sub-task: which binaries it keeps, and how a kept binary maps to a
+class index.
+
+Tasks (with the paper table they feed):
+
+    compiler   GCC vs Clang                            Tables 3, 6, 7
+    opt_hl     low (O0/O1) vs high (O2/O3)             Tables 3, 6
+    opt4       O0 vs O1 vs O2 vs O3                    Tables 4, 5, 7
+    opt_o0o1   O0 vs O1  (only O0/O1 binaries)         Table 4
+    opt_o2o3   O2 vs O3  (only O2/O3 binaries)         Tables 4, 6, 7
+    arch       instruction set architecture            §5.1(d)
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from typing import Callable, Sequence
+
+COMPILERS = ("gcc", "clang")
+OPT_LEVELS = ("O0", "O1", "O2", "O3")
+#: architectures present in BinKit; the paper evaluates the first four
+ARCHES = (
+    "x86_64",
+    "x86_32",
+    "arm_64",
+    "arm_32",
+    "mips_64",
+    "mips_32",
+    "mipseb_64",
+    "mipseb_32",
+)
+
+# ---------------------------------------------------------------------------
+# label extraction from a path
+# ---------------------------------------------------------------------------
+
+# BinKit's on-disk layout has changed between releases (and mirrors repack it),
+# so rather than hard-coding one directory template we scan every path
+# component for recognisable tokens. `scripts/build_corpus.py --dry-run` prints
+# what this produced so a new layout can be spotted before a long build.
+_ARCH_RE = re.compile(
+    r"(?<![a-z0-9])(x86|x64|i386|arm|aarch64|mips|mipseb|mipsel|ppc|riscv)"
+    r"(?:[-_]?(32|64))?(?![a-z0-9])",
+    re.I,
+)
+_COMPILER_RE = re.compile(
+    r"(?<![a-z])(gcc|g\+\+|clang|clang\+\+)(?:[-_]?([0-9]+(?:\.[0-9]+){0,2}))?(?![a-z])", re.I
+)
+_OPT_RE = re.compile(r"(?<![a-z0-9])-?O(0|1|2|3|s|fast)(?![a-z0-9])")
+_EXTRA_RE = re.compile(
+    r"(?<![a-z])(normal|nopie|pie|noinline|lto|sizeopt|obfus(?:2loop)?|bcf|fla|sub|all)(?![a-z])",
+    re.I,
+)
+# Obfuscator-LLVM transforms, reported separately in Table 7
+_OBF_TOKENS = {"bcf", "fla", "sub", "all"}
+
+
+@dataclass
+class Labels:
+    """Everything we know about how one binary was produced."""
+
+    arch: str | None = None
+    compiler: str | None = None
+    compiler_version: str | None = None
+    opt: str | None = None
+    extra: str = "normal"  # normal | pie | lto | noinline | sizeopt | ...
+    obfuscation: str | None = None  # bcf | fla | sub | all
+
+    def is_complete(self) -> bool:
+        """True when the binary can be used for every task in :data:`TASKS`."""
+        return bool(self.arch and self.compiler and self.opt)
+
+    def as_dict(self) -> dict:
+        return {
+            "arch": self.arch,
+            "compiler": self.compiler,
+            "compiler_version": self.compiler_version,
+            "opt": self.opt,
+            "extra": self.extra,
+            "obfuscation": self.obfuscation,
+        }
+
+
+def _norm_arch(base: str, width: str | None) -> str:
+    base = base.lower()
+    if base in ("x64",):
+        return "x86_64"
+    if base in ("i386",):
+        return "x86_32"
+    if base == "aarch64":
+        return "arm_64"
+    if base == "mipsel":
+        base = "mips"
+    if width:
+        return f"{base}_{width}"
+    # widths that are implied by the architecture name alone
+    return {"arm": "arm_32", "x86": "x86_32", "mips": "mips_32"}.get(base, base)
+
+
+def parse_labels(path: str) -> Labels:
+    """Derive provenance labels by scanning the components of ``path``.
+
+    Later components win, so a per-binary directory overrides a package-level
+    one. Compiler family/version are only taken together, to avoid pairing
+    "gcc" from one component with a version number from another.
+    """
+    lab = Labels()
+    parts = [p for p in str(path).replace("\\", "/").split("/") if p]
+    for part in parts:
+        if m := _ARCH_RE.search(part):
+            lab.arch = _norm_arch(m.group(1), m.group(2))
+        if m := _COMPILER_RE.search(part):
+            fam = m.group(1).lower().rstrip("+")
+            lab.compiler = "gcc" if fam in ("gcc", "g") else "clang"
+            if m.group(2):
+                lab.compiler_version = m.group(2)
+        if m := _OPT_RE.search(part):
+            lab.opt = "O" + m.group(1)
+        # finditer, not search: a component can carry both the family and the
+        # specific transform, e.g. "x86_64-clang-obfus-sub-O2"
+        for m in _EXTRA_RE.finditer(part):
+            tok = m.group(1).lower()
+            if tok in _OBF_TOKENS:
+                lab.obfuscation = tok
+                lab.extra = "obfus"
+            elif tok.startswith("obfus"):
+                lab.extra = "obfus"
+            else:
+                lab.extra = tok
+    return lab
+
+
+# ---------------------------------------------------------------------------
+# tasks
+# ---------------------------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class Task:
+    """A classification task: a filter plus a label map.
+
+    ``marginal_of``/``marginal_map`` describe a *factorized* task: one whose
+    classes are finer than the classes we actually want to score, together with
+    the map from its class index to the coarse one. Training on the finer split
+    and summing the probabilities back down is a different estimator of the same
+    quantity -- it hands the model the compiler as extra supervision instead of
+    making it marginalise the compiler out on its own.
+    """
+
+    name: str
+    classes: tuple[str, ...]
+    keep: Callable[[dict], bool]
+    to_class: Callable[[dict], str]
+    note: str = ""  # which paper table this task feeds
+    marginal_of: str | None = None
+    marginal_map: tuple[int, ...] = ()
+
+    @property
+    def num_labels(self) -> int:
+        return len(self.classes)
+
+    @property
+    def scored_classes(self) -> tuple[str, ...]:
+        """The classes metrics are reported over — coarse ones if factorized."""
+        if self.marginal_of is None:
+            return self.classes
+        return TASKS[self.marginal_of].classes
+
+    def label_of(self, labels: dict) -> int | None:
+        """Class index for a binary, or None if it is out of scope."""
+        if not self.keep(labels):
+            return None
+        cls = self.to_class(labels)
+        try:
+            return self.classes.index(cls)
+        except ValueError:
+            return None
+
+
+def _opt_in(*levels: str) -> Callable[[dict], bool]:
+    allowed = set(levels)
+    return lambda d: d.get("opt") in allowed
+
+
+TASKS: dict[str, Task] = {
+    "compiler": Task(
+        name="compiler",
+        classes=COMPILERS,
+        keep=lambda d: d.get("compiler") in COMPILERS,
+        to_class=lambda d: d["compiler"],
+        note="Table 3/6/7: GCC vs Clang",
+    ),
+    "opt_hl": Task(
+        name="opt_hl",
+        classes=("low", "high"),
+        keep=_opt_in(*OPT_LEVELS),
+        to_class=lambda d: "low" if d["opt"] in ("O0", "O1") else "high",
+        note="Table 3/6: O0,O1 vs O2,O3",
+    ),
+    "opt4": Task(
+        name="opt4",
+        classes=OPT_LEVELS,
+        keep=_opt_in(*OPT_LEVELS),
+        to_class=lambda d: d["opt"],
+        note="Table 4/5/7: O0/O1/O2/O3",
+    ),
+    "opt_o0o1": Task(
+        name="opt_o0o1",
+        classes=("O0", "O1"),
+        keep=_opt_in("O0", "O1"),
+        to_class=lambda d: d["opt"],
+        note="Table 4: O0 vs O1",
+    ),
+    "opt_o2o3": Task(
+        name="opt_o2o3",
+        classes=("O2", "O3"),
+        keep=_opt_in("O2", "O3"),
+        to_class=lambda d: d["opt"],
+        note="Table 4/6/7: O2 vs O3 (the hard one)",
+    ),
+    "opt_o2o3_x": Task(
+        name="opt_o2o3_x",
+        classes=("gcc_O2", "gcc_O3", "clang_O2", "clang_O3"),
+        keep=lambda d: d.get("opt") in ("O2", "O3") and d.get("compiler") in COMPILERS,
+        to_class=lambda d: f"{d['compiler']}_{d['opt']}",
+        note="O2 vs O3 factorized by compiler, marginalized back for scoring",
+        marginal_of="opt_o2o3",
+        marginal_map=(0, 1, 0, 1),
+    ),
+    "opt4_x": Task(
+        name="opt4_x",
+        classes=tuple(f"{c}_{o}" for c in COMPILERS for o in OPT_LEVELS),
+        keep=lambda d: d.get("opt") in OPT_LEVELS and d.get("compiler") in COMPILERS,
+        to_class=lambda d: f"{d['compiler']}_{d['opt']}",
+        note="O0/O1/O2/O3 factorized by compiler, marginalized back for scoring",
+        marginal_of="opt4",
+        marginal_map=(0, 1, 2, 3, 0, 1, 2, 3),
+    ),
+    "arch": Task(
+        name="arch",
+        classes=ARCHES,
+        keep=lambda d: d.get("arch") in ARCHES,
+        to_class=lambda d: d["arch"],
+        note="§5.1(d): instruction set architecture",
+    ),
+}
+
+
+def get_task(name: str) -> Task:
+    try:
+        return TASKS[name]
+    except KeyError:
+        raise SystemExit(
+            f"unknown task {name!r}; available: {', '.join(sorted(TASKS))}"
+        ) from None
+
+
+def toolchain_key(labels: dict) -> str:
+    """Stable identifier for a compilation configuration.
+
+    Used to balance the corpus across configurations and to report per-cell
+    numbers (e.g. Table 5, which splits optimization results by compiler).
+    """
+    parts: Sequence[str] = (
+        labels.get("arch") or "?",
+        labels.get("compiler") or "?",
+        labels.get("compiler_version") or "-",
+        labels.get("opt") or "?",
+        labels.get("obfuscation") or labels.get("extra") or "normal",
+    )
+    return "/".join(parts)

binprov/vocab.py

@@ -0,0 +1,66 @@
+"""Byte vocabulary for BinProv (paper §3.2, "Input representation E").
+
+The paper defines a vocabulary of 256 possible byte values plus 5 special
+tags: <PAD>, <S>, </S>, <UNK> and <MASK>, for a total of 261 tokens.
+
+Layout used here (ids are stable and must not be reshuffled once a model has
+been pre-trained, since they are baked into the embedding matrix):
+
+    0   <pad>
+    1   <s>
+    2   </s>
+    3   <unk>
+    4   <mask>
+    5   0x00
+    6   0x01
+    ...
+    260 0xff
+"""
+
+from __future__ import annotations
+
+PAD_ID = 0
+BOS_ID = 1
+EOS_ID = 2
+UNK_ID = 3
+MASK_ID = 4
+
+NUM_SPECIAL = 5
+BYTE_OFFSET = NUM_SPECIAL
+VOCAB_SIZE = NUM_SPECIAL + 256  # 261
+
+SPECIAL_TOKENS = ("<pad>", "<s>", "</s>", "<unk>", "<mask>")
+
+#: ids that are never a prediction target of the MLM task
+NON_BYTE_IDS = frozenset({PAD_ID, BOS_ID, EOS_ID, UNK_ID, MASK_ID})
+
+
+def byte_to_id(b: int) -> int:
+    """Map a raw byte value (0..255) to its token id."""
+    if not 0 <= b <= 255:
+        raise ValueError(f"not a byte value: {b}")
+    return b + BYTE_OFFSET
+
+
+def id_to_byte(i: int) -> int:
+    """Inverse of :func:`byte_to_id`. Raises for special-token ids."""
+    if i < BYTE_OFFSET or i >= VOCAB_SIZE:
+        raise ValueError(f"not a byte token id: {i}")
+    return i - BYTE_OFFSET
+
+
+def encode(data: bytes) -> list[int]:
+    """Encode raw bytes as token ids, without <s>/</s>."""
+    return [b + BYTE_OFFSET for b in data]
+
+
+def decode(ids) -> bytes:
+    """Decode token ids back to bytes, dropping special tokens."""
+    return bytes(i - BYTE_OFFSET for i in ids if i >= BYTE_OFFSET)
+
+
+def token_repr(i: int) -> str:
+    """Human-readable form of a token id, e.g. ``0x8b`` or ``<mask>``."""
+    if i < NUM_SPECIAL:
+        return SPECIAL_TOKENS[i]
+    return f"0x{i - BYTE_OFFSET:02x}"

binprov/vote.py

@@ -0,0 +1,125 @@
+"""Joint inference by majority voting (paper §3.4).
+
+A single 512-byte sequence is not a meaningful entity: it may hold a fragment of
+one function, or span several. The paper therefore aggregates sequence-level
+predictions over all sequences belonging to the same function, or the same
+binary, and takes the majority. That relies on the §2.3 measurement that >96% of
+real projects compile an individual binary with a single optimization level, so
+the sequences of one binary do share a label.
+
+Voting is where most of the paper's headline numbers come from: on the hard
+O2/O3 task it lifts accuracy from 83.6% (sequence) to 94.7% (function) to 99.8%
+(binary), Tables 4 and 6.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+
+def majority_vote(
+    group_ids: np.ndarray,
+    predictions: np.ndarray,
+    num_classes: int,
+    *,
+    probs: np.ndarray | None = None,
+    mode: str = "hard",
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+    """Aggregate per-sequence predictions into per-group predictions.
+
+    Args:
+        group_ids: dense group index per sequence (see
+            :meth:`~binprov.corpus.SequenceIndex.group_ids`).
+        predictions: predicted class per sequence.
+        num_classes: size of the label set.
+        probs: per-sequence class probabilities, required for ``mode="soft"``.
+        mode: ``"hard"`` counts votes, as the paper does. ``"soft"`` sums
+            probabilities, which breaks ties more gracefully when a group has
+            few sequences.
+
+    Returns:
+        ``(groups, voted, n_seqs)`` — the sorted unique group ids, the voted
+        class for each, and how many sequences each group had.
+    """
+    if len(group_ids) != len(predictions):
+        raise ValueError("group_ids and predictions must be the same length")
+    if len(group_ids) == 0:
+        empty_i = np.zeros(0, dtype=np.int64)
+        return empty_i, empty_i, empty_i
+
+    groups = np.unique(group_ids)
+    n_groups = int(groups.max()) + 1  # group_ids are dense, so this is exact
+
+    if mode == "hard":
+        flat = group_ids.astype(np.int64) * num_classes + predictions.astype(np.int64)
+        tally = np.bincount(flat, minlength=n_groups * num_classes).reshape(n_groups, num_classes)
+    elif mode == "soft":
+        if probs is None:
+            raise ValueError("mode='soft' needs probs")
+        tally = np.zeros((n_groups, num_classes), dtype=np.float64)
+        np.add.at(tally, group_ids.astype(np.int64), probs.astype(np.float64))
+    else:
+        raise ValueError(f"unknown mode {mode!r} (hard|soft)")
+
+    voted = tally.argmax(axis=1)
+    counts = np.bincount(group_ids.astype(np.int64), minlength=n_groups)
+    return groups, voted[groups], counts[groups]
+
+
+def group_truth(group_ids: np.ndarray, labels: np.ndarray) -> np.ndarray:
+    """The ground-truth label of each group.
+
+    Every sequence in a group must carry the same label — a group is one
+    function or one binary, and a function has exactly one provenance. A
+    mismatch means the corpus or the grouping is wrong, so this raises rather
+    than silently picking one.
+    """
+    groups = np.unique(group_ids)
+    n_groups = int(groups.max()) + 1 if len(groups) else 0
+    first = np.full(n_groups, -1, dtype=np.int64)
+    order = np.argsort(group_ids, kind="stable")
+    g_sorted = group_ids[order]
+    l_sorted = labels[order]
+    starts = np.searchsorted(g_sorted, groups, side="left")
+    first[groups] = l_sorted[starts]
+
+    expanded = first[group_ids]
+    bad = expanded != labels
+    if bad.any():
+        n = int(bad.sum())
+        raise ValueError(
+            f"{n} sequences disagree with their group's label; "
+            "the voting groups do not match the label granularity"
+        )
+    return first[groups]
+
+
+def vote_report(
+    group_ids: np.ndarray,
+    predictions: np.ndarray,
+    labels: np.ndarray,
+    num_classes: int,
+    *,
+    probs: np.ndarray | None = None,
+    mode: str = "hard",
+) -> dict:
+    """Voting accuracy plus the sequence-level accuracy it was built from."""
+    groups, voted, counts = majority_vote(
+        group_ids, predictions, num_classes, probs=probs, mode=mode
+    )
+    truth = group_truth(group_ids, labels)
+    correct = voted == truth
+    return {
+        "num_groups": int(len(groups)),
+        "num_sequences": int(len(predictions)),
+        "seq_accuracy": float((predictions == labels).mean()) if len(predictions) else 0.0,
+        "vote_accuracy": float(correct.mean()) if len(groups) else 0.0,
+        "mean_seqs_per_group": float(counts.mean()) if len(counts) else 0.0,
+        "median_seqs_per_group": float(np.median(counts)) if len(counts) else 0.0,
+        "mode": mode,
+        # kept for the per-binary drill-down in the analysis experiments
+        "_groups": groups,
+        "_voted": voted,
+        "_truth": truth,
+        "_counts": counts,
+    }

binprov_config.json

@@ -0,0 +1,10 @@
+{
+  "seq_bytes": 2048,
+  "hidden_size": 768,
+  "num_hidden_layers": 12,
+  "num_attention_heads": 12,
+  "intermediate_size": 3072,
+  "max_segments": 2,
+  "hidden_dropout_prob": 0.1,
+  "attention_probs_dropout_prob": 0.1
+}

environment.json

@@ -0,0 +1,8 @@
+{
+  "python": "3.12.1",
+  "platform": "Linux-4.18.0-553.137.1.el8_10.x86_64-x86_64-with-glibc2.28",
+  "sys.platform": "linux",
+  "torch": "2.8.0+cu128",
+  "transformers": "4.55.4",
+  "numpy": "2.3.3"
+}

evidence/args.json

@@ -0,0 +1,57 @@
+{
+  "corpus": "data/corpus/binkit_x86_64",
+  "task": "opt4",
+  "out": "results/hopper_release/opt4_wide_seed29/opt4",
+  "splits": "default",
+  "init_from": "results/hopper_release/opt4_wide_seed29/mlm2048",
+  "from_scratch": false,
+  "tag": "opt4_wide_seed29",
+  "arch": null,
+  "extra": null,
+  "compiler": null,
+  "version": null,
+  "test_version": null,
+  "train_compiler": null,
+  "max_seqs_per_binary": null,
+  "stride": 512,
+  "val_frac": 0.15,
+  "val_seed": 1234,
+  "split_mode": "program",
+  "seq_bytes": 2048,
+  "eval_window_bytes": 512,
+  "jitter": 0,
+  "pos_extend": "tile",
+  "pool": "border",
+  "pool_heads": 4,
+  "mil_tau": 1.0,
+  "taper": 32,
+  "dropout": null,
+  "classifier_dropout": 0.1,
+  "epochs": 6,
+  "batch_size": 4,
+  "grad_accum": 8,
+  "lr": 3e-05,
+  "head_lr": 0.0001,
+  "llrd": 1.0,
+  "weight_decay": 0.01,
+  "warmup_ratio": 0.06,
+  "clip_grad": 1.0,
+  "label_smoothing": 0.0,
+  "use_pairs": false,
+  "pair_loss": 0.0,
+  "pair_ce": 1.0,
+  "pair_margin": 1.0,
+  "pairs_per_batch": 32,
+  "pair_min_size": 32,
+  "pair_jitter": 64,
+  "workers": 6,
+  "seed": 29,
+  "resume": true,
+  "log_every": 100,
+  "val_max_seqs": 40000,
+  "tta": "none",
+  "tta_views": 4,
+  "tta_span": 128,
+  "save_probs": true,
+  "no_save_weights": false
+}

evidence/evaluation.json

@@ -0,0 +1,50 @@
+{
+  "profile": "opt4_wide_seed29",
+  "task": "opt4",
+  "scored_classes": [
+    "O0",
+    "O1",
+    "O2",
+    "O3"
+  ],
+  "sequence_accuracy": 0.842255,
+  "balanced_accuracy": 0.830951,
+  "binary_accuracy": 0.93883,
+  "num_sequences": 116321,
+  "num_binaries": 376,
+  "recorded_test_accuracy": 0.8422554826729481,
+  "recomputed_matches_recorded": true,
+  "checkpoint": "results/hopper_release/opt4_wide_seed29/opt4",
+  "result_json": {
+    "accuracy": 0.8422554826729481,
+    "balanced_accuracy": 0.8309510174499452,
+    "confusion": [
+      [
+        37645,
+        14,
+        34,
+        19
+      ],
+      [
+        10,
+        21157,
+        1149,
+        636
+      ],
+      [
+        1,
+        2449,
+        17545,
+        6190
+      ],
+      [
+        1,
+        1860,
+        5986,
+        21625
+      ]
+    ],
+    "n": 116321
+  },
+  "evaluated_at": "2026-09-06T03:48:35"
+}

evidence/probs.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58cc99e63e97eed6dc99c56e7e334671c8bdbd9353bf96d5af45ea12f5e86eaf
+size 1415873

evidence/result.json

@@ -0,0 +1,141 @@
+{
+  "tag": "opt4_wide_seed29",
+  "task": "opt4",
+  "scored_classes": [
+    "O0",
+    "O1",
+    "O2",
+    "O3"
+  ],
+  "args": {
+    "corpus": "data/corpus/binkit_x86_64",
+    "task": "opt4",
+    "out": "results/hopper_release/opt4_wide_seed29/opt4",
+    "splits": "default",
+    "init_from": "results/hopper_release/opt4_wide_seed29/mlm2048",
+    "from_scratch": false,
+    "tag": "opt4_wide_seed29",
+    "arch": null,
+    "extra": null,
+    "compiler": null,
+    "version": null,
+    "test_version": null,
+    "train_compiler": null,
+    "max_seqs_per_binary": null,
+    "stride": 512,
+    "val_frac": 0.15,
+    "val_seed": 1234,
+    "split_mode": "program",
+    "seq_bytes": 2048,
+    "eval_window_bytes": 512,
+    "jitter": 0,
+    "pos_extend": "tile",
+    "pool": "border",
+    "pool_heads": 4,
+    "mil_tau": 1.0,
+    "taper": 32,
+    "dropout": null,
+    "classifier_dropout": 0.1,
+    "epochs": 6,
+    "batch_size": 4,
+    "grad_accum": 8,
+    "lr": 3e-05,
+    "head_lr": 0.0001,
+    "llrd": 1.0,
+    "weight_decay": 0.01,
+    "warmup_ratio": 0.06,
+    "clip_grad": 1.0,
+    "label_smoothing": 0.0,
+    "use_pairs": false,
+    "pair_loss": 0.0,
+    "pair_ce": 1.0,
+    "pair_margin": 1.0,
+    "pairs_per_batch": 32,
+    "pair_min_size": 32,
+    "pair_jitter": 64,
+    "workers": 6,
+    "seed": 29,
+    "resume": true,
+    "log_every": 100,
+    "val_max_seqs": 40000,
+    "tta": "none",
+    "tta_views": 4,
+    "tta_span": 128,
+    "save_probs": true,
+    "no_save_weights": false
+  },
+  "best_epoch": 4,
+  "best_val_accuracy": 0.776125,
+  "test": {
+    "accuracy": 0.8422554826729481,
+    "balanced_accuracy": 0.8309510174499452,
+    "confusion": [
+      [
+        37645,
+        14,
+        34,
+        19
+      ],
+      [
+        10,
+        21157,
+        1149,
+        636
+      ],
+      [
+        1,
+        2449,
+        17545,
+        6190
+      ],
+      [
+        1,
+        1860,
+        5986,
+        21625
+      ]
+    ],
+    "n": 116321
+  },
+  "test_tta": null,
+  "history": [
+    {
+      "epoch": 0,
+      "val_accuracy": 0.714775,
+      "val_balanced_accuracy": 0.7049004306882054,
+      "val_n": 40000
+    },
+    {
+      "epoch": 1,
+      "val_accuracy": 0.75455,
+      "val_balanced_accuracy": 0.735723491217099,
+      "val_n": 40000
+    },
+    {
+      "epoch": 2,
+      "val_accuracy": 0.75315,
+      "val_balanced_accuracy": 0.7330662254462548,
+      "val_n": 40000
+    },
+    {
+      "epoch": 3,
+      "val_accuracy": 0.767475,
+      "val_balanced_accuracy": 0.7509352274830212,
+      "val_n": 40000
+    },
+    {
+      "epoch": 4,
+      "val_accuracy": 0.776125,
+      "val_balanced_accuracy": 0.7567212069146406,
+      "val_n": 40000
+    },
+    {
+      "epoch": 5,
+      "val_accuracy": 0.7684,
+      "val_balanced_accuracy": 0.7490872252223905,
+      "val_n": 40000
+    }
+  ],
+  "n_train_sequences": 264459,
+  "wall_clock": "6h54m"
+}

head.json

@@ -0,0 +1,8 @@
+{
+  "num_labels": 4,
+  "pool": "border",
+  "pool_heads": 4,
+  "mil_tau": 1.0,
+  "task": "opt4",
+  "tag": "opt4_wide_seed29"
+}

inference.py

@@ -0,0 +1,173 @@
+#!/usr/bin/env python3
+"""Minimal BinProv inference for a packaged export directory.
+
+This file is copied verbatim into every ``export_hf.py`` output and is the
+supported way to run a released checkpoint. It loads the BinProv-native model
+from ``model/`` next to this file, reads an ELF's ``.text`` with BinProv's own
+dependency-free reader (or raw ``.text`` bytes), replicates the training cut into
+``seq_bytes``-byte windows, predicts each window, and soft-votes per class over
+the windows.
+
+The end-to-end model is NOT a transformers AutoModel (the head is BinProv's), so
+this script needs the BinProv package installed::
+
+    pip install -r requirements.txt
+
+No remote code is executed anywhere; there is no ``trust_remote_code``.
+
+Example::
+
+    python inference.py --model model --elf /path/to/binary.elf
+    python inference.py --model model --text-bytes raw_text.bin
+    python inference.py --model model --elf a.elf --json   # machine-readable
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import numpy as np
+import torch
+
+EXPORT_ROOT = Path(__file__).resolve().parent
+DEFAULT_MODEL = EXPORT_ROOT / "model"
+
+
+def load_model(model_dir: Path):
+    from binprov.model import BinProvForProvenance
+
+    return BinProvForProvenance.load(model_dir)
+
+
+def cut_windows(data: bytes, seq_bytes: int, stride: int, min_bytes: int = 16):
+    """Cut overlapping model inputs at the evaluation stride."""
+    out = []
+    pos = 0
+    n = len(data)
+    i = 0
+    while pos < n:
+        ln = min(seq_bytes, n - pos)
+        if ln < min_bytes and pos > 0:
+            break
+        out.append((np.frombuffer(data[pos : pos + ln], dtype=np.uint8), i))
+        i += 1
+        if ln < seq_bytes:
+            break
+        pos += stride
+    return out
+
+
+def predict(model, data: bytes, *, stride: int, batch_size: int, device):
+    """Per-class mean probability over the binary's windows.
+
+    Returns ``(probabilities, num_windows)`` with probabilities the soft vote of
+    every non-overlapping ``seq_bytes`` window of ``data``.
+    """
+    from binprov.data import ClassificationCollator
+
+    seq = model.cfg.seq_bytes
+    coll = ClassificationCollator(model.cfg.seq_tokens)
+    windows = cut_windows(data, seq, stride)
+    if not windows:
+        raise ValueError("no usable windows: input is empty or shorter than 16 bytes")
+    model.to(device).eval()
+    total = np.zeros(model.num_labels, dtype=np.float64)
+    with torch.no_grad():
+        for start in range(0, len(windows), batch_size):
+            items = [(chunk, -1, idx) for chunk, idx in windows[start:start + batch_size]]
+            out = coll(items)
+            ids = out["input_ids"].to(device)
+            attn = out["attention_mask"].to(device)
+            types = out["token_type_ids"].to(device)
+            logits = model(ids, attn, types)["logits"]
+            total += logits.float().softmax(-1).sum(0).cpu().numpy()
+    return total / len(windows), len(windows)
+
+
+def choose_device(requested: str):
+    if requested != "auto":
+        return torch.device(requested)
+    if torch.cuda.is_available():
+        return torch.device("cuda")
+    if torch.backends.mps.is_available():
+        return torch.device("mps")
+    return torch.device("cpu")
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(
+        description="BinProv inference on an ELF or raw .text bytes",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=__doc__,
+    )
+    ap.add_argument("--model", default=str(DEFAULT_MODEL),
+                    help="model dir (model.pt + configs); default: ./model")
+    src = ap.add_mutually_exclusive_group(required=True)
+    src.add_argument("--elf", default=None, help="ELF binary to classify")
+    src.add_argument("--text-bytes", default=None,
+                     help="file containing raw .text bytes")
+    ap.add_argument("--json", action="store_true", help="print JSON only")
+    ap.add_argument("--stride", type=int, default=None,
+                    help="window stride in bytes; default: packaged evaluation stride")
+    ap.add_argument("--batch-size", type=int, default=1,
+                    help="inference batch size; default 1 limits memory use")
+    ap.add_argument("--device", default="auto",
+                    help="torch device (auto, cuda, mps, cpu); default: auto")
+    args = ap.parse_args()
+
+    if args.elf:
+        from binprov.elf import parse
+
+        buf = Path(args.elf).read_bytes()
+        text = parse(buf).data
+        source = f"ELF {args.elf} (.text, {len(text)} bytes)"
+    else:
+        text = Path(args.text_bytes).read_bytes()
+        source = f"{args.text_bytes} ({len(text)} bytes)"
+
+    model_dir = Path(args.model)
+    model, head = load_model(model_dir)
+    labels_file = model_dir / "labels.json"
+    labels = json.loads(labels_file.read_text()) if labels_file.is_file() else {}
+    classes = labels.get("classes") or []
+    inference_cfg_file = model_dir / "inference_config.json"
+    inference_cfg = json.loads(inference_cfg_file.read_text()) if inference_cfg_file.is_file() else {}
+    stride = args.stride or int(inference_cfg.get("stride", model.cfg.seq_bytes))
+    if stride <= 0 or args.batch_size <= 0:
+        ap.error("--stride and --batch-size must be positive")
+    device = choose_device(args.device)
+    prob, n_windows = predict(
+        model, text, stride=stride, batch_size=args.batch_size, device=device
+    )
+    pred = int(prob.argmax())
+    result = {
+        "source": source,
+        "num_windows": n_windows,
+        "seq_bytes": model.cfg.seq_bytes,
+        "stride": stride,
+        "device": str(device),
+        "classes": classes,
+        "probabilities": [round(float(x), 6) for x in prob],
+        "prediction": pred,
+        "predicted_label": classes[pred] if pred < len(classes) else str(pred),
+        "task": labels.get("task"),
+    }
+    if args.json:
+        print(json.dumps(result, indent=2))
+    else:
+        print(f"input: {source}")
+        print(f"task: {result['task']}   classes: {classes}")
+        print(f"windows: {n_windows}   input: {model.cfg.seq_bytes} B   stride: {stride} B")
+        print(f"device: {device}")
+        for i, p in enumerate(prob):
+            name = classes[i] if i < len(classes) else str(i)
+            print(f"  {name:>6}: {100 * p:.2f}%")
+        print(f"=> {result['predicted_label']}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

inference_config.json

@@ -0,0 +1,5 @@
+{
+  "seq_bytes": 2048,
+  "stride": 512,
+  "min_bytes": 16
+}

labels.json

@@ -0,0 +1,16 @@
+{
+  "task": "opt4",
+  "classes": [
+    "O0",
+    "O1",
+    "O2",
+    "O3"
+  ],
+  "label_map": {
+    "0": "O0",
+    "1": "O1",
+    "2": "O2",
+    "3": "O3"
+  },
+  "num_labels": 4
+}

metadata.json

@@ -0,0 +1,296 @@
+{
+  "args.json": {
+    "corpus": "data/corpus/binkit_x86_64",
+    "task": "opt4",
+    "out": "results/hopper_release/opt4_wide_seed29/opt4",
+    "splits": "default",
+    "init_from": "results/hopper_release/opt4_wide_seed29/mlm2048",
+    "from_scratch": false,
+    "tag": "opt4_wide_seed29",
+    "arch": null,
+    "extra": null,
+    "compiler": null,
+    "version": null,
+    "test_version": null,
+    "train_compiler": null,
+    "max_seqs_per_binary": null,
+    "stride": 512,
+    "val_frac": 0.15,
+    "val_seed": 1234,
+    "split_mode": "program",
+    "seq_bytes": 2048,
+    "eval_window_bytes": 512,
+    "jitter": 0,
+    "pos_extend": "tile",
+    "pool": "border",
+    "pool_heads": 4,
+    "mil_tau": 1.0,
+    "taper": 32,
+    "dropout": null,
+    "classifier_dropout": 0.1,
+    "epochs": 6,
+    "batch_size": 4,
+    "grad_accum": 8,
+    "lr": 3e-05,
+    "head_lr": 0.0001,
+    "llrd": 1.0,
+    "weight_decay": 0.01,
+    "warmup_ratio": 0.06,
+    "clip_grad": 1.0,
+    "label_smoothing": 0.0,
+    "use_pairs": false,
+    "pair_loss": 0.0,
+    "pair_ce": 1.0,
+    "pair_margin": 1.0,
+    "pairs_per_batch": 32,
+    "pair_min_size": 32,
+    "pair_jitter": 64,
+    "workers": 6,
+    "seed": 29,
+    "resume": true,
+    "log_every": 100,
+    "val_max_seqs": 40000,
+    "tta": "none",
+    "tta_views": 4,
+    "tta_span": 128,
+    "save_probs": true,
+    "no_save_weights": false
+  },
+  "result.json": {
+    "tag": "opt4_wide_seed29",
+    "task": "opt4",
+    "scored_classes": [
+      "O0",
+      "O1",
+      "O2",
+      "O3"
+    ],
+    "args": {
+      "corpus": "data/corpus/binkit_x86_64",
+      "task": "opt4",
+      "out": "results/hopper_release/opt4_wide_seed29/opt4",
+      "splits": "default",
+      "init_from": "results/hopper_release/opt4_wide_seed29/mlm2048",
+      "from_scratch": false,
+      "tag": "opt4_wide_seed29",
+      "arch": null,
+      "extra": null,
+      "compiler": null,
+      "version": null,
+      "test_version": null,
+      "train_compiler": null,
+      "max_seqs_per_binary": null,
+      "stride": 512,
+      "val_frac": 0.15,
+      "val_seed": 1234,
+      "split_mode": "program",
+      "seq_bytes": 2048,
+      "eval_window_bytes": 512,
+      "jitter": 0,
+      "pos_extend": "tile",
+      "pool": "border",
+      "pool_heads": 4,
+      "mil_tau": 1.0,
+      "taper": 32,
+      "dropout": null,
+      "classifier_dropout": 0.1,
+      "epochs": 6,
+      "batch_size": 4,
+      "grad_accum": 8,
+      "lr": 3e-05,
+      "head_lr": 0.0001,
+      "llrd": 1.0,
+      "weight_decay": 0.01,
+      "warmup_ratio": 0.06,
+      "clip_grad": 1.0,
+      "label_smoothing": 0.0,
+      "use_pairs": false,
+      "pair_loss": 0.0,
+      "pair_ce": 1.0,
+      "pair_margin": 1.0,
+      "pairs_per_batch": 32,
+      "pair_min_size": 32,
+      "pair_jitter": 64,
+      "workers": 6,
+      "seed": 29,
+      "resume": true,
+      "log_every": 100,
+      "val_max_seqs": 40000,
+      "tta": "none",
+      "tta_views": 4,
+      "tta_span": 128,
+      "save_probs": true,
+      "no_save_weights": false
+    },
+    "best_epoch": 4,
+    "best_val_accuracy": 0.776125,
+    "test": {
+      "accuracy": 0.8422554826729481,
+      "balanced_accuracy": 0.8309510174499452,
+      "confusion": [
+        [
+          37645,
+          14,
+          34,
+          19
+        ],
+        [
+          10,
+          21157,
+          1149,
+          636
+        ],
+        [
+          1,
+          2449,
+          17545,
+          6190
+        ],
+        [
+          1,
+          1860,
+          5986,
+          21625
+        ]
+      ],
+      "n": 116321
+    },
+    "test_tta": null,
+    "history": [
+      {
+        "epoch": 0,
+        "val_accuracy": 0.714775,
+        "val_balanced_accuracy": 0.7049004306882054,
+        "val_n": 40000
+      },
+      {
+        "epoch": 1,
+        "val_accuracy": 0.75455,
+        "val_balanced_accuracy": 0.735723491217099,
+        "val_n": 40000
+      },
+      {
+        "epoch": 2,
+        "val_accuracy": 0.75315,
+        "val_balanced_accuracy": 0.7330662254462548,
+        "val_n": 40000
+      },
+      {
+        "epoch": 3,
+        "val_accuracy": 0.767475,
+        "val_balanced_accuracy": 0.7509352274830212,
+        "val_n": 40000
+      },
+      {
+        "epoch": 4,
+        "val_accuracy": 0.776125,
+        "val_balanced_accuracy": 0.7567212069146406,
+        "val_n": 40000
+      },
+      {
+        "epoch": 5,
+        "val_accuracy": 0.7684,
+        "val_balanced_accuracy": 0.7490872252223905,
+        "val_n": 40000
+      }
+    ],
+    "n_train_sequences": 264459,
+    "wall_clock": "6h54m"
+  },
+  "head.json": {
+    "num_labels": 4,
+    "pool": "border",
+    "pool_heads": 4,
+    "mil_tau": 1.0,
+    "task": "opt4",
+    "tag": "opt4_wide_seed29"
+  },
+  "binprov_config.json": {
+    "seq_bytes": 2048,
+    "hidden_size": 768,
+    "num_hidden_layers": 12,
+    "num_attention_heads": 12,
+    "intermediate_size": 3072,
+    "max_segments": 2,
+    "hidden_dropout_prob": 0.1,
+    "attention_probs_dropout_prob": 0.1
+  },
+  "source_checkpoint": "results/hopper_release/opt4_wide_seed29/opt4",
+  "labels": {
+    "task": "opt4",
+    "classes": [
+      "O0",
+      "O1",
+      "O2",
+      "O3"
+    ],
+    "label_map": {
+      "0": "O0",
+      "1": "O1",
+      "2": "O2",
+      "3": "O3"
+    },
+    "num_labels": 4
+  },
+  "environment": {
+    "python": "3.12.1",
+    "platform": "Linux-4.18.0-553.137.1.el8_10.x86_64-x86_64-with-glibc2.28",
+    "sys.platform": "linux",
+    "torch": "2.8.0+cu128",
+    "transformers": "4.55.4",
+    "numpy": "2.3.3"
+  },
+  "summary": {
+    "model_name": "binprov",
+    "exported_from": "results/hopper_release/opt4_wide_seed29/opt4",
+    "created_at": "2026-09-06T07:48:37+00:00",
+    "files": [
+      "LICENSE",
+      "MODEL_CARD.md",
+      "README.md",
+      "binprov/__init__.py",
+      "binprov/corpus.py",
+      "binprov/data.py",
+      "binprov/discover.py",
+      "binprov/elf.py",
+      "binprov/metrics.py",
+      "binprov/model.py",
+      "binprov/provenance.py",
+      "binprov/vocab.py",
+      "binprov/vote.py",
+      "binprov_config.json",
+      "environment.json",
+      "evidence/args.json",
+      "evidence/evaluation.json",
+      "evidence/probs.npz",
+      "evidence/result.json",
+      "head.json",
+      "inference.py",
+      "inference_config.json",
+      "labels.json",
+      "model/binprov_config.json",
+      "model/head.json",
+      "model/inference_config.json",
+      "model/labels.json",
+      "model/model.pt",
+      "requirements.txt",
+      "metadata.json"
+    ],
+    "sequence_accuracy": 0.842255,
+    "binary_accuracy": 0.93883,
+    "transformers_compatible": false,
+    "note": "BinProv-native PyTorch checkpoint; use the bundled inference.py runtime.",
+    "verification": {
+      "verified": true,
+      "device": "cpu",
+      "windows": 4,
+      "mean_prob": [
+        0.5000002359981011,
+        0.4999849506011037,
+        6.91615203089746e-06,
+        7.897442639431818e-06
+      ],
+      "prediction": 0
+    }
+  }
+}

model/binprov_config.json

@@ -0,0 +1,10 @@
+{
+  "seq_bytes": 2048,
+  "hidden_size": 768,
+  "num_hidden_layers": 12,
+  "num_attention_heads": 12,
+  "intermediate_size": 3072,
+  "max_segments": 2,
+  "hidden_dropout_prob": 0.1,
+  "attention_probs_dropout_prob": 0.1
+}

model/head.json

@@ -0,0 +1,8 @@
+{
+  "num_labels": 4,
+  "pool": "border",
+  "pool_heads": 4,
+  "mil_tau": 1.0,
+  "task": "opt4",
+  "tag": "opt4_wide_seed29"
+}

model/inference_config.json

@@ -0,0 +1,5 @@
+{
+  "seq_bytes": 2048,
+  "stride": 512,
+  "min_bytes": 16
+}

model/labels.json

@@ -0,0 +1,16 @@
+{
+  "task": "opt4",
+  "classes": [
+    "O0",
+    "O1",
+    "O2",
+    "O3"
+  ],
+  "label_map": {
+    "0": "O0",
+    "1": "O1",
+    "2": "O2",
+    "3": "O3"
+  },
+  "num_labels": 4
+}

model/model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93efdc354956bd8e3f5abc254b97538947b856e185c737f891f00ddb95065247
+size 349795459

requirements.txt

@@ -0,0 +1,36 @@
+# BinProv runtime dependencies.
+#
+# Deliberately short. The original paper built the network with fairseq; this
+# re-implementation uses HuggingFace transformers instead, which is compatible
+# with current PyTorch (fairseq pins old torch versions and no longer builds
+# cleanly). The architecture is the same: a 12-layer, 768-hidden RoBERTa/BERT
+# encoder over a 261-token byte vocabulary.
+#
+# ELF parsing is done in-tree (binprov/elf.py) so corpus building needs no
+# native dependencies, and the metrics are hand-rolled so evaluation runs
+# anywhere the model runs.
+
+torch>=2.0
+transformers>=4.35
+numpy>=1.23
+
+# Optional, only for scripts/fetch_binkit.sh:
+#   gdown>=4.7    BinKit is hosted on Google Drive
+#   py7zr>=0.20   fallback extractor; BinKit ships .7z archives. Preferred is
+#                 bsdtar/libarchive, which is usually already installed and
+#                 reads .7z with the same wildcard syntax as tar.
+
+# ---------------------------------------------------------------------------
+# Versions the results in docs/ were produced with. The bounds above are what
+# the code needs; these are what it ran on. Accuracy numbers on this task move
+# by 1-2 points between seeds, so a different torch or transformers version is
+# a plausible source of a similar-sized shift -- pin these if you are comparing
+# against the tables rather than just running the pipeline.
+#
+#   python           3.12.11
+#   torch            2.8.0+cu128
+#   transformers     4.55.4
+#   numpy            2.3.3
+#
+# GPU: one NVIDIA H200 (141 GB). bf16 autocast throughout. The 2048-byte arms
+# need ~33 GB; the 512-byte arms ~26 GB at batch 128.