diff --git a/README.md b/README.md index f616d3f851d18222a00e2b9226f8c08126db4002..45a809ea91dad49466d466075ccd30691725bac7 100644 --- a/README.md +++ b/README.md @@ -8,147 +8,215 @@ tags: - coreml - weight-space-programming - handwritten-weights +- gradient-free - research - mill - succinct-labs - fork-and-flush inference: false +model_index: +- name: Blot Mini XS + results: + - task: + type: token-classification + name: Developer-secret location detection + dataset: + type: creddata + name: CredData full corpus (reused development data) + split: full + args: + revision: 86a918283f086baf9ca896a892519cd8462dfe0a + files: 11030 + metrics: + - type: f1 + name: Labeled-location F1 (%) + value: 49.64 + - type: precision + name: Labeled-location precision (%) + value: 58.0651 + - type: recall + name: Labeled-location recall (%) + value: 43.35 +- name: Blot Mini S + results: + - task: + type: token-classification + name: Developer-secret location detection + dataset: + type: creddata + name: CredData full corpus (reused development data) + split: full + args: + revision: 86a918283f086baf9ca896a892519cd8462dfe0a + files: 11030 + metrics: + - type: f1 + name: Labeled-location F1 (%) + value: 54.4316 + - type: precision + name: Labeled-location precision (%) + value: 71.1522 + - type: recall + name: Labeled-location recall (%) + value: 44.0743 +- name: Blot Mini M + results: + - task: + type: token-classification + name: Developer-secret location detection + dataset: + type: creddata + name: CredData full corpus (reused development data) + split: full + args: + revision: 86a918283f086baf9ca896a892519cd8462dfe0a + files: 11030 + metrics: + - type: f1 + name: Labeled-location F1 (%) + value: 65.4649 + - type: precision + name: Labeled-location precision (%) + value: 77.4917 + - type: recall + name: Labeled-location recall (%) + value: 56.6697 --- -# Blot Mini +# Blot Mini: XS, S and M
- Succinct Labs: Built with Mill + Built with Mill by Succinct Labs
Method: Direct weight authoring - Checkpoint: Precision A - Parameters: 2,283,666 - Inference assets: 9.17 MB - Format: Core ML FP32 + Variants: XS, S, M + Sizes: 1.05, 9.17, 24.57 MB + Format: FP32 + M full CredData F1: 65.46%
-**Built with [Mill](https://succinctlabs.ai/mill), the agentic research environment from [Succinct Labs](https://succinctlabs.ai).** Blot Mini finds API keys and developer-secret spans for downstream redaction. Agents wrote its neural weights directly, with development feedback and fork-and-flush iterations. No gradients, optimizer training, pretrained tensors or teacher distillation were used to construct this detector. +**Built with [Mill](https://succinctlabs.ai/mill), the agentic research environment from [Succinct Labs](https://succinctlabs.ai).** Blot Mini finds API keys and developer-secret spans in text for downstream redaction. Agents authored and edited its neural weights directly, using development feedback and fork-and-flush rounds. There was no gradient training, optimizer fitting, pretrained detector initialization or teacher distillation. -**The default download now contains the 9.17 MB precision winner**, checkpoint `precision-a-2026-11-04`, published November 5, 2026. Its source, NumPy weights and Core ML package are byte-identical to the candidate frozen before the final evaluation. It is a research release; publication does not establish production readiness or update the macOS test app. +**Choose a size, like a model family with several download options.** XS, S and M are distinct authored checkpoints with different network sizes and behavior. They are **not quantizations of one common model**: all three use FP32 coefficients, and no INT4/INT8 version is claimed. The default root download is now **M**, the frozen `uuid2` candidate evaluated on November 6, 2026. XS and S remain available in this same repository with their original inference files unchanged. -## Paper +## Choose a variant -**[Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights](https://rikiyat.github.io/blog/weight-space-programming/)** -Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, MIT CSAIL. Published November 2, 2026. +| Variant | Role | Checkpoint | Parameters | Inference assets | Download location | +|---|---|---|---:|---:|---| +| **Blot Mini XS** | Smallest | `r3a` | 257,751 | **1,044,930 bytes / 1.045 MB** | [`variants/xs/`](variants/xs) | +| **Blot Mini S** | Middle size | `precision-a-2026-11-04` | 2,283,666 | **9,166,928 bytes / 9.167 MB** | [`variants/s/`](variants/s) | +| **Blot Mini M** | Highest measured full-CredData F1; default | `uuid2-2026-11-05` | 6,129,296 | **24,574,583 bytes / 24.575 MB** | [Repository root](https://huggingface.co/pmarquees/blot-mini/tree/main) | -This research article inspired Blot Mini's direct weight authoring: agents construct and edit neural coefficients without gradient descent. Blot Mini applies that idea to secret detection with a fixed lexical frontend and frozen evaluation. It is our adaptation, not an official implementation or reproduction of every condition in the article. +Each variant includes its own Python frontend, frozen NumPy weights and Core ML FP32 package. The sizes above are **measured uncompressed inference assets**: `model.py`, `weights.npz` and `bundle/`. They exclude dependencies, documentation, examples, compiler caches and application packaging. They are not RAM measurements or the size of downloading the entire multi-variant repository. Every individual bundle is below the 30 MB research cap. A small compressed NPZ is not the deployed dense network size. -## From a 1 MB first release to the 9 MB winner +The networks have 5,153 input features and 50 / 443 / 1,189 hidden ReLU units for XS / S / M respectively. [variants.json](variants.json) provides paths, identities, sizes, hashes and results for programmatic selection. -Our first published model, **r3a**, packed 257,751 authored parameters into **1.045 MB** of inference assets. On the full pinned **CredData** corpus (11,030 files), it scored **49.64% labeled-location F1**, versus **59.57% for stock Gitleaks 8.30.1**: a gap of **9.93 percentage points**. This is about 16.7% lower in relative terms. r3a had already gone through three fork-and-flush rounds; it was not a one-shot result. CredData has known provenance overlap with its development data, so this is not an independent generalization claim. +## Same benchmark: all variants versus Gitleaks -We continued authoring the weights, and selected the **9.17 MB precision winner** now downloadable here. On a **separate frozen sample of 2,000 labeled GitHub issue candidates**, it scored **63.95% F1 versus Gitleaks' 52.49%**, a gain of **11.46 points**. Its precision was **96.18%**, versus Gitleaks' **98.35%**; its recall was **47.90%**, versus **35.80%**. It caught more labeled secrets, while Gitleaks remained more precise. +**Full pinned CredData: 11,030 files, 337 repositories, 15,188 positive annotated locations.** The primary metric is detection of unique annotated line ranges, called **labeled-location F1**. It is not token accuracy or exact-span F1. All rows below use the same corpus revision and scoring rules. -These are **different evaluations**, not a same-benchmark before-and-after curve. The new checkpoint has not been run on the full CredData corpus, and its larger size alone is not established as the cause of improvement. The issue sample measures localized labeled-candidate detection, not exhaustive document precision or universal scanner replacement. See the [historical CredData report](history/r3a/creddata/REPORT.md) and the current evaluation below. +| Frozen system | Precision | Recall | Labeled-location F1 | Conservative F1 | +|---|---:|---:|---:|---:| +| **Blot Mini XS — 1.045 MB** | 58.07% | 43.35% | **49.64%** | 42.27% | +| **Blot Mini S — 9.167 MB** | 71.15% | 44.07% | **54.43%** | 50.87% | +| **Blot Mini M — 24.575 MB** | 77.49% | **56.67%** | **65.46%** | **61.77%** | +| Stock **Gitleaks 8.30.1** | **85.99%** | 45.58% | **59.57%** | 58.96% | -| Current download | Value | -|---|---| -| Checkpoint | `precision-a-2026-11-04` | -| Parameters | 2,283,666 | -| Network | Authored two-layer ReLU scorer; 5,153 features, 443 hidden units, scalar readout | -| Inference assets | 9,166,928 bytes, approximately 9.17 MB | -| Dense FP32 coefficients | 9,134,664 bytes | -| Format | NumPy weights and Core ML FP32; no INT4/INT8 export claimed | -| Core ML target | iOS 18 / macOS 15 operations | -| Input pipeline | Required Python lexical candidate and feature frontend, followed by authored neural scoring | -| Output | `SECRET` spans with Unicode code-point offsets and exclusive ends | -| Status | Private research release | +- XS trails Gitleaks by **9.93 percentage points F1**, about 16.7% in relative terms. “Less than 10 points behind” does not mean “less than 10% lower.” +- S improves on XS by **4.79 points F1**, while remaining **5.14 points below Gitleaks** on this corpus. +- M improves on S by **11.03 points F1** and exceeds Gitleaks by **5.89 points F1**. Its conservative F1 is **2.81 points higher than Gitleaks**. Gitleaks remains more precise. -Asset size includes `model.py`, `weights.npz` and the Core ML package. It excludes runtime dependencies, documentation, historical artifacts, compiled caches and app packaging. It is not RAM usage or a complete application's size. A smaller ZIP or download reflects compression, not a smaller deployed network. +| System | True positives | Known false positives | False negatives | Unannotated finding lines | +|---|---:|---:|---:|---:| +| XS | 6,584 | 4,755 | 8,604 | 4,622 | +| S | 6,694 | 2,714 | 8,494 | 1,721 | +| M | 8,607 | 2,500 | 6,581 | 1,575 | +| Gitleaks 8.30.1 | 6,922 | 1,128 | 8,266 | 244 | -## Evaluation of the current downloadable checkpoint +Conservative F1 counts every finding line outside annotated ranges as a false positive. It is a pessimistic sensitivity view, not a claim that all unannotated findings are benign. For M, full labeled-value span coverage is **52.30%**, versus **38.75%** for S; detecting a labeled line does not guarantee complete redaction of its secret value. -Mill's research workflow continued direct weight authoring using two independent author branches with fresh conversations, adapting [Fork-and-Flush](https://arxiv.org/abs/2610.07447). Only the weight builder and its permitted prefix specification could change; candidate extraction, features, scoring threshold and decoder were frozen. No gradients, optimizer, pretrained tensors, distillation or automatic parameter sweep were used. Development feedback still guided the weight edits. +### What we can claim -The selected candidate added general weight-based inhibitors for ordinary web addresses, source paths, filename references and password placeholders, while preserving supporting evidence for credential-bearing formats. A competing branch exceeded 99% precision on author-development examples but lost too much CredData selection recall, so it was rejected before the final test. This is an adaptation of the research method, not evidence that branching itself outperforms a matched-budget single-agent control. +**M has the highest measured F1 in this frozen full-CredData comparison.** This is a corpus-specific result. CredData informed model development and selection, so the full run is **not a fresh independent holdout**. It does not establish scanner replacement or superiority on new repositories. -### Frozen issue-candidate comparison +The paired 95% repository-bootstrap interval for **M minus Gitleaks F1 is [−16.91, +44.13] percentage points**, which includes zero. Performance varies substantially by repository; the higher point estimate does not establish a reliable population-wide advantage. M minus S is **+11.03 points**, with interval **[+4.96, +21.36]**. Intervals use 2,000 paired repository resamples, seed 90272. -The held-out sample contains **2,000 labeled candidates from 2,000 distinct GitHub repositories**, with 1,000 positive and 1,000 negative upstream labels, drawn from the public [issue-report research dataset](https://doi.org/10.5281/zenodo.19622962). Its source repository is [Secret-Breach-Prevention-In-Software-Issue-Reports](https://github.com/nafiurahman00/Secret-Breach-Prevention-In-Software-Issue-Reports). This is a custom balanced subset, not a full CredData run or an industry-standard leaderboard. +Gitleaks uses its frozen stock 8.30.1 predictions, rescored with the unchanged evaluator; the results reproduced exactly. The scanner binary was not rerun during M confirmation. The M run completed all 11,030 files with zero file errors on an 8-vCPU Mill sandbox, resuming 5,646 verified saved predictions and evaluating 5,384 missing files. The model, thresholds, features, decoder, data and scorer remained frozen. No active credential verification occurred. -| Frozen system | TP | FP | FN | TN | Precision | Recall | F1 | -|---|---:|---:|---:|---:|---:|---:|---:| -| Immediate research parent, 6.98 MB; not published `r3a` | 480 | 40 | 520 | 960 | 92.31% | 48.00% | 63.16% | -| **Blot Mini precision A, 9.17 MB; current download** | 479 | 19 | 521 | 981 | **96.18%** | **47.90%** | **63.95%** | -| Stock Gitleaks 8.30.1 | 358 | 6 | 642 | 994 | 98.35% | 35.80% | 52.49% | +See [aggregate results and intervals](evaluation/creddata-2026-11-06/results.json), the [full confirmation report](evaluation/creddata-2026-11-06/REPORT.md), the [scorer](evaluation/creddata-2026-11-06/score.py), [selection protocol](evaluation/creddata-2026-11-06/protocol.json), and [resume provenance](evaluation/creddata-2026-11-06/resume-provenance.json). Raw benchmark text, secret values and per-case predictions are not included in this release. -Compared with the immediate research parent, the candidate reduced false positives on the labeled candidates from **40 to 19 (52.5%)**, while positive detections changed from **480 to 479**. Precision increased **3.88 percentage points**, with a paired 95% bootstrap interval **[+2.31, +5.56]**. Recall changed **−0.10 points**, interval **[−0.31, 0.00]**; F1 changed **+0.79 points**, interval **[+0.38, +1.22]**. The predeclared improvement gate passed: precision's lower interval endpoint >0, recall's lower endpoint ≥−2 points, and no F1 point-estimate regression. Intervals use 10,000 paired repository resamples, seed 20261104. +### Other historical evaluations -Against Gitleaks, the candidate had **higher primary F1 and recall, but lower precision**. The F1 difference was **+11.46 points**, 95% interval **[+8.63, +14.31]**. These figures measure localized labeled-candidate detection: a match requires overlap covering at least 50% of both the labeled target and predicted span. They do not estimate exhaustive document precision, production prevalence, exact-span accuracy or universal scanner replacement. Trained Blot, detect-secrets and TruffleHog were not rerun in this round. +S also scored **63.95% F1 versus Gitleaks' 52.49%** on a separate frozen, balanced sample of **2,000 GitHub issue candidates** from distinct repositories. That custom sample uses localized span overlap, not CredData's line-range metric. It is evidence for **S**, not an M result. XS's historical ProwlBench detection F1 was **86.19%** on a reused 24,603-snippet corpus; it is not comparable to the CredData percentages above. -Mandatory alternative views are included in the [complete results](research/precision-2026-11-04/RESULTS.md). Exact-boundary F1 was **24.78%** for both parent and candidate and **22.91%** for Gitleaks. Full-target coverage recall was **41.60%** for the candidate versus **35.80%** for Gitleaks. If every nonmatching output, including unannotated findings, is conservatively counted as false, candidate precision/F1 becomes **27.85% / 35.22%**, versus Gitleaks' **34.29% / 35.03%**. That sensitivity view is not complete ground truth and its small F1 difference does not establish superiority. +The [historical S card](history/precision-a-2026-11-04/README.md), [S issue-sample evidence](research/precision-2026-11-04/RESULTS.md), and [historical XS card](history/r3a/README.md) retain those details and the earlier overlap-audit correction. Trained Blot, detect-secrets and TruffleHog were not rerun in the M confirmation. The old overlap-excluded CredData slice is descriptive, not a restored clean holdout. -### Independence, integrity and correction +## How Mill made these models -The data and evaluator were frozen before candidate selection, and the selected model was frozen before one final inference pass per tool. All tools received the same complete normalized issue bodies. Models used 8,192-character windows with 512 overlap. Gitleaks used stock rules, default decoding and inline-allow behavior; all findings were mapped without unresolved or ambiguous locations. No active credential verification occurred. Raw texts, values and per-case predictions are not published. +This is a hybrid detector. A deterministic lexical frontend proposes spans and encodes character, shape and context features. A two-layer ReLU network scores those features using saved authored coefficients; a fixed threshold and overlap decoder select `SECRET` spans. The graph does not directly ingest strings. -The overlap audit rechecked 11,030 CredData files and 35 JSONL corpora and excluded all repositories from the preceding test. **An earlier audit had missed a Prowl span representation: 83 of the previous 1,000 cases contained previously seen labeled values. The earlier clean independent-test interpretation was withdrawn.** The new sample excludes the corrected exact overlaps and all previous test repositories. This does not prove that the old overlaps caused particular predictions. Procedural separation kept new test contents and feedback away from authors until candidate freeze; there was no OS sandbox, and unknown forks, paraphrases and author-model pretraining overlap cannot be completely certified. +1. **XS:** the first compact published checkpoint, built through three fork-and-flush rounds. Format and context conditions were encoded as weight-based conjunctions and inhibitory readout coefficients. +2. **S:** later direct-weight iterations added precision controls for benign URLs, paths, filenames, references and placeholders. Development labels informed the edits; the published S inference files are preserved byte for byte. +3. **M:** two additional fork-and-flush rounds on CredData. The first round compared independent precision, provider-recall and structural-recall branches. Fresh follow-up authors received the chosen precision parent and concise development-only lessons. A further authored UUID-format neuron won the predeclared selection gate. Only `authored_weights()` and its permitted prefix specification were editable; the rest of the model AST stayed unchanged from S. -The frozen candidate's Core ML and NumPy outputs matched on 353 invented control documents and logits matched on 256 arbitrary feature vectors. The full new holdout ran through the exact sparse NumPy matrices; it was not a separate full Core ML or physical-device evaluation. The 9.17 MB figure is uncompressed inference assets, excludes runtime dependencies, and remains below the 30 MB research cap. +M combines general source-reference inhibitors with a UUID-shaped-value detector requiring length and entropy features and suppressing explicit public/hash context. It does not look up benchmark literal values or paths. Benign UUIDs remain a real failure mode. -| Candidate identity | SHA-256 | -|---|---| -| `model.py` | `a881b7aed53b012c7cac1b2d532c26cf628a7748cdba81f148b023874de689bc` | -| `weights.npz` | `b0b4b1ce704651f3f4429cc83f1138537b4ae339df20dda0dad1a2e7a1b0b3ba` | -| New holdout | `7b2ea9d01ee6c4160954caad55011b1db6e78309ae18e5cd5642d758ecf58022` | +The selection split used 1,200 development files and 1,200 different selection files, restricted to files no larger than 128 KiB for iteration. Selection favored F1 while guarding conservative F1, recall and the 30 MB size cap. The final full run included large files too. These splits were separated by file, **not repository**, and the corpus was already reused. See the [round-one trace summary](evaluation/creddata-2026-11-06/round1-summary.json) and [round-two summary](evaluation/creddata-2026-11-06/round2-summary.json). + +No gradients does not mean no examples or no selection: feedback guided direct edits, and the authoring LLMs carry pretrained knowledge. We did not test an equivalent symbolic implementation or a matched-budget single-author control, so we do not claim that neural encoding or branching itself caused the improvement. Larger size alone is not established as the cause either. + +## Papers and research inspiration -See the [predeclared protocol](research/precision-2026-11-04/protocol.md), [aggregate metrics and intervals](research/precision-2026-11-04/results.json), [candidate freeze](research/precision-2026-11-04/candidate-freeze.json), [native validation](research/precision-2026-11-04/native-validation.json), [integrity receipt](research/precision-2026-11-04/postrun-integrity.json), and [old-test correction](research/precision-2026-11-04/old-holdout-corrected-prowl-audit.json). This holdout is now consumed; future optimization requires a new test for fresh generalization evidence. +**[Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights](https://rikiyat.github.io/blog/weight-space-programming/)** — Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, MIT CSAIL; November 2, 2026. This article inspired direct reasoning about and authoring of weight arrays without gradient descent. Blot Mini is our adaptation to a narrow secret-detection task, not an official implementation or reproduction of all the article's experiments. -## Run locally +**[Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents](https://arxiv.org/abs/2610.07447)** inspired the workflow of branching into fresh author contexts, evaluating proposals and continuing from the selected candidate. Our branch counts, tasks and evaluation protocol differ from the paper. -Use Python 3.12. For native Core ML inference, use macOS 15 or newer. Access to this private Hugging Face repository is required to download it; subsequent inference is local and does not verify whether credentials are active. +## Download and run a variant + +Download only the variant you want. For the **M default**, from a Hugging Face CLI installation: ```sh -hf auth login -hf download pmarquees/blot-mini --local-dir blot-mini -cd blot-mini +hf download pmarquees/blot-mini --local-dir blot-mini-m \ + --include "model.py" "weights.npz" "requirements.txt" "bundle/*" "examples/*" "frozen.json" +cd blot-mini-m python3.12 -m venv .venv source .venv/bin/activate python -m pip install -r requirements.txt -python model.py --coreml < examples/input.jsonl +python model.py < examples/input.jsonl ``` -Omit `--coreml` to run the exact sparse NumPy matrices. Each input line is JSON with a `text` field; each output line is a list of `{start, end, label}` objects. Offsets count Python Unicode code points, not UTF-8 bytes, UTF-16 code units or grapheme clusters. Scores are not calibrated probabilities. The [invented examples](examples/input.jsonl) have [expected outputs](examples/expected.jsonl), checked in both runtimes for this publication. - -The raw Core ML graph consumes extracted feature arrays, not strings. Keep `model.py`, `weights.npz` and `bundle/` together. This is not a Transformers `AutoModel` checkpoint or a native iOS SDK; deploying in an Apple app requires a frontend/decoder port and device validation. - -The CLI processes each supplied text directly. To reproduce the benchmark on long documents, use the evaluator's fixed **8,192-character windows with 512 overlap**, map outputs to original offsets and deduplicate identical spans. Direct full-document inference can differ at window boundaries; do not assume it reproduces the tiled benchmark. - -`python build.py` reconstructs the authored arrays and exports Core ML. Run it in a separate copy to retain the frozen release bytes: serialization metadata can change even when numerical weights agree. [frozen.json](frozen.json) identifies the benchmarked inference files; [SHA256SUMS](SHA256SUMS) covers the published files except itself. +For **XS** or **S**, select its directory instead: -## How Mill made it - -This experiment adapts [Sparks of Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming/) by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Agents wrote code that materializes the network's weight arrays. Prefix observations and contextual evidence become ReLU conjunctions, while negative readout coefficients inhibit placeholders, references and ordinary paths. Decisions use the saved matrices. - -The detector includes a deterministic lexical candidate/feature frontend. It is a hybrid system, not an end-to-end neural text encoder. The work does not establish that representing these conditions as weights beats an equivalent symbolic implementation. +```sh +hf download pmarquees/blot-mini --include "variants/xs/*" --local-dir blot-mini +cd blot-mini/variants/xs +# Create the environment and run model.py as above. +# Replace xs with s to select the 9 MB variant. +``` -Mill's workflow adapted [Fork-and-Flush](https://arxiv.org/abs/2610.07447): start independent authors in fresh conversations from a frozen parent and concise lessons, evaluate proposals on selection data, keep the eligible winner and discard unsuccessful branches. The original r3a release used three rounds with two authors per round. Later rounds improved recall and precision; the final precision round used two fresh authors with only the weight builder and permitted prefix specification editable. A separate evaluator held the final sample out until candidate freeze. +The repository is public. Inference runs on your machine after download and does not check whether credentials are live. Each input line is JSON with a `text` field; each output line is a list of `{start, end, label}` objects. Offsets count **Unicode code points**, with exclusive ends, not UTF-8 bytes or UTF-16 code units. Scores are not calibrated probabilities. -Development labels and feedback informed the edits. No gradient training does not mean no examples, no model selection or no pretrained knowledge in the authoring LLMs. There was no matched-budget single-author control, so this is not causal evidence for branching itself. +On macOS 15 or newer, `python model.py --coreml` uses the packaged Core ML graph. The graph uses iOS 18 / macOS 15 operations and accepts `(128, 5153)` extracted features. An Apple app still needs the frontend/decoder and appropriate offset conversion; these downloads are not a native iOS SDK or a Transformers `AutoModel` checkpoint. -## Release history and verification +The benchmark used **8,192-character windows with 512 overlap**, mapped predictions back to original offsets and deduplicated spans. The simple CLI processes each input text directly; use that same windowing to reproduce long-document benchmark behavior. Keep each variant's own `model.py`, `weights.npz` and `bundle/` together. -The previous 1.045 MB r3a checkpoint remains available at [its immutable revision](https://huggingface.co/pmarquees/blot-mini/tree/371716aaae546de275aa98bb58a3c22394b14f88): +## Packaging and verification -```sh -hf download pmarquees/blot-mini --revision 371716aaae546de275aa98bb58a3c22394b14f88 --local-dir blot-mini-r3a -``` +- **XS and S:** original inference-file hashes are preserved, with historical native/reference checks retained in each variant directory. Their historical validations are not new M checks. +- **M:** source and NumPy weights are byte-identical to the remotely benchmarked candidate. Its new Core ML package contains the same FP32 tensors and readout bias byte for byte, with the `linear → ReLU → linear` graph verified. **Native Core ML runtime parity and physical-device behavior were not tested for M.** The full benchmark ran through the sparse NumPy implementation. +- Packaging did not execute local inference or author new weights. See [Core ML packaging verification](evaluation/coreml-packaging.json) and [publication validation scope](evaluation/publication-smoke.json). -Its historical ProwlBench detection F1 was 86.19%, versus the previously recorded trained Blot v6a's 82.94%, on the same reused 24,603-snippet development corpus. These are r3a measurements, not results for the current 9.17 MB checkpoint. ProwlBench, CredData and the issue sample use different units and scoring rules; their numbers are not interchangeable. The [historical model card](history/r3a/README.md) and [r3a evaluation files](history/r3a/evaluation) preserve that evidence. +| M identity | SHA-256 | +|---|---| +| `model.py` | `2615330be6c58c0193cea6b4ba36b30f5ec2cb360480b7e81bed0373068e40db` | +| `weights.npz` | `39c316308e20e70f9a30d136eb78073554018854fc59f830c2f32ebb323f0514` | -Native Core ML and NumPy matched on 353 invented controls and 256 arbitrary binary feature vectors during candidate validation. Zeroing the readout removed every control output. The full issue holdout used NumPy, not a second full Core ML run. Publication additionally checks frozen hashes, regenerated numerical arrays, native/reference example parity and downloaded artifact integrity. See [publication validation](evaluation/publication-smoke.json). No physical-device test is claimed. +Each variant's `frozen.json` identifies its inference files. Root [SHA256SUMS](SHA256SUMS) covers the release files except itself. M's `build.py` re-exports saved tensors without regenerating the authored weights; serialization metadata can still change. The original XS revision [`371716a`](https://huggingface.co/pmarquees/blot-mini/tree/371716aaae546de275aa98bb58a3c22394b14f88) and S revision [`2845fcd`](https://huggingface.co/pmarquees/blot-mini/tree/2845fcd2b832b57ccec9f90b3ac57ee11d98baf7) remain available. Publishing this family does not update the separate macOS test app. ## Limitations and provenance -This release can miss credentials and flag benign text. It does not scan Git history, implement all scanner allowlists, verify live credentials or guarantee safe redaction. Exact-boundary results are much lower than the primary overlap metric, and full-target coverage is reported separately. Production traffic and complete annotation are needed before interpreting these results as operational precision. - -The supplied pattern workbook and Blot synthetic generators informed format hypotheses. References included [Gitleaks](https://github.com/01-gitleaks/gitleaks), [detect-secrets](https://github.com/Yelp/detect-secrets) and [SecretBench](https://github.com/michoo/secretbench). Neither scanner is the inference engine or a teacher. Trained Blot, detect-secrets and TruffleHog were not rerun in the precision comparison. +These are research checkpoints. They can miss secrets, flag benign text and produce incomplete boundaries. They do not scan Git history, reproduce every scanner rule or allowlist, verify credentials online or guarantee safe redaction. Real production prevalence and fully annotated independent traffic may produce very different precision and recall. -Development and evaluation used [ProwlBench](https://huggingface.co/datasets/Podric/prowl-secrets-corpus) revision `06f6d2cdf6a64c6d77ffb95a8dfb7abf7913b885` (CC BY-NC 4.0), [CredData](https://github.com/Samsung/CredData/tree/86a918283f086baf9ca896a892519cd8462dfe0a), and the [issue-report replication dataset](https://doi.org/10.5281/zenodo.19622962) (source receipt: CC BY 4.0). Dataset terms remain separate from this private model publication. Raw benchmark texts, secret values, per-case predictions and author conversations are not redistributed. +Development references included [Gitleaks](https://github.com/01-gitleaks/gitleaks), [detect-secrets](https://github.com/Yelp/detect-secrets), [SecretBench](https://github.com/michoo/secretbench), the supplied pattern workbook and Blot's synthetic generators. Neither Gitleaks nor detect-secrets is the inference engine or a distillation teacher. -`research/precision-2026-11-04/` contains the current checkpoint's frozen aggregate evidence and overlap correction. `history/r3a/` contains superseded evaluation records. Root inference files, examples and `frozen.json` refer to the current precision winner. +Dataset provenance includes [CredData](https://github.com/Samsung/CredData/tree/86a918283f086baf9ca896a892519cd8462dfe0a), [ProwlBench](https://huggingface.co/datasets/Podric/prowl-secrets-corpus) revision `06f6d2cdf6a64c6d77ffb95a8dfb7abf7913b885` (CC BY-NC 4.0), and the [issue-report replication dataset](https://doi.org/10.5281/zenodo.19622962) (source receipt: CC BY 4.0). Dataset terms are separate from model publication. This card does not grant new model or dataset license rights. No raw benchmark texts, secret values, per-case predictions or author conversations are redistributed in this family update. diff --git a/SHA256SUMS b/SHA256SUMS index d947b5485672ac7d16e8f6e220467d6d2829f68e..e73779005c35c49a74f20b08de0691dae4c756bb 100644 --- a/SHA256SUMS +++ b/SHA256SUMS @@ -1,15 +1,29 @@ 11ad7efa24975ee4b0c3c3a38ed18737f0658a5f75a0a96787b576a78a023361 .gitattributes 39d0076f167dba162127c04f842e5157e72a8a23247e94ba6cd5ebe34d47abea .gitignore -3ba2d6e8c980e63b29af5811a7aeb1818700f214cc84d53b72342d99ed8c72f6 README.md +9c94651833465ae0087fe5bb69075ea2d88c732bb7532db8e43655982a429a96 README.md fc6678ae19538ccd51a8d2b8f351c4e2952f44f89f23e823821b22e2a5583f61 assets/badges/built-with-mill.svg -8c7800cfd0a806900420891b69f3b26aac628559737eab1a186c45a987bea827 build.py -7dc6d0103bc02f79ca70e4d54228dc76140b0489e053562f7bc2b24aff58a763 bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/model.mlmodel -80c4594825ccd8ce17afc5bd439b0b33d31586ebc7f16332935bfc9dafa7f9a5 bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/weights/weight.bin -17e391ec6a22cb25baefe29889fc3bcd5f6995087a254b3b8b654373424b90e3 bundle/BlotHandwritten.mlpackage/Manifest.json -7e281a4ab191b7b6728f9bffa27caa6e88e525f90cb34a18a2048ef895bc194d evaluation/publication-smoke.json -312bcfca8678e925fdb39827c9b1caff8cd466ce53643f7d27e001dc3a21df9a examples/expected.jsonl +691064550c005ff2a0a384d7d84fa5313a0fbb0cb861ac157fe894fdb24fad60 build.py +6500869412433336ffc9799b0fd6677b01257f4fb1b395c175c122f2d6591564 bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/model.mlmodel +7477c65f0f1d0d8d29476ed23c180d3cf04338220121833ea56d23b48751112d bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/weights/weight.bin +bead02a307fe2306eefd6aaf9e3be7c5cda7e83a9eae3bf80b8f82fee4c5425d bundle/BlotHandwritten.mlpackage/Manifest.json +db1eb92b0a1f6ee66c9c8ffb6df840e64e1ea33108682b9ec59ff844b1b04eb9 evaluation/coreml-packaging.json +dcc3e4ae072e1a5777eb9db2348dd411149c35e156efef9196efd9ba311fae3b evaluation/creddata-2026-11-06/REPORT.md +d51a0493e6ee83fe2182ccf6d05df9d0112b15e2bb9585f526f9e80ba2446545 evaluation/creddata-2026-11-06/hill.py +851e72bb785f7107d27379089fb226e8485fdd0da1b060952924af656d923b48 evaluation/creddata-2026-11-06/protocol.json +62121efec09dca5a50ea621afeb043007c8667467353183fda1350a6eeca8365 evaluation/creddata-2026-11-06/results.json +fbbae3e30e590aca846ceb55463b17f7020edfeb4bf8d75af7b2fb8594d6c55b evaluation/creddata-2026-11-06/resume-provenance.json +6e60e6189fbf876f1f9a93725bf05c806b63ab3a25b056eda01b5c1df89dcc9a evaluation/creddata-2026-11-06/resume_final.py +d32b28c3e4bf970c5c7b72dfd038d0474f3e7ecc81f283e21da0da3ca9391222 evaluation/creddata-2026-11-06/round1-summary.json +a34fedb758cea805e4f5bddd78e22442acc243ccc7ec8b73475e3f7a205f93c9 evaluation/creddata-2026-11-06/round2-summary.json +9698b936534def4ef70ba37c99d9ac51179f1aa3ffe90c4a3590c500f9b27f99 evaluation/creddata-2026-11-06/score.py +3ce122b53140e75b47b7813b3955c832fdefc8b771824482fc55c5ec2a5b1b5a evaluation/package_coreml.py +943a53b5bd8e8d46fc6710a2a3bd49617b021c4d5b249d1914cfca5732ea06c5 evaluation/publication-smoke.json +d415d9fc62832449f46df08eebf14b85ec1b44e390afb95f7cfdd37c749ea5ea examples/README.md 218409b4c2341f13c599e635b8d0096b77855837701084019b66a6d6a1ebb27b examples/input.jsonl -3f31616444f6d0f201ea95dac60ccf7b70e20ec4ee81d7edcc1f57eb51457900 frozen.json +0f66e80239de49026044349a28b00de5621abec8fccb02437c36d0a683d83e81 frozen.json +00a2551cac2b44e5f883c99424b0170905e34c4c88a8d0b01efd5f63a3b738f6 history/precision-a-2026-11-04/README.md +3f31616444f6d0f201ea95dac60ccf7b70e20ec4ee81d7edcc1f57eb51457900 history/precision-a-2026-11-04/frozen.json +f3739fc2ddcc08e5e0274e8464f456b3ee9ea8dde208124fabb685d199af1dc4 history/precision-a-2026-11-04/release.json 4feb7a789c7631615f413ac922530223eac259613cd5ce55516f8f385a6d2a31 history/r3a/README.md 5927402ca3599042adec6d399947814ff14b5b3e2560a0ce346a2ae2662a1d0a history/r3a/creddata/REPORT.md ebfa00e8fed5e98d16828ced1f83a49085893fd6a281343119e9280f1621cf52 history/r3a/evaluation/causal-projection.json @@ -29,8 +43,8 @@ efbb61bd9ae874c7f921d56b336678d9ddff19ff9fda319090c167e9a36a5ef6 history/r3a/re 04a9fc8aa5e541d7affefd4098ebd7a2ff83e39ea5b30c766ec1254c042922a6 history/r3a/research/round3-decision.json 9053bf955f219c39c5d8ed25f6a34081644eb04d36aa103e0c3fd7eab587e0bc history/r3a/research/split-audit.json 4ffcc1aab0af2239b652f5066633da60e076f0992ff231a7143cc57f2f1642e7 history/r3a/research/trace.jsonl -a881b7aed53b012c7cac1b2d532c26cf628a7748cdba81f148b023874de689bc model.py -f3739fc2ddcc08e5e0274e8464f456b3ee9ea8dde208124fabb685d199af1dc4 release.json +2615330be6c58c0193cea6b4ba36b30f5ec2cb360480b7e81bed0373068e40db model.py +5b6a494b19a7dbf69176bec5bb3a9d8a4e51cafad7b883f7c31e9029dbb3702c release.json db536663fd350dad01449c067b63c67c27bf61d4a4f9cf5ba18e56bba9f31a3a requirements.txt 76177f526c7b5c3c1e1d2c81b85a28a81bbd32fec7930d3036621d066a35a0c0 research/precision-2026-11-04/RESULTS.md 75cdcd06e761ee8733d62c3bc556145dc0081c666c70e7ecf0aa880d5b8b12f7 research/precision-2026-11-04/candidate-freeze.json @@ -39,4 +53,29 @@ fd152e53fcd335f3d64e1bb5221dd4ded43ba355e940c62a24e73f73b1878694 research/preci 9d3a61275beffc8e61dd91148bde7485eea7e4b7bad3ada39e0326b6e56758fe research/precision-2026-11-04/postrun-integrity.json a959e91ae3d9ccb7c182668a086734b34cf2d23808c91b79bcb4bb72e25aba06 research/precision-2026-11-04/protocol.md 8e1b388ba007169e1d9939ebffceb404133003339cdc4222a4a65a9f640a9d86 research/precision-2026-11-04/results.json -b0b4b1ce704651f3f4429cc83f1138537b4ae339df20dda0dad1a2e7a1b0b3ba weights.npz +a5d2beb8a7710a3b0f91b334021e3a1e0bcf76e63b44089035f613942f3cb05a variants.json +d09134b4ae167743e407b6ce38da6dfa81f647d0178ddb44ecd79138d7d7109f variants/s/README.md +8c7800cfd0a806900420891b69f3b26aac628559737eab1a186c45a987bea827 variants/s/build.py +7dc6d0103bc02f79ca70e4d54228dc76140b0489e053562f7bc2b24aff58a763 variants/s/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/model.mlmodel +80c4594825ccd8ce17afc5bd439b0b33d31586ebc7f16332935bfc9dafa7f9a5 variants/s/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/weights/weight.bin +17e391ec6a22cb25baefe29889fc3bcd5f6995087a254b3b8b654373424b90e3 variants/s/bundle/BlotHandwritten.mlpackage/Manifest.json +312bcfca8678e925fdb39827c9b1caff8cd466ce53643f7d27e001dc3a21df9a variants/s/examples/expected.jsonl +218409b4c2341f13c599e635b8d0096b77855837701084019b66a6d6a1ebb27b variants/s/examples/input.jsonl +3f31616444f6d0f201ea95dac60ccf7b70e20ec4ee81d7edcc1f57eb51457900 variants/s/frozen.json +a881b7aed53b012c7cac1b2d532c26cf628a7748cdba81f148b023874de689bc variants/s/model.py +7e281a4ab191b7b6728f9bffa27caa6e88e525f90cb34a18a2048ef895bc194d variants/s/publication-validation.json +db536663fd350dad01449c067b63c67c27bf61d4a4f9cf5ba18e56bba9f31a3a variants/s/requirements.txt +b0b4b1ce704651f3f4429cc83f1138537b4ae339df20dda0dad1a2e7a1b0b3ba variants/s/weights.npz +ad0cf6fd4303563fb7cf5c8659443a0fa22b1fda1f5ae5d53bb05c331f1fb2b1 variants/xs/README.md +8c7800cfd0a806900420891b69f3b26aac628559737eab1a186c45a987bea827 variants/xs/build.py +8018d8706256963cb924ac967a502f421705fa4ae84bb05118e794088c09f4a4 variants/xs/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/model.mlmodel +386dfcdfad786e6895e04c548126cc9d3ec137cb4d29e875a035ccd0ed235192 variants/xs/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/weights/weight.bin +4e289c634c774e66d15a50049b34f623656f3efe5bcb0f7b15afd925dea412e4 variants/xs/bundle/BlotHandwritten.mlpackage/Manifest.json +312bcfca8678e925fdb39827c9b1caff8cd466ce53643f7d27e001dc3a21df9a variants/xs/examples/expected.jsonl +218409b4c2341f13c599e635b8d0096b77855837701084019b66a6d6a1ebb27b variants/xs/examples/input.jsonl +8f422b0aeafa66ece9d399ec0f561fda8afd566841be5b042eb5e460b2c85f85 variants/xs/frozen.json +8c7bc84a231fdb01769ba8a7364c1a5124f36ee43827dc9e1ff90bf028716d16 variants/xs/model.py +19024c08f73b3c9083875a02c4df5c896fa6dd6b75d824835dfe463ed55fb951 variants/xs/publication-validation.json +db536663fd350dad01449c067b63c67c27bf61d4a4f9cf5ba18e56bba9f31a3a variants/xs/requirements.txt +7fad5b4436f5598ce9e53ef4c58bab450b1e13100561a38519c30e35629f8e00 variants/xs/weights.npz +39c316308e20e70f9a30d136eb78073554018854fc59f830c2f32ebb323f0514 weights.npz diff --git a/build.py b/build.py index a49168e000d8fc62158ee13b21fbe72658f22b80..6708021c413a994fcc49434755cb91b06b4d07b6 100644 --- a/build.py +++ b/build.py @@ -1,12 +1,16 @@ -from model import * +"""Re-export the saved FP32 tensors without changing weights.npz.""" +from pathlib import Path +import numpy as np import coremltools as ct from coremltools.converters.mil import Builder as mb -W,b,o,names=authored_weights() -np.savez_compressed(ROOT/"weights.npz",W=W,b=b,o=o) -@mb.program(input_specs=[mb.TensorSpec(shape=(128,DIM))],opset_version=ct.target.iOS18) +ROOT=Path(__file__).resolve().parent +with np.load(ROOT/'weights.npz',allow_pickle=False) as z: + W,b,o=z['W'],z['b'],z['o'] +@mb.program(input_specs=[mb.TensorSpec(shape=(128,W.shape[1]))],opset_version=ct.target.iOS18) def network(features): - h=mb.relu(x=mb.linear(x=features,weight=W,bias=b)) - return mb.linear(x=h,weight=o[None,:],bias=np.array([-0.5],np.float32),name="scores") -m=ct.convert(network,minimum_deployment_target=ct.target.iOS18,compute_precision=ct.precision.FLOAT32) -(ROOT/"bundle").mkdir(exist_ok=True) -m.save(str(ROOT/"bundle/BlotHandwritten.mlpackage")) + h=mb.relu(x=mb.linear(x=features,weight=W,bias=b)) + return mb.linear(x=h,weight=o[None,:],bias=np.array([-0.5],np.float32),name='scores') +m=ct.convert(network,minimum_deployment_target=ct.target.iOS18, + compute_precision=ct.precision.FLOAT32,skip_model_load=True) +(ROOT/'bundle').mkdir(exist_ok=True) +m.save(str(ROOT/'bundle/BlotHandwritten.mlpackage')) diff --git a/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/model.mlmodel b/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/model.mlmodel index 6aa62b12cb3fa4e0c2e30b34917dcda424719e4f..10a05ba873719a808d58791cd86be6e193362dc6 100644 --- a/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/model.mlmodel +++ b/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/model.mlmodel @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7dc6d0103bc02f79ca70e4d54228dc76140b0489e053562f7bc2b24aff58a763 +oid sha256:6500869412433336ffc9799b0fd6677b01257f4fb1b395c175c122f2d6591564 size 1433 diff --git a/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/weights/weight.bin b/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/weights/weight.bin index b394a3aa7fc2c13e5d222e57da45b6324e2e1d57..a8184da588b7dbc09230cb19bae412ce177c86fe 100644 --- a/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/weights/weight.bin +++ b/bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/weights/weight.bin @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:80c4594825ccd8ce17afc5bd439b0b33d31586ebc7f16332935bfc9dafa7f9a5 -size 9134956 +oid sha256:7477c65f0f1d0d8d29476ed23c180d3cf04338220121833ea56d23b48751112d +size 24517524 diff --git a/bundle/BlotHandwritten.mlpackage/Manifest.json b/bundle/BlotHandwritten.mlpackage/Manifest.json index d59b08c4e124d3a566eda8f6350d4ead4fabd8de..3ec4732691492edeb922661c45abfb7e923aae8f 100644 --- a/bundle/BlotHandwritten.mlpackage/Manifest.json +++ b/bundle/BlotHandwritten.mlpackage/Manifest.json @@ -1,18 +1,18 @@ { "fileFormatVersion": "1.0.0", "itemInfoEntries": { - "AA18DC46-7430-459A-BE97-398C3275A073": { + "8048901E-30DD-4EA7-86C1-22CC15F7D626": { "author": "com.apple.CoreML", "description": "CoreML Model Specification", "name": "model.mlmodel", "path": "com.apple.CoreML/model.mlmodel" }, - "F916E05A-811A-457A-BCE4-567724FCE6ED": { + "97F1B9C7-A1E6-41A7-B42E-D30CDBFD2D44": { "author": "com.apple.CoreML", "description": "CoreML Model Weights", "name": "weights", "path": "com.apple.CoreML/weights" } }, - "rootModelIdentifier": "AA18DC46-7430-459A-BE97-398C3275A073" + "rootModelIdentifier": "8048901E-30DD-4EA7-86C1-22CC15F7D626" } diff --git a/evaluation/coreml-packaging.json b/evaluation/coreml-packaging.json new file mode 100644 index 0000000000000000000000000000000000000000..2735d988881f4f16d4b7e452f6c7868ea96d58f3 --- /dev/null +++ b/evaluation/coreml-packaging.json @@ -0,0 +1,31 @@ +{ + "checked_utc": "2026-11-06T12:30:58.162773+00:00", + "coremltools": "9.0", + "source_model_sha256": "2615330be6c58c0193cea6b4ba36b30f5ec2cb360480b7e81bed0373068e40db", + "source_weights_sha256": "39c316308e20e70f9a30d136eb78073554018854fc59f830c2f32ebb323f0514", + "parameters": 6129296, + "inference_asset_bytes": 24574583, + "graph": "linear -> ReLU -> linear", + "features": 5153, + "hidden_units": 1189, + "input_shape": [ + 128, + 5153 + ], + "output_shape": [ + 128, + 1 + ], + "all_serialized_weight_bytes_match_frozen_npz": true, + "readout_bias": -0.5, + "native_model_loaded": false, + "inference_executed": false, + "new_weights_authored": false, + "native_runtime_parity_tested": false, + "benchmark_rerun": false, + "files": { + "bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/model.mlmodel": "6500869412433336ffc9799b0fd6677b01257f4fb1b395c175c122f2d6591564", + "bundle/BlotHandwritten.mlpackage/Data/com.apple.CoreML/weights/weight.bin": "7477c65f0f1d0d8d29476ed23c180d3cf04338220121833ea56d23b48751112d", + "bundle/BlotHandwritten.mlpackage/Manifest.json": "bead02a307fe2306eefd6aaf9e3be7c5cda7e83a9eae3bf80b8f82fee4c5425d" + } +} diff --git a/evaluation/creddata-2026-11-06/REPORT.md b/evaluation/creddata-2026-11-06/REPORT.md new file mode 100644 index 0000000000000000000000000000000000000000..aca46a2757a94e7beef31391b412f1b78eaf5c03 --- /dev/null +++ b/evaluation/creddata-2026-11-06/REPORT.md @@ -0,0 +1,47 @@ +> Publication note: this report describes the research workspace. Only aggregate results, scripts and verification receipts are included here; the corpus, full predictions and private artifacts referenced below are not redistributed. The subsequently exported M bundle measures 24,574,583 bytes; its Core ML tensors match the benchmarked NumPy tensors, while native runtime parity remains untested. + +# Frozen Blot Mini versus Gitleaks: full CredData confirmation + +The frozen candidate beats Gitleaks on full-corpus labeled-location F1. + +**All 11,030 files across 337 repositories completed with zero file errors.** Evaluation ran entirely on a Mill Linux CPU sandbox. The model and scorer stayed frozen. + +| System | Precision | Recall | Labeled-location F1 | Conservative F1 | +|---|---:|---:|---:|---:| +| Published Blot Mini, 9.17 MB | 71.15% | 44.07% | 54.43% | 50.87% | +| Frozen uuid2 candidate, estimated 24.64 MB | 77.49% | 56.67% | 65.46% | 61.77% | +| Gitleaks 8.30.1 | 85.99% | 45.58% | 59.57% | 58.96% | + +Candidate minus Gitleaks: **+5.89 percentage points F1**; paired 95% repository-bootstrap interval **[-16.91, +44.13] points**. The paired repository-bootstrap interval includes zero; the point estimate does not establish a reliable advantage. + +| System | True positives | Known false positives | False negatives | Unannotated finding lines | +|---|---:|---:|---:|---:| +| Published Blot Mini | 6,694 | 2,714 | 8,494 | 1,721 | +| uuid2 candidate | 8,607 | 2,500 | 6,581 | 1,575 | +| Gitleaks | 6,922 | 1,128 | 8,266 | 244 | + +## What this result measures + +The primary metric is detection of unique annotated line ranges, not exact-span F1. Conservative F1 additionally treats all unannotated finding lines as false positives; that is a deliberately pessimistic metric, not a claim that every unannotated finding is benign. The unchanged scorer also reports full value-span coverage for Blot. + +Candidate value-span coverage: 52.30%; published parent: 38.75%. + +**CredData was reused during development and selection. This is a completed full-corpus comparison, not an independent fresh holdout and not proof of superiority on arbitrary repositories.** The historical overlap-excluded slice is descriptive only. Gitleaks was evaluated using its frozen 8.30.1 predictions, rescored exactly; the scanner binary was not rerun in this confirmation. + +## Frozen model and execution + +Candidate uuid2 adds an authored UUID-format detector to the precision branch. Its weights were authored without gradients; no model tuning occurred during this confirmation. Contextless benign UUIDs remain a known generalization risk. + +The model has 6,129,296 parameters. Estimated FP32 inference bundle: 24,637,729 bytes including export allowance. No new Core ML package or physical-device parity was verified. + +Source SHA256: `2615330be6c58c0193cea6b4ba36b30f5ec2cb360480b7e81bed0373068e40db`. Weights SHA256: `39c316308e20e70f9a30d136eb78073554018854fc59f830c2f32ebb323f0514`. + +The continuation validated 5,646 saved prediction rows, checked 20 fixed cached files against unchanged inference, and evaluated only the 5,384 missing files. It required exact manifest coverage and valid offsets before scoring. All original data, metadata, scorer, parent and candidate hashes were checked. Gitleaks rescoring reproduced the prior artifact exactly. Bootstrap comparisons use 2,000 paired repository resamples and seed 90272. + +Continuation inference: 392.8 seconds. Confirmation job including verification and scoring: 476.8 seconds. The previous partial job consumed 195 seconds before timing out; timings are not a fresh full-run latency benchmark. + +Run: `run_143a9bc10fe160f9e7f5ae6ffa87d57a`. Completed UTC: `2026-11-06T11:58:20.828855+00:00`. + +Metrics, per-repository scores, full predictions, provenance and frozen model are retained in `results/`. The retrieved artifact hashes and all 11,030 unique prediction paths were verified on the host without executing inference locally. Hugging Face and the Mac app were not modified. + +Mill artifact: `art_rymt4rgvj1ch`, delivered and verified on the host. Sandbox provider state confirmed stopped at 12:00:03.939 UTC, with no running jobs. diff --git a/evaluation/creddata-2026-11-06/hill.py b/evaluation/creddata-2026-11-06/hill.py new file mode 100644 index 0000000000000000000000000000000000000000..6499e78481073eb2f863dc327642ea75feb785bc --- /dev/null +++ b/evaluation/creddata-2026-11-06/hill.py @@ -0,0 +1,98 @@ +"""Remote-only gradient-free authored-weight selection and frozen evaluation.""" +from pathlib import Path +from datetime import datetime,timezone +import ast,bisect,collections,concurrent.futures,hashlib,importlib.util,json,os,sys,time,shutil +assert sys.platform=='linux' +R=Path(__file__).resolve().parent.parent;H=R/'hill' +sys.path.insert(0,str(R));import numpy as np;import score +def sha(p):return hashlib.sha256(p.read_bytes()).hexdigest() +def save(p,d):p.parent.mkdir(parents=True,exist_ok=True);p.write_text(json.dumps(d,indent=2)+'\n') +def loadmod(p,name): + s=importlib.util.spec_from_file_location(name,p);m=importlib.util.module_from_spec(s);s.loader.exec_module(m);return m +def compile_model(name): + p=H/'models'/name/'model.py';p.parent.mkdir(parents=True,exist_ok=True) + if name=='parent':shutil.copy2(R/'model/model.py',p) + def frozen(path): + t=ast.parse(path.read_text());t.body=[n for n in t.body if not(isinstance(n,ast.FunctionDef) and n.name=='authored_weights') and not(isinstance(n,ast.Assign) and any(isinstance(x,ast.Name) and x.id=='PREFIXES' for x in n.targets))];return ast.dump(t) + assert frozen(p)==frozen(R/'model/model.py'),name + m=loadmod(p,name);W,b,o,names=m.authored_weights() + assert all(np.all(np.isfinite(x)) for x in [W,b,o]) + np.savez_compressed(p.parent/'weights.npz',W=W,b=b,o=o) + size=(W.size+b.size+o.size+1)*4+p.stat().st_size+(p.parent/'weights.npz').stat().st_size+65536 + assert size<=30000000,(name,size) + d={'name':name,'source_sha256':sha(p),'weights_sha256':sha(p.parent/'weights.npz'),'parameters':W.size+b.size+o.size+1,'estimated_bundle_bytes_with_64kb_export_allowance':size,'gradients':False,'frontend_ast_unchanged':True,'frozen_utc':datetime.now(timezone.utc).isoformat()};save(p.parent/'frozen.json',d);return d +MODELS={};BASE=None +def init(names): + global MODELS,BASE + BASE=loadmod(R/'model/model.py','original');MODELS={n:loadmod(H/'models'/n/'model.py',n).Detector() for n in names} +def unit(entry,diagnose=False): + raw=(R/'CredData'/entry['path']).read_bytes();assert hashlib.sha256(raw).hexdigest()==entry['sha256'] + text=raw.decode('utf-8',errors='replace').replace('\r\n','\n').replace('\r','\n');starts=[0]+[i+1 for i,c in enumerate(text) if c=='\n'];out={n:set() for n in MODELS};diag=collections.defaultdict(collections.Counter) + for offset in range(0,len(text),7680): + chunk=text[offset:offset+8192];spans=BASE.candidates(chunk) + if not spans:continue + X=np.stack([BASE.feature(chunk,a,b) for a,b in spans]) + for name,engine in MODELS.items(): + scores=engine.scores(X);selected=[] + for i in sorted(range(len(spans)),key=lambda i:(-scores[i],-(spans[i][1]-spans[i][0]),spans[i][0])): + a,b=spans[i] + if scores[i]>0 and not any(ax for x,y in selected): + selected.append((a,b));out[name].add((offset+a,offset+b)) + if diagnose and name=='parent': + line=bisect.bisect_right(starts,offset+a);pos,neg=score.gold[entry['path']];kind='tp' if any(s<=line<=e for s,e in pos) else 'fp' if any(s<=line<=e for s,e in neg) else 'unlabeled' + flags=' '.join(f for f in BASE.FLAGS if X[i,BASE.F[f]]) + diag[flags][kind]+=1 + counts={} + for name,sp in out.items(): + lines=sorted({bisect.bisect_right(starts,a) for a,b in sp});pos,neg=score.gold[entry['path']];tp=sum(score.hit(lines,a,b) for a,b in pos);fp=sum(score.hit(lines,a,b) for a,b in neg);u=sum(not any(a<=line<=b for a,b in pos+neg) for line in lines) + counts[name]=[tp,fp,len(pos)-tp,len(neg)-fp,u,0,0] + return {'path':entry['path'],'counts':counts,'spans':{n:sorted(s) for n,s in out.items()},'diag':{k:dict(v) for k,v in diag.items()}} +def task(arg):return unit(*arg) +def split(): + eligible=[e for e in score.manifest['files'] if e['bytes']<=131072] + eligible.sort(key=lambda e:hashlib.sha256(('blot-weight-hill-20261105:'+e['path']).encode()).hexdigest()) + d={'dev':eligible[:1200],'selection':eligible[1200:2400],'method':'fixed path hash, 1200+1200 files <=128KiB, disjoint files; reused benchmark, not independent; large files reserved for full evaluation'} + save(H/'split.json',d);return d +def evaluate(label,names): + frozen={n:compile_model(n) for n in names};parts=split() if not (H/'split.json').exists() else json.loads((H/'split.json').read_text());all_results={};diagnostics=collections.defaultdict(collections.Counter);start=time.monotonic() + for part in ['dev','selection']: + sums={n:np.zeros(7,dtype=np.int64) for n in names};paths=[];rows=[] + with concurrent.futures.ProcessPoolExecutor(max_workers=8,initializer=init,initargs=(names,)) as pool: + for row in pool.map(task,[(e,part=='dev') for e in parts[part]],chunksize=8): + rows.append(row);paths.append(row['path']) + for n in names:sums[n]+=np.array(row['counts'][n]) + if part=='dev': + for k,v in row['diag'].items():diagnostics[k].update(v) + all_results[part]={n:score.rates(v) for n,v in sums.items()};save(H/f'{label}-{part}.json',all_results[part]) + if 'parent' in names: + original={json.loads(l)['path']:json.loads(l)['spans'] for l in (R/'runs/blot-mini-predictions.jsonl').open()} + assert all([list(s) for s in row['spans']['parent']]==original[row['path']] for row in rows),'cached execution parity failed' + print(json.dumps({'round':label,'partition':part,'results':all_results[part],'seconds':time.monotonic()-start}),flush=True) + save(H/f'{label}-diagnostics.json',sorted([{'flags':k,**dict(v)} for k,v in diagnostics.items()],key=lambda x:x.get('fp',0),reverse=True)[:60]) + save(H/f'{label}-summary.json',{'results':all_results,'models':frozen,'seconds':time.monotonic()-start,'run_id':os.environ.get('MILL_RUN_ID')}) +def full(name): + d=compile_model(name);D=H/'final';D.mkdir(exist_ok=True);(D/'runs').mkdir(exist_ok=True);start=time.monotonic();dest=D/'runs/blot-mini-predictions.jsonl';assert not dest.exists() + before=json.loads((R/'inputs-manifest.json').read_text()) + with dest.open('w') as out,concurrent.futures.ProcessPoolExecutor(max_workers=8,initializer=init,initargs=([name],)) as pool: + futures=[pool.submit(task,(e,False)) for e in score.manifest['files']] + for i,f in enumerate(concurrent.futures.as_completed(futures),1): + row=f.result();out.write(json.dumps({'path':row['path'],'spans':row['spans'][name]})+'\n');out.flush() + if i%100==0 or i==11030:print(json.dumps({'step':i,'total':11030,'errors':0,'seconds':round(time.monotonic()-start)}),flush=True) + inference_seconds=time.monotonic()-start + # Keep the original scorer byte-identical, only point its output/prediction directory at this run. + score.R=D;score.score('blot-mini');candidate=json.loads((D/'runs/blot-mini-scores.json').read_text());gitleaks=json.loads((R/'runs/gitleaks-scores.json').read_text());parent=json.loads((R/'results/comparison.json').read_text())['candidate'] + pairs={};keys=['f1_labeled','f1_conservative','precision_labeled','recall'];fields=['tp','fp_known','fn','tn_known','unlabeled_finding_lines','gold_value_spans','fully_covered_value_spans'] + for comparator,other in [('gitleaks',gitleaks),('parent',parent)]: + repos=sorted(candidate['by_repository']);a=np.array([[candidate['by_repository'][k][f] for f in fields] for k in repos]);b=np.array([[other['by_repository'][k][f] for f in fields] for k in repos]);rng=np.random.default_rng(90272);samples={k:[] for k in keys} + for _ in range(2000): + idx=rng.integers(0,len(repos),len(repos));ra=score.rates(a[idx].sum(0));rb=score.rates(b[idx].sum(0)) + for k in keys:samples[k].append(ra[k]-rb[k]) + pairs[comparator]={k:{'difference':candidate['all'][k]-other['all'][k],'ci95':np.quantile(samples[k],[.025,.975]).tolist()} for k in keys} + for path,item in before['files'].items():assert sha(R/path)==item['sha256'],path + assert sha(H/'models'/name/'model.py')==d['source_sha256'];assert sha(H/'models'/name/'weights.npz')==d['weights_sha256'] + shutil.copytree(H/'models'/name,D/'model',dirs_exist_ok=True) + save(D/'comparison.json',{'candidate':candidate,'gitleaks':gitleaks,'parent':parent,'paired':pairs,'freeze':d,'full_files':11030,'errors':0,'inference_seconds':inference_seconds,'elapsed_seconds':time.monotonic()-start,'run_id':os.environ.get('MILL_RUN_ID'),'all_original_inputs_and_scorer_unchanged':True,'completed_utc':datetime.now(timezone.utc).isoformat()}) + print(json.dumps({'completed':True,'candidate':candidate['all'],'gitleaks':gitleaks['all'],'seconds':time.monotonic()-start}),flush=True) +if __name__=='__main__': + if sys.argv[1]=='eval':evaluate(sys.argv[2],sys.argv[3:]) + elif sys.argv[1]=='full':full(sys.argv[2]) diff --git a/evaluation/creddata-2026-11-06/protocol.json b/evaluation/creddata-2026-11-06/protocol.json new file mode 100644 index 0000000000000000000000000000000000000000..94ce68d6b1336b0798ad107dff87a4390262a76c --- /dev/null +++ b/evaluation/creddata-2026-11-06/protocol.json @@ -0,0 +1,11 @@ +{ + "clock_start_utc": "2026-11-05T12:10:38+00:00", + "hard_stop_utc": "2026-11-05T12:30:38+00:00", + "search_freeze_no_later_utc": "2026-11-05T12:23:30+00:00", + "model_scope": "only PREFIXES and authored_weights; all other AST identical", + "data_scope": "path-hash dev1200/selection1200 files <=128KiB; full11030 final once; reused benchmark not fresh holdout", + "selection_rule": "maximize labeled-location F1 subject to >= parent F1, >= parent conservativeF1, recall >= parent minus 0.01, dense weights+source+NPZ+65536 export allowance <=30Mbytes", + "final_claim": "Report both full labeled-location F1 and conservative F1 versus frozen Gitleaks8.30.1; no generalization/statistical claim from point estimate alone", + "fork_flush": "three independent fresh authors then fresh follow-up authors with only DEV findings and concise accepted lessons; selection results withheld from authors", + "execution": "all model construction and inference remotely; local source edits and artifact IO only" +} diff --git a/evaluation/creddata-2026-11-06/results.json b/evaluation/creddata-2026-11-06/results.json new file mode 100644 index 0000000000000000000000000000000000000000..c091e21ea63cb53133b8d0ef0ca7ecee91845c4b --- /dev/null +++ b/evaluation/creddata-2026-11-06/results.json @@ -0,0 +1,219 @@ +{ + "dataset": "CredData", + "revision": "86a918283f086baf9ca896a892519cd8462dfe0a", + "files": 11030, + "repositories": 337, + "primary_metric": "labeled-location F1", + "reused_for_development": true, + "models": { + "xs": { + "tp": 6584, + "fp_known": 4755, + "fn": 8604, + "tn_known": 44751, + "unlabeled_finding_lines": 4622, + "precision_labeled": 0.5806508510450658, + "recall": 0.4335001316829076, + "f1_labeled": 0.49639989444716703, + "false_alarm_rate_labeled": 0.09604896376196824, + "precision_conservative": 0.4125054821126496, + "f1_conservative": 0.42274230312369576, + "gold_value_spans": 14626, + "fully_covered_value_spans": 5560, + "value_span_coverage": 0.3801449473540271, + "repositories": 337, + "files": 11030, + "repository_bootstrap_95ci": { + "f1_labeled": [ + 0.4331984868264032, + 0.5629172482619 + ], + "f1_conservative": [ + 0.34974768846978893, + 0.48012795975435846 + ], + "precision_labeled": [ + 0.4339294437249924, + 0.7121587664723148 + ], + "recall": [ + 0.36516678055934193, + 0.5647255829571748 + ] + } + }, + "s": { + "tp": 6694, + "fp_known": 2714, + "fn": 8494, + "tn_known": 46792, + "unlabeled_finding_lines": 1721, + "precision_labeled": 0.7115221088435374, + "recall": 0.4407426915986305, + "f1_labeled": 0.5443161489673117, + "false_alarm_rate_labeled": 0.054821637781279034, + "precision_conservative": 0.6014915985263726, + "f1_conservative": 0.5087205988524528, + "gold_value_spans": 14626, + "fully_covered_value_spans": 5668, + "value_span_coverage": 0.3875290578421988, + "repositories": 337, + "files": 11030, + "repository_bootstrap_95ci": { + "f1_labeled": [ + 0.4948538126203439, + 0.6198633625346651 + ], + "f1_conservative": [ + 0.4574686865136793, + 0.5743092360549852 + ], + "precision_labeled": [ + 0.5968971541060741, + 0.8067035668437251 + ], + "recall": [ + 0.3714226674903767, + 0.5759559427762218 + ] + } + }, + "m": { + "tp": 8607, + "fp_known": 2500, + "fn": 6581, + "tn_known": 47006, + "unlabeled_finding_lines": 1575, + "precision_labeled": 0.7749167191860988, + "recall": 0.5666973926784303, + "f1_labeled": 0.6546491728465488, + "false_alarm_rate_labeled": 0.05049892942269624, + "precision_conservative": 0.6786784418861378, + "f1_conservative": 0.6176533907427342, + "gold_value_spans": 14626, + "fully_covered_value_spans": 7649, + "value_span_coverage": 0.5229727881854233, + "repositories": 337, + "files": 11030, + "repository_bootstrap_95ci": { + "f1_labeled": [ + 0.5788048743310074, + 0.7490982013497618 + ], + "f1_conservative": [ + 0.5531796541591388, + 0.702748117048343 + ], + "precision_labeled": [ + 0.6807618559778611, + 0.8475060645715197 + ], + "recall": [ + 0.449137698960733, + 0.7347152247365508 + ] + } + }, + "gitleaks_8.30.1": { + "tp": 6922, + "fp_known": 1128, + "fn": 8266, + "tn_known": 48378, + "unlabeled_finding_lines": 244, + "precision_labeled": 0.8598757763975156, + "recall": 0.45575454306031077, + "f1_labeled": 0.5957483432309149, + "false_alarm_rate_labeled": 0.022785116955520542, + "precision_conservative": 0.8345792138895587, + "f1_conservative": 0.5895579592879653, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null, + "repositories": 337, + "files": 11030, + "repository_bootstrap_95ci": { + "f1_labeled": [ + 0.29500609280277446, + 0.7532335575108697 + ], + "f1_conservative": [ + 0.2903965053234994, + 0.7493405557164543 + ], + "precision_labeled": [ + 0.8456379925771637, + 0.9294971884351182 + ], + "recall": [ + 0.17626056091175676, + 0.6780529398566875 + ] + } + } + }, + "paired_m_vs_s_and_gitleaks": { + "gitleaks": { + "f1_labeled": { + "difference": 0.05890082961563392, + "ci95": [ + -0.16910640779453773, + 0.44131914919700144 + ] + }, + "f1_conservative": { + "difference": 0.028095431454768893, + "ci95": [ + -0.1875603840348245, + 0.39732066354996176 + ] + }, + "precision_labeled": { + "difference": -0.08495905721141672, + "ci95": [ + -0.23029529662554707, + 0.0002818848342839422 + ] + }, + "recall": { + "difference": 0.11094284961811957, + "ci95": [ + -0.22615725986504562, + 0.5445705430374552 + ] + } + }, + "parent": { + "f1_labeled": { + "difference": 0.11033302387923705, + "ci95": [ + 0.0495919612075952, + 0.21364664979402745 + ] + }, + "f1_conservative": { + "difference": 0.10893279189028138, + "ci95": [ + 0.05008070602674289, + 0.20813984885392645 + ] + }, + "precision_labeled": { + "difference": 0.06339461034256144, + "ci95": [ + 0.03054581298248636, + 0.11685935994797664 + ] + }, + "recall": { + "difference": 0.12595470107979984, + "ci95": [ + 0.04987876539125093, + 0.2721193307161686 + ] + } + } + }, + "gitleaks_original_predictions_rescored_exactly": true, + "full_m_run_id": "run_143a9bc10fe160f9e7f5ae6ffa87d57a", + "completed_utc": "2026-11-06T11:58:20.828855+00:00" +} diff --git a/evaluation/creddata-2026-11-06/resume-provenance.json b/evaluation/creddata-2026-11-06/resume-provenance.json new file mode 100644 index 0000000000000000000000000000000000000000..0288b681ba833fafc45b501096d3733faaf80c52 --- /dev/null +++ b/evaluation/creddata-2026-11-06/resume-provenance.json @@ -0,0 +1,16 @@ +{ + "started_utc": "2026-11-06T11:51:09.105849+00:00", + "run_id": "run_143a9bc10fe160f9e7f5ae6ffa87d57a", + "partial_source_sha256": "f8a5a2a0cd4d588ef3f620ac0386cc801e2651921c677de7733e9afb8d5e28ee", + "previous_run_id": "run_3508d0b1f496acd5f774606d34c71b0b", + "original_cached_rows": 5646, + "already_complete_rows": 5646, + "remaining_files_this_dispatch": 5384, + "cached_parity_files": 20, + "source_sha256": "2615330be6c58c0193cea6b4ba36b30f5ec2cb360480b7e81bed0373068e40db", + "weights_sha256": "39c316308e20e70f9a30d136eb78073554018854fc59f830c2f32ebb323f0514", + "original_input_hashes_verified": 11381, + "model_and_decoder_changed": false, + "training_or_gradients": false, + "benchmark_reused_for_development": true +} diff --git a/evaluation/creddata-2026-11-06/resume_final.py b/evaluation/creddata-2026-11-06/resume_final.py new file mode 100644 index 0000000000000000000000000000000000000000..8a7caaab0bca6d3c372f6165c6d852ad069c2b99 --- /dev/null +++ b/evaluation/creddata-2026-11-06/resume_final.py @@ -0,0 +1,158 @@ +"""Remote-only frozen CredData continuation; no weight edits or local execution.""" +import sys +assert sys.platform == 'linux', 'This evaluator must run in the remote sandbox.' +from pathlib import Path +from datetime import datetime, timezone +import concurrent.futures, hashlib, json, os, shutil, time +import numpy as np +import hill + +R, H, score = hill.R, hill.H, hill.score +D = H / 'confirmation-2026-11-06' +NAME = 'uuid2' +SOURCE_SHA = '2615330be6c58c0193cea6b4ba36b30f5ec2cb360480b7e81bed0373068e40db' +WEIGHTS_SHA = '39c316308e20e70f9a30d136eb78073554018854fc59f830c2f32ebb323f0514' +PARTIAL_SHA = 'f8a5a2a0cd4d588ef3f620ac0386cc801e2651921c677de7733e9afb8d5e28ee' +sha, save = hill.sha, hill.save + +def verify_original_inputs(): + manifest = json.loads((R / 'inputs-manifest.json').read_text()) + for path, item in manifest['files'].items(): + assert sha(R / path) == item['sha256'], ('input changed', path) + assert sha(H / 'models' / NAME / 'model.py') == SOURCE_SHA + assert sha(H / 'models' / NAME / 'weights.npz') == WEIGHTS_SHA + return len(manifest['files']) + +def load_rows(path, entries): + rows = {} + for raw in path.open(): + row = json.loads(raw) + key = row['path'] + assert key in entries and key not in rows and not row.get('error'), key + spans = row['spans'] + assert spans == [list(s) for s in sorted(set(map(tuple, spans)))], key + assert all(isinstance(a, int) and isinstance(b, int) and 0 <= a < b for a, b in spans), key + rows[key] = row + return rows + +def main(): + started = time.monotonic() + D.mkdir(exist_ok=True) + (D / 'runs').mkdir(exist_ok=True) + entries = {e['path']: e for e in score.manifest['files']} + assert len(entries) == 11030 + input_count = verify_original_inputs() + old = H / 'final/runs/blot-mini-predictions.jsonl' + assert sha(old) == PARTIAL_SHA + old_rows = load_rows(old, entries) + assert len(old_rows) == 5646 + dest = D / 'runs/blot-mini-predictions.jsonl' + if not dest.exists(): + shutil.copy2(old, dest) + rows = load_rows(dest, entries) + assert all(rows.get(k) == v for k, v in old_rows.items()), 'Cached rows changed' + + # Recheck a fixed, hash-selected sample against unchanged actual inference. + parity_paths = sorted(old_rows, key=lambda p: hashlib.sha256(p.encode()).hexdigest())[:20] + hill.init([NAME]) + for path in parity_paths: + result = hill.unit(entries[path]) + assert [list(s) for s in result['spans'][NAME]] == old_rows[path]['spans'], path + missing = [e for e in score.manifest['files'] if e['path'] not in rows] + # Scheduling only: start large files first to avoid a long serial tail. + missing.sort(key=lambda e: (-e['bytes'], e['path'])) + save(D / 'resume-provenance.json', { + 'started_utc': datetime.now(timezone.utc).isoformat(), + 'run_id': os.environ.get('MILL_RUN_ID'), 'partial_source_sha256': PARTIAL_SHA, + 'previous_run_id': 'run_3508d0b1f496acd5f774606d34c71b0b', + 'original_cached_rows': len(old_rows), 'already_complete_rows': len(rows), + 'remaining_files_this_dispatch': len(missing), 'cached_parity_files': len(parity_paths), + 'source_sha256': SOURCE_SHA, 'weights_sha256': WEIGHTS_SHA, + 'original_input_hashes_verified': input_count, + 'model_and_decoder_changed': False, 'training_or_gradients': False, + 'benchmark_reused_for_development': True, + }) + print(json.dumps({'phase': 'verified', 'cached': len(rows), 'remaining': len(missing), 'parity': 20}), flush=True) + inference_start = time.monotonic() + initial_count = len(rows) + with dest.open('a') as out, concurrent.futures.ProcessPoolExecutor( + max_workers=8, initializer=hill.init, initargs=([NAME],) + ) as pool: + pending = {pool.submit(hill.task, (e, False)): e['path'] for e in missing} + completed = initial_count + last_heartbeat = time.monotonic() + while pending: + done, _ = concurrent.futures.wait(pending, timeout=20, return_when=concurrent.futures.FIRST_COMPLETED) + for future in done: + path = pending.pop(future) + result = future.result() + assert result['path'] == path and path not in rows + row = {'path': path, 'spans': result['spans'][NAME]} + out.write(json.dumps(row) + '\n') + out.flush() + rows[path] = row + completed += 1 + if completed == len(entries) or time.monotonic() - last_heartbeat >= 20: + print(json.dumps({'phase': 'inference', 'step': completed, 'total': len(entries), + 'remaining': len(pending), 'seconds': round(time.monotonic() - started), 'errors': 0}), flush=True) + last_heartbeat = time.monotonic() + inference_seconds = time.monotonic() - inference_start + checked = load_rows(dest, entries) + assert set(checked) == set(entries), 'Full coverage required before scoring' + # Validate all saved offsets against the same normalized text. + for path, row in checked.items(): + text = (R / 'CredData' / path).read_bytes().decode('utf-8', errors='replace').replace('\r\n', '\n').replace('\r', '\n') + assert all(b <= len(text) for a, b in row['spans']), path + print(json.dumps({'phase': 'scoring', 'files': len(checked)}), flush=True) + score.R = D + score.score('blot-mini') + (D / 'raw').mkdir(exist_ok=True) + gitleaks_raw = D / 'raw/gitleaks.json' + if not gitleaks_raw.exists(): + shutil.copy2(R / 'raw/gitleaks.json', gitleaks_raw) + assert sha(gitleaks_raw) == sha(R / 'raw/gitleaks.json') + score.score('gitleaks') + candidate = json.loads((D / 'runs/blot-mini-scores.json').read_text()) + gitleaks = json.loads((D / 'runs/gitleaks-scores.json').read_text()) + previous_gitleaks = json.loads((R / 'runs/gitleaks-scores.json').read_text()) + assert gitleaks == previous_gitleaks, 'Gitleaks reproduction changed' + parent = json.loads((R / 'results/comparison.json').read_text())['candidate'] + pairs = {} + keys = ['f1_labeled', 'f1_conservative', 'precision_labeled', 'recall'] + fields = ['tp', 'fp_known', 'fn', 'tn_known', 'unlabeled_finding_lines', 'gold_value_spans', 'fully_covered_value_spans'] + for label, comparator in [('gitleaks', gitleaks), ('parent', parent)]: + repos = sorted(candidate['by_repository']) + assert set(repos) == set(comparator['by_repository']) + a = np.array([[candidate['by_repository'][k][f] for f in fields] for k in repos]) + b = np.array([[comparator['by_repository'][k][f] for f in fields] for k in repos]) + rng = np.random.default_rng(90272) + samples = {k: [] for k in keys} + for _ in range(2000): + idx = rng.integers(0, len(repos), len(repos)) + ra, rb = score.rates(a[idx].sum(0)), score.rates(b[idx].sum(0)) + for key in keys: + samples[key].append(ra[key] - rb[key]) + pairs[label] = {key: {'difference': candidate['all'][key] - comparator['all'][key], + 'ci95': np.quantile(samples[key], [.025, .975]).tolist()} for key in keys} + assert verify_original_inputs() == input_count + assert sha(old) == PARTIAL_SHA + shutil.copytree(H / 'models' / NAME, D / 'model', dirs_exist_ok=True) + freeze = json.loads((D / 'model/frozen.json').read_text()) + save(D / 'comparison.json', { + 'candidate': candidate, 'gitleaks': gitleaks, 'parent': parent, 'paired': pairs, + 'freeze': freeze, 'full_files': len(checked), 'errors': 0, + 'resumed_inference_seconds': inference_seconds, 'elapsed_seconds': time.monotonic() - started, + 'previous_partial_timeout_seconds': 195, 'original_cached_rows': len(old_rows), + 'new_rows_this_dispatch': len(missing), 'run_id': os.environ.get('MILL_RUN_ID'), + 'all_original_inputs_and_scorer_unchanged': True, + 'gitleaks_frozen_predictions_rescored_exactly': True, + 'benchmark_reused_for_development': True, + 'completed_utc': datetime.now(timezone.utc).isoformat(), + }) + save(D / 'SHA256SUMS.json', {str(p.relative_to(D)): sha(p) for p in sorted(D.rglob('*')) + if p.is_file() and p.name != 'SHA256SUMS.json'}) + print(json.dumps({'completed': True, 'candidate': candidate['all'], 'gitleaks': gitleaks['all'], + 'paired': pairs, 'seconds': time.monotonic() - started}), flush=True) + +if __name__ == '__main__': + main() diff --git a/evaluation/creddata-2026-11-06/round1-summary.json b/evaluation/creddata-2026-11-06/round1-summary.json new file mode 100644 index 0000000000000000000000000000000000000000..89dff6f8224cfcfde710bb52f4752f44adc752dd --- /dev/null +++ b/evaluation/creddata-2026-11-06/round1-summary.json @@ -0,0 +1,180 @@ +{ + "results": { + "dev": { + "parent": { + "tp": 646, + "fp_known": 277, + "fn": 580, + "tn_known": 4242, + "unlabeled_finding_lines": 128, + "precision_labeled": 0.6998916576381365, + "recall": 0.5269168026101142, + "f1_labeled": 0.6012098650535133, + "false_alarm_rate_labeled": 0.061296747067935385, + "precision_conservative": 0.6146527117031398, + "f1_conservative": 0.5674132630654369, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "precision1": { + "tp": 645, + "fp_known": 248, + "fn": 581, + "tn_known": 4271, + "unlabeled_finding_lines": 111, + "precision_labeled": 0.7222844344904815, + "recall": 0.5261011419249593, + "f1_labeled": 0.6087777253421425, + "false_alarm_rate_labeled": 0.0548793980969241, + "precision_conservative": 0.6424302788844621, + "f1_conservative": 0.57847533632287, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "recall1": { + "tp": 652, + "fp_known": 412, + "fn": 574, + "tn_known": 4107, + "unlabeled_finding_lines": 260, + "precision_labeled": 0.6127819548872181, + "recall": 0.531810766721044, + "f1_labeled": 0.5694323144104804, + "false_alarm_rate_labeled": 0.09117061296747068, + "precision_conservative": 0.49244712990936557, + "f1_conservative": 0.5113725490196078, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "structure1": { + "tp": 691, + "fp_known": 397, + "fn": 535, + "tn_known": 4122, + "unlabeled_finding_lines": 1074, + "precision_labeled": 0.6351102941176471, + "recall": 0.5636215334420881, + "f1_labeled": 0.5972342264477096, + "false_alarm_rate_labeled": 0.08785129453418898, + "precision_conservative": 0.31961147086031455, + "f1_conservative": 0.4079102715466352, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + } + }, + "selection": { + "parent": { + "tp": 627, + "fp_known": 218, + "fn": 555, + "tn_known": 3975, + "unlabeled_finding_lines": 185, + "precision_labeled": 0.7420118343195267, + "recall": 0.5304568527918782, + "f1_labeled": 0.6186482486433152, + "false_alarm_rate_labeled": 0.051991414261865015, + "precision_conservative": 0.6087378640776699, + "f1_conservative": 0.5669077757685352, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "precision1": { + "tp": 627, + "fp_known": 205, + "fn": 555, + "tn_known": 3988, + "unlabeled_finding_lines": 171, + "precision_labeled": 0.7536057692307693, + "recall": 0.5304568527918782, + "f1_labeled": 0.6226415094339622, + "false_alarm_rate_labeled": 0.048891008824230864, + "precision_conservative": 0.6251246261216351, + "f1_conservative": 0.5739130434782609, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "recall1": { + "tp": 630, + "fp_known": 329, + "fn": 552, + "tn_known": 3864, + "unlabeled_finding_lines": 374, + "precision_labeled": 0.656934306569343, + "recall": 0.5329949238578681, + "f1_labeled": 0.5885100420364315, + "false_alarm_rate_labeled": 0.07846410684474124, + "precision_conservative": 0.47261815453863465, + "f1_conservative": 0.5009940357852882, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "structure1": { + "tp": 660, + "fp_known": 337, + "fn": 522, + "tn_known": 3856, + "unlabeled_finding_lines": 1568, + "precision_labeled": 0.6619859578736209, + "recall": 0.5583756345177665, + "f1_labeled": 0.6057824690224873, + "false_alarm_rate_labeled": 0.0803720486525161, + "precision_conservative": 0.2573099415204678, + "f1_conservative": 0.3522818254603683, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + } + } + }, + "models": { + "parent": { + "name": "parent", + "source_sha256": "a881b7aed53b012c7cac1b2d532c26cf628a7748cdba81f148b023874de689bc", + "weights_sha256": "b0b4b1ce704651f3f4429cc83f1138537b4ae339df20dda0dad1a2e7a1b0b3ba", + "parameters": 2283666, + "estimated_bundle_bytes_with_64kb_export_allowance": 9230122, + "gradients": false, + "frontend_ast_unchanged": true, + "frozen_utc": "2026-11-05T12:15:25.252885+00:00" + }, + "precision1": { + "name": "precision1", + "source_sha256": "88407d3697039487253e13fbf2bb667431a9c3a11302df1b6e4b20abc538314b", + "weights_sha256": "eed25ff39181574fb2e51735bceb8e48de834c2f9ea594345c3dac22c10dec6d", + "parameters": 6124141, + "estimated_bundle_bytes_with_64kb_export_allowance": 24616863, + "gradients": false, + "frontend_ast_unchanged": true, + "frozen_utc": "2026-11-05T12:15:25.322767+00:00" + }, + "recall1": { + "name": "recall1", + "source_sha256": "ee41b15d72cbf9d98e729122090bdd01b2237ad57f201ed193fcc8dc99913186", + "weights_sha256": "e0b78de266c58e5965c8f84bfa59acd52329e86dc4db855d51f020c0ada393ad", + "parameters": 3015676, + "estimated_bundle_bytes_with_64kb_export_allowance": 12166534, + "gradients": false, + "frontend_ast_unchanged": true, + "frozen_utc": "2026-11-05T12:15:25.354620+00:00" + }, + "structure1": { + "name": "structure1", + "source_sha256": "40f3a2466457435da56859c3bdbbf347c8e7aa4432e84060c27df482a2a7c82b", + "weights_sha256": "9ba8876eef98cb615de6786fd1042e50226850f7ff63c1c642780b867b60560f", + "parameters": 3036296, + "estimated_bundle_bytes_with_64kb_export_allowance": 12247129, + "gradients": false, + "frontend_ast_unchanged": true, + "frozen_utc": "2026-11-05T12:15:25.384853+00:00" + } + }, + "seconds": 21.541285481000045, + "run_id": "run_70a6866cbf1f1a385b7e47e58d73e2c4" +} diff --git a/evaluation/creddata-2026-11-06/round2-summary.json b/evaluation/creddata-2026-11-06/round2-summary.json new file mode 100644 index 0000000000000000000000000000000000000000..d088e60dce62f0ecaa304aa47b0928c7daead811 --- /dev/null +++ b/evaluation/creddata-2026-11-06/round2-summary.json @@ -0,0 +1,264 @@ +{ + "results": { + "dev": { + "parent": { + "tp": 646, + "fp_known": 277, + "fn": 580, + "tn_known": 4242, + "unlabeled_finding_lines": 128, + "precision_labeled": 0.6998916576381365, + "recall": 0.5269168026101142, + "f1_labeled": 0.6012098650535133, + "false_alarm_rate_labeled": 0.061296747067935385, + "precision_conservative": 0.6146527117031398, + "f1_conservative": 0.5674132630654369, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "precision1": { + "tp": 645, + "fp_known": 248, + "fn": 581, + "tn_known": 4271, + "unlabeled_finding_lines": 111, + "precision_labeled": 0.7222844344904815, + "recall": 0.5261011419249593, + "f1_labeled": 0.6087777253421425, + "false_alarm_rate_labeled": 0.0548793980969241, + "precision_conservative": 0.6424302788844621, + "f1_conservative": 0.57847533632287, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "precision2": { + "tp": 629, + "fp_known": 238, + "fn": 597, + "tn_known": 4281, + "unlabeled_finding_lines": 108, + "precision_labeled": 0.7254901960784313, + "recall": 0.5130505709624796, + "f1_labeled": 0.6010511227902532, + "false_alarm_rate_labeled": 0.05266651914140297, + "precision_conservative": 0.6451282051282051, + "f1_conservative": 0.5715583825533849, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "short2": { + "tp": 663, + "fp_known": 299, + "fn": 563, + "tn_known": 4220, + "unlabeled_finding_lines": 119, + "precision_labeled": 0.6891891891891891, + "recall": 0.5407830342577488, + "f1_labeled": 0.6060329067641682, + "false_alarm_rate_labeled": 0.06616508077008187, + "precision_conservative": 0.6133209990749307, + "f1_conservative": 0.5747724317295189, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "provider2": { + "tp": 645, + "fp_known": 248, + "fn": 581, + "tn_known": 4271, + "unlabeled_finding_lines": 111, + "precision_labeled": 0.7222844344904815, + "recall": 0.5261011419249593, + "f1_labeled": 0.6087777253421425, + "false_alarm_rate_labeled": 0.0548793980969241, + "precision_conservative": 0.6424302788844621, + "f1_conservative": 0.57847533632287, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "uuid2": { + "tp": 786, + "fp_known": 254, + "fn": 440, + "tn_known": 4265, + "unlabeled_finding_lines": 111, + "precision_labeled": 0.7557692307692307, + "recall": 0.6411092985318108, + "f1_labeled": 0.6937334510150044, + "false_alarm_rate_labeled": 0.05620712547023678, + "precision_conservative": 0.6828844483058211, + "f1_conservative": 0.6613378207824989, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + } + }, + "selection": { + "parent": { + "tp": 627, + "fp_known": 218, + "fn": 555, + "tn_known": 3975, + "unlabeled_finding_lines": 185, + "precision_labeled": 0.7420118343195267, + "recall": 0.5304568527918782, + "f1_labeled": 0.6186482486433152, + "false_alarm_rate_labeled": 0.051991414261865015, + "precision_conservative": 0.6087378640776699, + "f1_conservative": 0.5669077757685352, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "precision1": { + "tp": 627, + "fp_known": 205, + "fn": 555, + "tn_known": 3988, + "unlabeled_finding_lines": 171, + "precision_labeled": 0.7536057692307693, + "recall": 0.5304568527918782, + "f1_labeled": 0.6226415094339622, + "false_alarm_rate_labeled": 0.048891008824230864, + "precision_conservative": 0.6251246261216351, + "f1_conservative": 0.5739130434782609, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "precision2": { + "tp": 626, + "fp_known": 201, + "fn": 556, + "tn_known": 3992, + "unlabeled_finding_lines": 167, + "precision_labeled": 0.7569528415961306, + "recall": 0.5296108291032149, + "f1_labeled": 0.6231956197112991, + "false_alarm_rate_labeled": 0.04793703792034343, + "precision_conservative": 0.6297786720321932, + "f1_conservative": 0.5753676470588235, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "short2": { + "tp": 639, + "fp_known": 266, + "fn": 543, + "tn_known": 3927, + "unlabeled_finding_lines": 173, + "precision_labeled": 0.7060773480662983, + "recall": 0.5406091370558376, + "f1_labeled": 0.6123622424532822, + "false_alarm_rate_labeled": 0.06343906510851419, + "precision_conservative": 0.5927643784786641, + "f1_conservative": 0.5654867256637168, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "provider2": { + "tp": 627, + "fp_known": 205, + "fn": 555, + "tn_known": 3988, + "unlabeled_finding_lines": 171, + "precision_labeled": 0.7536057692307693, + "recall": 0.5304568527918782, + "f1_labeled": 0.6226415094339622, + "false_alarm_rate_labeled": 0.048891008824230864, + "precision_conservative": 0.6251246261216351, + "f1_conservative": 0.5739130434782609, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + }, + "uuid2": { + "tp": 696, + "fp_known": 207, + "fn": 486, + "tn_known": 3986, + "unlabeled_finding_lines": 171, + "precision_labeled": 0.770764119601329, + "recall": 0.5888324873096447, + "f1_labeled": 0.6676258992805756, + "false_alarm_rate_labeled": 0.049367994276174575, + "precision_conservative": 0.6480446927374302, + "f1_conservative": 0.6170212765957447, + "gold_value_spans": 0, + "fully_covered_value_spans": 0, + "value_span_coverage": null + } + } + }, + "models": { + "parent": { + "name": "parent", + "source_sha256": "a881b7aed53b012c7cac1b2d532c26cf628a7748cdba81f148b023874de689bc", + "weights_sha256": "b0b4b1ce704651f3f4429cc83f1138537b4ae339df20dda0dad1a2e7a1b0b3ba", + "parameters": 2283666, + "estimated_bundle_bytes_with_64kb_export_allowance": 9230122, + "gradients": false, + "frontend_ast_unchanged": true, + "frozen_utc": "2026-11-05T12:21:56.866899+00:00" + }, + "precision1": { + "name": "precision1", + "source_sha256": "88407d3697039487253e13fbf2bb667431a9c3a11302df1b6e4b20abc538314b", + "weights_sha256": "eed25ff39181574fb2e51735bceb8e48de834c2f9ea594345c3dac22c10dec6d", + "parameters": 6124141, + "estimated_bundle_bytes_with_64kb_export_allowance": 24616863, + "gradients": false, + "frontend_ast_unchanged": true, + "frozen_utc": "2026-11-05T12:21:56.962032+00:00" + }, + "precision2": { + "name": "precision2", + "source_sha256": "c77c21ba4222a779a011605705a3aede46936442c7f79d36101133a8af6c536f", + "weights_sha256": "52c09eb6063104b87414bb5beb519e6d28a226fda5a23d9771425df580e8cde3", + "parameters": 6134451, + "estimated_bundle_bytes_with_64kb_export_allowance": 24659295, + "gradients": false, + "frontend_ast_unchanged": true, + "frozen_utc": "2026-11-05T12:21:57.028442+00:00" + }, + "short2": { + "name": "short2", + "source_sha256": "e55dd4b8b6f7a53a3e0ccbcf0cfe80b174184a0b6402cc527f944133aab08bd9", + "weights_sha256": "2fc2d98158e483a0da1897ffacd7a6c191432932cd27b4a0835b840a5ac03884", + "parameters": 6134451, + "estimated_bundle_bytes_with_64kb_export_allowance": 24659587, + "gradients": false, + "frontend_ast_unchanged": true, + "frozen_utc": "2026-11-05T12:21:57.104168+00:00" + }, + "provider2": { + "name": "provider2", + "source_sha256": "da1edf4fcec4a0bcbf1515e01d65403c1d9455f6efe5061258eaa190484db066", + "weights_sha256": "86fbf4f92b3375785db485a2e7ae45bb8783d487be15a8e6743ed86df810fded", + "parameters": 6227241, + "estimated_bundle_bytes_with_64kb_export_allowance": 25031134, + "gradients": false, + "frontend_ast_unchanged": true, + "frozen_utc": "2026-11-05T12:21:57.170697+00:00" + }, + "uuid2": { + "name": "uuid2", + "source_sha256": "2615330be6c58c0193cea6b4ba36b30f5ec2cb360480b7e81bed0373068e40db", + "weights_sha256": "39c316308e20e70f9a30d136eb78073554018854fc59f830c2f32ebb323f0514", + "parameters": 6129296, + "estimated_bundle_bytes_with_64kb_export_allowance": 24637729, + "gradients": false, + "frontend_ast_unchanged": true, + "frozen_utc": "2026-11-05T12:21:57.237869+00:00" + } + }, + "seconds": 48.83860467399995, + "run_id": "run_98487e7b0ac6f897639a8cdc48bb26f4" +} diff --git a/evaluation/creddata-2026-11-06/score.py b/evaluation/creddata-2026-11-06/score.py new file mode 100644 index 0000000000000000000000000000000000000000..7d7e2e8a77512f82b8ded8c9b7fc6789d1300856 --- /dev/null +++ b/evaluation/creddata-2026-11-06/score.py @@ -0,0 +1,83 @@ +from pathlib import Path +import json,csv,collections,bisect,hashlib +import numpy as np +R=Path(__file__).resolve().parent;C=R/'CredData';manifest=json.loads((R/'data-manifest.json').read_text());files={x['path']:x for x in manifest['files']};excluded=set(json.loads((R/'overlap-audit.json').read_text())['exclude_paths']) +def merge(ranges): + out=[] + for a,b in sorted(set(map(tuple,ranges))): + if out and a<=out[-1][1]:out[-1]=(out[-1][0],max(b,out[-1][1])) + else:out.append((a,b)) + return out +def hit(lines,a,b): + i=bisect.bisect_left(lines,a);return i=0 and ve>=0):continue + a,b=starts[ls-1]+vs,starts[le-1]+ve + if not 0<=a