Add files using upload-large-folder tool

benchmarks/csn-3.12-licensed/DATA-CARD.md

@@ -0,0 +1,209 @@
+---
+license: other
+license_name: mixed-permissive-per-row
+license_link: https://huggingface.co/datasets/BlazingCustoms/pybytecode-csn-3.12-licensed/blob/main/NOTICES.md
+task_categories:
+  - text-generation
+language:
+  - en
+tags:
+  - decompilation
+  - python
+  - bytecode
+  - code
+  - reverse-engineering
+size_categories:
+  - n<1K
+configs:
+  - config_name: default
+    data_files: bench.jsonl
+---
+
+# Data card — CSN-3.12-licensed decompilation benchmark
+
+600 real Python functions from GitHub, compiled to Python 3.12 bytecode, each paired with the
+source that produced it. Every row carries full provenance and a licence resolved at the exact
+commit. Built 2026-08-04 by `tools/build_csn_licensed.py`.
+
+**Redistributable**, provided `NOTICES.md` ships alongside. That file is not optional: MIT, BSD
+and Apache-2.0 require the notice to travel with the copy.
+
+---
+
+## The set
+
+| | |
+|---|---|
+| Rows | **600** |
+| Source repositories | **117** |
+| Max rows from any one repo | **6 (1.00%)** |
+| Median rows per repo | 6 |
+| Distinct commit SHAs | 117 |
+| Rows with complete provenance | 600 / 600 |
+| Rows with a resolved SPDX identifier | 600 / 600 |
+| Disassembly length | mean 87.3 lines, median 59, max 1,622 |
+| Rows with annotations | 63 |
+| Python | 3.12, `optimize=0` |
+
+Licences, all resolved at the row's own commit:
+
+| SPDX | rows | share |
+|---|---:|---:|
+| MIT | 391 | 65.2% |
+| BSD-3-Clause | 118 | 19.7% |
+| Apache-2.0 | 64 | 10.7% |
+| BSD-2-Clause | 15 | 2.5% |
+| ISC | 12 | 2.0% |
+
+### Why this replaces the previous 400-row set
+
+The first CSN benchmark drew 400 rows from **24** repositories with **15% from one**
+(`Karaage-Cluster/python-tldap`), recorded no commit SHA, and captured no licence. Auditing it
+afterwards: **180 of its 400 rows (45%) cannot be redistributed** — 83 GPL-3.0 (including all 60
+from that top repository) and 97 from repositories publishing no licence at all. Concentration at
+that level also means per-repository idiosyncrasy was a large fraction of what the benchmark
+measured. Details in `../../LICENSING-DETERMINATION.md` §2.
+
+The cap was set at 6 rows per repo — about 1% of the set — chosen so that no single codebase can
+move the headline number by more than a rounding error, while still admitting enough rows per
+repo to be worth resolving a licence for. At n=600 the 95% confidence interval on an accuracy
+near 93% is roughly ±2.0 points, against ±2.5 at n=400.
+
+---
+
+## Per-row fields
+
+```jsonc
+{
+  "i": 0,
+  "input":  "CODE <module>()\n  RESUME 0\n ...",   // model input: normalised disassembly
+  "expected": "def f(x):\n    ...",                 // reference source
+  "src_path": "src/00000.py",                       // RELATIVE to this directory
+  "pyc_path": "pyc/00000.pyc",
+  "provenance": {
+    "repo": "owner/name",
+    "repo_url": "https://github.com/owner/name",
+    "file_path": "pkg/module.py",                   // path within the repo
+    "commit_sha": "1b38e7cd...",                    // the commit the function was taken at
+    "func_name": "Class.method",
+    "line_start": 323, "line_end": 372,
+    "permalink": "https://github.com/.../blob/<sha>/pkg/module.py#L323-L372",
+    "dataset": "code-search-net/code_search_net (python, train split)"
+  },
+  "license": {
+    "spdx": "BSD-3-Clause",
+    "file": "LICENSE", "blob_sha": "3d9cc0c2...",
+    "resolved_at_commit": "1b38e7cd...",
+    "resolved_via": "GitHub GET /repos/{repo}/license?ref={sha}"
+  },
+  "n_instr": 49, "has_annotation": false
+}
+```
+
+Paths are relative, so the set works from any checkout location. Nothing points at `/tmp`.
+
+## How rows were selected
+
+Source: `code-search-net/code_search_net`, python train split, read in file order from a local
+parquet copy (`tools/fetch_csn.sh`; sha256
+`ad9e3a4ab10c2c1d8926d2b26ca2bfcc3aadda1477ba29a933391f93806b9fed`).
+
+3,138 rows were read to keep 600. In order:
+
+| Gate | Effect |
+|---|---|
+| Repo cap (6) | 1,838 rows skipped — the mechanism that spreads the set across 117 repos |
+| Repo blacklisted on licence | 668 rows skipped, from the 25 repos rejected below |
+| **Licence not permissive / not resolvable** | **25 repos rejected** |
+| Train-contamination (identifier-blind fingerprint) | 3 rows dropped |
+| Unparseable | 4 rows dropped |
+
+Repos rejected on licence: GPL-3.0 ×8, no licence file at that commit ×7, AGPL-3.0 ×3,
+LGPL-3.0 ×2, GPL-2.0 ×2, `NOASSERTION` ×2, LGPL-2.1 ×1.
+
+**A licence is resolved, never guessed.** `GET /repos/{owner}/{repo}/license?ref={sha}` must
+return a concrete SPDX identifier on the allowlist (`tools/resolve_licenses.py:ALLOWED_SPDX`).
+`NOASSERTION` and `Other` count as unresolved and the row is dropped. We never infer a licence
+from a README, a `setup.py` classifier, or a package index.
+
+## Normalisation
+
+Each function is reduced to a normal form before compiling: every docstring is replaced by the
+literal `'pass'`, then the tree is re-emitted with `ast.unparse`. Idempotence is checked per row
+and a row that is not a fixed point is dropped.
+
+This build uses `tools/normalize.py`, a stdlib-only implementation. **The GPL-3.0 tool that built
+the previous benchmark is not in this pipeline.** The two agree byte-for-byte on 400/400 rows of
+the previous benchmark (`tools/verify_equivalence.py`); the reasoning is in
+`../../LICENSING-DETERMINATION.md` §1.
+
+Because docstrings are normalised away, **this benchmark measures nothing about docstring
+recovery** — `samples_with_real_docstring` is 0 by construction. Use the held-out set for that.
+
+## Compilation
+
+`py_compile` at `optimize=0` for the reference `.pyc`, and `compile(..., dont_inherit=True,
+optimize=0)` for the model's input. `dont_inherit` is deliberate: the training corpus was
+compiled with PEP-563 stringised annotations inherited, and real `.pyc` files are not. Handing
+the model its training-time distribution would flatter it, so we do not. See
+`../../DATA-CARD-training-corpus.md` §7.
+
+## Harness soundness on this set
+
+`grade.py --self-test-only`: pre-flight **600/600 = 100%**, mutation kill rate **116/116 = 100%**,
+0 survivors. Both are prerequisites for quoting any score — but note that **pre-flight is trivial
+by construction and proves nothing about soundness** (it compares `compile(x)` with `compile(x)`).
+See `../../ORACLE-LIMITS.md` §1.
+
+## Unit size — read this before comparing scores against another benchmark
+
+An aggregate on this set is largely a statement about **small units**, so the size distribution is
+part of the result rather than a footnote to it. Size is measured in **representation lines** —
+the number of lines in the `input` field, i.e. the disassembly text a model is actually given.
+
+| min | p25 | median | p75 | p90 | p99 | max |
+|---|---|---|---|---|---|---|
+| 19 | 39 | **59** | 101 | 167 | 478 | 1,622 |
+
+Certification falls off sharply with size (v3 greedy, this set; intervals omitted below 30 rows /
+10 repositories rather than printed at a misleading width):
+
+| rep lines | rows | v3 greedy | 95% CI | v3 best-of-32 | 95% CI |
+|---|---|---|---|---|---|
+| 0–49 | 224 | 95.98% | [93.01, 98.51] | 98.21% | [96.31, 99.58] |
+| 50–99 | 224 | 89.73% | [85.17, 93.93] | 98.21% | [96.26, 99.57] |
+| 100–199 | 112 | 65.18% | [55.36, 74.14] | 85.71% | [77.57, 92.98] |
+| 200–299 | 27 | 51.85% | — | 77.78% | — |
+| 300–399 | 5 | 60.00% | — | 100.00% | — |
+| 400–599 | 5 | 0.00% | — | 0.00% | — |
+| 600+ | 3 | 0.00% | — | 0.00% | — |
+
+**96.64%** of v3's greedy certifications on this set (95.37% at best-of-32) come from units under
+200 rep lines, and above ~400 rep lines nothing certified even at 32 samples. The
+300–399 bucket reading above the one below it is n=5 noise, not a recovery. Recompute all of it
+with `harness/size_curve.py`; the stored output is `../../results/size_curve_csn600.json`.
+
+## Limitations
+
+- **Measured scores (2026-08-04, L1 strict oracle):** PyBytecode v3 greedy
+  **506/600 = 84.33%**, repo-clustered 95% CI [80.48, 87.94]; verified best-of-32
+  **562/600 = 93.67%**, CI [90.86, 96.08]. Untuned `Qwen2.5-Coder-1.5B-Instruct` control:
+  **4/600 = 0.67%**, CI [0.16, 1.35]. Per-row verdicts: `../../results/rows_with_base.jsonl`.
+- GitHub's licence detection is repository-level. A repository can vendor third-party files under
+  other terms; we do not detect that, and no automated tool reliably does.
+- A licence resolved at a commit is the licence *of that repository snapshot*. It does not prove
+  the specific file was contributed under it.
+- CodeSearchNet is a 2019-era snapshot, so the code skews toward pre-3.7 idiom. It contains no
+  `match` statements, no walrus operators, and few modern typing constructs.
+- Rows are capped, not sampled uniformly at random: the set is the first 6 admissible functions
+  per repository in dataset order. That is reproducible and unbiased with respect to difficulty,
+  but it is not a random sample of Python.
+- 117 repositories is enough to defeat single-repo dominance; it is not a representative sample
+  of the language ecosystem.
+
+## Provenance and licence of this benchmark
+
+Derived from CodeSearchNet (`code-search-net/code_search_net`). The underlying functions remain
+under their original licences and copyright, listed per row and reproduced in full in
+`NOTICES.md` (117 repositories, 272 KB of licence text). The assembly, normalisation and
+disassembly are ours.

benchmarks/csn-3.12-licensed/build_stats.json

@@ -0,0 +1,110 @@
+{
+  "n": 600,
+  "repos": 117,
+  "cap_rows_per_repo": 6,
+  "max_rows_from_one_repo": 6,
+  "max_share_pct": 1.0,
+  "median_rows_per_repo": 6,
+  "licences": {
+    "MIT": 391,
+    "BSD-3-Clause": 118,
+    "Apache-2.0": 64,
+    "BSD-2-Clause": 15,
+    "ISC": 12
+  },
+  "top_repos": [
+    [
+      "mjirik/imcut",
+      6
+    ],
+    [
+      "disqus/gutter",
+      6
+    ],
+    [
+      "kaste/mockito-python",
+      6
+    ],
+    [
+      "developersociety/django-glitter",
+      6
+    ],
+    [
+      "quizl/quizler",
+      6
+    ],
+    [
+      "MatterMiners/cobald",
+      6
+    ],
+    [
+      "quantmind/agile-toolkit",
+      6
+    ],
+    [
+      "aroberge/experimental",
+      6
+    ],
+    [
+      "alixedi/palal",
+      6
+    ],
+    [
+      "astroduff/commah",
+      6
+    ],
+    [
+      "fchorney/rpI2C",
+      6
+    ],
+    [
+      "jamescooke/flake8-aaa",
+      6
+    ],
+    [
+      "mozilla-services/amo2kinto",
+      6
+    ],
+    [
+      "josiah-wolf-oberholtzer/uqbar",
+      6
+    ],
+    [
+      "fredrike/pypoint",
+      6
+    ]
+  ],
+  "scanned": 3138,
+  "repos_rejected_on_licence": 25,
+  "drops": {
+    "drop_contaminated": 3,
+    "drop_licence": 25,
+    "drop_parse": 4,
+    "skip_blacklisted_repo": 668,
+    "skip_repo_cap": 1838
+  },
+  "licence_reject_reasons": {
+    "no licence file at that commit, or repo unavailable": 7,
+    "GPL-3.0": 8,
+    "LGPL-3.0": 2,
+    "AGPL-3.0": 3,
+    "licence not identified by GitHub (spdx='NOASSERTION')": 2,
+    "LGPL-2.1": 1,
+    "GPL-2.0": 2
+  },
+  "allowed_spdx": [
+    "0BSD",
+    "Apache-2.0",
+    "BSD-2-Clause",
+    "BSD-3-Clause",
+    "BSD-3-Clause-Clear",
+    "CC0-1.0",
+    "ISC",
+    "MIT",
+    "MIT-0",
+    "PostgreSQL",
+    "Python-2.0",
+    "Unlicense",
+    "Zlib"
+  ]
+}