| # PR quality/multilingual/tags fastText classifiers |
|
|
| Four fastText classifiers that score or tag a GitHub PR from its diff + touched-file |
| content, distilled from Jev (`typesafe/jev-1.13`) annotations on mined GitHub Archive |
| PRs. Each subfolder is self-contained: `model.bin` (weights), `train.txt`/`val.txt` |
| (the exact fastText training data), `train.py` (the exact training call — rerun in |
| place to reproduce `model.bin` from `train.txt`), and `infer.py` (standalone scoring |
| script — fetches a live PR via the GitHub API and classifies it, no dependency on the |
| original mining pipeline). |
|
|
| ## Usage |
|
|
| ```bash |
| pip install fasttext==0.9.3 "numpy<2" requests # numpy<2 is required -- see note below |
| export GITHUB_PAT=... # any GitHub token (read-only is fine) |
| python quality/infer.py https://github.com/owner/repo/pull/123 |
| python multilingual/infer.py https://github.com/owner/repo/pull/123 |
| python research_novelty/infer.py https://github.com/owner/repo/pull/123 |
| python tags/infer.py https://github.com/owner/repo/pull/123 # multi-label |
| python tags/infer.py --threshold 0.3 https://github.com/owner/repo/pull/123 |
| ``` |
|
|
| To retrain a model from its own `train.txt` in place: `cd <subfolder> && python train.py`. |
|
|
| `quality`/`multilingual`/`research_novelty` each print a `soft` score (probability- |
| weighted expected value, the better-correlating of the two — use this) and a `hard` |
| score (argmax level), both on a 0-10 scale. `tags` instead prints every tag (of 424) |
| whose predicted probability clears `--threshold` (default 0.5), sorted by confidence — |
| see its own section below for why it's one multi-label model, not 424 separate ones. |
|
|
| **Note on fastText + NumPy 2.x:** the prebuilt `fasttext-wheel` can silently corrupt |
| batch-`predict()` probabilities under NumPy 2.x, and even a from-source build's Python |
| wrapper crashes outright on `np.array(..., copy=False)`. Pin `numpy<2`. |
|
|
| ## `quality/` — Jev's strict 0-10 quality rubric |
|
|
| Rates the PR's diff on a strict, sensibility-gated 0-10 scale: 0-5 is gated on whether |
| the codebase/diff is coherent and credible (a simple-but-clean change tops out at 5); 6+ |
| requires genuine engineering or conceptual complexity on top of that. See |
| `annotate_pr_quality.py`'s `QUALITY_LEVELS` for the full rubric text. |
|
|
| - **Hyperparameters**: dim=64, epoch=10, lr=0.3, wordNgrams=2, minCount=3, bucket=2,000,000 |
| - **Train**: 254,143 PRs (`train.txt`) |
| - **Val**: 28,238 PRs, 1,766 at level≥6 (`val.txt`) |
|
|
| **This is the plain dim64/epoch10 baseline** — every variant tried (2-4x capacity, class- |
| rebalanced train sets, a ~1.55x-larger combined corpus with a separately-mined |
| higher-credibility PR pool) came out flat or slightly worse on a held-out 10,000-PR |
| tracking set (685 at quality≥6). More data/capacity does not fix this classifier; |
| treat it as a known ceiling, not a bug to chase. |
|
|
| **Metrics** (10,000-PR tracking holdout, rule = `quality >= 6`): |
|
|
| | metric | value | |
| |---|---| |
| | rule accuracy | 93.73% | |
| | precision | 74.17% | |
| | **recall** | **12.99%** | |
| | exact-level accuracy | 78.01% | |
| | MAE | 0.531 | |
| | soft Pearson / Spearman | 0.7252 / 0.6339 | |
|
|
| **Read this before using it as a filter**: it misses ~87% of genuinely good PRs |
| (quality≥6) to keep false positives to 0.33% of negatives, and its own predictions never |
| exceed ~6.8-7 on any PR we've tested — it is a high-precision, very-low-recall filter, |
| not a general-purpose quality scorer. It also gave near-identical predictions (4.4-5.3) |
| to real, high-quality ML/research PRs (e.g. lucidrains' `x-transformers`, `imagen-pytorch`) |
| as to trivial personal-repo PRs — it is not sensitive to genuine technical depth. Lowering |
| the decision threshold to `>=5` makes this *worse*, not better (accuracy drops to 37%, |
| since ~80% of real PRs are already `>=5` and the model's own predicted mass sits below |
| that line). |
|
|
| ## `multilingual/` — non-English comment/string content |
|
|
| Purely descriptive 0-10 scale: how much non-English natural-language text (comments, |
| docstrings, string literals) appears in the diff and touched files, independent of the |
| programming language itself. See `annotate_pr_quality.py`'s `MULTILINGUAL_LEVELS`. |
|
|
| - **Hyperparameters**: dim=128, epoch=25, lr=0.3, wordNgrams=2, minCount=3, bucket=2,000,000 |
| (best of a capacity sweep vs dim64/epoch10 baseline and dim256/epoch25 — dim128 won on |
| every axis, dim256 added nothing further for 2x the model size) |
| - **Train**: 254,143 PRs (`train.txt`) |
| - **Val**: 28,238 PRs (`val.txt`) |
|
|
| **Metrics** (100-PR live holdout, 79 true positives, rule = `multilingual <= 1`): |
|
|
| | metric | value | |
| |---|---| |
| | rule accuracy | 96.00% | |
| | precision | 96.30% | |
| | **recall** | **98.73%** | |
| | MAE | 0.586 | |
| | Pearson / Spearman | 0.906 / 0.721 | |
|
|
| This one is genuinely reliable as a pre-filter — high recall *and* precision. If you only |
| trust one of these two classifiers for a filtering decision, trust this one. |
|
|
| ## `research_novelty/` — research/IP content (⚠️ still recall≈0 at the `>=6` rule) |
| |
| 0-10 scale rating the extent to which the PR implements technical algorithms, ideas from |
| research papers, or original design/IP, as opposed to routine engineering — independent |
| of how difficult or high-quality the engineering itself is. See `annotate_pr_quality.py`'s |
| `RESEARCH_NOVELTY_LEVELS`. |
| |
| - **Hyperparameters**: dim=64, epoch=10 (same baseline as `quality`/`multilingual`) |
| - **Train**: 96,634 PRs, 297 at level≥6 (`train.txt`) |
| - **Val**: 10,772 PRs, 35 at level≥6 (`val.txt`) |
| - **Same train/val PR ids as `tags/`** (both come from the same annotation batch; see |
| Provenance) — comparable across the two. |
| |
| **Updated 2026-09-30**: retrained on a new 107,406-PR annotation batch (up from the |
| original 39,081-PR/98-positive pass), joined back to `quality`/`multilingual`'s existing |
| `text` field so the same 254K+-PR pool's diff/file content is reused rather than |
| re-mined. Labels use the field's own fixed 0-10 scale (int truncation), same as |
| `quality`/`multilingual` — not sample quantiles. |
| |
| **Metrics** (own val.txt, 10,772 PRs): |
| |
| | metric | value | |
| |---|---| |
| | exact-level accuracy | 82.45% | |
| | within-1-level accuracy | 94.06% | |
| | MAE | 0.2745 | |
| | hard Pearson / Spearman | 0.5679 / 0.5069 | |
| | soft Pearson / Spearman | 0.6662 / 0.5470 | |
| | rule (`>=6`) accuracy | 99.68% | |
| | rule (`>=6`) precision | undefined (0 predicted positive) | |
| | **rule (`>=6`) recall** | **0.00%** (0/35) | |
| |
| **Read this before using it as a filter**: correlation improved modestly over the old |
| diagnostic pass (soft Pearson 0.60→0.67) and the accuracy/MAE numbers look good, but that's |
| almost entirely the near-100%-easy "correctly predict low novelty" majority class — at the |
| actual decision boundary that matters (`>=6`), it now has enough val data to say |
| confidently (35 true positives, not 2) that it **misses every single one**, the same |
| recall-collapse pattern as `quality`. Its own predictions rarely if ever clear ~6-7 (see |
| the Megatron-LM spot-check below). Treat it as a continuous novelty *signal* (the soft |
| score correlates reasonably, 0.67) rather than a working `>=6` binary classifier. |
| |
| **Spot-check**: `NVIDIA/Megatron-LM#7762` (an Engram/CUDA synchronization change) scores |
| soft=2.46, hard=1 — plausible as "some genuine ML-systems depth but not paper-derived," |
| though still likely underrating it given the classifier's known insensitivity to real |
| technical depth (see `quality`'s note above; the same failure mode shows up here). |
| |
| ## `tags/` — multi-label topic tagging (424 tags) |
| |
| Multi-label classification over the topic vocabulary in `/ansh/harbor-tags.txt` (424 |
| distinct tags after dedup — e.g. `python`, `machine-learning`, `web-security`, |
| `gpu`, `formal methods`) — any number of tags can apply to one PR, independently. |
| Each tag is an independent Jev "noul" (boolean-probability) claim ("this PR's diff or |
| touched files involve code/content related to topic X"), bundled into one Jev call per |
| PR alongside `research_novelty` (see `annotate_tags_research.py`). |
|
|
| **One joint multi-label model, not 424 separate binary classifiers.** fastText's native |
| multi-label mode (`loss="ova"`, one-vs-all) shares a single embedding/vocabulary across |
| all 424 tags and trains in one pass — dramatically cheaper than 424x the training |
| time/storage for what is fundamentally the same text signal, at the cost of slightly |
| less per-tag threshold flexibility (mitigated by tuning `--threshold` in `infer.py` |
| per use case; default 0.5). |
|
|
| - **Hyperparameters**: dim=64, epoch=10, lr=0.3, wordNgrams=2, minCount=3, |
| bucket=2,000,000, **loss=ova** |
| - **Train**: 96,604 PRs (`train.txt`) / **Val**: 10,769 PRs (`val.txt`) — same PR ids as |
| `research_novelty/`'s updated split (both from the same annotation batch) |
| - 33 PRs with zero tags above threshold were dropped (fastText needs ≥1 label/line) |
|
|
| **Metrics** (10,769-PR val set, pooled across all 10,769×424 PR-tag pairs, threshold=0.5): |
|
|
| | metric | value | |
| |---|---| |
| | **precision** | **81.95%** | |
| | **recall** | **57.17%** | |
| | f1 | 67.35% | |
| | pooled Pearson r (vs. Jev's raw `tag_probs`) | **0.8055** | |
| | pooled Spearman ρ | **0.7145** | |
|
|
| Precision stays high across common tags (0.79-0.99) with recall degrading gracefully as |
| per-tag support drops — normal multi-label behavior, not a red flag. This is the |
| best-behaved of the four classifiers relative to its task difficulty (a 424-way |
| multi-label problem vs. a single 0-10 scale). |
|
|
| **Known data-hygiene issue**: `harbor-tags.txt` has a couple of near-duplicate tag pairs |
| that survived as distinct labels (`data processing`/`data-processing`, `data |
| engineering`/`data_engineering` — the latter pair also collided post-slugification and |
| one was suffixed `_dup` in `tag_vocab.json` to disambiguate). Worth deduping the source |
| file; not fixed here since it would invalidate the current label vocabulary/model. |
| |
| **Spot-check**: `vercel/next.js#99499` (a tag-revalidation fix) → 30 tags incl. |
| `typescript`, `react`, `nextjs`, `frontend`, `routing`, `concurrency`, `web-server`. |
| `NVIDIA/Megatron-LM#7762` → 36 tags incl. `machine-learning`, `pytorch`, `cuda`, `gpu`, |
| `distributed-systems`, `performance-optimization`, `deeply-technical-ip`. Both match |
| what a human skim of the PRs would tag. |
| |
| ## Provenance |
| |
| - Distilled from `typesafe/jev-1.13` (via OpenRouter) on GitHub Archive-mined PRs. |
| - Full annotation/training pipeline: `/ansh/prism-datagen/annotate_pr_quality.py` |
| (rubric + Jev calls, incl. `HARBOR_TAGS`/`TAG_CLAIMS`), `/ansh/fasttext-stuff/run.py`/ |
| `train_sweep.py` (quality/multilingual training — each subfolder's `train.py` is the |
| relevant one of these, trimmed to a single model), `/ansh/fasttext-stuff/evaluate.py`/ |
| `evaluate_sweep.py`/`compare_pre_post.py` (eval). |
| - `quality`'s known ceiling was diagnosed, not guessed: a dim/epoch capacity sweep |
| (64→128→256), a class-rebalancing ablation (10%/15% minority oversampling), and a |
| ~1.55x larger combined training corpus (adding a separately GitHub-credibility-filtered |
| PR pool skewed toward higher quality) were all tried and none moved recall past ~14%. |
| - `research_novelty`/`tags` come from a separate, lighter-weight annotation pass: |
| `/ansh/prism-datagen/annotate_tags_research.py` re-streams the same S3 source |
| `quality`/`multilingual` were mined from, filtered to PR ids already in that |
| 254K+-PR corpus, and asks Jev only for `research_novelty` + the 424 tag claims |
| (bundled into one call per PR, not 424 separate calls) — reusing that corpus's |
| existing `text` field rather than re-annotating everything from scratch. Dataset |
| build: `/ansh/fasttext-stuff/build_research_novelty_v2_dataset.py` and |
| `build_tags_dataset.py` (both replicate `quality`/`multilingual`'s exact train/val |
| split — same seed, same 10%, same PR ids — for cross-classifier comparability). |
| Training/eval: each subfolder's own `train.py`/`evaluate.py`. |
|
|
