# PR quality/multilingual/tags fastText classifiers

Four fastText classifiers that score or tag a GitHub PR from its diff + touched-file
content, distilled from Jev (`typesafe/jev-1.13`) annotations on mined GitHub Archive
PRs. Each subfolder is self-contained: `model.bin` (weights), `train.txt`/`val.txt`
(the exact fastText training data), `train.py` (the exact training call — rerun in
place to reproduce `model.bin` from `train.txt`), and `infer.py` (standalone scoring
script — fetches a live PR via the GitHub API and classifies it, no dependency on the
original mining pipeline).

## Usage

```bash
pip install fasttext==0.9.3 "numpy<2" requests   # numpy<2 is required -- see note below
export GITHUB_PAT=...                             # any GitHub token (read-only is fine)
python quality/infer.py https://github.com/owner/repo/pull/123
python multilingual/infer.py https://github.com/owner/repo/pull/123
python research_novelty/infer.py https://github.com/owner/repo/pull/123
python tags/infer.py https://github.com/owner/repo/pull/123               # multi-label
python tags/infer.py --threshold 0.3 https://github.com/owner/repo/pull/123
```

To retrain a model from its own `train.txt` in place: `cd <subfolder> && python train.py`.

`quality`/`multilingual`/`research_novelty` each print a `soft` score (probability-
weighted expected value, the better-correlating of the two — use this) and a `hard`
score (argmax level), both on a 0-10 scale. `tags` instead prints every tag (of 424)
whose predicted probability clears `--threshold` (default 0.5), sorted by confidence —
see its own section below for why it's one multi-label model, not 424 separate ones.

**Note on fastText + NumPy 2.x:** the prebuilt `fasttext-wheel` can silently corrupt
batch-`predict()` probabilities under NumPy 2.x, and even a from-source build's Python
wrapper crashes outright on `np.array(..., copy=False)`. Pin `numpy<2`.

## `quality/` — Jev's strict 0-10 quality rubric

Rates the PR's diff on a strict, sensibility-gated 0-10 scale: 0-5 is gated on whether
the codebase/diff is coherent and credible (a simple-but-clean change tops out at 5); 6+
requires genuine engineering or conceptual complexity on top of that. See
`annotate_pr_quality.py`'s `QUALITY_LEVELS` for the full rubric text.

- **Hyperparameters**: dim=64, epoch=10, lr=0.3, wordNgrams=2, minCount=3, bucket=2,000,000
- **Train**: 254,143 PRs (`train.txt`)
- **Val**: 28,238 PRs, 1,766 at level≥6 (`val.txt`)

**This is the plain dim64/epoch10 baseline** — every variant tried (2-4x capacity, class-
rebalanced train sets, a ~1.55x-larger combined corpus with a separately-mined
higher-credibility PR pool) came out flat or slightly worse on a held-out 10,000-PR
tracking set (685 at quality≥6). More data/capacity does not fix this classifier;
treat it as a known ceiling, not a bug to chase.

**Metrics** (10,000-PR tracking holdout, rule = `quality >= 6`):

| metric | value |
|---|---|
| rule accuracy | 93.73% |
| precision | 74.17% |
| **recall** | **12.99%** |
| exact-level accuracy | 78.01% |
| MAE | 0.531 |
| soft Pearson / Spearman | 0.7252 / 0.6339 |

**Read this before using it as a filter**: it misses ~87% of genuinely good PRs
(quality≥6) to keep false positives to 0.33% of negatives, and its own predictions never
exceed ~6.8-7 on any PR we've tested — it is a high-precision, very-low-recall filter,
not a general-purpose quality scorer. It also gave near-identical predictions (4.4-5.3)
to real, high-quality ML/research PRs (e.g. lucidrains' `x-transformers`, `imagen-pytorch`)
as to trivial personal-repo PRs — it is not sensitive to genuine technical depth. Lowering
the decision threshold to `>=5` makes this *worse*, not better (accuracy drops to 37%,
since ~80% of real PRs are already `>=5` and the model's own predicted mass sits below
that line).

## `multilingual/` — non-English comment/string content

Purely descriptive 0-10 scale: how much non-English natural-language text (comments,
docstrings, string literals) appears in the diff and touched files, independent of the
programming language itself. See `annotate_pr_quality.py`'s `MULTILINGUAL_LEVELS`.

- **Hyperparameters**: dim=128, epoch=25, lr=0.3, wordNgrams=2, minCount=3, bucket=2,000,000
  (best of a capacity sweep vs dim64/epoch10 baseline and dim256/epoch25 — dim128 won on
  every axis, dim256 added nothing further for 2x the model size)
- **Train**: 254,143 PRs (`train.txt`)
- **Val**: 28,238 PRs (`val.txt`)

**Metrics** (100-PR live holdout, 79 true positives, rule = `multilingual <= 1`):

| metric | value |
|---|---|
| rule accuracy | 96.00% |
| precision | 96.30% |
| **recall** | **98.73%** |
| MAE | 0.586 |
| Pearson / Spearman | 0.906 / 0.721 |

This one is genuinely reliable as a pre-filter — high recall *and* precision. If you only
trust one of these two classifiers for a filtering decision, trust this one.

## `research_novelty/` — research/IP content (⚠️ still recall≈0 at the `>=6` rule)

0-10 scale rating the extent to which the PR implements technical algorithms, ideas from
research papers, or original design/IP, as opposed to routine engineering — independent
of how difficult or high-quality the engineering itself is. See `annotate_pr_quality.py`'s
`RESEARCH_NOVELTY_LEVELS`.

- **Hyperparameters**: dim=64, epoch=10 (same baseline as `quality`/`multilingual`)
- **Train**: 96,634 PRs, 297 at level≥6 (`train.txt`)
- **Val**: 10,772 PRs, 35 at level≥6 (`val.txt`)
- **Same train/val PR ids as `tags/`** (both come from the same annotation batch; see
  Provenance) — comparable across the two.

**Updated 2026-09-30**: retrained on a new 107,406-PR annotation batch (up from the
original 39,081-PR/98-positive pass), joined back to `quality`/`multilingual`'s existing
`text` field so the same 254K+-PR pool's diff/file content is reused rather than
re-mined. Labels use the field's own fixed 0-10 scale (int truncation), same as
`quality`/`multilingual` — not sample quantiles.

**Metrics** (own val.txt, 10,772 PRs):

| metric | value |
|---|---|
| exact-level accuracy | 82.45% |
| within-1-level accuracy | 94.06% |
| MAE | 0.2745 |
| hard Pearson / Spearman | 0.5679 / 0.5069 |
| soft Pearson / Spearman | 0.6662 / 0.5470 |
| rule (`>=6`) accuracy | 99.68% |
| rule (`>=6`) precision | undefined (0 predicted positive) |
| **rule (`>=6`) recall** | **0.00%** (0/35) |

**Read this before using it as a filter**: correlation improved modestly over the old
diagnostic pass (soft Pearson 0.60→0.67) and the accuracy/MAE numbers look good, but that's
almost entirely the near-100%-easy "correctly predict low novelty" majority class — at the
actual decision boundary that matters (`>=6`), it now has enough val data to say
confidently (35 true positives, not 2) that it **misses every single one**, the same
recall-collapse pattern as `quality`. Its own predictions rarely if ever clear ~6-7 (see
the Megatron-LM spot-check below). Treat it as a continuous novelty *signal* (the soft
score correlates reasonably, 0.67) rather than a working `>=6` binary classifier.

**Spot-check**: `NVIDIA/Megatron-LM#7762` (an Engram/CUDA synchronization change) scores
soft=2.46, hard=1 — plausible as "some genuine ML-systems depth but not paper-derived,"
though still likely underrating it given the classifier's known insensitivity to real
technical depth (see `quality`'s note above; the same failure mode shows up here).

## `tags/` — multi-label topic tagging (424 tags)

Multi-label classification over the topic vocabulary in `/ansh/harbor-tags.txt` (424
distinct tags after dedup — e.g. `python`, `machine-learning`, `web-security`,
`gpu`, `formal methods`) — any number of tags can apply to one PR, independently.
Each tag is an independent Jev "noul" (boolean-probability) claim ("this PR's diff or
touched files involve code/content related to topic X"), bundled into one Jev call per
PR alongside `research_novelty` (see `annotate_tags_research.py`).

**One joint multi-label model, not 424 separate binary classifiers.** fastText's native
multi-label mode (`loss="ova"`, one-vs-all) shares a single embedding/vocabulary across
all 424 tags and trains in one pass — dramatically cheaper than 424x the training
time/storage for what is fundamentally the same text signal, at the cost of slightly
less per-tag threshold flexibility (mitigated by tuning `--threshold` in `infer.py`
per use case; default 0.5).

- **Hyperparameters**: dim=64, epoch=10, lr=0.3, wordNgrams=2, minCount=3,
  bucket=2,000,000, **loss=ova**
- **Train**: 96,604 PRs (`train.txt`) / **Val**: 10,769 PRs (`val.txt`) — same PR ids as
  `research_novelty/`'s updated split (both from the same annotation batch)
- 33 PRs with zero tags above threshold were dropped (fastText needs ≥1 label/line)

**Metrics** (10,769-PR val set, pooled across all 10,769×424 PR-tag pairs, threshold=0.5):

| metric | value |
|---|---|
| **precision** | **81.95%** |
| **recall** | **57.17%** |
| f1 | 67.35% |
| pooled Pearson r (vs. Jev's raw `tag_probs`) | **0.8055** |
| pooled Spearman ρ | **0.7145** |

Precision stays high across common tags (0.79-0.99) with recall degrading gracefully as
per-tag support drops — normal multi-label behavior, not a red flag. This is the
best-behaved of the four classifiers relative to its task difficulty (a 424-way
multi-label problem vs. a single 0-10 scale).

**Known data-hygiene issue**: `harbor-tags.txt` has a couple of near-duplicate tag pairs
that survived as distinct labels (`data processing`/`data-processing`, `data
engineering`/`data_engineering` — the latter pair also collided post-slugification and
one was suffixed `_dup` in `tag_vocab.json` to disambiguate). Worth deduping the source
file; not fixed here since it would invalidate the current label vocabulary/model.

**Spot-check**: `vercel/next.js#99499` (a tag-revalidation fix) → 30 tags incl.
`typescript`, `react`, `nextjs`, `frontend`, `routing`, `concurrency`, `web-server`.
`NVIDIA/Megatron-LM#7762` → 36 tags incl. `machine-learning`, `pytorch`, `cuda`, `gpu`,
`distributed-systems`, `performance-optimization`, `deeply-technical-ip`. Both match
what a human skim of the PRs would tag.

## Provenance

- Distilled from `typesafe/jev-1.13` (via OpenRouter) on GitHub Archive-mined PRs.
- Full annotation/training pipeline: `/ansh/prism-datagen/annotate_pr_quality.py`
  (rubric + Jev calls, incl. `HARBOR_TAGS`/`TAG_CLAIMS`), `/ansh/fasttext-stuff/run.py`/
  `train_sweep.py` (quality/multilingual training — each subfolder's `train.py` is the
  relevant one of these, trimmed to a single model), `/ansh/fasttext-stuff/evaluate.py`/
  `evaluate_sweep.py`/`compare_pre_post.py` (eval).
- `quality`'s known ceiling was diagnosed, not guessed: a dim/epoch capacity sweep
  (64→128→256), a class-rebalancing ablation (10%/15% minority oversampling), and a
  ~1.55x larger combined training corpus (adding a separately GitHub-credibility-filtered
  PR pool skewed toward higher quality) were all tried and none moved recall past ~14%.
- `research_novelty`/`tags` come from a separate, lighter-weight annotation pass:
  `/ansh/prism-datagen/annotate_tags_research.py` re-streams the same S3 source
  `quality`/`multilingual` were mined from, filtered to PR ids already in that
  254K+-PR corpus, and asks Jev only for `research_novelty` + the 424 tag claims
  (bundled into one call per PR, not 424 separate calls) — reusing that corpus's
  existing `text` field rather than re-annotating everything from scratch. Dataset
  build: `/ansh/fasttext-stuff/build_research_novelty_v2_dataset.py` and
  `build_tags_dataset.py` (both replicate `quality`/`multilingual`'s exact train/val
  split — same seed, same 10%, same PR ids — for cross-classifier comparability).
  Training/eval: each subfolder's own `train.py`/`evaluate.py`.