v17

README.md

@@ -31,7 +31,9 @@ The course permits external data provided it is declared. This model uses three
 | + transcript-level TrEMBL as a fast second pass, weight 0.1 (v16, shipped) | **0.8467** |
 | train + full Swiss-Prot (v13) | 0.9634 |
 
-The labels for this check come from Swiss-Prot. They agree with the official scoring: our train-only model scores L1 0.962 here and 0.960 on the leaderboard.
+On **official labels**, the same idea gives a larger honest gain. On the 13,440 hard training proteins above, the vote scores 0.732 with the training reference only and **0.801** with the external references added. Training sequences are never part of the external sets, so every hit is a different protein.
+
+The labels for the validation check come from Swiss-Prot. They agree with the official scoring: our train-only model scores L1 0.962 here and 0.960 on the leaderboard.
 
 No external data was used for training any learned component. ESM-2 and other pretrained protein language models were used only in experiments and are not part of this submission.
 
@@ -40,6 +42,7 @@ No external data was used for training any learned component. ESM-2 and other pr
 2. **Second pass.** Inputs with no hit, or a best hit of 30% identity or less, are searched again with `--ultra-sensitive` when the time budget clearly allows it.
 3. **Extra reference pass (v16).** All inputs are also searched against the transcript-level TrEMBL database in DIAMOND's `--fast` mode, about 25 s. This runs only if less than 330 s have passed. It only adds hits and is skipped on any error. For the hard validation proteins not in Swiss-Prot, it doubles the number with a relative at 80% identity or more (53 to 103, with validation sequences excluded).
 4. **Vote.** Each EC gets `sum over the top 50 hits of (bitscore / best bitscore)^12 x w_source`, split evenly across a hit's ECs. `w_source` is 1 for train and Swiss-Prot references, 0.2 for protein-level TrEMBL references and 0.1 for transcript-level TrEMBL references, whose automatic annotations are noisier. The highest-scoring EC is predicted, with ties broken by training frequency. Settings are in `weights/mode.json`. They were tuned on the validation inputs that have Swiss-Prot labels, with every validation sequence removed from the external reference. Tuning used alternate halves: the best setting chosen on one half improved the other half by the same margin (+0.5 points over k=20, power 8, unweighted).
+5. **Learned re-ranker (v17).** For inputs whose best hit is below 80% identity, a gradient-boosted tree model (scikit-learn HistGradientBoosting, exported to pure numpy in `weights/ranker_v2.npz`) re-orders the vote's top 10 ECs. It uses 29 features per candidate (`rerank2.py`): vote shares at several powers, share of the vote from each source (train / Swiss-Prot / TrEMBL), best-hit identity, coverage and rank, support counts, EC-hierarchy agreement, EC frequency and query difficulty. It was trained on **official training labels**: 13,440 hard training proteins (one per 50%-identity cluster, best training hit below 58% identity), searched against the other cluster folds plus the external references. In grouped 5-fold CV on that set it improves L4 by about 0.3 points (vote 0.7982 / 0.8031 → re-ranker 0.8011 / 0.8064 on the two halves). The gain comes below 80% identity; above that it is neutral, hence the cutoff.
 
 ## Robustness
 - The evaluator caps every written file at 16 MiB (RLIMIT_FSIZE) and allows only 256 open files (RLIMIT_NOFILE). DIAMOND runs with `--bin 128` (derived from RLIMIT_NOFILE) and batches of at most 8,000 queries. It halves the bin count on "Too many open files" and splits a batch on SIGXFSZ.

predict.py

@@ -1,4 +1,4 @@
-"""Offline CPU inference: DIAMOND homology search against train + UniProt references, bitscore vote (mode.json).
+"""Offline CPU inference: DIAMOND homology search against train + UniProt references, bitscore vote + learned re-ranker (mode.json).
 
 1. Search every unique input against the training set (diamond blastp
    --very-sensitive, 50 hits).
@@ -328,6 +328,35 @@ def homology_predictions(seqs, labels, freq):
                     votes[e] = votes.get(e, 0.0) + w / len(ecs)
             if votes:
                 preds[i] = sorted(votes, key=lambda e: (-votes[e], -freq.get(e, 0), e))[:10]
+        # v17: learned re-ranker over the vote's top 10 candidates, used only when the best hit is below
+        # rerank_max_pid identity (it was trained and validated on official-label hard proteins there).
+        rr_path = ROOT / 'weights' / 'ranker_v2.npz'
+        if mode_cfg.get('reranker') and rr_path.exists():
+            try:
+                from rerank2 import candidate_features
+                rr = TreeRanker(rr_path)
+                srcw = mode_cfg.get('source_weights', {})
+                qs, blocks, cand_of = [], [], {}
+                for i in sorted(hits):
+                    if not hits[i] or max(x[1] for x in hits[i]) >= float(mode_cfg.get('rerank_max_pid', 80)):
+                        continue
+                    cands, X = candidate_features(hits[i], labels, freq, srcw, int(mode_cfg['vote_k']),
+                                                  float(mode_cfg['vote_power']))
+                    if len(cands) > 1:
+                        qs.append(i)
+                        cand_of[i] = cands
+                        blocks.append(np.asarray(X, dtype=np.float64))
+                if blocks:
+                    scores = rr.score(np.vstack(blocks))
+                    pos = 0
+                    for i in qs:
+                        c = cand_of[i]
+                        sc = scores[pos:pos + len(c)]
+                        pos += len(c)
+                        preds[i] = [c[j] for j in np.argsort(-sc, kind='stable')]
+                log(f're-ranker v2 applied to {len(qs)} inputs')
+            except Exception as exc:
+                log(f're-ranker v2 skipped, using the vote: {exc!r}')
         return preds
     ranker = TreeRanker(ROOT / 'weights' / 'ranker.npz')
     rel = EcReliability(ROOT / 'weights' / 'te_tables.json.gz')

rerank2.py

@@ -0,0 +1,67 @@
+"""Re-ranker v2: candidate-EC features from the merged hit list (train + UniProt references).
+
+The same function builds the training rows (official-label proxy, see experiments/external) and the
+inference rows, so features cannot drift. Candidates are the top 10 ECs of the weighted vote.
+"""
+import math
+
+FEATURES = ['v12w', 'v12', 'v8', 'v4', 'v1', 'rank', 's_tr', 's_sp', 's_un', 's_u2', 'best_bits_rel', 'best_pid',
+            'best_rank', 'best_src', 'qcov', 'scov', 'n10', 'n50', 'l3', 'l2', 'lfreq', 'top_pid', 'top_bits',
+            'top_src', 'n_hits', 'n_cand', 'margin', 'pid_gap', 'nec_best']
+SRC = {'sp_': 1, 'un_': 2, 'u2_': 3}
+
+
+def source(s):
+    """0 = training protein (ids tr_<digits>), 1 = Swiss-Prot, 2 = TrEMBL protein-level, 3 = TrEMBL transcript-level."""
+    return 0 if s[3:].isdigit() else SRC.get(s[:3], 0)
+
+
+def candidate_features(hits, labels, freq, weights, k=50, power=12):
+    """hits: (sseqid, pid, qlen, slen, evalue, bits, qstart, qend, sstart, send) tuples, any order.
+    Returns (candidate ECs ordered by weighted vote, feature rows)."""
+    h = sorted(hits, key=lambda x: -x[5])[:k]
+    if not h:
+        return [], []
+    top = h[0][5]
+    st, l3, l2 = {}, {}, {}
+    tot = {'w': 0.0, '12': 0.0, '8': 0.0, '4': 0.0, '1': 0.0}
+    for r, (s, pid, ql, sl, ev, b, qs, qe, ss, se) in enumerate(h):
+        ecs = labels[s]
+        n = len(ecs)
+        rb = b / top
+        src = source(s)
+        w = rb ** power * (1.0 if src == 0 else weights.get(s[:3], 1.0))
+        tot['w'] += w
+        tot['12'] += rb ** 12
+        tot['8'] += rb ** 8
+        tot['4'] += rb ** 4
+        tot['1'] += rb
+        for e in ecs:
+            d = st.get(e)
+            if d is None:
+                d = st[e] = dict(w=0.0, p12=0.0, p8=0.0, p4=0.0, p1=0.0, src=[0.0, 0.0, 0.0, 0.0], bb=b, bp=pid, br=r,
+                                 bs=src, qc=(qe - qs + 1) / ql, sc=(se - ss + 1) / sl, n10=0, n50=0, nec=n)
+            d['w'] += w / n
+            d['p12'] += rb ** 12 / n
+            d['p8'] += rb ** 8 / n
+            d['p4'] += rb ** 4 / n
+            d['p1'] += rb / n
+            d['src'][src] += w / n
+            d['n10'] += r < 10
+            d['n50'] += 1
+            p3 = e.rsplit('.', 1)[0]
+            p2 = '.'.join(e.split('.')[:2])
+            l3[p3] = l3.get(p3, 0.0) + w / n
+            l2[p2] = l2.get(p2, 0.0) + w / n
+    cands = sorted(st, key=lambda e: (-st[e]['w'], -freq.get(e, 0), e))[:10]
+    ws = [st[e]['w'] for e in cands]
+    margin = (ws[0] - (ws[1] if len(ws) > 1 else 0.0)) / tot['w']
+    rows = []
+    for i, e in enumerate(cands):
+        d = st[e]
+        rows.append([d['w'] / tot['w'], d['p12'] / tot['12'], d['p8'] / tot['8'], d['p4'] / tot['4'], d['p1'] / tot['1'], i,
+                     *[x / tot['w'] for x in d['src']], d['bb'] / top, d['bp'], d['br'], d['bs'], d['qc'], d['sc'],
+                     d['n10'], d['n50'], l3[e.rsplit('.', 1)[0]] / tot['w'], l2['.'.join(e.split('.')[:2])] / tot['w'],
+                     math.log1p(freq.get(e, 0)), h[0][1], top, source(h[0][0]), len(h), len(st), margin,
+                     h[0][1] - d['bp'], d['nec']])
+    return cands, rows

weights/mode.json

@@ -1 +1 @@
-{"scoring": "vote", "vote_k": 50, "vote_power": 12, "source_weights": {"sp_": 1.0, "un_": 0.2, "u2_": 0.1}, "extra_db": "trembl_ev2.dmnd", "extra_mode": "fast", "extra_guard": 330}
+{"scoring": "vote", "vote_k": 50, "vote_power": 12, "source_weights": {"sp_": 1.0, "un_": 0.2, "u2_": 0.1}, "extra_db": "trembl_ev2.dmnd", "extra_mode": "fast", "extra_guard": 330, "reranker": true, "rerank_max_pid": 80}

weights/ranker_v2.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8909167aa4c5233416f390f70f93ac1cce2627aaaf0cb6a14535e7d3bc9b3875
+size 193919