Upload 12 files
Browse files- Dockerfile +9 -0
- README.md +59 -0
- db.parquet +3 -0
- ec_vocabulary.json +0 -0
- gnn.npz +3 -0
- gnn.py +82 -0
- inputs.py +26 -0
- kmer.py +67 -0
- predict.py +154 -0
- requirements.txt +7 -0
- task.json +19 -0
- train.py +31 -0
Dockerfile
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
FROM python:3.11-slim-bookworm
|
| 2 |
+
RUN useradd --create-home --uid 1000 user
|
| 3 |
+
WORKDIR /app
|
| 4 |
+
COPY requirements.txt .
|
| 5 |
+
RUN pip install --no-cache-dir -r requirements.txt
|
| 6 |
+
COPY --chown=user:user . .
|
| 7 |
+
USER user
|
| 8 |
+
ENV OMP_NUM_THREADS=2 OPENBLAS_NUM_THREADS=2
|
| 9 |
+
CMD ["python", "/app/predict.py", "--help"]
|
README.md
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
tags:
|
| 4 |
+
- enzyme-classification
|
| 5 |
+
- ec-number
|
| 6 |
+
- graph-neural-network
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
# Leak CURBER enzyme classification — homology-graph neural network
|
| 10 |
+
|
| 11 |
+
A graph attention network predicts one EC number per protein from the protein's homology neighbourhood.
|
| 12 |
+
Retrieval supplies the graph; the neural network makes every prediction.
|
| 13 |
+
|
| 14 |
+
## Pipeline
|
| 15 |
+
1. **Retrieval (builds the graph)**: TF-IDF k-mer search over the 186k unique training enzymes (top 1,000 by
|
| 16 |
+
5-mers + top 300 by 4-mers), then Smith-Waterman local alignment (BLOSUM62, gap 11/1, `pyopal`).
|
| 17 |
+
The query and its 20 best hits form a graph.
|
| 18 |
+
2. **Graph neural network** (`gnn.py`): each neighbour is a node with 11 features:
|
| 19 |
+
- 7 edge features from the alignment: score / query self-score, score / neighbour self-score,
|
| 20 |
+
log(score / best score), rank, log length ratio, best score / query self-score, log(1 + score);
|
| 21 |
+
- 4 agreement features: how much of the neighbourhood shares this neighbour's EC at levels 1, 2, 3, 4.
|
| 22 |
+
|
| 23 |
+
Features -> MLP (11 -> 128 -> 128) -> 2 transformer layers (4 heads, pre-norm) that pass messages between
|
| 24 |
+
all neighbours -> a weight per neighbour (output head plus a residual alpha * log(score/best), alpha learned).
|
| 25 |
+
The EC with the largest total softmax weight is the prediction.
|
| 26 |
+
|
| 27 |
+
## Training
|
| 28 |
+
- Graph for every training enzyme: its top-60 Smith-Waterman neighbours among the other training enzymes.
|
| 29 |
+
- To mimic test conditions (validation enzymes are distant from train: median best alignment = 0.38 of
|
| 30 |
+
self-score, vs 0.92 inside train), each training example hides neighbours closer than a threshold drawn from
|
| 31 |
+
the validation distribution, then keeps the 20 best remaining.
|
| 32 |
+
- Loss: negative log of the softmax weight on neighbours carrying a true EC. AdamW, lr 1e-3, 5 epochs, PyTorch;
|
| 33 |
+
weights exported to NumPy (`gnn.npz`, 1.1 MB, ~280k parameters). The residual starts the network at a hand-tuned
|
| 34 |
+
similarity vote, so training learns corrections to it.
|
| 35 |
+
|
| 36 |
+
## Local validation (4,915 distant training enzymes, weighted to match validation)
|
| 37 |
+
| Metric | Hand-tuned vote (v2) | Graph network |
|
| 38 |
+
|---|---|---|
|
| 39 |
+
| L1 | 0.947 | 0.951 |
|
| 40 |
+
| L2 | 0.917 | 0.920 |
|
| 41 |
+
| L3 | 0.903 | 0.906 |
|
| 42 |
+
| L4 | 0.749 | 0.749 |
|
| 43 |
+
|
| 44 |
+
Leaderboard (public validation): v2 vote L4 0.773.
|
| 45 |
+
|
| 46 |
+
Alternatives tested: ESM2-8M embeddings (too slow on 2 CPUs: ~76 min for validation); a k-mer
|
| 47 |
+
embedding-bag classifier alone (L4 0.53 locally).
|
| 48 |
+
|
| 49 |
+
## Runtime
|
| 50 |
+
11,367 validation inputs in 357 s (graph network step: 9 s), 4.1 GB RAM, 2 vCPU. Queries not aligned by 480 s keep a
|
| 51 |
+
5-mer nearest-neighbour prediction, so the run stays under 10 minutes.
|
| 52 |
+
|
| 53 |
+
## Files
|
| 54 |
+
- `predict.py` — inference entry point
|
| 55 |
+
- `gnn.py`, `gnn.npz` — graph neural network (NumPy forward pass, weights)
|
| 56 |
+
- `kmer.py` — sparse k-mer index
|
| 57 |
+
- `db.parquet` — unique training sequences with their ECs
|
| 58 |
+
- `train.py` — builds `db.parquet` from `train.parquet`
|
| 59 |
+
- `inputs.py`, `task.json`, `ec_vocabulary.json` — from the starter
|
db.parquet
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2f4bacae315f0a143f72523b455ae5d92f028c0c9ef567e8ead2b1e2c5e8b501
|
| 3 |
+
size 26879165
|
ec_vocabulary.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
gnn.npz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e6f2c8ebfe3a72806b2e1a899d97d7f37f8e3e75a600bc61b1ea124366fdb458
|
| 3 |
+
size 1142832
|
gnn.py
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""NumPy inference for the homology-graph attention network (trained in PyTorch, weights in gnn.npz).
|
| 2 |
+
|
| 3 |
+
For each query, the graph is the query plus its K=20 best Smith-Waterman hits among the training enzymes.
|
| 4 |
+
Each neighbour becomes a token of 11 features:
|
| 5 |
+
7 edge features from the alignment (score / query self-score, score / neighbour self-score, log(score / best),
|
| 6 |
+
rank, log length ratio, best / query self-score, log(1 + score)) and
|
| 7 |
+
4 agreement features (vote-weighted share of the neighbourhood sharing this neighbour's EC at levels 1..4).
|
| 8 |
+
Tokens -> MLP(11 -> 128 -> 128) -> 2 transformer layers (4 heads, pre-norm) that pass messages between
|
| 9 |
+
neighbours -> per-neighbour weight = head(h) + alpha * log(score / best). The EC with the largest total
|
| 10 |
+
softmax weight is the prediction.
|
| 11 |
+
"""
|
| 12 |
+
import numpy as np
|
| 13 |
+
from scipy.special import erf
|
| 14 |
+
|
| 15 |
+
K = 20
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def _ln(x, w, b, eps=1e-5):
|
| 19 |
+
mu = x.mean(-1, keepdims=True)
|
| 20 |
+
var = ((x - mu) ** 2).mean(-1, keepdims=True)
|
| 21 |
+
return (x - mu) / np.sqrt(var + eps) * w + b
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _gelu(x):
|
| 25 |
+
return 0.5 * x * (1 + erf(x / np.sqrt(2)))
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def _softmax(x, axis=-1):
|
| 29 |
+
x = x - x.max(axis, keepdims=True)
|
| 30 |
+
e = np.exp(x)
|
| 31 |
+
return e / e.sum(axis, keepdims=True)
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def features(S, selfq, selfn, lq, ln, valid):
|
| 35 |
+
"""S: (B,K) alignment scores sorted descending. Returns edge features (B,K,7) and log(score/best) (B,K)."""
|
| 36 |
+
S = np.where(valid, np.maximum(S, 1e-3), 1e-3)
|
| 37 |
+
mx = S[:, :1]
|
| 38 |
+
rel = np.log(S / mx)
|
| 39 |
+
f = np.stack([S / selfq[:, None], S / np.maximum(selfn, 1), rel / 5,
|
| 40 |
+
np.broadcast_to(np.arange(K) / K, S.shape), np.log(np.maximum(ln, 1) / lq[:, None]),
|
| 41 |
+
np.broadcast_to(mx / selfq[:, None], S.shape), np.log1p(S) / 10], -1)
|
| 42 |
+
f[~valid] = 0
|
| 43 |
+
return f.astype(np.float32), np.where(valid, rel, -30).astype(np.float32)
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def agreement(prefixes, rel, valid):
|
| 47 |
+
"""prefixes: (B,K,4) int ids of the first EC's L1..L4 prefixes. Returns (B,K,4)."""
|
| 48 |
+
wv = _softmax(16 * np.where(valid, rel, -30))
|
| 49 |
+
same = (prefixes[:, :, None, :] == prefixes[:, None, :, :]) & valid[:, None, :, None]
|
| 50 |
+
return (same * wv[:, None, :, None]).sum(2).astype(np.float32)
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
class HomologyGNN:
|
| 54 |
+
def __init__(self, path):
|
| 55 |
+
w = np.load(path)
|
| 56 |
+
self.w = {k: w[k] for k in w.files}
|
| 57 |
+
self.alpha = float(self.w['alpha'])
|
| 58 |
+
self.nl = sum(1 for k in self.w if k.endswith('self_attn.in_proj_weight'))
|
| 59 |
+
|
| 60 |
+
def _layer(self, x, i, keymask):
|
| 61 |
+
w = self.w; p = f'enc.layers.{i}.'
|
| 62 |
+
B, T, D = x.shape; H = 4; hd = D // H
|
| 63 |
+
a = _ln(x, w[p + 'norm1.weight'], w[p + 'norm1.bias'])
|
| 64 |
+
qkv = a @ w[p + 'self_attn.in_proj_weight'].T + w[p + 'self_attn.in_proj_bias']
|
| 65 |
+
q, k, v = (qkv[..., j * D:(j + 1) * D].reshape(B, T, H, hd).transpose(0, 2, 1, 3) for j in range(3))
|
| 66 |
+
s = q @ k.transpose(0, 1, 3, 2) / np.sqrt(hd)
|
| 67 |
+
s = np.where(keymask[:, None, None, :], s, -np.inf)
|
| 68 |
+
ctx = (_softmax(s) @ v).transpose(0, 2, 1, 3).reshape(B, T, D)
|
| 69 |
+
x = x + ctx @ w[p + 'self_attn.out_proj.weight'].T + w[p + 'self_attn.out_proj.bias']
|
| 70 |
+
b = _ln(x, w[p + 'norm2.weight'], w[p + 'norm2.bias'])
|
| 71 |
+
ff = np.maximum(b @ w[p + 'linear1.weight'].T + w[p + 'linear1.bias'], 0)
|
| 72 |
+
return x + ff @ w[p + 'linear2.weight'].T + w[p + 'linear2.bias']
|
| 73 |
+
|
| 74 |
+
def neighbour_weights(self, f, rel, prefixes, valid):
|
| 75 |
+
w = self.w
|
| 76 |
+
x = np.concatenate([f, agreement(prefixes, rel, valid)], -1)
|
| 77 |
+
x = _gelu(x @ w['fin.0.weight'].T + w['fin.0.bias']) @ w['fin.2.weight'].T + w['fin.2.bias']
|
| 78 |
+
for i in range(self.nl):
|
| 79 |
+
x = self._layer(x, i, valid)
|
| 80 |
+
a = (_ln(x, w['head.0.weight'], w['head.0.bias']) @ w['head.1.weight'].T)[..., 0] + w['head.1.bias'][0]
|
| 81 |
+
a = a + self.alpha * rel
|
| 82 |
+
return _softmax(np.where(valid, a, -1e4))
|
inputs.py
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Prepare exactly the unique input rows used by the evaluator."""
|
| 2 |
+
import argparse
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
import pandas as pd
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def prepare_inputs(frame):
|
| 10 |
+
config = json.loads((Path(__file__).parent / 'task.json').read_text())
|
| 11 |
+
columns = ['id'] + config['input_columns']
|
| 12 |
+
if not set(columns) <= set(frame.columns):
|
| 13 |
+
raise ValueError('Missing input columns: ' + ', '.join(columns))
|
| 14 |
+
if frame.id.isna().any() or frame.id.duplicated().any():
|
| 15 |
+
raise ValueError('Input IDs must be present and unique.')
|
| 16 |
+
return frame[columns].drop_duplicates(config['input_columns'], keep='first').reset_index(drop=True)
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
if __name__ == '__main__':
|
| 20 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 21 |
+
parser.add_argument('--input', required=True)
|
| 22 |
+
parser.add_argument('--output', required=True)
|
| 23 |
+
args = parser.parse_args()
|
| 24 |
+
frame = prepare_inputs(pd.read_parquet(args.input))
|
| 25 |
+
frame.to_parquet(args.output, index=False)
|
| 26 |
+
print(f'Wrote {len(frame)} unique inputs to {args.output}')
|
kmer.py
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Sparse k-mer index: TF-IDF-weighted, L2-normalised presence vectors for cosine search."""
|
| 2 |
+
import numpy as np
|
| 3 |
+
import scipy.sparse as sp
|
| 4 |
+
|
| 5 |
+
AA = 'ACDEFGHIKLMNPQRSTVWY'
|
| 6 |
+
LUT = np.full(256, 20, dtype=np.int64)
|
| 7 |
+
for i, c in enumerate(AA):
|
| 8 |
+
LUT[ord(c)] = i
|
| 9 |
+
LUT[ord(c.lower())] = i
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
def kmer_ids(seq, k):
|
| 13 |
+
a = LUT[np.frombuffer(seq.encode('ascii', 'replace'), dtype=np.uint8)]
|
| 14 |
+
if len(a) < k:
|
| 15 |
+
return np.empty(0, dtype=np.int64)
|
| 16 |
+
ids = np.zeros(len(a) - k + 1, dtype=np.int64)
|
| 17 |
+
for j in range(k):
|
| 18 |
+
ids = ids * 21 + a[j:len(a) - k + 1 + j]
|
| 19 |
+
return np.unique(ids)
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def presence_matrix(seqs, k):
|
| 23 |
+
rows, cols = [], []
|
| 24 |
+
for i, s in enumerate(seqs):
|
| 25 |
+
ids = kmer_ids(s, k)
|
| 26 |
+
rows.append(np.full(len(ids), i, dtype=np.int32))
|
| 27 |
+
cols.append(ids)
|
| 28 |
+
rows = np.concatenate(rows)
|
| 29 |
+
cols = np.concatenate(cols)
|
| 30 |
+
data = np.ones(len(rows), dtype=np.float32)
|
| 31 |
+
return sp.csr_matrix((data, (rows, cols)), shape=(len(seqs), 21 ** k))
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def weight(m, idf):
|
| 35 |
+
m = m.multiply(idf[None, :]).tocsr().astype(np.float32)
|
| 36 |
+
norms = np.sqrt(np.asarray(m.multiply(m).sum(axis=1)).ravel())
|
| 37 |
+
norms[norms == 0] = 1
|
| 38 |
+
return sp.diags(1 / norms).dot(m).tocsr().astype(np.float32)
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
class KmerIndex:
|
| 42 |
+
def __init__(self, seqs, k=5):
|
| 43 |
+
self.k = k
|
| 44 |
+
m = presence_matrix(seqs, k)
|
| 45 |
+
df = np.bincount(m.indices, minlength=m.shape[1]).astype(np.float32)
|
| 46 |
+
self.idf = np.log((1 + m.shape[0]) / (1 + df)).astype(np.float32)
|
| 47 |
+
self.T = weight(m, self.idf).T.tocsr() # (features x n_train)
|
| 48 |
+
|
| 49 |
+
def topk(self, seqs, k=50, chunk=256):
|
| 50 |
+
"""Return (indices, scores) arrays of shape (len(seqs), k)."""
|
| 51 |
+
n = len(seqs)
|
| 52 |
+
idx = np.full((n, k), -1, dtype=np.int64)
|
| 53 |
+
sc = np.zeros((n, k), dtype=np.float32)
|
| 54 |
+
for start in range(0, n, chunk):
|
| 55 |
+
q = weight(presence_matrix(seqs[start:start + chunk], self.k), self.idf)
|
| 56 |
+
s = (q @ self.T).tocsr()
|
| 57 |
+
for r in range(s.shape[0]):
|
| 58 |
+
lo, hi = s.indptr[r], s.indptr[r + 1]
|
| 59 |
+
if hi == lo:
|
| 60 |
+
continue
|
| 61 |
+
d, c = s.data[lo:hi], s.indices[lo:hi]
|
| 62 |
+
m = min(k, hi - lo)
|
| 63 |
+
part = np.argpartition(-d, m - 1)[:m]
|
| 64 |
+
part = part[np.argsort(-d[part])]
|
| 65 |
+
idx[start + r, :m] = c[part]
|
| 66 |
+
sc[start + r, :m] = d[part]
|
| 67 |
+
return idx, sc
|
predict.py
ADDED
|
@@ -0,0 +1,154 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Homology-graph neural network for EC prediction (offline, 2 CPUs).
|
| 2 |
+
|
| 3 |
+
1. Retrieval: k-mer prefilter (5-mers + 4-mers) finds candidate training enzymes; Smith-Waterman alignment
|
| 4 |
+
(BLOSUM62) scores them; the 20 best hits form the query's homology graph.
|
| 5 |
+
2. Neural network: a graph attention network (gnn.py) reads the 20 neighbours (alignment features + EC agreement),
|
| 6 |
+
passes messages between them with two transformer layers and assigns each neighbour a weight.
|
| 7 |
+
The EC carrying the most weight is the prediction.
|
| 8 |
+
"""
|
| 9 |
+
import argparse
|
| 10 |
+
import gc
|
| 11 |
+
import json
|
| 12 |
+
import os
|
| 13 |
+
import time
|
| 14 |
+
from collections import Counter
|
| 15 |
+
from concurrent.futures import ThreadPoolExecutor
|
| 16 |
+
from pathlib import Path
|
| 17 |
+
|
| 18 |
+
import numpy as np
|
| 19 |
+
import pandas as pd
|
| 20 |
+
import pyopal
|
| 21 |
+
import scoring_matrices
|
| 22 |
+
|
| 23 |
+
from gnn import K, HomologyGNN, features
|
| 24 |
+
from inputs import prepare_inputs
|
| 25 |
+
from kmer import KmerIndex
|
| 26 |
+
|
| 27 |
+
ROOT = Path(__file__).parent
|
| 28 |
+
T0 = time.time()
|
| 29 |
+
BUDGET = float(os.environ.get('TIME_BUDGET', 480)) # seconds; hard limit is 600
|
| 30 |
+
N5 = 1000 # 5-mer candidates per query
|
| 31 |
+
N4 = 300 # extra 4-mer candidates per query (merged, duplicates removed)
|
| 32 |
+
MAX_LEN_RATIO = 3 # skip 4-mer extras longer than 3x the query (slow to align, rarely homologs)
|
| 33 |
+
THREADS = 2
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def log(msg):
|
| 37 |
+
print(f'[{time.time() - T0:6.1f}s] {msg}', flush=True)
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def find(name):
|
| 41 |
+
for p in (ROOT / 'weights' / name, ROOT / name):
|
| 42 |
+
if p.exists():
|
| 43 |
+
return p
|
| 44 |
+
raise FileNotFoundError(name)
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def load_db():
|
| 48 |
+
db = pd.read_parquet(find('db.parquet'))
|
| 49 |
+
vocab = set(json.loads((ROOT / 'ec_vocabulary.json').read_text()))
|
| 50 |
+
ecs = [[e for e in row if e in vocab] for row in db.ecs]
|
| 51 |
+
keep = [i for i, e in enumerate(ecs) if e]
|
| 52 |
+
seqs = [db.sequence.iat[i] for i in keep]
|
| 53 |
+
ecs = [ecs[i] for i in keep]
|
| 54 |
+
fallback = Counter(e for row in ecs for e in row).most_common(1)[0][0]
|
| 55 |
+
return seqs, ecs, fallback
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def self_scores(seqs):
|
| 59 |
+
m = scoring_matrices.ScoringMatrix.from_name('BLOSUM62')
|
| 60 |
+
diag = np.zeros(256)
|
| 61 |
+
for i, ch in enumerate(m.alphabet):
|
| 62 |
+
diag[ord(ch)] = m[ch][i]
|
| 63 |
+
return np.array([diag[np.frombuffer(s.encode('ascii', 'replace'), np.uint8)].sum() for s in seqs], np.float32)
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def prefix_ids(ecs):
|
| 67 |
+
"""(N,4) ids of the L1..L4 prefixes of each enzyme's first EC."""
|
| 68 |
+
pid = {}
|
| 69 |
+
return np.array([[pid.setdefault('.'.join(row[0].split('.')[:L]) + f'#{L}', len(pid)) for L in (1, 2, 3, 4)]
|
| 70 |
+
for row in ecs], np.int64)
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def candidates(seqs, queries, k, n):
|
| 74 |
+
index = KmerIndex(seqs, k=k)
|
| 75 |
+
idx, _ = index.topk(queries, n)
|
| 76 |
+
del index
|
| 77 |
+
gc.collect()
|
| 78 |
+
log(f'{k}-mer candidates ready')
|
| 79 |
+
return idx
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def predict_file(input_path, output_path):
|
| 83 |
+
frame = prepare_inputs(pd.read_parquet(input_path))
|
| 84 |
+
queries = [s if isinstance(s, str) else '' for s in frame.sequence]
|
| 85 |
+
log(f'{len(queries)} unique inputs')
|
| 86 |
+
|
| 87 |
+
seqs, ecs, fallback = load_db()
|
| 88 |
+
c5 = candidates(seqs, queries, 5, N5)
|
| 89 |
+
# Default prediction: 5-mer nearest neighbour (used only if time runs short).
|
| 90 |
+
preds = [ecs[i[0]][0] if i[0] >= 0 else fallback for i in c5]
|
| 91 |
+
c4 = candidates(seqs, queries, 4, N4) if time.time() - T0 < BUDGET / 3 else None
|
| 92 |
+
|
| 93 |
+
aligner = pyopal.Aligner('BLOSUM62', gap_open=11, gap_extend=1)
|
| 94 |
+
database = pyopal.Database(seqs)
|
| 95 |
+
lengths = np.array([len(s) for s in seqs])
|
| 96 |
+
|
| 97 |
+
def align(q):
|
| 98 |
+
if time.time() - T0 > BUDGET or not queries[q]:
|
| 99 |
+
return q, None
|
| 100 |
+
pool = c5[q]
|
| 101 |
+
if c4 is not None:
|
| 102 |
+
extra = c4[q][c4[q] >= 0]
|
| 103 |
+
extra = extra[lengths[extra] <= MAX_LEN_RATIO * len(queries[q])]
|
| 104 |
+
pool = np.concatenate([pool, extra])
|
| 105 |
+
cand = np.array([int(c) for c in dict.fromkeys(pool.tolist()) if c >= 0])
|
| 106 |
+
if len(cand) == 0:
|
| 107 |
+
return q, None
|
| 108 |
+
scores = np.array([r.score for r in aligner.align(queries[q], database.extract(cand.tolist()))], np.float32)
|
| 109 |
+
if scores.max() <= 0:
|
| 110 |
+
return q, None
|
| 111 |
+
o = np.argsort(-scores)[:K]
|
| 112 |
+
return q, (cand[o], scores[o])
|
| 113 |
+
|
| 114 |
+
NI = np.full((len(queries), K), -1, np.int64)
|
| 115 |
+
NS = np.zeros((len(queries), K), np.float32)
|
| 116 |
+
with ThreadPoolExecutor(THREADS) as pool:
|
| 117 |
+
for q, res in pool.map(align, range(len(queries))):
|
| 118 |
+
if res is not None:
|
| 119 |
+
NI[q, :len(res[0])], NS[q, :len(res[1])] = res
|
| 120 |
+
aligned = np.where(NI[:, 0] >= 0)[0]
|
| 121 |
+
log(f'aligned {len(aligned)}/{len(queries)} queries')
|
| 122 |
+
|
| 123 |
+
# Homology graph neural network
|
| 124 |
+
net = HomologyGNN(find('gnn.npz'))
|
| 125 |
+
dself, dpref = self_scores(seqs), prefix_ids(ecs)
|
| 126 |
+
qself = self_scores(queries)
|
| 127 |
+
qlen = np.array([max(len(s), 1) for s in queries], np.float32)
|
| 128 |
+
for s in range(0, len(aligned), 1024):
|
| 129 |
+
b = aligned[s:s + 1024]
|
| 130 |
+
ni, valid = NI[b].clip(0), NI[b] >= 0
|
| 131 |
+
f, rel = features(NS[b], qself[b], dself[ni], qlen[b], lengths[ni].astype(np.float32), valid)
|
| 132 |
+
w = net.neighbour_weights(f, rel, dpref[ni], valid)
|
| 133 |
+
for r, q in enumerate(b):
|
| 134 |
+
mass = {}
|
| 135 |
+
for j in range(K):
|
| 136 |
+
if valid[r, j]:
|
| 137 |
+
row = ecs[NI[q, j]]
|
| 138 |
+
for e in row:
|
| 139 |
+
mass[e] = mass.get(e, 0) + w[r, j] / len(row)
|
| 140 |
+
preds[q] = max(mass, key=mass.get)
|
| 141 |
+
log(f'graph network predicted {len(aligned)} queries')
|
| 142 |
+
|
| 143 |
+
with Path(output_path).open('w') as out:
|
| 144 |
+
for key, p in zip(frame.id, preds):
|
| 145 |
+
out.write(json.dumps({'id': key, 'prediction': [p]}) + '\n')
|
| 146 |
+
log(f'wrote {output_path}')
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
if __name__ == '__main__':
|
| 150 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 151 |
+
parser.add_argument('--input', required=True)
|
| 152 |
+
parser.add_argument('--output', required=True)
|
| 153 |
+
args = parser.parse_args()
|
| 154 |
+
predict_file(args.input, args.output)
|
requirements.txt
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
numpy==2.4.4
|
| 2 |
+
pandas==2.3.3
|
| 3 |
+
pyarrow==23.0.1
|
| 4 |
+
scipy==1.17.1
|
| 5 |
+
pyopal==0.7.3
|
| 6 |
+
scoring-matrices==0.3.5
|
| 7 |
+
archspec==0.2.6
|
task.json
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"title": "Leak CURBER enzyme classification",
|
| 3 |
+
"slug": "leak-curber-enzyme-classification",
|
| 4 |
+
"benchmark_id": "leak-curber-enzyme-classification-20261023",
|
| 5 |
+
"input_columns": [
|
| 6 |
+
"sequence"
|
| 7 |
+
],
|
| 8 |
+
"label_column": "ec_number",
|
| 9 |
+
"output_format": "jsonl",
|
| 10 |
+
"primary_metric": "accuracy_l4",
|
| 11 |
+
"metrics": [
|
| 12 |
+
"accuracy_l4",
|
| 13 |
+
"accuracy_l3",
|
| 14 |
+
"accuracy_l2",
|
| 15 |
+
"accuracy_l1"
|
| 16 |
+
],
|
| 17 |
+
"data_dir": "data/enzyme_tasks/classification",
|
| 18 |
+
"task_id": "enzyme-classification"
|
| 19 |
+
}
|
train.py
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Build the reference database: one row per unique training sequence with its ECs (most frequent first)."""
|
| 2 |
+
import argparse
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
import pandas as pd
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def build(train_path, output_path):
|
| 10 |
+
root = Path(__file__).parent
|
| 11 |
+
vocab = set(json.loads((root / 'ec_vocabulary.json').read_text()))
|
| 12 |
+
frame = pd.read_parquet(train_path)[['sequence', 'ec_number']].dropna()
|
| 13 |
+
bad = sorted(set(frame.ec_number) - vocab)
|
| 14 |
+
if bad:
|
| 15 |
+
raise ValueError(f'{len(bad)} training ECs not in vocabulary, e.g. {bad[:5]}')
|
| 16 |
+
counts = frame.ec_number.value_counts()
|
| 17 |
+
frame = frame.drop_duplicates()
|
| 18 |
+
frame['n'] = frame.ec_number.map(counts)
|
| 19 |
+
frame = frame.sort_values(['sequence', 'n', 'ec_number'], ascending=[True, False, True])
|
| 20 |
+
db = frame.groupby('sequence', sort=False).ec_number.agg(list).reset_index(name='ecs')
|
| 21 |
+
Path(output_path).parent.mkdir(parents=True, exist_ok=True)
|
| 22 |
+
db.to_parquet(output_path, index=False, compression='zstd')
|
| 23 |
+
print(f'Wrote {len(db)} sequences, {frame.ec_number.nunique()} ECs to {output_path}')
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
if __name__ == '__main__':
|
| 27 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 28 |
+
parser.add_argument('--train', required=True)
|
| 29 |
+
parser.add_argument('--output', default=str(Path(__file__).parent / 'weights/db.parquet'))
|
| 30 |
+
args = parser.parse_args()
|
| 31 |
+
build(args.train, args.output)
|