| --- |
| tags: |
| - leak-curber |
| - enzyme |
| - leaderboard |
| --- |
| # Leak CURBER enzyme classification model starter |
|
|
| This is a starter model for the enzyme classification task. You train a model on your computer and upload it to a public Hugging Face (HF) model repository. Then you submit that repository to the Leak CURBER leaderboard. A free HF account is enough. |
|
|
| ## Task |
|
|
| The input is a Parquet file with the columns `id` and `sequence` (the enzyme amino acid sequence). Predict one EC number for each unique input. The output is JSON Lines with the keys `id` and `prediction`. The prediction is a list of distinct ECs, best first: |
|
|
| ```json |
| {"id":"v_0000001","prediction":["1.1.1.1"]} |
| ``` |
|
|
| The public data are `train.parquet` (with the `ec_number` label) and `val_inputs.parquet` (inputs only). Validation labels and test data are private. `ec_vocabulary.json` lists the permitted ECs. |
|
|
| An input can have several recorded ECs. `train.parquet` has one input–EC pair per row. We group identical inputs within each split, keep the first ID, and score each unique input once. Predict only the first ID of each unique input; do not predict repeat IDs. `inputs.py` applies the same grouping. Do not normalize sequences. |
|
|
| Only the first predicted EC counts. You can list up to 10, but the others do not change the score. We report top-1 accuracy at EC levels L4, L3, L2, and L1, and rank by L4. Equal scores favor the earlier submission. |
|
|
| If an input has n recorded ECs, the first prediction earns 1/n for each recorded EC whose first L components match it. We average this over unique inputs. With recorded ECs 1.1.1.1 and 1.1.1.2, the prediction 1.1.1.1 scores 0.5 at L4 and 1.0 at L3, L2, and L1. An empty prediction scores zero. |
|
|
| `ec_vocabulary.json` comes from ExPASy ENZYME, release 02-Sep-2026. SIB provides it under CC BY 4.0: https://enzyme.expasy.org/ |
|
|
| ## Baseline |
|
|
| The baseline ignores the input. It predicts the most frequent training EC for every input. It uses only the training split. |
|
|
| ## Run the baseline |
|
|
| ```bash |
| python -m pip install huggingface-hub==2.2.0 |
| hf download starkAhmed43/leak-curber-enzyme-classification-starter --local-dir model |
| hf download starkAhmed43/leak-curber-enzyme-classification train.parquet val_inputs.parquet --repo-type dataset --local-dir data |
| cd model |
| python -m pip install -r requirements.txt |
| python predict.py --input ../data/val_inputs.parquet --output predictions.jsonl |
| ``` |
|
|
| To retrain the baseline: |
|
|
| ```bash |
| python train.py --train ../data/train.parquet --output weights/baseline.json |
| ``` |
|
|
| To write the unique validation inputs to a file: |
|
|
| ```bash |
| python inputs.py --input ../data/val_inputs.parquet --output unique_validation.parquet |
| ``` |
|
|
| ## Check with Docker |
|
|
| Run these commands in the `model` directory: |
|
|
| ```bash |
| mkdir -p output && chmod 777 output |
| docker build --platform linux/amd64 -t my-enzyme-model . |
| docker run --rm --platform linux/amd64 --user 1000:1000 --network none --cpus 2 --memory 16g \ |
| -v "$PWD/../data:/input:ro" -v "$PWD/output:/output" \ |
| my-enzyme-model python /app/predict.py \ |
| --input /input/val_inputs.parquet --output /output/predictions.jsonl |
| ``` |
|
|
| `output/predictions.jsonl` must contain every unique input ID exactly once. |
|
|
| ## Replace the baseline |
|
|
| Use any model or library. Train wherever you like, but the model runs on CPU only. The evaluator gives it 2 vCPUs, 16 GB RAM, and 15 minutes for all inputs. This is the maximum and will not be raised. It covers image pull, model load and inference. It has no GPU and no network. Change `Dockerfile`, `requirements.txt`, and `predict.py`. Put the weights in your repository, with all other files that inference needs. Do not add the downloaded data or the `output` directory. |
|
|
| We build the image from your `Dockerfile`. The image must meet these limits: |
|
|
| - Linux x86-64, with Python at `/usr/local/bin/python` and working directory `/app`. |
| - The command is `python /app/predict.py --input INPUT --output OUTPUT`. |
| - The build must finish in 30 minutes. The repository must be 4 GiB or less. |
| - No file that `predict.py` writes may exceed 16 MiB. |
|
|
| We start Python directly. Docker entrypoints and shell startup hooks do not run. Use `/tmp` for temporary files. |
|
|
| An evaluation fails in these cases: |
|
|
| - A line is blank, or is not a JSON object with exactly the keys `id` and `prediction`. |
| - A unique input ID is missing, duplicated, or added. |
| - A prediction has a duplicate EC, more than 10 ECs, or an EC that is not in `ec_vocabulary.json`. |
| - Inference needs more than 15 minutes, or needs the network. |
|
|
| ## Submit |
|
|
| Choose the task assigned to you in [this sheet](https://docs.google.com/spreadsheets/d/12c2PaYcNDpa6bA0JPmKi6tHsdV-EdFe2IYQ70RqpGRI/edit?usp=sharing). |
|
|
| 1. Sign in: `hf auth login`. |
| 2. Upload the `model` directory to a public model repository: |
|
|
| ```bash |
| hf upload YOUR_USERNAME/YOUR_MODEL . --repo-type model --no-private --exclude ".cache/**" "output/**" "predictions.jsonl" |
| ``` |
|
|
| For an existing repository, make sure it is public. The `--no-private` flag does not change an existing repository. |
| The repository must contain `Dockerfile`, `predict.py`, and `README.md`. |
| 3. Open the [leaderboard](https://huggingface.co/spaces/starkAhmed43/leak-curber-leaderboard), select the Enzyme classification tab, and sign in with your registered account. |
| 4. Under Submit model, enter the repository ID, the revision (default `main`), and a model name. Select Submit for validation. |
| 5. Watch the progress under My submissions. |
|
|
| A submission records a fixed commit. Later changes to the repository do not change it. The validation score appears when the evaluation ends. |
|
|
| ## Limits and deadline |
|
|
| - Each team has 2 submissions per day and 20 in total for each task. |
| - Failed model runs count against these limits. |
| - Only one of your evaluations can be active at a time. |
| - Under My submissions, nominate one completed submission for the test evaluation. Each task has its own nomination. If you do not nominate, we use your best validation score. |
| - Your best and your nominated submissions stay protected. Other successful archives expire after 7 days. |
| - Submit and nominate before November 7, 2026, 23:59 IST. We will publish test scores after that. |
|
|
| ## Help |
|
|
| - If the leaderboard says your account is not approved, post your HF username in the Moodle course discussion forum. |
| - If the leaderboard says the evaluator is unavailable, refresh shortly. Post in the Moodle course discussion forum if the error persists. |
| - Post any other question in the Moodle course discussion forum. |
|
|
