DL_Lab / GUIDE.md

Upload folder using huggingface_hub

d346d4e verified 10 days ago

7.53 kB

# Submit a model to Leak CURBER

Use the [Leak CURBER leaderboard](https://huggingface.co/spaces/starkAhmed43/leak-curber-leaderboard).
Choose a task tab before submitting. You need a free HF account and a public model repository.
You do not need to host a Space or buy a paid plan.

## Choose your task

| Task | Input | Prediction | Ranking metric |
|---|---|---|---|
| `kcat` | SMILES and enzyme sequence | log10 kcat | Spearman |
| `enzyme-classification` | Enzyme sequence | Set of EC numbers | Mean per-input F1 |
| `enzyme-retrieval` | Reaction SMILES | Ranked list of up to 10 ECs | MAP@10 |

The dataset names are `leak-curber-kcat`, `leak-curber-enzyme-classification`, and `leak-curber-enzyme-retrieval`.
Their model starters use the same name with `-starter` added.
All current repositories belong to `starkAhmed43`.

Training inputs and labels are public. Validation inputs are public.
Validation labels and all test data remain private.
Create your own development split from training data to select models and thresholds.

## Download a starter and data

1. Install the HF client.
2. Choose the dataset name for your task.
3. Download the starter and public data.

```bash
pip install huggingface_hub
TASK=leak-curber-enzyme-classification
hf download "starkAhmed43/${TASK}-starter" --local-dir model
hf download "starkAhmed43/${TASK}" train.parquet val_inputs.parquet \
  --repo-type dataset --local-dir data
cd model
pip install -r requirements.txt
python train.py --train ../data/train.parquet
```

For either enzyme task, run:

```bash
python predict.py --input ../data/val_inputs.parquet --output predictions.jsonl
```

For kcat, run:

```bash
python predict.py --input ../data/val_inputs.parquet --output predictions.csv
```

The kcat starter fits a small regression model.
The enzyme starters predict frequent training ECs and ignore the input. Replace these simple baselines with your own models.

## Understand enzyme labels and grouping

An input can have several recorded EC labels. Training data contains one input–EC pair per row.
Gather the labels for each exact input when training a multilabel model.

During evaluation, each unique sequence or reaction counts once.
The evaluator groups identical inputs within the current split and keeps the first existing ID.
It scores predictions against all recorded ECs for that input within that split.
The starter's `inputs.py` uses the same grouping rule.
Do not normalize sequences or canonicalize reactions when preparing evaluation IDs.

To create the unique validation inputs explicitly, run:

```bash
python inputs.py --input ../data/val_inputs.parquet --output unique_validation.parquet
```

`predict.py` also applies this grouping automatically.
Grouping does not move rows between train, validation, and test.
Recorded annotations may omit valid enzyme activities. Scores measure agreement with the recorded annotations.

## Write predictions

For kcat, write CSV with exactly `id,prediction`. Predict log10 kcat, rather than raw kcat.
Return one finite number for every input ID.

For enzyme tasks, write JSON Lines: one JSON object per unique input.
Each object must contain exactly `id` and `prediction`.

```json
{"id":"v_0000001","prediction":["1.1.1.1","1.1.1.2"]}
{"id":"v_0000008","prediction":[]}
```

Use distinct, fully specified EC strings from `ec_vocabulary.json`.
An empty list is allowed and receives zero for that input.
Missing, extra, or duplicate IDs fail evaluation. Unknown ECs and duplicate ECs also fail evaluation.

For classification, predict the applicable EC set. Array order does not matter.
F1 balances precision and recall. We average F1 across unique sequences to determine rank.
The leaderboard also shows mean precision, mean recall, and exact-set accuracy.

For retrieval, put your best EC first. Return at most 10 ECs.
Average precision rewards correct ECs near the top of the list.
For each reaction, its denominator is the smaller of 10 and its number of recorded correct ECs.
MAP@10 averages this score across unique reactions.
The leaderboard also shows MRR@10, Recall@10, and Hit@1.

Higher primary scores rank first. Equal primary scores favor the earlier submission.

## Test your Docker image

The evaluator runs the following command inside your image:

```bash
python /app/predict.py --input INPUT --output OUTPUT
```

Provide `/usr/local/bin/python` and `/app/predict.py`.
Package all weights, dependencies, tokenizers, and other inference files in the image.
Inference has no internet access. Any CPU-compatible model or library is allowed.

Build and test an enzyme starter locally:

```bash
docker build --platform linux/amd64 -t leak-curber-model .
mkdir -p ../output
docker run --rm --platform linux/amd64 --network none --cpus 2 --memory 16g \
  -v "$(realpath ../data):/input:ro" \
  -v "$(realpath ../output):/output" \
  --entrypoint /usr/local/bin/python leak-curber-model /app/predict.py \
  --input /input/val_inputs.parquet --output /output/predictions.jsonl
```

For kcat, change the output filename to `predictions.csv`.
Inference has 10 minutes, 2 vCPU, and 16 GB RAM. Builds have 30 minutes.
Your model repository must fit within 4 GiB. Prediction output must fit within 16 MiB.

## Publish and submit

1. Sign in with `hf auth login`.
2. Create a public HF model repository under your approved account or team owner.
3. Upload your model directory.
4. Open the matching task tab on the leaderboard.
5. Sign in with HF.
6. Enter the model repository, revision, and model name.
7. Select **Submit for validation**.
8. Open **My submissions** to check progress.

```bash
hf repo create YOUR_USERNAME/YOUR_MODEL --repo-type model
hf upload YOUR_USERNAME/YOUR_MODEL . . --repo-type model
```

The organizer freezes the resolved commit and builds it on HF.
Editing your repository later does not change an accepted submission.
Do not upload downloaded datasets or local development outputs into your model repository.

Each team has **2 submissions per day and 20 total per task**.
Team members share quotas. Infrastructure failures are refunded.
The evaluator processes one submission at a time across tasks.

## Nominate a final model

1. Open **My submissions** in the correct task tab.
2. Copy a completed submission ID.
3. Enter that ID under **Nominate**.
4. Select **Nominate** before the deadline.

Each task has its own nomination. Without a nomination, we select your best successful validation submission.
The deadline is **November 6, 2026, at 23:59 IST**.
TAs evaluate the frozen selection on private test data and publish the test leaderboard later.

Best, nominated, and frozen submission archives remain protected.
Other successful archives expire after 7 days. Failed artifacts expire after 2 days.
Scores and audit records remain available.

## Resolve common errors

| Message | Action |
|---|---|
| Account is not approved | Ask a TA to check your HF username and pilot roster. |
| Repository owner is not registered | Submit under an approved account or team owner. |
| Build failed | Build your Dockerfile locally and check its dependencies. |
| Inference failed | Run the offline Docker command and check paths and packaged weights. |
| Invalid predictions | Check IDs, output format, EC vocabulary, and duplicate records. |
| Time limit exceeded | Reduce inference cost and load the model only once. |
| Submission limit reached | Check the remaining quota in the matching task tab. |
| Evaluator unavailable | Refresh shortly; contact a TA if the error persists. |

	# Submit a model to Leak CURBER

	Use the [Leak CURBER leaderboard](https://huggingface.co/spaces/starkAhmed43/leak-curber-leaderboard).
	Choose a task tab before submitting. You need a free HF account and a public model repository.
	You do not need to host a Space or buy a paid plan.

	## Choose your task

	\| Task \| Input \| Prediction \| Ranking metric \|
	\|---\|---\|---\|---\|
	\| `kcat` \| SMILES and enzyme sequence \| log10 kcat \| Spearman \|
	\| `enzyme-classification` \| Enzyme sequence \| Set of EC numbers \| Mean per-input F1 \|
	\| `enzyme-retrieval` \| Reaction SMILES \| Ranked list of up to 10 ECs \| MAP@10 \|

	The dataset names are `leak-curber-kcat`, `leak-curber-enzyme-classification`, and `leak-curber-enzyme-retrieval`.
	Their model starters use the same name with `-starter` added.
	All current repositories belong to `starkAhmed43`.

	Training inputs and labels are public. Validation inputs are public.
	Validation labels and all test data remain private.
	Create your own development split from training data to select models and thresholds.

	## Download a starter and data

	1. Install the HF client.
	2. Choose the dataset name for your task.
	3. Download the starter and public data.

	```bash
	pip install huggingface_hub
	TASK=leak-curber-enzyme-classification
	hf download "starkAhmed43/${TASK}-starter" --local-dir model
	hf download "starkAhmed43/${TASK}" train.parquet val_inputs.parquet \
	--repo-type dataset --local-dir data
	cd model
	pip install -r requirements.txt
	python train.py --train ../data/train.parquet
	```

	For either enzyme task, run:

	```bash
	python predict.py --input ../data/val_inputs.parquet --output predictions.jsonl
	```

	For kcat, run:

	```bash
	python predict.py --input ../data/val_inputs.parquet --output predictions.csv
	```

	The kcat starter fits a small regression model.
	The enzyme starters predict frequent training ECs and ignore the input. Replace these simple baselines with your own models.

	## Understand enzyme labels and grouping

	An input can have several recorded EC labels. Training data contains one input–EC pair per row.
	Gather the labels for each exact input when training a multilabel model.

	During evaluation, each unique sequence or reaction counts once.
	The evaluator groups identical inputs within the current split and keeps the first existing ID.
	It scores predictions against all recorded ECs for that input within that split.
	The starter's `inputs.py` uses the same grouping rule.
	Do not normalize sequences or canonicalize reactions when preparing evaluation IDs.

	To create the unique validation inputs explicitly, run:

	```bash
	python inputs.py --input ../data/val_inputs.parquet --output unique_validation.parquet
	```

	`predict.py` also applies this grouping automatically.
	Grouping does not move rows between train, validation, and test.
	Recorded annotations may omit valid enzyme activities. Scores measure agreement with the recorded annotations.

	## Write predictions

	For kcat, write CSV with exactly `id,prediction`. Predict log10 kcat, rather than raw kcat.
	Return one finite number for every input ID.

	For enzyme tasks, write JSON Lines: one JSON object per unique input.
	Each object must contain exactly `id` and `prediction`.

	```json
	{"id":"v_0000001","prediction":["1.1.1.1","1.1.1.2"]}
	{"id":"v_0000008","prediction":[]}
	```

	Use distinct, fully specified EC strings from `ec_vocabulary.json`.
	An empty list is allowed and receives zero for that input.
	Missing, extra, or duplicate IDs fail evaluation. Unknown ECs and duplicate ECs also fail evaluation.

	For classification, predict the applicable EC set. Array order does not matter.
	F1 balances precision and recall. We average F1 across unique sequences to determine rank.
	The leaderboard also shows mean precision, mean recall, and exact-set accuracy.

	For retrieval, put your best EC first. Return at most 10 ECs.
	Average precision rewards correct ECs near the top of the list.
	For each reaction, its denominator is the smaller of 10 and its number of recorded correct ECs.
	MAP@10 averages this score across unique reactions.
	The leaderboard also shows MRR@10, Recall@10, and Hit@1.

	Higher primary scores rank first. Equal primary scores favor the earlier submission.

	## Test your Docker image

	The evaluator runs the following command inside your image:

	```bash
	python /app/predict.py --input INPUT --output OUTPUT
	```

	Provide `/usr/local/bin/python` and `/app/predict.py`.
	Package all weights, dependencies, tokenizers, and other inference files in the image.
	Inference has no internet access. Any CPU-compatible model or library is allowed.

	Build and test an enzyme starter locally:

	```bash
	docker build --platform linux/amd64 -t leak-curber-model .
	mkdir -p ../output
	docker run --rm --platform linux/amd64 --network none --cpus 2 --memory 16g \
	-v "$(realpath ../data):/input:ro" \
	-v "$(realpath ../output):/output" \
	--entrypoint /usr/local/bin/python leak-curber-model /app/predict.py \
	--input /input/val_inputs.parquet --output /output/predictions.jsonl
	```

	For kcat, change the output filename to `predictions.csv`.
	Inference has 10 minutes, 2 vCPU, and 16 GB RAM. Builds have 30 minutes.
	Your model repository must fit within 4 GiB. Prediction output must fit within 16 MiB.

	## Publish and submit

	1. Sign in with `hf auth login`.
	2. Create a public HF model repository under your approved account or team owner.
	3. Upload your model directory.
	4. Open the matching task tab on the leaderboard.
	5. Sign in with HF.
	6. Enter the model repository, revision, and model name.
	7. Select Submit for validation.
	8. Open My submissions to check progress.

	```bash
	hf repo create YOUR_USERNAME/YOUR_MODEL --repo-type model
	hf upload YOUR_USERNAME/YOUR_MODEL . . --repo-type model
	```

	The organizer freezes the resolved commit and builds it on HF.
	Editing your repository later does not change an accepted submission.
	Do not upload downloaded datasets or local development outputs into your model repository.

	Each team has 2 submissions per day and 20 total per task.
	Team members share quotas. Infrastructure failures are refunded.
	The evaluator processes one submission at a time across tasks.

	## Nominate a final model

	1. Open My submissions in the correct task tab.
	2. Copy a completed submission ID.
	3. Enter that ID under Nominate.
	4. Select Nominate before the deadline.

	Each task has its own nomination. Without a nomination, we select your best successful validation submission.
	The deadline is November 6, 2026, at 23:59 IST.
	TAs evaluate the frozen selection on private test data and publish the test leaderboard later.

	Best, nominated, and frozen submission archives remain protected.
	Other successful archives expire after 7 days. Failed artifacts expire after 2 days.
	Scores and audit records remain available.

	## Resolve common errors

	\| Message \| Action \|
	\|---\|---\|
	\| Account is not approved \| Ask a TA to check your HF username and pilot roster. \|
	\| Repository owner is not registered \| Submit under an approved account or team owner. \|
	\| Build failed \| Build your Dockerfile locally and check its dependencies. \|
	\| Inference failed \| Run the offline Docker command and check paths and packaged weights. \|
	\| Invalid predictions \| Check IDs, output format, EC vocabulary, and duplicate records. \|
	\| Time limit exceeded \| Reduce inference cost and load the model only once. \|
	\| Submission limit reached \| Check the remaining quota in the matching task tab. \|
	\| Evaluator unavailable \| Refresh shortly; contact a TA if the error persists. \|