JoyXiangLab/rnaseek-full

RNASeek: models, benchmarks, generation, and figures

This repository contains RNASeek checkpoints and fixed datasets for ribozyme catalytic-efficiency prediction, 3′ UTR-mediated RNA stability prediction, supervised sequence generation, and reinforcement-learning fine-tuning. It also includes encoder/GEMORNA benchmark artifacts and the underlying analysis data.

Installation

The complete archive is approximately 475 GB, plus space for environments and download caches. Use a filesystem with sufficient free space and quota. In a separate downloader environment with the Hugging Face CLI installed (python -m pip install huggingface_hub==0.34.4):

hf auth login
hf download JoyXiangLab/rnaseek-full --local-dir rnaseek
cd rnaseek

Authentication is recommended for this large public archive: anonymous requests can be rate-limited. Enter credentials interactively, not in scripts. If a transfer is interrupted, rerun the same download command into the same directory to reuse completed files. For an immutable snapshot, add --revision COMMIT_SHA. Downloading published caches is expected; regenerating their scientific contents is not.

Tested platform: Linux x86-64, Python 3.11, NVIDIA CUDA GPUs. Windows users should use WSL2 with NVIDIA GPU support. Native Windows/macOS GPU training has not been validated. CPU figure rendering and regression inference are supported, but full-model training is intended for a suitable NVIDIA GPU. GPU smoke tests used approximately 48-GB devices; full training memory requirements depend on checkpoint, sequence length, and batch size.

Install Miniforge or Miniconda first, then run from the downloaded repository:

bash setup_env.sh rnaseek

The script creates three new environments, installs pinned project dependencies, checks package consistency, and tests key imports:

Environment	Use
rnaseek	RNASeek regression, supervised generation, GRPO, reward APIs, figures
rnaseek-benchmark	Encoder benchmarks and GEMORNA benchmark training
rnaseek-evo2	NVIDIA BF16-adapted Evo 2 1B regression benchmarks

The separation is intentional: the generation stack uses Transformers 4.57.6 / TRL 0.23.1; the encoder benchmark stack uses Transformers 5.5.4 / MultiMolecule 0.2.1; Evo 2 uses Torch 2.7.1 / Vortex 1.1.0 with a matching prebuilt FlashAttention wheel. Do not combine the requirements files into one environment. The Evo 2 backend requires Linux x86-64 and an Ampere-or-newer NVIDIA GPU; it is not a CPU inference backend. The script installs its matching binary wheel automatically, without requiring a local CUDA compiler.

An existing environment is never modified by this script. Choose another prefix to make a separate installation. To install only one profile:

bash setup_env.sh another-name core
bash setup_env.sh another-name benchmark
bash setup_env.sh another-name evo2

Use bash setup_env.sh another-name both for just the core and encoder environments. The default all profile includes Evo 2. All checkpoint inference uses local files; installation itself requires internet access.

For RNASeek regression and both GRPO tasks, only the core profile is needed. The installer disables Python user-site packages during installation and saves PYTHONNOUSERSITE=1 in each new Conda environment; activate the environment before use so unrelated packages cannot shadow its pinned dependencies. It also clears inherited PYTHONPATH during installation. If your shell or notebook sets a custom PYTHONPATH, unset it before running this project.

On a cluster, run GPU checks inside a scheduler allocation. To keep environments and package caches on job-local storage, set CONDA_ENVS_PATH, CONDA_PKGS_DIRS, PIP_CACHE_DIR, and HF_HOME to separate directories beneath the scheduler-provided scratch directory before running the installer. Scratch may be deleted when the job ends: copy validation reports to persistent storage first. Do not replace existing result directories or regenerate shipped caches.

Internet access is required for installation. Published model inference and cache-only figures do not require model downloads. A compatible NVIDIA driver must already be installed; the Python packages supply their CUDA runtime dependencies, not the system driver.

Repository map

Location	Contents
RNASeek-regression-pretrained/	Pretrained initialization for both regression tasks
RNASeek-lm-pretrained/	Pretrained causal-language-model initializer and custom tokenizer for generation
efficiency_figure2/	Efficiency datasets, regression checkpoint, benchmark results, figure inputs
regression_stability_functionalviral/	Stability data, regression/reward checkpoints, benchmark and design-space results
ribozymegen-figure7/	Ribozyme supervised-generation data and selected checkpoints
utrgen/	UTR supervised-generation data and selected checkpoints
ribozymegen-rl-figure7/	Ribozyme GRPO, fixed prompts, efficiency and directive reward services
utrgen-rl/	UTR GRPO, fixed prompt splits, stability reward service, immutable generation/analysis caches
validation/	Measured release evaluation and smoke-test outputs
checkpoint_manifest.json	Retained checkpoint locations and selection reasons
pretraining/	Preserved pretraining subproject; follow its own documentation

Raw sequencing inputs, processed alignments, indexes, and fixed generation/analysis caches are retained. This is a large research archive: downloading only the subprojects needed for a particular experiment is usually preferable to cloning every artifact.

Pretraining

RNASeek uses a Qwen-derived, Qwen2 causal-language-model architecture and a custom RNASeek tokenizer. Its language-model objective is next-token prediction. Pretraining is performed separately from the downstream regression, supervised generation, and RLHF workflows described below, using the LLaMA Factory graphical interface. Refer to the pretraining README for a short GUI guide and links to the official instructions. The READMEs inside the bundled LLaMA Factory and ExLlamaV2 directories document those third-party tools, not RNASeek's experimental methods.

The pretrained generation checkpoint supplied for downstream generation is RNASeek-lm-pretrained/, with its matching custom tokenizer. It predicts tokens rather than scalar regression scores. This is the project's designated generation initializer; its retained history also includes ribozyme RLHF, as explained below. It should not be interpreted as an untouched, pretraining-only export or as evidence of a particular zero-shot benchmark.

The separate RNASeek-regression-pretrained/ checkpoint is the pretrained initialization designated for regression only. Regression uses a regression-adapted RNASeek language-model backbone, with the LM head replaced by a scalar regression head. RNASeek-lm-pretrained is the generation-side model.

Which workflow and checkpoint do I need?

For downstream generation, the pretrained RNA language-model initializer is RNASeek-lm-pretrained, used with its own tokenizer. For regression, use the separate RNASeek-regression-pretrained initialization. The regression workflow replaces the next-token output with a scalar regression head; task-specific fine-tuning produces the predictors used for the reported regression results. A classification head is theoretically possible, but classification adaptation has not been tested in this release.

Regression predicts a number; generation produces a sequence. They use different output heads and different trained checkpoints. SFT means supervised fine-tuning on directive/sequence examples. RLHF here is GRPO training of a generative policy using reward services.

Why call SFT “sequence generation” if it is not reinforcement learning? Generation describes what the model does: it produces an RNA sequence in response to a directive. SFT and RLHF describe how that model is trained. SFT learns to imitate the supplied directive/sequence examples, teaching sequence patterns and instruction following; an SFT policy can already generate sequences without RLHF. RLHF then updates that policy to favor outputs scored more highly by the reward functions, including the separate regression predictor. In this project's SFT-to-RLHF workflow, both stages contribute to the final policy, but they are distinct training stages. SFT does not itself learn a scalar quality score, and a higher predicted reward does not guarantee better experimental performance.

Regression training -> trained regression checkpoint -> reward service ----+
                                                                         |
Generation initializer -> SFT -> SFT policy checkpoint -> GRPO/RLHF <------+
                                                           |
                                                           v
                                                  updated generation policy

The reward service scores generated sequences; it does not generate them and its regression weights are not updated by GRPO. The policy generates sequences and is the model updated by GRPO. A normal training run can save multiple intermediate checkpoints; the paths below identify its usable output artifact, not a promise of exactly one checkpoint file.

Stage	Entry point	Input model	Output
Efficiency or stability regression	train_regression.py efficiency / stability	Defaults to RNASeek-regression-pretrained; a fresh regression head is added	<output>/model/: backbone plus trained scalar regression head
Supervised sequence generation	train_sft.py PRESET	Explicit --checkpoint: a compatible causal-language model or existing generative policy	<output>/best_model/: SFT generation policy and tokenizer
UTR RLHF	utrgen-rl/train_grpo.py	SFT policy via --policy-ckpt; separate stability predictor loaded by the reward service	<output-dir>/final/: updated generation policy
Ribozyme RLHF	ribozymegen-rl-figure7/train_grpo.py	SFT policy via --checkpoint; separate efficiency predictor and directive scorer	Trainer-selected checkpoint-* directories beneath --output: updated generation policies

Use the regression output for scalar inference or as a fixed reward model. Use the SFT output as the policy initializer for RLHF. Use an RLHF output for sequence generation or a separate continuation experiment. These outputs are not interchangeable.

These are the engineering roles of the stages, not a claim that every historical run followed one linear training sequence. Some historical runs reused a policy already trained with RLHF as the initializer for another SFT run; the RNASeek-lm-pretrained case is explained below. The utrgen/ and ribozymegen-figure7/ folders contain generation workflows, not regression training.

Path conventions and checkpoint formats

The root entry points (validate_release.py, train_regression.py, train_sft.py, and benchmark_checkpoint.py) resolve relative paths against the repository root, independent of your shell's working directory. GRPO scripts and reward services resolve relative checkpoint/data/output paths against their own subproject directory. Absolute paths may also be supplied for your own external checkpoint.

Use a checkpoint together with its matching tokenizer and model configuration. RNASeek regression checkpoints include both the backbone and regression head; a backbone alone is not a complete predictor. The loader accepts the shipped separate regression_head.pt layout and the combined backbone.* / regression_head.* tensor layout, using strict state-dictionary loading.

RNASeek-regression-pretrained/ is the pretrained regression initialization, shared by efficiency and stability regression. New regression training defaults to this root-level model and initializes a fresh regression head. The serialized initialization retains the next-token model layout; the regression loader performs the conversion to a scalar predictor. It is not a trained task predictor: evaluation and RLHF reward services still default to their selected, task-specific regression checkpoints. Its supported role in this guide is regression only.

Generation checkpoints use the causal-language-model layout. train_sft.py defaults to the tokenizer alongside the supplied checkpoint. The ribozyme GRPO script instead defaults to its explicitly supplied, archived no-EOS tokenizer; use --tokenizer when training a different compatible policy.

Regression evaluation

conda activate rnaseek
python validate_release.py efficiency --output my_runs/efficiency-evaluation
python validate_release.py stability --precision autocast --output my_runs/stability-evaluation

Default checkpoints are the publication efficiency checkpoint and the wave-29 stability model. To evaluate a compatible checkpoint of your own:

python validate_release.py efficiency \
  --checkpoint my_checkpoints/efficiency \
  --tokenizer my_checkpoints/efficiency \
  --output my_runs/custom-efficiency

Use --device cpu for CPU inference, or --device cuda:1 to select a visible GPU. --limit 8 is a smoke test, not a full validation score. Each output directory must be new.

Full release evaluations measured:

Task	Records	Metric	Reference	Measured
Efficiency	5,756	R²	0.389	0.380351
Stability	2,935	all-validation Spearman R	0.342	0.340927

A fresh Conda core installation was also tested on a Slurm-managed A100 80GB. Full offline inference reproduced efficiency R² 0.380351 and stability Spearman 0.340374. Both GRPO workflows completed two optimizer updates with their live reward services sharing that single GPU. The eight fixed prompt files were unchanged. These are trainability smoke tests, not full-training convergence results. See cluster validation evidence for exact scores, versions, and input hashes.

The evaluation script reports actual predictions and all three metrics (R², Pearson, Spearman). Its default warning threshold is an absolute decrease of more than 0.01 from the reference metric, configurable with --allowed-decrease. Stability scores reported here use all validation records.

Actual regression training

For a new MSE fine-tuning run from the shared pretrained base, supply fixed data (--checkpoint defaults to RNASeek-regression-pretrained):

python train_regression.py efficiency \
  --train-data efficiency_figure2/evenBetterDataFolded-tr.json \
  --valid-data efficiency_figure2/evenBetterDataFolded-vl.json \
  --output my_runs/efficiency-finetune --max-steps 1000

The same pretrained default is used for stability:

python train_regression.py stability \
  --train-data regression_stability_functionalviral/training_seq_score_consistent_from_alignment_zscore_split.tsv \
  --valid-data regression_stability_functionalviral/training_seq_score_consistent_from_alignment_zscore_split.tsv \
  --output my_runs/stability-finetune --max-steps 1000

These are actual training commands: they use the full supplied training split, run 1,000 optimizer updates, evaluate the validation split, and save the trained predictor to <output>/model/. Set --max-steps to your intended training budget; this entry point is step-based, not an automatic train-to-convergence procedure. Do not add --limit or --skip-model-save for an ordinary run. For a two-update smoke test only, use a different output directory and add --max-steps 2 --limit 4 --skip-model-save.

Pass --checkpoint my_checkpoints/efficiency (or your stability checkpoint) to fine-tune an existing predictor instead; its trained head is retained. Efficiency data are JSON objects mapping complete prompt strings to numeric labels. Stability data are TSV tables with text and label, or seq and score; if a split column is present, training and validation records are selected separately. This portable MSE entry point starts a new optimizer run. Its default settings do not reproduce the full historical model-selection procedure or guarantee the published scores from a new training run. Historical stability weighting/ranking experiment configurations and their dedicated research trainer are retained under rnaseek_abs20_tuning/; that trainer also defaults to RNASeek-regression-pretrained.

Supervised sequence generation

train_sft.py trains a model to generate a sequence from a directive. It does not train a scalar regression head and does not perform RLHF. It replaces the separate, largely shared training implementations previously maintained in the four directories below.

The first argument selects a data/training preset, not a checkpoint:

First argument	What it actually trains on	Default training file	Default validation file
ribozyme	Ribozyme directive-to-sequence generation	ribozymegen-figure7/train.txt	ribozymegen-figure7/valid.txt
utr	UTR directive-to-sequence generation	utrgen/train.txt	utrgen/valid.txt
ribozyme-wet	Ribozyme generation using the preserved filteredwet split	ribozymegen-figure7/filteredwet/train.txt	ribozymegen-figure7/filteredwet/valid.txt
utr-wet	UTR generation using the prepared directive/sequence split, with the small-batch, one-epoch preset	utrgen/train.txt	utrgen/valid.txt

UTR data mapping: both utr and utr-wet read the same existing, prepared generation files: utrgen/train.txt and utrgen/valid.txt. They contain UTR directives and target sequences, not scalar regression labels. Neither preset converts regression scores into SFT examples or regenerates a split. They differ in training defaults: utr uses batch size 16 and 50,000 steps; utr-wet uses batch size 2 and one epoch. Both use a 512-token maximum length. These are four CLI presets, not four independent biological datasets; the legacy -wet preset name does not select the labeled regression table.

The old utrgen/filteredwet/train.txt and valid.txt contain ribozyme examples and are retained only as historical artifacts; utr-wet no longer selects them. Likewise, checkpoints under that legacy directory are not evidence of training on the corrected UTR split. Scalar stability regression remains a separate workflow using the labeled training_seq_score_consistent_from_alignment_zscore_split.tsv table, not these directive/sequence text files.

Actual SFT training: select the starting checkpoint explicitly

--checkpoint is the model before this SFT run; --output is a new directory for this run. The preset does not choose the model for you. These four examples use shipped generative initializers: the ribozyme example uses RNASeek-lm-pretrained, while both UTR examples use the adapted UTR policy rather than the legacy utrgen/filteredwet model. They are not reconstructions of every historical run's original initializer:

conda activate rnaseek

python train_sft.py ribozyme \
  --checkpoint RNASeek-lm-pretrained \
  --output my_runs/ribozyme-sft

python train_sft.py utr \
  --checkpoint utrgen/runs/lm_finetune_adamw8bit/checkpoint-4956 \
  --output my_runs/utr-sft

python train_sft.py ribozyme-wet \
  --checkpoint ribozymegen-figure7/filteredwet/runs/lm_finetune/checkpoint-7098 \
  --output my_runs/ribozyme-wet-sft

python train_sft.py utr-wet \
  --checkpoint utrgen/runs/lm_finetune_adamw8bit/checkpoint-4956 \
  --output my_runs/utr-wet-sft

Run only the command for the preset you need. Each preset selects the listed files and its recorded sequence-length, batch-size, schedule, and other training defaults. All paths in these train_sft.py commands are repository-root-relative, even if your shell is elsewhere. Replace the value after --checkpoint to change the initializer; replace --output to name your new run. To use your own data or tokenizer, add --train-data, --valid-data, and --tokenizer; otherwise the tokenizer comes from the supplied checkpoint.

To use the designated pretrained generation initializer for a new UTR SFT run, supply --checkpoint RNASeek-lm-pretrained, as shown below. Keep RNASeek-regression-pretrained for the regression workflow. Do not substitute a task-fine-tuned scalar regression checkpoint into train_sft.py or into the RLHF policy argument.

What is RNASeek-lm-pretrained?

This is the designated pretrained generation checkpoint for downstream use. Here, “pretrained” identifies its role as an existing trained initializer; it does not mean the file contains only the initial next-token-pretraining stage. Its model weights are byte-identical (SHA-256 9c7e359ff2ffab1b9718b3318fd1fed5df7018de900a026063866eaa89ad40ee) to ribozymegen-rl-figure7/runs/5e-06/tb-grpo-ribozyme-rmXdirective-zscore-add/checkpoint-55000. Its saved trainer state also records that GRPO run and step 55,000.

It is therefore an already ribozyme-RLHF-trained generation policy, provided at the repository root as an initializer for subsequent SFT. It is not the shared pretrained regression base, a trained regression predictor, or merely an SFT-only checkpoint waiting for its first RLHF stage. The historical transfer is:

ribozyme GRPO policy at step 55,000 -> UTR SFT -> UTR generation policy -> UTR GRPO

To explicitly choose that transferred initializer for a new UTR SFT run:

python train_sft.py utr \
  --checkpoint RNASeek-lm-pretrained \
  --output my_runs/utr-sft-from-transferred-policy

The published UTR RLHF command below instead starts from the post-SFT policy utrgen/runs/lm_finetune_adamw8bit/checkpoint-4956. A checkpoint's directory location or numeric step alone does not tell you its training stage.

The text format is one record per line, with literal \n separating the directive and target sequence. The trainer converts that separator to a newline, ensures the archived </s> terminator, and masks prompt tokens using the original trainers' convention. It rejects a tokenizer/checkpoint vocabulary mismatch rather than silently resizing the model.

Smoke test only: short training/evaluation check

The following command checks execution only; it does not produce a trained policy for downstream RLHF:

CUDA_VISIBLE_DEVICES=0 python train_sft.py ribozyme-wet \
  --checkpoint ribozymegen-figure7/filteredwet/runs/lm_finetune/checkpoint-7098 \
  --output my_runs/ribozyme-wet-smoke \
  --max-steps 2 --limit 4 --batch-size 1 --eval-steps 2 \
  --gradient-checkpointing --optimizer adamw_bnb_8bit --skip-model-save

--skip-model-save is for smoke tests. Normal runs save model weights, tokenizer, evaluation metrics, and TensorBoard logs beneath the new output directory. Smaller physical batches or gradient checkpointing can reduce memory requirements. Short smoke tests do not establish training convergence.

To run actual training, use one of the four SFT commands above. For example, the corresponding ribozyme-wet training command is:

conda activate rnaseek
CUDA_VISIBLE_DEVICES=0 python train_sft.py ribozyme-wet \
  --checkpoint ribozymegen-figure7/filteredwet/runs/lm_finetune/checkpoint-7098 \
  --output my_runs/ribozyme-wet-training \
  --gradient-checkpointing --optimizer adamw_bnb_8bit

This uses the full fixed train/validation splits, the preset batch size of 2, and one training epoch, and saves the policy/tokenizer to my_runs/ribozyme-wet-training/best_model/. Unlike the smoke command, it omits --max-steps 2, --limit 4, --batch-size 1, --eval-steps 2, and --skip-model-save. The normal schedules are 1,295,200 updates for ribozyme, 50,000 updates for utr, and one epoch for either -wet preset. Supply --max-steps N to choose a different update budget. Always use a new output directory; existing results are protected.

Pass your own trained models into RLHF

If you run the SFT examples above normally, their policy output is my_runs/<run-name>/best_model/. The regression trainer's output is my_runs/<run-name>/model/. Supply those different artifacts to the policy trainer and reward service respectively. The RLHF scripts/services resolve paths relative to their own subproject, so a repository-root my_runs/... path becomes ../my_runs/... in their arguments:

Workflow component	Replace this argument with your output
UTR RLHF policy	--policy-ckpt ../my_runs/utr-sft/best_model
UTR stability reward service	--rm-ckpt ../my_runs/stability-finetune/model
Ribozyme RLHF policy	--checkpoint ../my_runs/ribozyme-wet-sft/best_model --tokenizer ../my_runs/ribozyme-wet-sft/best_model
Ribozyme efficiency reward service	RM_MODEL_PATH=../my_runs/efficiency-finetune/model RM_TOKENIZER_PATH=../my_runs/efficiency-finetune/model before its Python command

Start reward services separately, as shown below. They remain fixed while RLHF updates the policy. The ribozyme workflow also needs its CPU directive scorer. To reproduce archived reward calculations, keep the published default reward checkpoints rather than substituting a newly trained predictor. In particular, UTR RLHF uses checkpoint-44040_best, not the wave-29 stability evaluation checkpoint.

UTR reinforcement learning

Start the stability reward service in one terminal:

conda activate rnaseek
CUDA_VISIBLE_DEVICES=1 python utrgen-rl/rm_evaluator_api.py --no-print-rm-seen

Then run actual policy training in another terminal (100,000 updates by default; change with --max-steps N):

conda activate rnaseek
CUDA_VISIBLE_DEVICES=0 python utrgen-rl/train_grpo.py \
  --policy-ckpt ../utrgen/runs/lm_finetune_adamw8bit/checkpoint-4956 \
  --output-dir ../my_runs/utr-grpo

The service defaults to port 8008 and the archived checkpoint-44040_best reward model, which is retained because it was used to score the fixed generation analyses. It is not the wave-29 prediction benchmark checkpoint. Override service paths with --rm-ckpt / --rm-tokenizer; a custom tokenizer defaults to the selected checkpoint when omitted. The policy accepts --policy-tokenizer and --rm-api-url.

The examples place the reward model and policy on separate visible GPUs. With one sufficiently large GPU, use CUDA_VISIBLE_DEVICES=0 for both processes instead (both models must fit together). Within a scheduler allocation, these indices refer only to the GPUs made visible to that job. The same rule applies to the ribozyme workflow below.

The fixed 47,500/2,500 prompt splits are read from utrgen-rl/runs/grpo_cpu_random170_iupac_50k_splits/. Missing splits produce an error; training never regenerates them. Add --smoke-test --skip-preview for two short training updates with two completions per group and the normal completion-length limit.

Ribozyme reinforcement learning

This workflow uses an efficiency model service and a CPU directive scorer. Start each in its own terminal:

conda activate rnaseek
CUDA_VISIBLE_DEVICES=1 python ribozymegen-rl-figure7/ribozyme_efficiency_api.py

conda activate rnaseek
python -m uvicorn ribozyme_directive_score_api:app \
  --app-dir ribozymegen-rl-figure7 --host 127.0.0.1 --port 8004

Run actual policy training with a new output directory (100,000 updates by default; change with --max-steps N):

conda activate rnaseek
CUDA_VISIBLE_DEVICES=0 python ribozymegen-rl-figure7/train_grpo.py \
  --checkpoint ../ribozymegen-figure7/filteredwet/runs/lm_finetune/checkpoint-7098 \
  --output ../my_runs/ribozyme-grpo

Add --smoke-test for two updates. The efficiency service listens on port 8003; override its paths with RM_MODEL_PATH / RM_TOKENIZER_PATH and its port with RM_API_PORT. These paths are relative to ribozymegen-rl-figure7/. RIBO_RM_URL and RIBO_RULE_URL select alternative services. GPU selection is controlled by your CUDA_VISIBLE_DEVICES, not overwritten by the script. Services bind to localhost by default.

Both GRPO workflows keep the original reward calculations and fixed prompt data. New rollouts produced during training belong to the new run; they do not replace any published analysis cache. Existing output directories are rejected.

Encoder benchmarks

All 66 retained combined encoder-checkpoint exports passed strict local loading, prediction, and a finite-gradient training update. These tests used an empty Hugging Face cache and a working directory outside the repository; details are recorded in validation/benchmark_test_results.json.

They also passed a subsequent full-table inference recheck in a newly created Conda environment, again offline and from outside the repository. Each applicable R², Pearson, and Spearman metric agreed with its archived prediction-table metric within an absolute tolerance of 0.01. The per-checkpoint evidence is in validation/pi-update/full_benchmark_results.json. These checks validate the retained exports; they do not replace the historical benchmark scores with refitted results.

Here, “full-table” means every row of each checkpoint's archived validation prediction table. One legacy RNABert export has only 32 archived rows; its recheck is not a full-dataset benchmark. The report records each table's actual sample count.

Fresh-environment RNASeek evaluations and two-update generation checks are recorded in fresh_core_results.json and fresh_grpo_results.json. The generation checks used new output directories and the existing fixed inputs/caches; they demonstrate trainability, not convergence. Additional MSE and GEMORNA training checks are recorded in other_training_results.json.

conda activate rnaseek-benchmark
python benchmark_checkpoint.py \
  --checkpoint efficiency_figure2/benchmark_shipped_encoders/rerun_200ep_patience20/runs/RNABert/best_model \
  --data efficiency_figure2/benchmark_shipped_encoders/rerun_200ep_patience20/runs/RNABert/valid_predictions.tsv \
  --output my_runs/rnabert-check --limit 4 --batch-size 1 --smoke-train

This strictly loads the local fine-tuned encoder and regression head. --smoke-train performs one update on a fresh in-memory copy without saving over the original checkpoint. Omit both --limit and --smoke-train to evaluate the full table. Data must contain the archived seq_model and label columns. Local configurations and required custom model code are included with the selected checkpoint exports.

Keep each RNA-MSM export's configuration alongside its weights: rnaseek_position_id_start preserves the positional-index convention used by that historical training run. The efficiency export uses the upstream one-based buffer; the two stability exports retain their training code's zero-based buffer. These conventions are explicit checkpoint metadata, not inferred from file locations, and the model weights and archived prediction tables are unchanged.

For full fine-tuning, use regression_stability_functionalviral/benchmark_shipped_encoders/benchmark_shipped_encoders.py. Its manifest specifies model labels, checkpoint locations/Hub IDs, sequence alphabets, and hyperparameters. Local checkpoint paths in a manifest resolve relative to the manifest's directory. CLI data/output paths resolve relative to your working directory; the supplied run_benchmark.sh wrappers set the working directory to their subproject. Use a new parent experiment directory for --runs-dir; existing results are protected. Hub IDs require their original model cache or --no-local-files-only for a download. Local fine-tuned checkpoints do not need that cache.

--effective-batch-size uses gradient accumulation when the physical batch is smaller. The MSE loss is normalized across all examples in the accumulated update, including uneven final microbatches; a regression test checks equivalence to a physical batch. --max-steps 2 enables a short training check without claiming convergence. Always select a new --runs-dir.

Completed RiNALMo-Giga and Evo 2 benchmarks

The two new models were fitted with a 200-epoch cap and patience 20, evaluating once per epoch. Historical checkpoint-selection settings remain recorded in the run artifacts; the table below reports the full-validation metrics. All four completed exports passed full-table offline inference from a foreign working directory in fresh environments; metric differences were below 0.0001. See new_model_export_results.json.

Model	Efficiency Pearson	Efficiency R²	Stability all-validation Spearman
RiNALMo-Giga	0.630390	0.349659	0.386481
Evo 2 1B (BF16-adapted)	0.555053	0.296765	0.310780

The revised Figure 2 panel C uses efficiency Pearson. Figure 4's PowerPoint panel B uses all-validation Spearman. The notebook also retains supplementary analyses; these are not substitutes for the manuscript panels.

The reviewed editable decks are Figure 2 and Figure 4. Their generating code and saved outputs are in figures.ipynb.

The starting weights are in benchmark_backbones/. The new fine-tuned predictors are under the following paths, each ending in best_model:

Task/model	Checkpoint
Efficiency / Giga	efficiency_figure2/benchmark_shipped_encoders/pi_additions_v2/runs/RiNALMo-giga/best_model
Stability / Giga	regression_stability_functionalviral/benchmark_shipped_encoders/pi_additions_v3/runs/RiNALMo-giga/best_model
Efficiency / Evo 2	efficiency_figure2/benchmark_shipped_encoders/pi_evo_additions_v1/runs/Evo_2_1B_BF16-adapted/best_model
Stability / Evo 2	regression_stability_functionalviral/benchmark_shipped_encoders/pi_evo_additions_v1/runs/Evo_2_1B_BF16-adapted/best_model

Use benchmark_checkpoint.py as above, activating rnaseek-evo2 for Evo 2 or rnaseek-benchmark for Giga. Use the selected run's adjacent valid_predictions.tsv as --data; omit --limit for full validation.

To reproduce Giga efficiency fine-tuning, run from the repository root:

conda activate rnaseek-benchmark
python regression_stability_functionalviral/benchmark_shipped_encoders/benchmark_shipped_encoders.py \
  --manifest efficiency_figure2/benchmark_shipped_encoders/pi_rinalmo_manifest.json \
  --runs-dir my_runs/giga-efficiency/runs --data-kind efficiency_json \
  --train-json efficiency_figure2/evenBetterDataFolded-tr.json \
  --valid-json efficiency_figure2/evenBetterDataFolded-vl.json \
  --num-epochs 200 --early-stopping-patience 20 --effective-batch-size 16 \
  --dataloader-num-workers 2 --report-to tensorboard

For stability, use the stability project's pi_rinalmo_manifest.json, set --effective-batch-size 64, and replace the three efficiency data options with --data-tsv regression_stability_functionalviral/benchmark_shipped_encoders/rerun_200ep_patience20_train_pruned_similarity50_top25/training_seq_score_train_pruned_similarity50.tsv. Always choose a new experiment directory.

For Evo 2, activate rnaseek-evo2, select the task's pi_evo_manifest.json, and add --gradient-checkpointing. The manifests specify physical batch 16 for efficiency and 8 for Evo 2 stability; effective batches remain 16 and 64. For a trainability-only check, add --max-steps 2 --max-train-samples 128 --max-eval-samples 16 and use another new output directory. These checks do not replace completed benchmark results.

Evo 2 implementation details

The Evo 2 addition uses Evo 2 1B (BF16-adapted), NVIDIA's evo2/1b-8k-bf16:1.0 variant. It is not the original FP8-sensitive Arc 1B checkpoint. NVIDIA's model documentation describes the BF16-adapted weights; their attribution and license are included under third_party/evo2.

evo2_backbone.py exposes a fully trainable backbone through the same mean-pooling and regression-head workflow as the other benchmarks. It uses Vortex blocks with the source checkpoint's RMSNorm, causal FP32 FFT-convolution arithmetic, original trainable modal parameters, and fixed medium-filter decay buffers. Loading is strict. DNA characters retain their byte-level token IDs, with no added boundary tokens; batches use right padding. This adapter supports regression, not cached autoregressive generation. The separate RNASeek generation workflows are unchanged.

The converted backbone was compared with the unmodified NVIDIA BioNeMo 2.6.1 predictor on the same source weights and an 82-base test sequence: next-token loss was 0.835361 for the adapter versus 0.835213 natively; relative logit L2 error was 0.002705. This is a single-sequence numerical compatibility check, not bitwise equivalence or a regression score. The sequence, source checksum, tolerances, and measured results are recorded in evo_native_reference.json.

GEMORNA's benchmark-specific entry points remain in the two benchmark subdirectories and use the bundled predictor implementation. Its generation comparisons use the published pregenerated sequences, not regenerated samples.

Figures, caches, and reproducibility

Open figures.ipynb to view the figures and validation tables. Executed outputs are saved in the notebook, so viewing them does not require running a model. Running the notebook redraws figures from published, fixed inputs; it does not generate replacement sequences, refit cached projections, or rerun motif discovery.

Fast checks can be rerun with python -m unittest discover -s tests -v. The historical UTR syntax-monitor snapshots are packaged under their original run as syntax_monitor/snapshots.tar.gz, with a per-file SHA-256 manifest. Their JSONL contents are preserved byte-for-byte; extract them to a new directory if needed.

One historical efficiency TensorBoard file ends with an incomplete record from the original run. Its complete events and original trailing bytes are retained; use the saved metrics and prediction tables for reproducible numerical analysis.

conda activate rnaseek
jupyter lab figures.ipynb

Start Jupyter at the repository root, or set RNASEEK_ROOT to the downloaded root. Run all cells. Every execution creates a new figure_outputs/run-* directory. The notebook checks input hashes before and after rendering and saves an input manifest with repository-relative paths.

The UTR comparisons deliberately distinguish raw-sequence stability scores from adapter-cleaned folding/embedding analyses. Cached sampling settings, GC-matched motif backgrounds, token-position masks, and high-N analysis outputs must remain unchanged. If an input is missing, restore it from this release; do not regenerate it as a substitute.

Best/reference checkpoints and their tokenizers are retained; redundant intermediate checkpoints and serialized optimizer/process states are excluded. Fine-tuning from a selected checkpoint starts a new optimizer trajectory, not an exact resume of an archived training process. Recorded scalar results and fixed scientific inputs remain available for comparison.

Third-party model implementations, pretrained weights, and datasets retain their upstream attribution and licensing requirements. Consult their included documentation before redistribution or use. The preserved pretraining subproject is separate from the locally tested release workflows described above.