How to correctly select and reuse the best hyperparameters after cc.validate(n_hyperopt_trials=20)?

#595
by Mashuaiii - opened

Hi Geneformer team,

After running hyperparameter search with Classifier.validate(n_hyperopt_trials=20), I tried to follow the tutorial and evaluate the model on the held-out test set using cc.evaluate_saved_model(...), but it fails with:

text
Error: no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory ...

What the output directory looks like

The tutorial suggests passing:

python
all_metrics_test = cc.evaluate_saved_model(
    model_directory=f"{output_dir}/{datestamp_min}_geneformer_cellClassifier_{output_prefix}/ksplit1/",
    id_class_dict_file=f"{output_dir}/{output_prefix}_id_class_dict.pkl",
    test_data_file=f"{output_dir}/{output_prefix}_labeled_test.dataset",
    output_directory=output_dir,
    output_prefix=output_prefix,
)

But after n_hyperopt_trials=20, ksplit1/ does not contain pytorch_model.bin directly. Instead, it contains one subdirectory per trial, e.g.:

text
ksplit1/
β”œβ”€β”€ _objective_2026-10-20_22-40-39/   # Ray Tune metadata
β”œβ”€β”€ run-1307681c/
β”œβ”€β”€ run-22af5413/
β”œβ”€β”€ run-2493eba9/
β”œβ”€β”€ ...
└── run-d2aa0900/

Each run-*/ subdirectory appears to correspond to one trial's saved model, but ksplit1/ itself has no model weights file, so evaluate_saved_model cannot find one.

Questions

When n_hyperopt_trials > 0, is evaluate_saved_model supposed to be pointed at a specific run-*/ subdirectory (the best trial), rather than at ksplit1/? If so, is there a recommended way to programmatically obtain the path of the best trial, or should users manually pick it from the Ray Tune results table (e.g. by eval_macro_f1)?

Is there a built-in function or attribute that returns the best trial's model directory after validate(n_hyperopt_trials > 0) completes? The tutorial example seems to assume a single model is saved directly under ksplit1/, which does not match the per-trial run-*/ layout produced by hyperparameter search.

For context, my best trial by eval_macro_f1 is _objective_9a43b606, and I can evaluate it successfully by passing .../ksplit1/run-9a43b606/ as model_directory. But I would like to confirm whether this manual selection is the intended workflow, and whether the tutorial's ksplit1/ path is only valid when n_hyperopt_trials=0.

Thanks for your help!

Sign up or log in to comment