Delower/dpp4-molgemma-finetuned

MolGemma — DPP-4 fine-tuned (epoch 4)

Fine-tune of Delower/MolGemma on the DPP-4 potent+moderate+FDA-12 set (9,338 SMILES, 90/10 split, seed 42; AdamW lr 2e-5, batch 32, DeepChem-591 tokenizer).

Training (Kaggle CPU, notebook delower2021/dpp-4-molgemma-fine-tune V3/V4) was stopped at the owner's instruction after epoch 4:

epoch	train loss	val loss
1	0.4910	0.3331
2	0.2820	0.2467
3	0.2235	0.2053
4	0.1914	0.1800

Files

file	what it is
molgemma_finetuned_dpp4.keras	Full fine-tuned model — architecture + epoch-4 weights in one file; loads standalone (see below). Verified after upload preparation: reloaded in a fresh process and sampled (5/8 RDKit-valid in an 8-sample draw @ T0.8; larger draws from these weights run ~72–81% valid — raw samples can include salt mixtures, which the validation pipeline's cleaning pass handles).
molgemma_best.weights.h5	Epoch-4 best weights only (366 MB). Load the architecture (from the full model above, or from Delower/MolGemma) and call model.load_weights(...).
vocab_deepchem_2.4.0.txt	The 591-token DeepChem 2.4.0 vocabulary; token id = line index. Required to tokenize anything for this model.
tokenizer.py	The exact tokenizer used in training/generation (DeepChemTokenizer: tokenize / encode / decode), with a self-test (python tokenizer.py). Behavior-verified token-for-token against the training implementation on all 9,338 fine-tune rows.
generate_molgemma.py	The project's generation script — the exact sampler behind the project runs (2,000 @ T0.8 seed 101 + 2,000 @ T1.0 seed 202, GEN_STEPS 86, inline validity/novelty reporting). Only paths/header were adapted for this repo.
validate_rdkit.py	The project's RDKit validation pipeline as a standalone CLI: canonicalize → validity → unique → novelty (vs fine-tune set, and vs FDA-12 separately) → largest-fragment cleaning, printing the funnel. Reproduces the project funnels exactly (e.g. pretrained-baseline file: 500 → 491 valid → 490 unique → 490 novel).
config.json	Architecture, fine-tune provenance, generation parameters, and file inventory in machine-readable form.
training_log.csv	Per-epoch train/val loss of the fine-tune run (the table above).

Model summary

• Keras Functional model, inputs token_ids + padding_mask, output logits over the 591-token vocab; 30,498,304 parameters.
• ReversibleEmbedding input_dim 591 → output_dim 512 (tied weights); 8 × GemmaDecoderBlock (hidden 512, intermediate 768, head_dim 256, 8 query heads, 4 KV heads). Pretrained in mixed_float16.
• Tokenizer: DeepChem 2.4.0 regex SMILES tokenizer — not a generic text tokenizer. Special ids: [PAD]=0, [UNK]=11, [CLS]=12, [SEP]=13.

How to run inference

Generation protocol (identical to the project's generation runs): sequences are seeded with the single token C, sampled autoregressively for 86 steps (p99 train token length + 5) at temperature 0.8 (seed 101) or 1.0 (seed 202), then detokenized with [PAD]/[CLS]/[SEP] stripped. No BOS/EOS tokens are ever added — training used raw token ids.

import numpy as np
import keras, keras_nlp  # keras_nlp registers GemmaDecoderBlock
from tokenizer import DeepChemTokenizer, PAD_ID

tok = DeepChemTokenizer("vocab_deepchem_2.4.0.txt")
model = keras.models.load_model("molgemma_finetuned_dpp4.keras")

def generate(n=8, temperature=0.8, steps=86, seed=101):
    r = np.random.default_rng(seed)
    x = np.full((n, 1), tok.stoi["C"], dtype=np.int32)
    for _ in range(steps):
        logits = model({"token_ids": x,
                        "padding_mask": (x != PAD_ID).astype(np.int32)},
                       training=False).numpy()
        lg = logits[:, -1, :].astype(np.float64) / temperature
        lg -= lg.max(axis=-1, keepdims=True)
        p = np.exp(lg); p /= p.sum(axis=-1, keepdims=True)
        nxt = np.array([r.choice(p.shape[-1], p=row) for row in p],
                       dtype=np.int32)[:, None]
        x = np.concatenate([x, nxt], axis=1)
    return [tok.decode(row) for row in x]

Using the weights-only file instead:

from huggingface_hub import hf_hub_download
arch = hf_hub_download("Delower/MolGemma",
                       "output-gemma-seed42-fp16-last.model.keras")
model = keras.models.load_model(arch)
model.load_weights("molgemma_best.weights.h5")

Notes:

• Loading the base .keras or the weights file can print an optimizer-variable-count warning left over from pretraining; it is harmless for inference.
• Generated SMILES quality is assessed with RDKit (validity, uniqueness, novelty vs the fine-tune set and vs the FDA-12, reported separately).

Reproduce the pipeline

1. Generate — python generate_molgemma.py pulls the weights and vocab from this repo (and the architecture from Delower/MolGemma), samples 2,000 @ T0.8 + 2,000 @ T1.0, and writes the generations CSVs + metrics.json (with its own inline validity/novelty report). Needs the fine-tune CSV alongside for GEN_STEPS and the novelty reference.
2. Validate — python validate_rdkit.py generations_finetuned_T0.8.csv --reference-csv dpp4_finetune_potent_moderate_plus_fda12.csv prints the full funnel (n → valid → unique → novel → cleaned-unique) and writes a per-row validated CSV including the largest-fragment cleaned SMILES used for scoring.
3. Score — cleaned novel molecules are scored for DPP-4 potency with the tuned LTN pIC50 model (Delower/dpp4-pic50-ltn, 4,608 fingerprint features; sanity gate: alogliptin must reproduce pIC50 7.7408). Potent = predicted pIC50 ≥ 7. The scorer lives in the DPP-4 predictor repo, not here.

Provenance

• Base model: Delower/MolGemma (a true SMILES model; DeepChem-591 vocab recovered from the reference notebook MolMistral-Moses-KerasNLP.ipynb).
• Fine-tune data: dpp4_finetune_potent_moderate_plus_fda12.csv — 5,747 potent + 3,579 moderate DPP-4 compounds + the 12 FDA-approved gliptins; 9,338 rows, 0 out-of-vocab rows under this tokenizer.
• The delivered checkpoint is the best-val epoch-4 weights (val loss 0.1800). V3 and V4 Kaggle runs produced byte-identical weights (same SHA-256).

Full-pool generation results (2026-11-12)

The delivered epoch-4 weights were run generation-only (no retraining) in the Kaggle notebook delower2021/dpp-4-molgemma-generation-only, sampling 2,000 @ T0.8 (seed 101) + 2,000 @ T1.0 (seed 202) from molgemma_best.weights.h5:

• T0.8: 1,506/2,000 RDKit-valid (75.3%) → 1,380 unique → 1,173 novel
• T1.0: 1,184/2,000 RDKit-valid (59.2%) → 1,154 unique → 1,036 novel
• Combined cleaned-unique novel pool: 2,179 molecules
• LTN-scored (tuned DPP-4 LTN, gate alogliptin 7.7408): 793 potent (pIC50 ≥ 7), median predicted pIC50 6.6744, best 9.7650

Weights verification (2026-11-12): the molgemma_best.weights.h5 in this repo is byte-identical (SHA-256 97e4b4d3e8eec37d23c84f3a01f44d3cf6ad520a489801cb73f2ffe29573af29) to the file the generation run loaded, and all 75 model tensors stored in molgemma_finetuned_dpp4.keras are exactly equal to that weights file — the .keras in this repo provably carries the exact weights that produced the 793-potent pool.