| --- |
| library_name: transformers |
| tags: |
| - bulk RNA-seq |
| - DNA methylation |
| - biology |
| - transcriptomics |
| - epigenomics |
| - multimodal |
| --- |
| |
| # MOJO |
|
|
| MOJO (MultiOmics JOint representation learning) is a model that learns joint representations of bulk RNA-seq and DNA methylation through bimodal masked language modeling and is tailored for cancer-type classification and survival analysis on the TCGA dataset. |
|
|
| **Developed by:** [InstaDeep](https://huggingface.co/InstaDeepAI) |
|
|
| ### Model Sources |
|
|
| <!-- Provide the basic links for the model. --> |
|
|
| - [**Repository**](https://github.com/instadeepai/multiomics-open-research) |
| - **Paper:** [Bimodal masked language modeling for bulk RNA-seq and DNA methylation |
| representation learning](https://www.biorxiv.org/content/10.1101/2025.06.25.661237v1) |
|
|
| ### How to use |
|
|
| Until its next release, the transformers library needs to be installed from source with |
| the following command in order to use the models. |
| PyTorch should also be installed. |
|
|
| ``` |
| pip install --upgrade git+https://github.com/huggingface/transformers.git |
| pip install torch |
| ``` |
|
|
| ## Other notes |
| We also provide the params for the MOJO jax model in `jax_params`. |
|
|
| A small snippet of code is provided below to run inference with the model using bulk RNA-seq and DNA methylation samples from the [TCGA](https://portal.gdc.cancer.gov/) dataset. |
|
|
| ``` |
| import numpy as np |
| import pandas as pd |
| from transformers import AutoModel, AutoTokenizer |
| from huggingface_hub import hf_hub_download |
| |
| tokenizer = AutoTokenizer.from_pretrained("InstaDeepAI/MOJO", trust_remote_code=True) |
| model = AutoModel.from_pretrained( |
| "InstaDeepAI/MOJO", |
| trust_remote_code=True, |
| ) |
| |
| n_examples = 4 |
| omic_dict = {} |
| |
| for omic in ["rnaseq", "methylation"]: |
| csv_path = hf_hub_download( |
| repo_id="InstaDeepAI/MOJO", |
| filename=f"data/tcga_{omic}_sample.csv", |
| repo_type="model", |
| ) |
| omic_array = pd.read_csv(csv_path).drop(["identifier", "cohort"], axis=1).to_numpy()[:n_examples, :] |
| if omic == "rnaseq": |
| omic_array = np.log10(1 + omic_array) |
| assert omic_array.shape[1] == model.config.sequence_length |
| omic_dict[omic] = omic_array |
| |
| omic_ids = { |
| omic: tokens["input_ids"] |
| for omic, tokens in tokenizer.batch_encode_plus(omic_dict, pad_to_fixed_length=True, return_tensors="pt").items() |
| } |
| |
| omic_mean_embeddings = model(omic_ids)["after_transformer_embedding"].mean(axis=1) # embeddings can be used for downstream tasks. |
| ``` |
|
|
|
|
| ### Citing our work |
|
|
| ``` |
| @article {G{\'e}lard2025.06.25.661237, |
| author = {G{\'e}lard, Maxence and Benkirane, Hakim and Pierrot, Thomas and Richard, Guillaume and Courn{\`e}de, Paul-Henry}, |
| title = {Bimodal masked language modeling for bulk RNA-seq and DNA methylation representation learning}, |
| elocation-id = {2025.06.25.661237}, |
| year = {2025}, |
| doi = {10.1101/2025.06.25.661237}, |
| publisher = {Cold Spring Harbor Laboratory}, |
| URL = {https://www.biorxiv.org/content/early/2025/06/27/2025.06.25.661237}, |
| journal = {bioRxiv} |
| } |
| ``` |
|
|
