|
Download README.md from NYUMedML/RAVEN: direct link, hf CLI and curl.
- Browser
- Download file 14.7 kB
-
https://huggingface.co/NYUMedML/RAVEN/resolve/main/README.md
- Command line
-
hf download hf://NYUMedML/RAVEN/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/NYUMedML/RAVEN/resolve/main/README.md
14.7 kB
| library_name: pytorch | |
| license: cc-by-nc-nd-4.0 | |
| extra_gated_heading: Request access to RAVEN | |
| extra_gated_prompt: >- | |
| RAVEN weights are licensed under CC BY-NC-ND 4.0; the accompanying code is | |
| licensed under MIT. Access to the model is limited to non-commercial academic | |
| research with proper attribution and is subject to the access terms below. | |
| Clinical, diagnostic, and therapeutic use is not permitted. Commercial use, | |
| sale, or other monetization of RAVEN, adapted models, models trained on its | |
| outputs, or datasets generated using it is prohibited without prior written | |
| approval from the authors. Commercial entities must contact the authors. | |
| Register through your own Hugging Face account and set its primary email to | |
| your institutional email. Provide your institution, principal investigator, | |
| and research purpose. By requesting access and downloading the model, you | |
| attest that this information is accurate and up-to-date and agree to the | |
| license, access terms, and safety guidance. Do not distribute, publish, or | |
| reproduce model weights for other users. Every user, including colleagues in | |
| your organization, must register individually and receive approval. Keep access | |
| tokens private. Do not attempt to re-identify training patients or reconstruct | |
| their records. | |
| extra_gated_fields: | |
| Institution: text | |
| Principal investigator name: text | |
| Research purpose: text | |
| I have read and agree to the license and access terms: checkbox | |
| My affiliation and research information are accurate and up-to-date: checkbox | |
| I will use the model only for non-commercial academic research with attribution: checkbox | |
| I will not distribute, publish, or reproduce model weights for other users: checkbox | |
| I will not attempt to identify training patients or recover their records: checkbox | |
| tags: | |
| - electronic-health-records | |
| - clinical-records | |
| - longitudinal-prediction | |
| - raven | |
| - safetensors | |
| # RAVEN: Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction | |
| [**Preprint**](https://arxiv.org/abs/2603.24562) / | |
| [**GitHub**](https://github.com/xgao0o/RAVEN) | |
| RAVEN is a transformer for longitudinal, structured electronic health records. | |
| This model uses a temporal decay of 0.5 and was trained for **600,000 microbatch | |
| iterations / 150,000 optimizer updates**, with **42,351 token IDs** and | |
| **33,193 supervised outputs**. | |
| It predicts clinical features at visit separator positions using clinical tokens | |
| and elapsed days. Inputs are structured records, not free text. | |
| ## Requesting access | |
| Request access through your own Hugging Face account using your institutional | |
| email as the account's primary email. Provide your affiliation, principal | |
| investigator, and research purpose. Requests are reviewed individually for | |
| non-commercial academic research; each collaborator must submit their own request | |
| and receive approval. By requesting access and downloading the model, you attest | |
| that your information is accurate and up-to-date and accept the access terms. | |
| Read the [License](#license) and [Access terms and responsible use](#access-terms-and-responsible-use) | |
| before requesting access. Commercial entities should contact the authors listed | |
| in [Contact](#contact) to discuss permission before using the model. | |
| ## Load and run | |
| Use Python 3.10 and install the validated dependencies from this folder: | |
| ```bash | |
| python -m pip install -r requirements.txt | |
| python example_inference.py --device cpu | |
| ``` | |
| The example uses fabricated feature IDs and dates. For CUDA, pass `--device cuda` | |
| after installing a compatible PyTorch/torchvision build. | |
| ```python | |
| import torch | |
| from load_model import load_model | |
| from example_inference import synthetic_inputs | |
| model = load_model(device="cpu") | |
| with torch.inference_mode(): | |
| logits, _, _, _ = model(**synthetic_inputs()) | |
| print(logits.shape) # [1, 4, 42351] | |
| ``` | |
| After downloading the entire Hugging Face repository, run the scripts from that | |
| downloaded folder. `load_model()` resolves metadata relative to its own file. | |
| The weights use the included native PyTorch architecture; a Transformers | |
| `AutoModel`, text-generation pipeline, and `AutoTokenizer` are not implemented. | |
| **Use all-zero `patient_ids` for a single patient.** `prepare_inputs()` supplies | |
| these automatically. This argument represents the packed-window position, | |
| not a real patient identifier. | |
| Omitting it removes a positional embedding used in training and changes outputs. | |
| Use `data_files/loss_mask_0_perc.npy` to select supervised output IDs. Other output | |
| rows exist in the vocabulary but were not directly supervised as targets. | |
| ## Model and training | |
| | Property | Value | | |
| | --- | --- | | |
| | Architecture | 8 transformer blocks, 8 attention heads, hidden width 1,024 | | |
| | Parameters | 144,048,640, including fixed rotary frequencies | | |
| | Context | 512 tokens | | |
| | Position encoding | Day-based rotary embeddings; XPos disabled | | |
| | Embedding/output weights | Tied | | |
| | Vocabulary | 42,345 retained columns plus 6 sequence controls | | |
| | Supervised outputs | 33,193, occurring in more than 100 distinct patients | | |
| | Loss | Sigmoid focal loss, gamma 2.0, alpha 0.995 | | |
| | Temporal decay | 0.5; repeated positive targets have weights 1, 0.5, 0.25, 0.125, then 0.1 | | |
| | Training | From scratch on NYU structured EHR data; multi-window packing | | |
| | Optimizer | AdamW; learning rate 0.00022; weight decay 0.01 | | |
| | Training compute | Two A100 GPUs; bfloat16 autocast | | |
| | Released weights | Original FP32 tensors; no quantization | | |
| The decay is applied to repeated positive occurrences of each target within a | |
| sampled window, with a minimum weight of 0.1. The relative-day column and six | |
| sequence controls are excluded from the loss. Eligible outputs comprise 23,937 diagnoses, 6,804 medications, | |
| 2,408 lab bins, and 44 demographic targets. | |
| `config.json` records the architecture and loss settings, temporal decay, | |
| training iteration counts, and tensor aliases needed to load this release. | |
| ## Using your own data | |
| Map your clinical records directly to the frozen | |
| [model vocabulary](data_files/token_headers.json). Each JSON list | |
| index corresponds to model token ID **index + 6**. `load_vocabulary()` adds the | |
| six sequence controls so its returned list can be indexed directly by token ID. | |
| The label order is fixed by the weights and must not be changed. | |
| Group the mapped clinical IDs into visits and supply their elapsed days. | |
| `prepare_inputs()` adds separators, day embeddings, attention masks, and the | |
| single-patient position indices expected by the model: | |
| ```python | |
| import torch | |
| from load_model import load_model, load_vocabulary | |
| from prepare_inputs import prepare_inputs | |
| labels = load_vocabulary() | |
| token_to_id = {label: token_id for token_id, label in enumerate(labels)} | |
| # Example only: replace this with features mapped from your own records. | |
| visits = [[token_to_id["Diagnosis:Essential (primary) hypertension I10"]]] | |
| inputs = prepare_inputs(visits, visit_days=[0], prediction_day=365) | |
| model = load_model() | |
| with torch.inference_mode(): | |
| logits, _, _, _ = model(**inputs) | |
| forecast_logits = logits[0, -1] # Last SEP predicts at prediction_day. | |
| ``` | |
| Your mapping must account for clinical codes, medication identities, lab units and | |
| normalization, age bins, and missing observations. See | |
| [PREPROCESSING.md](PREPROCESSING.md) for the encoding rules and input contract. | |
| The helper operates on already-mapped IDs; it does not infer equivalence between | |
| your local codes and the vocabulary labels. | |
| The only files in `data_files/` are the vocabulary and the supervised-output mask. | |
| The mask distinguishes the 33,193 trained prediction targets from other vocabulary | |
| slots; clinical features outside that target set may still be used as inputs. | |
| ## Evaluation | |
| The table below summarizes performance on six EHRSHOT tasks. | |
| [EHRSHOT condition headers](evaluation/ehrshot_condition_headers.json) and their | |
| [model token IDs](evaluation/ehrshot_condition_token_ids.json) define the disease | |
| scores. | |
| Evaluation used FP32, context 512, batch size 16, all-zero patient-position | |
| indices, direct prediction at 365 days, and sums of disease logits. The six | |
| tasks cover 11,387 label instances; counts are task-specific and should not be | |
| treated as unique patients across tasks. | |
| | Condition | AUROC | AUPRC | | |
| | --- | --- | --- | | |
| | Heart attack | 0.6862 | 0.1370 | | |
| | Pancreatic cancer | 0.7838 | 0.1857 | | |
| | Hypertension | 0.7022 | 0.2523 | | |
| | Hyperlipidemia | 0.6262 | 0.2045 | | |
| | Lupus | 0.6707 | 0.0412 | | |
| | Celiac disease | 0.5720 | 0.0116 | | |
| ## Intended use | |
| The intended use is research on longitudinal clinical representation and prediction. | |
| Examples include studying clinical representations, evaluating future-feature | |
| prediction, and comparing research methods on appropriately governed datasets. | |
| ## Limitations & Safety | |
| RAVEN is a research model. This release does not support clinical, diagnostic, | |
| or therapeutic use, and its evaluation does not establish regulatory clearance | |
| or clinical effectiveness. Do not use its outputs to make patient-care decisions. | |
| The NYU training cohort and its documentation practices can introduce demographic | |
| and clinical biases. Performance may change across institutions, patient groups, | |
| coding systems, lab units, observation schedules, and missing-data patterns. | |
| The six reported EHRSHOT tasks do not establish performance for every population, | |
| condition, or use case. Evaluate subgroup performance and the intended setting | |
| independently before drawing conclusions. | |
| Outputs are uncalibrated scores and can be wrong. Rare-outcome estimates have few | |
| positive patients; assess uncertainty in the setting where you evaluate the model. | |
| Scores do not establish that a patient has, or will develop, a condition. | |
| If scores are incorporated into an LLM or another system, independently check | |
| the resulting claims; plausible prose does not validate a prediction. | |
| The release contains no patient-level records. Model weights are not a guarantee | |
| of privacy: do not attempt to identify training patients or reconstruct their | |
| records. Use only data you are authorized to process, and do not send identifiable | |
| patient information through public issues, discussion threads, or support email. | |
| ## Included files | |
| `model.safetensors` contains only model tensors. The output projection shares the | |
| embedding weights; `config.json` records the alias and `load_model.py` restores it | |
| before strict state loading. The optimizer and training checkpoint pickle are | |
| omitted. This follows the safetensors treatment of | |
| [shared tensors](https://huggingface.co/docs/safetensors/torch_shared_tensors). | |
| `model.py`, `focal.py`, and `rotary_embedding_torch.py` implement the trained | |
| architecture. `load_model.py` loads the weights and vocabulary; `prepare_inputs.py` | |
| builds inputs from mapped visits using `special_tokens_map.json` for sequence | |
| controls. `example_inference.py` provides a runnable synthetic example, and | |
| `PREPROCESSING.md` describes how to encode your own data. `requirements.txt` | |
| records the validated dependencies. `data_files/` contains the vocabulary and | |
| output mask, and `evaluation/` contains EHRSHOT condition headers and their | |
| model token IDs. | |
| The source project acknowledges nanoGPT; the bundled rotary implementation is | |
| derived from rotary-embedding-torch. Their notices are retained in | |
| [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). | |
| ## Contact | |
| For research questions, access requests, or licensing inquiries, contact: | |
| - Narges Razavian: [Narges.Razavian@nyulangone.org](mailto:Narges.Razavian@nyulangone.org) | |
| - Xiang Gao: [x.gao@nyu.edu](mailto:x.gao@nyu.edu) | |
| ## License | |
| - **Model weights:** [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/), | |
| with the full text in [LICENSE](LICENSE). | |
| - **Code:** [MIT License](LICENSE-CODE), with upstream notices retained. | |
| The code's MIT license applies to the accompanying code; the weights have their | |
| own license. The full license texts govern copyright permissions. Access to the | |
| hosted model also requires acceptance of the terms below. Contact the authors | |
| for commercial-use requests or other licensing inquiries. | |
| ## Access terms and responsible use | |
| By requesting access and downloading the model from this repository, you agree | |
| to the following access terms: | |
| 1. **Permitted use and attribution:** Use RAVEN only for non-commercial academic | |
| research. Please provide proper attribution to the original authors, | |
| including, but not limited to, citing the paper below, | |
| identifying the model version in research reports, | |
| and retaining the applicable license and attribution notices. Clinical, | |
| diagnostic, and therapeutic use is not permitted. | |
| 2. **Commercial prohibition:** Commercial use, sale, or other monetization of | |
| RAVEN requires prior written approval from the authors. This restriction also | |
| applies to adapted models, models trained on RAVEN outputs, and datasets | |
| generated using RAVEN. Commercial entities must contact the authors before use. | |
| 3. **Access, registration, and accuracy:** Register on Hugging Face using your | |
| own account, with your institutional email as its primary email, and accept | |
| the license and these terms before downloading. Provide your institution, | |
| principal investigator, and research purpose. You attest that your submitted | |
| information is accurate and up-to-date and will keep it current. | |
| 4. **Non-distribution:** Do not distribute, publish, or reproduce copies of the | |
| model weights for other users. This includes sharing weights with colleagues | |
| within your organization. Direct prospective users to request access here. | |
| 5. **Individual access:** Approval is personal. Every user must register | |
| independently, accept these terms, and receive approval. Keep access tokens | |
| private; do not share accounts, credentials, or access approval. | |
| 6. **Data privacy:** Do not attempt to re-identify training participants, | |
| reconstruct or extract their records, or use the model to disclose private | |
| patient information. Process only data you are authorized to use. | |
| 7. **Safety and warranties:** Follow the safety guidance above. The model is | |
| supplied without warranties of accuracy or fitness for a particular purpose, | |
| subject to the full license terms. | |
| ## Citation | |
| Please cite the following paper when using RAVEN in your research: | |
| ```bibtex | |
| @article{gao2026raven, | |
| title={Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction}, | |
| author={Gao, Xiang and Rajamohan, Haresh Rengaraj and Zhu, Weicheng and Huang, Shih-Lun and Chen, Long and Schulman, Gabe and Jin, Huizhen and Li, Shengduo and Wang, Yixuan and Yang, Huidi and others}, | |
| journal={arXiv preprint arXiv:2603.24562}, | |
| year={2026} | |
| } | |
| ``` | |