RAVEN / README.md
xgaoo's picture
Update RAVEN title and paper and code links
ccb3621
|
Raw History Blame Contribute Delete
14.7 kB
---
library_name: pytorch
license: cc-by-nc-nd-4.0
extra_gated_heading: Request access to RAVEN
extra_gated_prompt: >-
RAVEN weights are licensed under CC BY-NC-ND 4.0; the accompanying code is
licensed under MIT. Access to the model is limited to non-commercial academic
research with proper attribution and is subject to the access terms below.
Clinical, diagnostic, and therapeutic use is not permitted. Commercial use,
sale, or other monetization of RAVEN, adapted models, models trained on its
outputs, or datasets generated using it is prohibited without prior written
approval from the authors. Commercial entities must contact the authors.
Register through your own Hugging Face account and set its primary email to
your institutional email. Provide your institution, principal investigator,
and research purpose. By requesting access and downloading the model, you
attest that this information is accurate and up-to-date and agree to the
license, access terms, and safety guidance. Do not distribute, publish, or
reproduce model weights for other users. Every user, including colleagues in
your organization, must register individually and receive approval. Keep access
tokens private. Do not attempt to re-identify training patients or reconstruct
their records.
extra_gated_fields:
Institution: text
Principal investigator name: text
Research purpose: text
I have read and agree to the license and access terms: checkbox
My affiliation and research information are accurate and up-to-date: checkbox
I will use the model only for non-commercial academic research with attribution: checkbox
I will not distribute, publish, or reproduce model weights for other users: checkbox
I will not attempt to identify training patients or recover their records: checkbox
tags:
- electronic-health-records
- clinical-records
- longitudinal-prediction
- raven
- safetensors
---
# RAVEN: Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction
[**Preprint**](https://arxiv.org/abs/2603.24562) /
[**GitHub**](https://github.com/xgao0o/RAVEN)
RAVEN is a transformer for longitudinal, structured electronic health records.
This model uses a temporal decay of 0.5 and was trained for **600,000 microbatch
iterations / 150,000 optimizer updates**, with **42,351 token IDs** and
**33,193 supervised outputs**.
It predicts clinical features at visit separator positions using clinical tokens
and elapsed days. Inputs are structured records, not free text.
## Requesting access
Request access through your own Hugging Face account using your institutional
email as the account's primary email. Provide your affiliation, principal
investigator, and research purpose. Requests are reviewed individually for
non-commercial academic research; each collaborator must submit their own request
and receive approval. By requesting access and downloading the model, you attest
that your information is accurate and up-to-date and accept the access terms.
Read the [License](#license) and [Access terms and responsible use](#access-terms-and-responsible-use)
before requesting access. Commercial entities should contact the authors listed
in [Contact](#contact) to discuss permission before using the model.
## Load and run
Use Python 3.10 and install the validated dependencies from this folder:
```bash
python -m pip install -r requirements.txt
python example_inference.py --device cpu
```
The example uses fabricated feature IDs and dates. For CUDA, pass `--device cuda`
after installing a compatible PyTorch/torchvision build.
```python
import torch
from load_model import load_model
from example_inference import synthetic_inputs
model = load_model(device="cpu")
with torch.inference_mode():
logits, _, _, _ = model(**synthetic_inputs())
print(logits.shape) # [1, 4, 42351]
```
After downloading the entire Hugging Face repository, run the scripts from that
downloaded folder. `load_model()` resolves metadata relative to its own file.
The weights use the included native PyTorch architecture; a Transformers
`AutoModel`, text-generation pipeline, and `AutoTokenizer` are not implemented.
**Use all-zero `patient_ids` for a single patient.** `prepare_inputs()` supplies
these automatically. This argument represents the packed-window position,
not a real patient identifier.
Omitting it removes a positional embedding used in training and changes outputs.
Use `data_files/loss_mask_0_perc.npy` to select supervised output IDs. Other output
rows exist in the vocabulary but were not directly supervised as targets.
## Model and training
| Property | Value |
| --- | --- |
| Architecture | 8 transformer blocks, 8 attention heads, hidden width 1,024 |
| Parameters | 144,048,640, including fixed rotary frequencies |
| Context | 512 tokens |
| Position encoding | Day-based rotary embeddings; XPos disabled |
| Embedding/output weights | Tied |
| Vocabulary | 42,345 retained columns plus 6 sequence controls |
| Supervised outputs | 33,193, occurring in more than 100 distinct patients |
| Loss | Sigmoid focal loss, gamma 2.0, alpha 0.995 |
| Temporal decay | 0.5; repeated positive targets have weights 1, 0.5, 0.25, 0.125, then 0.1 |
| Training | From scratch on NYU structured EHR data; multi-window packing |
| Optimizer | AdamW; learning rate 0.00022; weight decay 0.01 |
| Training compute | Two A100 GPUs; bfloat16 autocast |
| Released weights | Original FP32 tensors; no quantization |
The decay is applied to repeated positive occurrences of each target within a
sampled window, with a minimum weight of 0.1. The relative-day column and six
sequence controls are excluded from the loss. Eligible outputs comprise 23,937 diagnoses, 6,804 medications,
2,408 lab bins, and 44 demographic targets.
`config.json` records the architecture and loss settings, temporal decay,
training iteration counts, and tensor aliases needed to load this release.
## Using your own data
Map your clinical records directly to the frozen
[model vocabulary](data_files/token_headers.json). Each JSON list
index corresponds to model token ID **index + 6**. `load_vocabulary()` adds the
six sequence controls so its returned list can be indexed directly by token ID.
The label order is fixed by the weights and must not be changed.
Group the mapped clinical IDs into visits and supply their elapsed days.
`prepare_inputs()` adds separators, day embeddings, attention masks, and the
single-patient position indices expected by the model:
```python
import torch
from load_model import load_model, load_vocabulary
from prepare_inputs import prepare_inputs
labels = load_vocabulary()
token_to_id = {label: token_id for token_id, label in enumerate(labels)}
# Example only: replace this with features mapped from your own records.
visits = [[token_to_id["Diagnosis:Essential (primary) hypertension I10"]]]
inputs = prepare_inputs(visits, visit_days=[0], prediction_day=365)
model = load_model()
with torch.inference_mode():
logits, _, _, _ = model(**inputs)
forecast_logits = logits[0, -1] # Last SEP predicts at prediction_day.
```
Your mapping must account for clinical codes, medication identities, lab units and
normalization, age bins, and missing observations. See
[PREPROCESSING.md](PREPROCESSING.md) for the encoding rules and input contract.
The helper operates on already-mapped IDs; it does not infer equivalence between
your local codes and the vocabulary labels.
The only files in `data_files/` are the vocabulary and the supervised-output mask.
The mask distinguishes the 33,193 trained prediction targets from other vocabulary
slots; clinical features outside that target set may still be used as inputs.
## Evaluation
The table below summarizes performance on six EHRSHOT tasks.
[EHRSHOT condition headers](evaluation/ehrshot_condition_headers.json) and their
[model token IDs](evaluation/ehrshot_condition_token_ids.json) define the disease
scores.
Evaluation used FP32, context 512, batch size 16, all-zero patient-position
indices, direct prediction at 365 days, and sums of disease logits. The six
tasks cover 11,387 label instances; counts are task-specific and should not be
treated as unique patients across tasks.
| Condition | AUROC | AUPRC |
| --- | --- | --- |
| Heart attack | 0.6862 | 0.1370 |
| Pancreatic cancer | 0.7838 | 0.1857 |
| Hypertension | 0.7022 | 0.2523 |
| Hyperlipidemia | 0.6262 | 0.2045 |
| Lupus | 0.6707 | 0.0412 |
| Celiac disease | 0.5720 | 0.0116 |
## Intended use
The intended use is research on longitudinal clinical representation and prediction.
Examples include studying clinical representations, evaluating future-feature
prediction, and comparing research methods on appropriately governed datasets.
## Limitations & Safety
RAVEN is a research model. This release does not support clinical, diagnostic,
or therapeutic use, and its evaluation does not establish regulatory clearance
or clinical effectiveness. Do not use its outputs to make patient-care decisions.
The NYU training cohort and its documentation practices can introduce demographic
and clinical biases. Performance may change across institutions, patient groups,
coding systems, lab units, observation schedules, and missing-data patterns.
The six reported EHRSHOT tasks do not establish performance for every population,
condition, or use case. Evaluate subgroup performance and the intended setting
independently before drawing conclusions.
Outputs are uncalibrated scores and can be wrong. Rare-outcome estimates have few
positive patients; assess uncertainty in the setting where you evaluate the model.
Scores do not establish that a patient has, or will develop, a condition.
If scores are incorporated into an LLM or another system, independently check
the resulting claims; plausible prose does not validate a prediction.
The release contains no patient-level records. Model weights are not a guarantee
of privacy: do not attempt to identify training patients or reconstruct their
records. Use only data you are authorized to process, and do not send identifiable
patient information through public issues, discussion threads, or support email.
## Included files
`model.safetensors` contains only model tensors. The output projection shares the
embedding weights; `config.json` records the alias and `load_model.py` restores it
before strict state loading. The optimizer and training checkpoint pickle are
omitted. This follows the safetensors treatment of
[shared tensors](https://huggingface.co/docs/safetensors/torch_shared_tensors).
`model.py`, `focal.py`, and `rotary_embedding_torch.py` implement the trained
architecture. `load_model.py` loads the weights and vocabulary; `prepare_inputs.py`
builds inputs from mapped visits using `special_tokens_map.json` for sequence
controls. `example_inference.py` provides a runnable synthetic example, and
`PREPROCESSING.md` describes how to encode your own data. `requirements.txt`
records the validated dependencies. `data_files/` contains the vocabulary and
output mask, and `evaluation/` contains EHRSHOT condition headers and their
model token IDs.
The source project acknowledges nanoGPT; the bundled rotary implementation is
derived from rotary-embedding-torch. Their notices are retained in
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
## Contact
For research questions, access requests, or licensing inquiries, contact:
- Narges Razavian: [Narges.Razavian@nyulangone.org](mailto:Narges.Razavian@nyulangone.org)
- Xiang Gao: [x.gao@nyu.edu](mailto:x.gao@nyu.edu)
## License
- **Model weights:** [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/),
with the full text in [LICENSE](LICENSE).
- **Code:** [MIT License](LICENSE-CODE), with upstream notices retained.
The code's MIT license applies to the accompanying code; the weights have their
own license. The full license texts govern copyright permissions. Access to the
hosted model also requires acceptance of the terms below. Contact the authors
for commercial-use requests or other licensing inquiries.
## Access terms and responsible use
By requesting access and downloading the model from this repository, you agree
to the following access terms:
1. **Permitted use and attribution:** Use RAVEN only for non-commercial academic
research. Please provide proper attribution to the original authors,
including, but not limited to, citing the paper below,
identifying the model version in research reports,
and retaining the applicable license and attribution notices. Clinical,
diagnostic, and therapeutic use is not permitted.
2. **Commercial prohibition:** Commercial use, sale, or other monetization of
RAVEN requires prior written approval from the authors. This restriction also
applies to adapted models, models trained on RAVEN outputs, and datasets
generated using RAVEN. Commercial entities must contact the authors before use.
3. **Access, registration, and accuracy:** Register on Hugging Face using your
own account, with your institutional email as its primary email, and accept
the license and these terms before downloading. Provide your institution,
principal investigator, and research purpose. You attest that your submitted
information is accurate and up-to-date and will keep it current.
4. **Non-distribution:** Do not distribute, publish, or reproduce copies of the
model weights for other users. This includes sharing weights with colleagues
within your organization. Direct prospective users to request access here.
5. **Individual access:** Approval is personal. Every user must register
independently, accept these terms, and receive approval. Keep access tokens
private; do not share accounts, credentials, or access approval.
6. **Data privacy:** Do not attempt to re-identify training participants,
reconstruct or extract their records, or use the model to disclose private
patient information. Process only data you are authorized to use.
7. **Safety and warranties:** Follow the safety guidance above. The model is
supplied without warranties of accuracy or fitness for a particular purpose,
subject to the full license terms.
## Citation
Please cite the following paper when using RAVEN in your research:
```bibtex
@article{gao2026raven,
title={Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction},
author={Gao, Xiang and Rajamohan, Haresh Rengaraj and Zhu, Weicheng and Huang, Shih-Lun and Chen, Long and Schulman, Gabe and Jin, Huizhen and Li, Shengduo and Wang, Yixuan and Yang, Huidi and others},
journal={arXiv preprint arXiv:2603.24562},
year={2026}
}
```