| --- |
| license: unknown |
| tags: |
| - tabular-classification |
| - feature-tokenizer-transformer |
| - distilbert |
| - cardiovascular-disease |
| - uk-biobank |
| pipeline_tag: tabular-classification |
| --- |
| |
| # RAIC — FTT (Feature Tokenizer Transformer) for UK Biobank CVD risk classification |
|
|
| A DistilBERT-style transformer adapted to **tabular** clinical data. Each row (participant) |
| is treated as a "sentence" whose "tokens" are its features: |
|
|
| - **Continuous features** are embedded via a per-feature learned weight vector, scaled by |
| the (min-max normalized) value (`num_input_val * variable_weights_embeddings[num_input_ids]`). |
| - **Categorical features** are embedded via a per-value lookup table |
| (`variable_weights_embeddings[cat_input_ids]`). |
| - A separate **variable/bias embedding** table encodes which feature slot ("variable") each |
| token corresponds to, acting like a position embedding |
| (`variable_bias_embeddings[num_variable_ids | cat_variable_ids]`). |
|
|
| The model was trained to predict **5-year incident cardiovascular disease (CVD)** from UK |
| Biobank baseline assessment data (`YEAR_5_CVD`, binary), using 69 continuous and 76 |
| categorical features. |
|
|
| This checkpoint corresponds to the local experiment `20240603_1_UKB_CVD_cls_hp_search`. |
|
|
| > **Data/label note:** No participant-level data is included in this repository — only model |
| > weights, config, tokenizers, and aggregate (variable-level) min/max normalization |
| > statistics used at training time. UK Biobank data governance means this model should only |
| > be applied to appropriately approved data/use cases. |
|
|
| ## Model details |
|
|
| - Architecture: `FTTForSequenceClassification` (DistilBERT encoder, 8 layers, 16 heads, |
| hidden dim 128, feed-forward dim 1024) |
| - Task: binary sequence classification (`num_labels=2`) |
| - `0`: no 5-year CVD event |
| - `1`: 5-year CVD event |
| - Custom modeling code lives in [`modeling_ftt.py`](modeling_ftt.py) and is loaded |
| automatically via `trust_remote_code=True` (see `auto_map` in `config.json`). |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |---|---| |
| | `config.json` | Model hyperparameters + `auto_map` for custom-code loading | |
| | `modeling_ftt.py` | `FTTConfig`, `FTTModel`, `FTTForSequenceClassification`, `FTTForMaskedLM` | |
| | `model.safetensors` | Fine-tuned classification weights | |
| | `ftt_variable_weights_tokenizer_20240530/` | Maps each continuous-variable name and each categorical (variable, value) pair to a token id used for the *weight* embedding table | |
| | `ftt_variable_bias_tokenizer_20240530/` | Maps each variable name to a token id used for the *bias/position* embedding table | |
| | `train_continuous_variables_min_max_values.json` | Per-continuous-variable `[min, max]` from the training set, used to min-max normalize raw values into `[1, 3]` before scoring (see `test_upload.py`) | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoConfig, AutoModelForSequenceClassification, AutoTokenizer |
| |
| repo_id = "thinkhong/RAIC" |
| |
| config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True) |
| model = AutoModelForSequenceClassification.from_pretrained(repo_id, trust_remote_code=True) |
| model.eval() |
| |
| weights_tok = AutoTokenizer.from_pretrained(repo_id, subfolder="ftt_variable_weights_tokenizer_20240530") |
| bias_tok = AutoTokenizer.from_pretrained(repo_id, subfolder="ftt_variable_bias_tokenizer_20240530") |
| ``` |
|
|
| The model's `forward()` does **not** take `input_ids`/text. Instead it expects: |
|
|
| - `num_input_val` (float): normalized continuous values, `[CLS]` slot = `1.0` |
| - `num_input_ids` (long): weight-tokenizer ids for each continuous variable's *name*, `[CLS]`-prefixed |
| - `num_variable_ids` (long): bias-tokenizer ids for each continuous variable's *name*, `[CLS]`-prefixed |
| - `cat_input_ids` (long): weight-tokenizer ids for each categorical `variable_value` pair present |
| - `cat_variable_ids` (long): bias-tokenizer ids for each categorical variable's *name* |
| - `attention_mask` (long): `1` for real tokens (`1 CLS + n_continuous + n_categorical`), `0` for padding |
|
|
| See [`test_upload.py`](test_upload.py) for a full worked (dummy-value) example, including how |
| to normalize a raw continuous value with `train_continuous_variables_min_max_values.json`. |
|
|
| ## Testing |
|
|
| Two test scripts are included: |
|
|
| - **`test_upload.py`** — a runnable demo/smoke test. Downloads the model from the Hub with |
| `trust_remote_code=True`, builds a full synthetic example, and prints the logits/predicted class. |
| ```bash |
| pip install transformers huggingface_hub torch |
| python test_upload.py --repo-id thinkhong/RAIC |
| ``` |
| - **`test_ftt_hub.py`** — a `pytest` suite with pass/fail assertions (architecture, param count, |
| tokenizer vocab sizes, forward-pass shape, determinism, and an exact-match check against the |
| reference logits recorded at publish time). |
| ```bash |
| pip install pytest transformers huggingface_hub torch |
| pytest test_ftt_hub.py -v |
| ``` |
| |
| ## Verification |
| |
| This checkpoint was uploaded from the original training output directory and verified to |
| produce bit-identical logits to the source checkpoint on an identical input before publishing |
| (see `test_upload.py`). |
| |
