---
license: unknown
tags:
  - tabular-classification
  - feature-tokenizer-transformer
  - distilbert
  - cardiovascular-disease
  - uk-biobank
pipeline_tag: tabular-classification
---

# RAIC — FTT (Feature Tokenizer Transformer) for UK Biobank CVD risk classification

A DistilBERT-style transformer adapted to **tabular** clinical data. Each row (participant)
is treated as a "sentence" whose "tokens" are its features:

- **Continuous features** are embedded via a per-feature learned weight vector, scaled by
  the (min-max normalized) value (`num_input_val * variable_weights_embeddings[num_input_ids]`).
- **Categorical features** are embedded via a per-value lookup table
  (`variable_weights_embeddings[cat_input_ids]`).
- A separate **variable/bias embedding** table encodes which feature slot ("variable") each
  token corresponds to, acting like a position embedding
  (`variable_bias_embeddings[num_variable_ids | cat_variable_ids]`).

The model was trained to predict **5-year incident cardiovascular disease (CVD)** from UK
Biobank baseline assessment data (`YEAR_5_CVD`, binary), using 69 continuous and 76
categorical features.

This checkpoint corresponds to the local experiment `20240603_1_UKB_CVD_cls_hp_search`.

> **Data/label note:** No participant-level data is included in this repository — only model
> weights, config, tokenizers, and aggregate (variable-level) min/max normalization
> statistics used at training time. UK Biobank data governance means this model should only
> be applied to appropriately approved data/use cases.

## Model details

- Architecture: `FTTForSequenceClassification` (DistilBERT encoder, 8 layers, 16 heads,
  hidden dim 128, feed-forward dim 1024)
- Task: binary sequence classification (`num_labels=2`)
  - `0`: no 5-year CVD event
  - `1`: 5-year CVD event
- Custom modeling code lives in [`modeling_ftt.py`](modeling_ftt.py) and is loaded
  automatically via `trust_remote_code=True` (see `auto_map` in `config.json`).

## Files

| File | Purpose |
|---|---|
| `config.json` | Model hyperparameters + `auto_map` for custom-code loading |
| `modeling_ftt.py` | `FTTConfig`, `FTTModel`, `FTTForSequenceClassification`, `FTTForMaskedLM` |
| `model.safetensors` | Fine-tuned classification weights |
| `ftt_variable_weights_tokenizer_20240530/` | Maps each continuous-variable name and each categorical (variable, value) pair to a token id used for the *weight* embedding table |
| `ftt_variable_bias_tokenizer_20240530/` | Maps each variable name to a token id used for the *bias/position* embedding table |
| `train_continuous_variables_min_max_values.json` | Per-continuous-variable `[min, max]` from the training set, used to min-max normalize raw values into `[1, 3]` before scoring (see `test_upload.py`) |

## Usage

```python
from transformers import AutoConfig, AutoModelForSequenceClassification, AutoTokenizer

repo_id = "thinkhong/RAIC"

config = AutoConfig.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

weights_tok = AutoTokenizer.from_pretrained(repo_id, subfolder="ftt_variable_weights_tokenizer_20240530")
bias_tok = AutoTokenizer.from_pretrained(repo_id, subfolder="ftt_variable_bias_tokenizer_20240530")
```

The model's `forward()` does **not** take `input_ids`/text. Instead it expects:

- `num_input_val` (float): normalized continuous values, `[CLS]` slot = `1.0`
- `num_input_ids` (long): weight-tokenizer ids for each continuous variable's *name*, `[CLS]`-prefixed
- `num_variable_ids` (long): bias-tokenizer ids for each continuous variable's *name*, `[CLS]`-prefixed
- `cat_input_ids` (long): weight-tokenizer ids for each categorical `variable_value` pair present
- `cat_variable_ids` (long): bias-tokenizer ids for each categorical variable's *name*
- `attention_mask` (long): `1` for real tokens (`1 CLS + n_continuous + n_categorical`), `0` for padding

See [`test_upload.py`](test_upload.py) for a full worked (dummy-value) example, including how
to normalize a raw continuous value with `train_continuous_variables_min_max_values.json`.

## Testing

Two test scripts are included:

- **`test_upload.py`** — a runnable demo/smoke test. Downloads the model from the Hub with
  `trust_remote_code=True`, builds a full synthetic example, and prints the logits/predicted class.
  ```bash
  pip install transformers huggingface_hub torch
  python test_upload.py --repo-id thinkhong/RAIC
  ```
- **`test_ftt_hub.py`** — a `pytest` suite with pass/fail assertions (architecture, param count,
  tokenizer vocab sizes, forward-pass shape, determinism, and an exact-match check against the
  reference logits recorded at publish time).
  ```bash
  pip install pytest transformers huggingface_hub torch
  pytest test_ftt_hub.py -v
  ```

## Verification

This checkpoint was uploaded from the original training output directory and verified to
produce bit-identical logits to the source checkpoint on an identical input before publishing
(see `test_upload.py`).