---
license: agpl-3.0
language: multilingual
base_model: FacebookAI/xlm-roberta-large
library_name: onnx
pipeline_tag: text-classification
tags:
  - onnx
  - xlm-roberta
  - research-data
  - tabular-metadata
  - pytacheck
  - metacheck
---

# datacheck-concepts v1

This model tells what a column of a research data file measures. It reads the column's name, its
file, its neighbouring columns, summary statistics and up to 200 of its values, and returns one of
12 concepts. [pytacheck](https://github.com/scienceverse/pytacheck)'s `data_check` uses it
offline to fill in the concepts that
[metacheck](https://github.com/scienceverse/metacheck)'s rules leave blank. metacheck asks an LLM
for those columns; without an LLM they stay blank.

```bash
pip install "pytacheck[concepts]"
```

With that extra installed, `data_check` downloads this model once (tag `v1`, ~840 MB) and uses it
by default. `concepts="cascade"` sends the columns it is least sure of (confidence below 0.92) to
the configured LLM, and `concepts="llm"` is metacheck's behaviour.

## Labels

`reaction_time`, `accuracy`, `age`, `gender`, `race`, `likert`, `condition`, `id`, `date`,
`timestamp`, `measure`, `other`, in the order of `concept_model.json`. `measure` (some other
measured variable) and `other` mean "no concept": pytacheck leaves those columns blank, as it does
when the LLM gives these answers.

## Input

The model was trained on text in exactly this form (pytacheck's `concept_text()` builds it):

```
column: {name} | file: {basename} | neighbours: {3 before, 2 after} | n=.. missing=.. unique=.. min=.. max=.. mean=.. sd=.. type={numeric|text|...} | values: {k} distinct in sample: {v1} ; {v2} ; ...
```

- The values are up to 200 evenly spaced non-missing values, each cut to 200 characters. The
  first 40 distinct ones are listed, cut to 40 characters.
- Statistics are printed to 4 significant digits.
- The input is truncated at 256 tokens.

## Files

- `model.onnx`: inputs are `input_ids` and `attention_mask` (int64, batch × tokens). The output
  `probs` holds the softmax probabilities (float32, batch × 12).
  - The fp32 export (opset 18) has its matmuls quantised weight-only: MatMulNBits, 8 bits, symmetric
    blocks of 32, `accuracy_level` 1, so the arithmetic stays fp32.
  - The embedding tables are stored as fp16 and cast back to fp32 after the lookup.
  - Against the PyTorch fp32 model it agrees on the top label for 320/320 reference columns (mean
    absolute difference of the top probability 0.0023). Its confidences are therefore usable for
    the cascade threshold.
  - onnxruntime's dynamic int8 quantisation was not used: XLM-R's outlier activations break it,
    and it cost 0.023 F1 in the cascade.
- `tokenizer.json`: XLM-RoBERTa's tokenizer, for the `tokenizers` library.
- `concept_model.json`: labels, `max_len`, `pad_id`, the default cascade threshold, the
  quantisation and the training settings.

On CPU it takes about 0.1 s per column with 8 threads and about 2 GB of memory.

## Training

- Base model: [FacebookAI/xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large),
  fine-tuned for sequence classification.
- Settings: 2 epochs, learning rate 1.5e-5, batch size 32, max length 256, seed 0, bf16, on one GPU
  (10 minutes).
- Data: 107,643 columns of public research data files. The sources are ResearchBox boxes and OSF,
  GitHub and Zenodo repositories linked from psychology papers. They were collected by running
  metacheck's `data_check` with its LLM calls recorded instead of sent.
- Labels: Muse Spark 1.3 Contributor, at reasoning effort xhigh, with the labelling prompt. For
  the 2,145 benchmark columns below, the benchmark's gold labels replace Muse's.

## Evaluation

The same recipe was scored without the evaluation repositories:
- **RB test:** 98 ResearchBox boxes, 961 columns, 5-fold cross-fit by box.
- **OOD:** 86 OSF/GitHub/Zenodo repositories, 812 columns, none of them in training.

The F1 combines the precision of the concepts written with the recall of the columns that have
one. `measure` and `other` count as blank, and each repository is weighted equally.

| system | RB test | OOD |
|---|---|---|
| this model (the recipe, trained without the evaluation repositories) | 0.795 | 0.736-0.761 (two runs) |
| Muse Spark 1.3 Contributor, metacheck's prompt | 0.784 | 0.740 |
| Gemini 3.5 Flash-Lite, metacheck's prompt | 0.762 | 0.694 |
| gpt-oss-20b, metacheck's prompt | 0.613 | 0.515 |
| this model, then Muse Spark below confidence 0.92 (~37% of columns sent) | 0.852 | 0.786-0.801 |

On its own the model is level with Muse Spark: the differences (+0.011 RB, +0.021 OOD) are within
their repository-clustered 95% confidence intervals. What it adds is that it runs offline, costs
nothing to run, and gives the same answer every time.

## Limitations

- Most training columns come from English-language psychology data. The base model is
  multilingual, but other languages and fields are less tested.
- The model leans on column names. For example, a lowercase `rt` column next to
  `subject, block, trial` can come out as `id`. metacheck's rules label such columns before the
  model sees them.
- The labels are an LLM's (Muse Spark), not people's, and the model inherits its mistakes.
- The input contains values from the data file. The model runs locally, and nothing is sent
  anywhere.

## License

AGPL-3.0-or-later, as pytacheck and metacheck. The base model, XLM-RoBERTa-large, is MIT-licensed.