README.md

---
license: mit
language:
- ar
- it
- de
- en
- hi
- fi
- fr
- tr
- es
- pt
- pl
base_model:
- jhu-clsp/mmBERT-base
pipeline_tag: token-classification
tags:
- ner
- pii
- de-identification
- multilingual
- modernbert
- crf
- flert
---

# Multilingual DialogPII NER

A fine-tuned [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) model with a CRF layer for **Personally Identifiable Information (PII) detection** in multilingual dialogues across 11 languages.

## Model Description

This model performs token-level Named Entity Recognition (NER) to identify and classify PII entities in dialogue text. It was trained on synthetic multilingual de-identification of conversational data.

- **Architecture:** mmBERT-base (ModernBERT) + CRF head with FLERT context windowing
- **Training:** Fine-tuned on all 11 languages jointly (multilingual training) using FLERT-style document context
- **Loss:** Cross-Entropy
- **Hyperparameters:** lr=2e-05, batch_size=32, max_length=2048, dropout=0.1, epochs=10
- **Context window:** 2 sentences left + 2 sentences right, separated by `[SEP]` markers
- **Decoding:** Viterbi decoding via CRF layer

## Supported Languages

| Code | Language |
|------|----------|
| AR | Arabic |
| DE | German |
| EN | English |
| FI | Finnish |
| FR | French |
| HI | Hindi |
| IT | Italian |
| PL | Polish |
| PT | Portuguese |
| SP | Spanish |
| TR | Turkish |

## Entity Types

The model recognizes 19 PII entity types using BIO tagging:

| Entity | Description |
|--------|-------------|
| `PERSON` | Person names |
| `PERSON_EMAIL` | Email addresses |
| `PERSON_SOCIAL_RELATION` | Social relations (e.g., "my wife") |
| `ORG` | Organizations |
| `LOC_CITY` | Cities |
| `LOC_COUNTRY` | Countries |
| `LOC_STREET` | Street names |
| `LOC_ZIP` | ZIP/postal codes |
| `LOC_HOUSENUMBER` | House numbers |
| `LOC_OTHER` | Other locations |
| `DATETIME` | Dates and times |
| `DATETIME_AGE` | Ages |
| `CODE` | ID numbers, reference codes |
| `CODE_PHONE` | Phone numbers |
| `CODE_URL` | URLs |
| `PROFESSION` | Professions |
| `PRODUCT` | Product names |
| `QUANTITY` | Quantities |
| `MISC` | Miscellaneous PII |

## Performance

Evaluated on held-out test sets per language (type-aware micro scores):

| Language | Len P | Len R | Len F1 | Len F2 | Ex P | Ex R | Ex F1 | Ex F2 |
|----------|-------|-------|--------|--------|------|------|-------|-------|
| AR | 87.87 | 73.15 | 79.84 | 75.69 | 84.45 | 70.30 | 76.73 | 72.74 |
| DE | 94.12 | 90.66 | 92.36 | 91.33 | 93.33 | 89.90 | 91.58 | 90.56 |
| EN | 94.93 | 93.45 | 94.18 | 93.74 | 92.41 | 90.97 | 91.69 | 91.25 |
| FI | 91.36 | 88.46 | 89.89 | 89.03 | 89.93 | 87.07 | 88.48 | 87.63 |
| FR | 90.91 | 88.09 | 89.48 | 88.64 | 87.66 | 84.94 | 86.28 | 85.47 |
| HI | 87.55 | 82.33 | 84.86 | 83.33 | 83.37 | 78.40 | 80.81 | 79.35 |
| IT | 93.57 | 87.81 | 90.60 | 88.90 | 90.72 | 85.13 | 87.84 | 86.19 |
| PL | 90.11 | 90.31 | 90.21 | 90.27 | 87.41 | 87.61 | 87.51 | 87.57 |
| PT | 91.10 | 90.69 | 90.90 | 90.77 | 89.28 | 88.88 | 89.08 | 88.96 |
| SP | 93.06 | 91.47 | 92.26 | 91.79 | 91.30 | 89.74 | 90.51 | 90.05 |
| TR | 89.13 | 86.53 | 87.81 | 87.04 | 85.79 | 83.29 | 84.52 | 83.78 |
| **AVG** | **91.25** | **87.54** | **89.31** | **88.23** | **88.70** | **85.11** | **86.82** | **85.78** |

## Usage

This model uses a custom CRF architecture with FLERT-style context windowing and cannot be loaded directly with `AutoModelForTokenClassification`. You need to use the custom `ModernBertCRF` class.

> **Note:** The `config.json` in this repo exists solely for Hugging Face download tracking. For model loading, use `crf_config.json` and `flert_config.json` instead.

### Setup

```python
import torch
import json
import re
import torch.nn as nn
import spacy
from transformers import AutoModel, AutoTokenizer
from torchcrf import CRF
from huggingface_hub import snapshot_download

class ModernBertCRF(nn.Module):
    def __init__(self, base_model_name, num_labels, id2label, label2id):
        super().__init__()
        self.num_labels = num_labels
        self.id2label = id2label
        self.label2id = label2id
        self.transformer = AutoModel.from_pretrained(base_model_name)
        hidden_size = self.transformer.config.hidden_size
        self.classifier = nn.Linear(hidden_size, num_labels)
        self.dropout = nn.Dropout(0.1)
        self.crf = CRF(num_labels, batch_first=True)

    def forward(self, input_ids, attention_mask, labels=None, **kwargs):
        kwargs.pop("token_type_ids", None)
        outputs = self.transformer(input_ids=input_ids, attention_mask=attention_mask)
        sequence_output = self.dropout(outputs.last_hidden_state)
        emissions = self.classifier(sequence_output)
        return {"logits": emissions}

    def decode(self, emissions, mask):
        return self.crf.decode(emissions, mask=mask)

# Load model
model_dir = snapshot_download("DFKI-SLT/multilingual_DialogPII_NER")

with open(f"{model_dir}/crf_config.json") as f:
    config = json.load(f)

with open(f"{model_dir}/flert_config.json") as f:
    flert_config = json.load(f)

model = ModernBertCRF(
    base_model_name=config["base_model_name"],
    num_labels=config["num_labels"],
    id2label=config["id2label"],
    label2id=config["label2id"],
)
model.load_state_dict(torch.load(f"{model_dir}/pytorch_model.bin", map_location="cpu"))
model.eval()

tokenizer = AutoTokenizer.from_pretrained(model_dir)
id2label = {int(k): v for k, v in config["id2label"].items()}

context_window = flert_config["context_window"]     # 2
use_sep_marker = flert_config["context_sep_marker"]  # True
```

### Preprocessing: Sentence Splitting

The model was trained using FLERT-style context windowing over **sentence-level** input. Each sentence is predicted with surrounding context sentences. For best results, split your input into sentences using [spaCy](https://spacy.io/) before inference.

```python
nlp = spacy.blank("en")          # use "de" for German, "xx" for multilingual
nlp.add_pipe("sentencizer")

def split_dialogue(text, nlp):
    sentences = []
    for line in text.strip().splitlines():
        m = re.match(r"^(SPEAKER_\d+)\s*:\s*(.*)", line.strip())
        if m:
            speaker, rest = m.group(1), m.group(2)
            sentences.append([speaker, ":"])
            line = rest
        if not line:
            continue
        doc = nlp(line)
        for sent in doc.sents:
            tokens = [tok.text for tok in sent if not tok.is_space]
            if tokens:
                sentences.append(tokens)
    return sentences

# Example
raw = """SPEAKER_00: Hello, my name is Peter.
SPEAKER_01: Hello, my name is Peter as well. Okay, and where do you come from? I come from Chicago."""

sentences = split_dialogue(raw, nlp)
```

### Inference with FLERT Context Windowing

The key difference from standard token classification: each sentence is predicted within a window of surrounding context sentences, joined by `[SEP]` tokens. Only labels for the target sentence are extracted.

```python
def predict_dialogue(sentences, model, tokenizer, id2label,
                     context_window=2, use_sep_marker=True, device="cpu"):
    sep = tokenizer.sep_token
    all_labels = []
    for i, target_tokens in enumerate(sentences):
        left  = sentences[max(0, i - context_window):i]
        right = sentences[i + 1:i + 1 + context_window]

        flat_tokens = []
        for s in left:
            flat_tokens.extend(s)
        if use_sep_marker and left:
            flat_tokens.append(sep)

        tgt_start = len(flat_tokens)
        flat_tokens.extend(target_tokens)
        tgt_end = len(flat_tokens)

        if use_sep_marker and right:
            flat_tokens.append(sep)
        for s in right:
            flat_tokens.extend(s)

        enc = tokenizer(flat_tokens, is_split_into_words=True,
                        return_tensors="pt", truncation=False).to(device)
        word_ids = enc.word_ids(batch_index=0)

        with torch.no_grad():
            emissions = model(**enc)["logits"]
            mask = enc["attention_mask"].bool()
            preds = model.decode(emissions, mask)[0]

        word_labels = ["O"] * len(target_tokens)
        seen = set()
        for idx, wid in enumerate(word_ids):
            if wid is None or wid in seen:
                continue
            seen.add(wid)
            if tgt_start <= wid < tgt_end:
                word_labels[wid - tgt_start] = id2label[preds[idx]]

        all_labels.append(word_labels)
    return all_labels


# Run prediction
results = predict_dialogue(sentences, model, tokenizer, id2label,
                           context_window=context_window,
                           use_sep_marker=use_sep_marker)

for sent_tokens, sent_labels in zip(sentences, results):
    for token, label in zip(sent_tokens, sent_labels):
        if label != "O":
            print(f"{token:20s} -> {label}")
```

### Single-sentence inference

For isolated sentences without dialogue context, pass them with `context_window=0`:

```python
tokens = ["My", "name", "is", "John", "Smith", "and", "I", "live", "in", "Berlin", "."]

results = predict_dialogue([tokens], model, tokenizer, id2label,
                           context_window=0, use_sep_marker=False)

for token, label in zip(tokens, results[0]):
    if label != "O":
        print(f"{token:20s} -> {label}")
```

## Training Data

The model was trained on synthetic multilingual dialogue data covering various domains (medical anamnesis, customer support, police reports, therapy sessions, etc.). The data was generated and annotated as part of a thesis project on multilingual PII de-identification.

## Limitations

- Trained on synthetic dialogue data; performance on real-world data may vary
- Optimized for dialogue/conversational text; may underperform on formal documents
- Arabic and Hindi show lower performance compared to European languages
- Requires `pytorch-crf` package for inference

## Citation

If you use this model, please cite:

```
@misc{roller2026multilingual,
  title={DialogPII: A multilingual dataset of synthetic dialog transcripts to detect personal information},
  author={Roland Roller and Vera Czehmann and Derya Erman and Luke Flanagan and Ibrahim Baroud and Fr{\'e}d{\'e}ric Blain and Viviana Cotik and Eletta Giusto and Akhil Juneja and Mariana Neves and Maria S{\l}owi{\'n}ska and Christine Hovhannisyan and Aaron Louis Eidt and Lisa Raithel and Sebastian M{\"o}ller and Maija Poikela},
  year={2026},
  institution={DFKI SLT}
}
```