pii-detect interim (ckpt-4000, epoch 0.23) + public-eval leaderboard

README.md

@@ -0,0 +1,130 @@
+---
+license: other
+language: [en, es, fr, de, it, pt, nl, ru, ar, hi, ja, ko, zh, th, id, vi, tr, pl]
+pipeline_tag: token-classification
+tags: [pii, ner, token-classification, lfm2, multilingual, privacy]
+---
+
+# LiquidAI/pii-detect
+
+Multilingual PII detection — a **full fine-tune of the bidirectional LFM2.5-350M MLM
+encoder** (`v13_encoder_mlm`) with a BIOES token-classification head. Detects ~27
+canonical PII types across 7 domains and 18 languages.
+
+> **Status:** pilot / interim checkpoint (trained on 10M synthetic multilingual docs).
+> Private — LiquidAI internal.
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForTokenClassification
+import torch
+
+tok = AutoTokenizer.from_pretrained("LiquidAI/pii-detect", trust_remote_code=True)
+model = AutoModelForTokenClassification.from_pretrained(
+    "LiquidAI/pii-detect", trust_remote_code=True).eval()
+
+text = "Email Jane Doe at jane@acme.com or call +1-202-555-0142."
+enc = tok(text, return_offsets_mapping=True, return_tensors="pt")
+off = enc.pop("offset_mapping")[0]
+with torch.no_grad():
+    pred = model(**enc).logits[0].argmax(-1)
+labels = [model.config.id2label[i] for i in pred.tolist()]
+# merge adjacent same-type tokens into spans using `off` + BIOES `labels`
+```
+
+- **Labels:** 109 BIOES classes (27 types × B/I/E/S + O). See `id2label` in `config.json`.
+- **Backbone:** bidirectional LFM2.5-350M (MLM-pretrained), full fine-tune (no LoRA).
+- **Training data:** 10M synthetic multilingual PII documents (entity-first generation).
+
+## Public benchmark leaderboard
+
+Type-level **micro** scores, **partial-match**. P=precision, R=recall (PII caught),
+F1, F2 (recall-weighted β=2 — the metric that matters most for PII). Ours is an
+**interim checkpoint (≈¼ of epoch 1)** vs **fully-trained** released baselines.
+
+### SPY  (969 samples)
+_clean · synthetic · tests PII-vs-NER discrimination_
+
+| Model | P | R | F1 | F2 |
+|---|---|---|---|---|
+| **LiquidAI/pii-detect (ours)** | 0.387 | 0.799 | 0.521 | 0.658 |
+| regex baseline | 0.259 | 0.579 | 0.358 | 0.464 |
+| openai/privacy-filter | 0.169 | 0.595 | 0.264 | 0.396 |
+| VAGOsolutions/SauerkrautLM-GLiNER | 0.160 | 0.513 | 0.244 | 0.356 |
+| iiiorg/piiranha-v1 | 0.159 | 0.422 | 0.230 | 0.317 |
+| OpenMed/privacy-filter-nemotron | 0.143 | 0.545 | 0.226 | 0.348 |
+| LiquidAI demo (LFM2.5-PII-Demo) | 0.132 | 0.350 | 0.192 | 0.263 |
+
+### GRETEL  (500 samples)
+_clean · synthetic English_
+
+| Model | P | R | F1 | F2 |
+|---|---|---|---|---|
+| LiquidAI demo (LFM2.5-PII-Demo) | 0.887 | 0.735 | 0.804 | 0.761 |
+| OpenMed/privacy-filter-nemotron | 0.762 | 0.773 | 0.767 | 0.770 |
+| VAGOsolutions/SauerkrautLM-GLiNER | 0.722 | 0.686 | 0.703 | 0.693 |
+| **LiquidAI/pii-detect (ours)** | 0.705 | 0.621 | 0.660 | 0.636 |
+| iiiorg/piiranha-v1 | 0.633 | 0.490 | 0.552 | 0.513 |
+| openai/privacy-filter | 0.455 | 0.462 | 0.459 | 0.461 |
+| regex baseline | 0.500 | 0.254 | 0.337 | 0.281 |
+
+### TAB  (127 samples)
+_clean · human gold · English legal (ECHR)_
+
+| Model | P | R | F1 | F2 |
+|---|---|---|---|---|
+| OpenMed/privacy-filter-nemotron | 0.728 | 0.625 | 0.673 | 0.643 |
+| VAGOsolutions/SauerkrautLM-GLiNER | 0.984 | 0.406 | 0.575 | 0.460 |
+| openai/privacy-filter | 0.933 | 0.383 | 0.543 | 0.434 |
+| LiquidAI demo (LFM2.5-PII-Demo) | 0.995 | 0.325 | 0.490 | 0.375 |
+| iiiorg/piiranha-v1 | 0.721 | 0.160 | 0.262 | 0.189 |
+| **LiquidAI/pii-detect (ours)** | 0.857 | 0.138 | 0.238 | 0.166 |
+| regex baseline | 0.000 | 0.000 | 0.000 | 0.000 |
+
+### MAPA  (370 samples)
+_clean · human gold · multilingual EU legal_
+
+| Model | P | R | F1 | F2 |
+|---|---|---|---|---|
+| VAGOsolutions/SauerkrautLM-GLiNER | 0.527 | 0.194 | 0.283 | 0.222 |
+| iiiorg/piiranha-v1 | 0.465 | 0.181 | 0.261 | 0.206 |
+| openai/privacy-filter | 0.372 | 0.186 | 0.248 | 0.207 |
+| LiquidAI demo (LFM2.5-PII-Demo) | 0.274 | 0.140 | 0.185 | 0.155 |
+| OpenMed/privacy-filter-nemotron | 0.081 | 0.180 | 0.111 | 0.144 |
+| **LiquidAI/pii-detect (ours)** | 0.325 | 0.058 | 0.099 | 0.070 |
+| regex baseline | 0.000 | 0.000 | 0.000 | 0.000 |
+
+
+### Contaminated benchmarks (reference only)
+### NEMOTRON  (1000 samples)
+_⚠ in-distribution for OpenMed-nemotron & nvidia GLiNER (read as ceilings)_
+
+| Model | P | R | F1 | F2 |
+|---|---|---|---|---|
+| OpenMed/privacy-filter-nemotron | 0.970 | 0.872 | 0.918 | 0.890 |
+| LiquidAI demo (LFM2.5-PII-Demo) | 0.894 | 0.518 | 0.656 | 0.566 |
+| iiiorg/piiranha-v1 | 0.841 | 0.538 | 0.656 | 0.580 |
+| VAGOsolutions/SauerkrautLM-GLiNER | 0.815 | 0.533 | 0.644 | 0.572 |
+| openai/privacy-filter | 0.699 | 0.484 | 0.572 | 0.516 |
+| **LiquidAI/pii-detect (ours)** | 0.754 | 0.429 | 0.547 | 0.470 |
+| regex baseline | 0.645 | 0.227 | 0.335 | 0.260 |
+
+### AI4PRIVACY  (660 samples)
+_⚠ in OUR backbone's pretraining — contaminated for us; reference only_
+
+| Model | P | R | F1 | F2 |
+|---|---|---|---|---|
+| iiiorg/piiranha-v1 | 0.969 | 0.924 | 0.946 | 0.932 |
+| LiquidAI demo (LFM2.5-PII-Demo) | 0.533 | 0.464 | 0.496 | 0.476 |
+| VAGOsolutions/SauerkrautLM-GLiNER | 0.399 | 0.614 | 0.484 | 0.554 |
+| OpenMed/privacy-filter-nemotron | 0.341 | 0.541 | 0.418 | 0.484 |
+| openai/privacy-filter | 0.316 | 0.492 | 0.385 | 0.443 |
+| **LiquidAI/pii-detect (ours)** | 0.280 | 0.301 | 0.290 | 0.296 |
+| regex baseline | 0.269 | 0.147 | 0.190 | 0.162 |
+
+
+## Notes
+- **Strength:** best-in-class on **SPY** (PII-vs-NER discrimination) with the highest recall of any model tested.
+- **Gap:** legal-domain benchmarks (TAB/MAPA) — high precision but low recall; targeted for the next data iteration.
+- Benchmarks use differing entity taxonomies; our fine-grained types are mapped to each via an identity/crosswalk, so type-level F1 understates raw detection (binary detection is higher).

config.json

@@ -0,0 +1,289 @@
+{
+  "architectures": [
+    "Lfm2BidirForTokenClassification"
+  ],
+  "auto_map": {
+    "AutoModel": "modeling_lfm2_tc.Lfm2BidirModel",
+    "AutoModelForTokenClassification": "modeling_lfm2_tc.Lfm2BidirForTokenClassification"
+  },
+  "block_auto_adjust_ff_dim": true,
+  "block_dim": 1024,
+  "block_ffn_dim_multiplier": 1.0,
+  "block_mlp_init_scale": 1.0,
+  "block_multiple_of": 256,
+  "block_norm_eps": 1e-05,
+  "block_out_init_scale": 1.0,
+  "block_use_swiglu": true,
+  "block_use_xavier_init": true,
+  "bos_token_id": 1,
+  "classifier_dropout": 0.1,
+  "conv_L_cache": 3,
+  "conv_bias": false,
+  "conv_dim": 1024,
+  "conv_dim_out": 1024,
+  "conv_use_xavier_init": true,
+  "dtype": "float32",
+  "eos_token_id": 7,
+  "full_attn_idxs": null,
+  "hidden_size": 1024,
+  "id2label": {
+    "0": "O",
+    "1": "B-contact.address",
+    "2": "I-contact.address",
+    "3": "E-contact.address",
+    "4": "S-contact.address",
+    "5": "B-contact.email",
+    "6": "I-contact.email",
+    "7": "E-contact.email",
+    "8": "S-contact.email",
+    "9": "B-contact.ip_address",
+    "10": "I-contact.ip_address",
+    "11": "E-contact.ip_address",
+    "12": "S-contact.ip_address",
+    "13": "B-contact.phone",
+    "14": "I-contact.phone",
+    "15": "E-contact.phone",
+    "16": "S-contact.phone",
+    "17": "B-contact.postal_code",
+    "18": "I-contact.postal_code",
+    "19": "E-contact.postal_code",
+    "20": "S-contact.postal_code",
+    "21": "B-credential.api_key",
+    "22": "I-credential.api_key",
+    "23": "E-credential.api_key",
+    "24": "S-credential.api_key",
+    "25": "B-device.mac_address",
+    "26": "I-device.mac_address",
+    "27": "E-device.mac_address",
+    "28": "S-device.mac_address",
+    "29": "B-financial.bank_account",
+    "30": "I-financial.bank_account",
+    "31": "E-financial.bank_account",
+    "32": "S-financial.bank_account",
+    "33": "B-financial.credit_card",
+    "34": "I-financial.credit_card",
+    "35": "E-financial.credit_card",
+    "36": "S-financial.credit_card",
+    "37": "B-financial.iban",
+    "38": "I-financial.iban",
+    "39": "E-financial.iban",
+    "40": "S-financial.iban",
+    "41": "B-financial.swift_bic",
+    "42": "I-financial.swift_bic",
+    "43": "E-financial.swift_bic",
+    "44": "S-financial.swift_bic",
+    "45": "B-healthcare.condition",
+    "46": "I-healthcare.condition",
+    "47": "E-healthcare.condition",
+    "48": "S-healthcare.condition",
+    "49": "B-healthcare.medical_record",
+    "50": "I-healthcare.medical_record",
+    "51": "E-healthcare.medical_record",
+    "52": "S-healthcare.medical_record",
+    "53": "B-healthcare.medication",
+    "54": "I-healthcare.medication",
+    "55": "E-healthcare.medication",
+    "56": "S-healthcare.medication",
+    "57": "B-identity.date_of_birth",
+    "58": "I-identity.date_of_birth",
+    "59": "E-identity.date_of_birth",
+    "60": "S-identity.date_of_birth",
+    "61": "B-identity.drivers_license",
+    "62": "I-identity.drivers_license",
+    "63": "E-identity.drivers_license",
+    "64": "S-identity.drivers_license",
+    "65": "B-identity.national_id",
+    "66": "I-identity.national_id",
+    "67": "E-identity.national_id",
+    "68": "S-identity.national_id",
+    "69": "B-identity.passport",
+    "70": "I-identity.passport",
+    "71": "E-identity.passport",
+    "72": "S-identity.passport",
+    "73": "B-identity.person_name",
+    "74": "I-identity.person_name",
+    "75": "E-identity.person_name",
+    "76": "S-identity.person_name",
+    "77": "B-identity.ssn",
+    "78": "I-identity.ssn",
+    "79": "E-identity.ssn",
+    "80": "S-identity.ssn",
+    "81": "B-legal.case_number",
+    "82": "I-legal.case_number",
+    "83": "E-legal.case_number",
+    "84": "S-legal.case_number",
+    "85": "B-location.gps_coordinates",
+    "86": "I-location.gps_coordinates",
+    "87": "E-location.gps_coordinates",
+    "88": "S-location.gps_coordinates",
+    "89": "B-online.url",
+    "90": "I-online.url",
+    "91": "E-online.url",
+    "92": "S-online.url",
+    "93": "B-online.username",
+    "94": "I-online.username",
+    "95": "E-online.username",
+    "96": "S-online.username",
+    "97": "B-org.company_name",
+    "98": "I-org.company_name",
+    "99": "E-org.company_name",
+    "100": "S-org.company_name",
+    "101": "B-special.political",
+    "102": "I-special.political",
+    "103": "E-special.political",
+    "104": "S-special.political",
+    "105": "B-special.religion",
+    "106": "I-special.religion",
+    "107": "E-special.religion",
+    "108": "S-special.religion"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 6656,
+  "label2id": {
+    "B-contact.address": 1,
+    "B-contact.email": 5,
+    "B-contact.ip_address": 9,
+    "B-contact.phone": 13,
+    "B-contact.postal_code": 17,
+    "B-credential.api_key": 21,
+    "B-device.mac_address": 25,
+    "B-financial.bank_account": 29,
+    "B-financial.credit_card": 33,
+    "B-financial.iban": 37,
+    "B-financial.swift_bic": 41,
+    "B-healthcare.condition": 45,
+    "B-healthcare.medical_record": 49,
+    "B-healthcare.medication": 53,
+    "B-identity.date_of_birth": 57,
+    "B-identity.drivers_license": 61,
+    "B-identity.national_id": 65,
+    "B-identity.passport": 69,
+    "B-identity.person_name": 73,
+    "B-identity.ssn": 77,
+    "B-legal.case_number": 81,
+    "B-location.gps_coordinates": 85,
+    "B-online.url": 89,
+    "B-online.username": 93,
+    "B-org.company_name": 97,
+    "B-special.political": 101,
+    "B-special.religion": 105,
+    "E-contact.address": 3,
+    "E-contact.email": 7,
+    "E-contact.ip_address": 11,
+    "E-contact.phone": 15,
+    "E-contact.postal_code": 19,
+    "E-credential.api_key": 23,
+    "E-device.mac_address": 27,
+    "E-financial.bank_account": 31,
+    "E-financial.credit_card": 35,
+    "E-financial.iban": 39,
+    "E-financial.swift_bic": 43,
+    "E-healthcare.condition": 47,
+    "E-healthcare.medical_record": 51,
+    "E-healthcare.medication": 55,
+    "E-identity.date_of_birth": 59,
+    "E-identity.drivers_license": 63,
+    "E-identity.national_id": 67,
+    "E-identity.passport": 71,
+    "E-identity.person_name": 75,
+    "E-identity.ssn": 79,
+    "E-legal.case_number": 83,
+    "E-location.gps_coordinates": 87,
+    "E-online.url": 91,
+    "E-online.username": 95,
+    "E-org.company_name": 99,
+    "E-special.political": 103,
+    "E-special.religion": 107,
+    "I-contact.address": 2,
+    "I-contact.email": 6,
+    "I-contact.ip_address": 10,
+    "I-contact.phone": 14,
+    "I-contact.postal_code": 18,
+    "I-credential.api_key": 22,
+    "I-device.mac_address": 26,
+    "I-financial.bank_account": 30,
+    "I-financial.credit_card": 34,
+    "I-financial.iban": 38,
+    "I-financial.swift_bic": 42,
+    "I-healthcare.condition": 46,
+    "I-healthcare.medical_record": 50,
+    "I-healthcare.medication": 54,
+    "I-identity.date_of_birth": 58,
+    "I-identity.drivers_license": 62,
+    "I-identity.national_id": 66,
+    "I-identity.passport": 70,
+    "I-identity.person_name": 74,
+    "I-identity.ssn": 78,
+    "I-legal.case_number": 82,
+    "I-location.gps_coordinates": 86,
+    "I-online.url": 90,
+    "I-online.username": 94,
+    "I-org.company_name": 98,
+    "I-special.political": 102,
+    "I-special.religion": 106,
+    "O": 0,
+    "S-contact.address": 4,
+    "S-contact.email": 8,
+    "S-contact.ip_address": 12,
+    "S-contact.phone": 16,
+    "S-contact.postal_code": 20,
+    "S-credential.api_key": 24,
+    "S-device.mac_address": 28,
+    "S-financial.bank_account": 32,
+    "S-financial.credit_card": 36,
+    "S-financial.iban": 40,
+    "S-financial.swift_bic": 44,
+    "S-healthcare.condition": 48,
+    "S-healthcare.medical_record": 52,
+    "S-healthcare.medication": 56,
+    "S-identity.date_of_birth": 60,
+    "S-identity.drivers_license": 64,
+    "S-identity.national_id": 68,
+    "S-identity.passport": 72,
+    "S-identity.person_name": 76,
+    "S-identity.ssn": 80,
+    "S-legal.case_number": 84,
+    "S-location.gps_coordinates": 88,
+    "S-online.url": 92,
+    "S-online.username": 96,
+    "S-org.company_name": 100,
+    "S-special.political": 104,
+    "S-special.religion": 108
+  },
+  "label_smoothing": 0.1,
+  "layer_types": [
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv",
+    "full_attention",
+    "conv"
+  ],
+  "max_position_embeddings": 128000,
+  "model_type": "lfm2",
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_heads": 16,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.11.0",
+  "use_cache": false,
+  "use_pos_enc": true,
+  "vocab_size": 65536
+}

label_schema.json

@@ -0,0 +1,255 @@
+{
+  "num_labels": 109,
+  "scheme": "BIOES",
+  "types_in_order": [
+    "contact.address",
+    "contact.email",
+    "contact.ip_address",
+    "contact.phone",
+    "contact.postal_code",
+    "credential.api_key",
+    "device.mac_address",
+    "financial.bank_account",
+    "financial.credit_card",
+    "financial.iban",
+    "financial.swift_bic",
+    "healthcare.condition",
+    "healthcare.medical_record",
+    "healthcare.medication",
+    "identity.date_of_birth",
+    "identity.drivers_license",
+    "identity.national_id",
+    "identity.passport",
+    "identity.person_name",
+    "identity.ssn",
+    "legal.case_number",
+    "location.gps_coordinates",
+    "online.url",
+    "online.username",
+    "org.company_name",
+    "special.political",
+    "special.religion"
+  ],
+  "id2label": {
+    "0": "O",
+    "1": "B-contact.address",
+    "2": "I-contact.address",
+    "3": "E-contact.address",
+    "4": "S-contact.address",
+    "5": "B-contact.email",
+    "6": "I-contact.email",
+    "7": "E-contact.email",
+    "8": "S-contact.email",
+    "9": "B-contact.ip_address",
+    "10": "I-contact.ip_address",
+    "11": "E-contact.ip_address",
+    "12": "S-contact.ip_address",
+    "13": "B-contact.phone",
+    "14": "I-contact.phone",
+    "15": "E-contact.phone",
+    "16": "S-contact.phone",
+    "17": "B-contact.postal_code",
+    "18": "I-contact.postal_code",
+    "19": "E-contact.postal_code",
+    "20": "S-contact.postal_code",
+    "21": "B-credential.api_key",
+    "22": "I-credential.api_key",
+    "23": "E-credential.api_key",
+    "24": "S-credential.api_key",
+    "25": "B-device.mac_address",
+    "26": "I-device.mac_address",
+    "27": "E-device.mac_address",
+    "28": "S-device.mac_address",
+    "29": "B-financial.bank_account",
+    "30": "I-financial.bank_account",
+    "31": "E-financial.bank_account",
+    "32": "S-financial.bank_account",
+    "33": "B-financial.credit_card",
+    "34": "I-financial.credit_card",
+    "35": "E-financial.credit_card",
+    "36": "S-financial.credit_card",
+    "37": "B-financial.iban",
+    "38": "I-financial.iban",
+    "39": "E-financial.iban",
+    "40": "S-financial.iban",
+    "41": "B-financial.swift_bic",
+    "42": "I-financial.swift_bic",
+    "43": "E-financial.swift_bic",
+    "44": "S-financial.swift_bic",
+    "45": "B-healthcare.condition",
+    "46": "I-healthcare.condition",
+    "47": "E-healthcare.condition",
+    "48": "S-healthcare.condition",
+    "49": "B-healthcare.medical_record",
+    "50": "I-healthcare.medical_record",
+    "51": "E-healthcare.medical_record",
+    "52": "S-healthcare.medical_record",
+    "53": "B-healthcare.medication",
+    "54": "I-healthcare.medication",
+    "55": "E-healthcare.medication",
+    "56": "S-healthcare.medication",
+    "57": "B-identity.date_of_birth",
+    "58": "I-identity.date_of_birth",
+    "59": "E-identity.date_of_birth",
+    "60": "S-identity.date_of_birth",
+    "61": "B-identity.drivers_license",
+    "62": "I-identity.drivers_license",
+    "63": "E-identity.drivers_license",
+    "64": "S-identity.drivers_license",
+    "65": "B-identity.national_id",
+    "66": "I-identity.national_id",
+    "67": "E-identity.national_id",
+    "68": "S-identity.national_id",
+    "69": "B-identity.passport",
+    "70": "I-identity.passport",
+    "71": "E-identity.passport",
+    "72": "S-identity.passport",
+    "73": "B-identity.person_name",
+    "74": "I-identity.person_name",
+    "75": "E-identity.person_name",
+    "76": "S-identity.person_name",
+    "77": "B-identity.ssn",
+    "78": "I-identity.ssn",
+    "79": "E-identity.ssn",
+    "80": "S-identity.ssn",
+    "81": "B-legal.case_number",
+    "82": "I-legal.case_number",
+    "83": "E-legal.case_number",
+    "84": "S-legal.case_number",
+    "85": "B-location.gps_coordinates",
+    "86": "I-location.gps_coordinates",
+    "87": "E-location.gps_coordinates",
+    "88": "S-location.gps_coordinates",
+    "89": "B-online.url",
+    "90": "I-online.url",
+    "91": "E-online.url",
+    "92": "S-online.url",
+    "93": "B-online.username",
+    "94": "I-online.username",
+    "95": "E-online.username",
+    "96": "S-online.username",
+    "97": "B-org.company_name",
+    "98": "I-org.company_name",
+    "99": "E-org.company_name",
+    "100": "S-org.company_name",
+    "101": "B-special.political",
+    "102": "I-special.political",
+    "103": "E-special.political",
+    "104": "S-special.political",
+    "105": "B-special.religion",
+    "106": "I-special.religion",
+    "107": "E-special.religion",
+    "108": "S-special.religion"
+  },
+  "label2id": {
+    "O": 0,
+    "B-contact.address": 1,
+    "I-contact.address": 2,
+    "E-contact.address": 3,
+    "S-contact.address": 4,
+    "B-contact.email": 5,
+    "I-contact.email": 6,
+    "E-contact.email": 7,
+    "S-contact.email": 8,
+    "B-contact.ip_address": 9,
+    "I-contact.ip_address": 10,
+    "E-contact.ip_address": 11,
+    "S-contact.ip_address": 12,
+    "B-contact.phone": 13,
+    "I-contact.phone": 14,
+    "E-contact.phone": 15,
+    "S-contact.phone": 16,
+    "B-contact.postal_code": 17,
+    "I-contact.postal_code": 18,
+    "E-contact.postal_code": 19,
+    "S-contact.postal_code": 20,
+    "B-credential.api_key": 21,
+    "I-credential.api_key": 22,
+    "E-credential.api_key": 23,
+    "S-credential.api_key": 24,
+    "B-device.mac_address": 25,
+    "I-device.mac_address": 26,
+    "E-device.mac_address": 27,
+    "S-device.mac_address": 28,
+    "B-financial.bank_account": 29,
+    "I-financial.bank_account": 30,
+    "E-financial.bank_account": 31,
+    "S-financial.bank_account": 32,
+    "B-financial.credit_card": 33,
+    "I-financial.credit_card": 34,
+    "E-financial.credit_card": 35,
+    "S-financial.credit_card": 36,
+    "B-financial.iban": 37,
+    "I-financial.iban": 38,
+    "E-financial.iban": 39,
+    "S-financial.iban": 40,
+    "B-financial.swift_bic": 41,
+    "I-financial.swift_bic": 42,
+    "E-financial.swift_bic": 43,
+    "S-financial.swift_bic": 44,
+    "B-healthcare.condition": 45,
+    "I-healthcare.condition": 46,
+    "E-healthcare.condition": 47,
+    "S-healthcare.condition": 48,
+    "B-healthcare.medical_record": 49,
+    "I-healthcare.medical_record": 50,
+    "E-healthcare.medical_record": 51,
+    "S-healthcare.medical_record": 52,
+    "B-healthcare.medication": 53,
+    "I-healthcare.medication": 54,
+    "E-healthcare.medication": 55,
+    "S-healthcare.medication": 56,
+    "B-identity.date_of_birth": 57,
+    "I-identity.date_of_birth": 58,
+    "E-identity.date_of_birth": 59,
+    "S-identity.date_of_birth": 60,
+    "B-identity.drivers_license": 61,
+    "I-identity.drivers_license": 62,
+    "E-identity.drivers_license": 63,
+    "S-identity.drivers_license": 64,
+    "B-identity.national_id": 65,
+    "I-identity.national_id": 66,
+    "E-identity.national_id": 67,
+    "S-identity.national_id": 68,
+    "B-identity.passport": 69,
+    "I-identity.passport": 70,
+    "E-identity.passport": 71,
+    "S-identity.passport": 72,
+    "B-identity.person_name": 73,
+    "I-identity.person_name": 74,
+    "E-identity.person_name": 75,
+    "S-identity.person_name": 76,
+    "B-identity.ssn": 77,
+    "I-identity.ssn": 78,
+    "E-identity.ssn": 79,
+    "S-identity.ssn": 80,
+    "B-legal.case_number": 81,
+    "I-legal.case_number": 82,
+    "E-legal.case_number": 83,
+    "S-legal.case_number": 84,
+    "B-location.gps_coordinates": 85,
+    "I-location.gps_coordinates": 86,
+    "E-location.gps_coordinates": 87,
+    "S-location.gps_coordinates": 88,
+    "B-online.url": 89,
+    "I-online.url": 90,
+    "E-online.url": 91,
+    "S-online.url": 92,
+    "B-online.username": 93,
+    "I-online.username": 94,
+    "E-online.username": 95,
+    "S-online.username": 96,
+    "B-org.company_name": 97,
+    "I-org.company_name": 98,
+    "E-org.company_name": 99,
+    "S-org.company_name": 100,
+    "B-special.political": 101,
+    "I-special.political": 102,
+    "E-special.political": 103,
+    "S-special.political": 104,
+    "B-special.religion": 105,
+    "I-special.religion": 106,
+    "E-special.religion": 107,
+    "S-special.religion": 108
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ad4d7a4e9e465578b475961d312c4928a2a97448d9a67271be8536e691ade16
+size 1250624876

modeling_lfm2_tc.py

@@ -0,0 +1,449 @@
+"""
+Self-contained inference modeling for the Lfm2 bidirectional encoder MLM.
+
+Designed to be SHIPPED ALONGSIDE THE CHECKPOINT via trust_remote_code so that:
+
+  >>> from transformers import AutoModelForMaskedLM, AutoTokenizer, AutoModel
+  >>> tok = AutoTokenizer.from_pretrained("LiquidAI/v13_encoder_mlm_s131200", trust_remote_code=True)
+  >>> mlm = AutoModelForMaskedLM.from_pretrained("LiquidAI/v13_encoder_mlm_s131200", trust_remote_code=True)
+  >>> body = AutoModel.from_pretrained("LiquidAI/v13_encoder_mlm_s131200", trust_remote_code=True)
+
+works on a clean transformers install WITHOUT importing the training-time
+`encoder_lfm` package, AND without dead/unused parameters from the stripped
+`shortconv.in_proj` / `shortconv.out_proj` weights.
+
+The trained Lfm2ForMaskedLM (training-time) wraps upstream Lfm2Model + a
+runtime monkey-patch that replaces Lfm2ShortConv.forward with a symmetric
+depthwise conv on hidden_states — completely bypassing `in_proj`, `B*x`
+gating, `c*conv_out` gating, and `out_proj`. Those projections receive zero
+gradient during training and stay at random init. They are not used at
+inference. This file makes that explicit by declaring a fresh
+`Lfm2ShortConvBidir` module that has ONLY `conv.weight`, and swaps it into
+every shortconv slot at __init__.
+
+Bidirectional attention: replace the upstream `create_causal_mask` with a
+padding-only additive mask (no triangular component). MLM forward never has
+KV cache.
+
+Compatible with:
+  - transformers >= 5.0 (uses processing_class etc.)
+  - sentence-transformers (via AutoModel + mean pooling)
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Optional
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel
+from transformers.models.lfm2 import Lfm2Config
+from transformers.models.lfm2 import modeling_lfm2 as _lfm2_mod
+from transformers.modeling_outputs import BaseModelOutput, MaskedLMOutput
+
+# --------------------------------------------------------------------------- #
+#  Bidirectional attention mask                                               #
+# --------------------------------------------------------------------------- #
+
+
+def _bidirectional_mask(*args, **kwargs):
+    """Replacement for `transformers.models.lfm2.modeling_lfm2.create_causal_mask`.
+
+    Builds a NON-causal (padding-only) mask. On transformers >=5.1 this delegates
+    to the official, attention-impl-aware `create_bidirectional_mask` (handles
+    eager / sdpa / flash mask formats correctly). On older versions it falls back
+    to a hand-rolled (B, 1, Q, K) additive bias (zero everywhere, -1e9 at padding
+    key positions; no triangular component).
+
+    Accepts arbitrary args/kwargs because the upstream `create_causal_mask` call
+    signature changed across transformers releases (5.11 passes `inputs_embeds=`,
+    `position_ids=` by keyword).
+    """
+    try:
+        from transformers.masking_utils import create_bidirectional_mask
+        return create_bidirectional_mask(*args, **kwargs)
+    except ImportError:
+        pass
+    # ---- fallback for transformers without create_bidirectional_mask ----
+    config = kwargs.get("config", args[0] if len(args) > 0 else None)
+    input_embeds = (kwargs.get("inputs_embeds") or kwargs.get("input_embeds")
+                    or (args[1] if len(args) > 1 else None))
+    attention_mask = kwargs.get("attention_mask", args[2] if len(args) > 2 else None)
+    cache_position = kwargs.get("cache_position")
+    bsz, q_len = input_embeds.shape[0], input_embeds.shape[1]
+    past = int(cache_position[0]) if cache_position is not None else 0
+    kv_len = past + q_len
+    device, dtype = input_embeds.device, input_embeds.dtype
+    mask = torch.zeros((bsz, 1, q_len, kv_len), device=device, dtype=dtype)
+    if attention_mask is not None:
+        if attention_mask.dim() == 2:
+            pad_vec = (attention_mask == 0).to(dtype) * -1e9
+            mask = mask + pad_vec[:, None, None, :]
+        elif attention_mask.dim() == 4:
+            mask = attention_mask.to(dtype)
+    return mask
+
+
+# --------------------------------------------------------------------------- #
+#  Symmetric depthwise shortconv — replaces upstream Lfm2ShortConv            #
+# --------------------------------------------------------------------------- #
+
+
+class Lfm2ShortConvBidir(nn.Module):
+    """Inference-only bidirectional shortconv.
+
+    Holds ONLY `conv.weight` (depthwise kernel) — no in_proj, no out_proj, no
+    chunking, no B*x or c*conv_out gating. Symmetric center-pad so each
+    position sees a [t-K/2, ..., t+K/2] window.
+
+    Tensor name compatibility: state_dict keys remain `conv.weight` (matching
+    the trained safetensors) so checkpoints saved by the training-time class
+    load directly into this class without renaming.
+    """
+
+    def __init__(self, config: Lfm2Config, layer_idx: int):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        self.L_cache = int(config.conv_L_cache)
+        self.conv = nn.Conv1d(
+            in_channels=config.hidden_size,
+            out_channels=config.hidden_size,
+            kernel_size=self.L_cache,
+            groups=config.hidden_size,
+            bias=False,
+            padding=self.L_cache // 2,
+        )
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        past_key_values=None,
+        cache_position=None,
+        attention_mask=None,
+    ):  # signature mirrors upstream Lfm2ShortConv.forward
+        x = hidden_states  # (B, T, C)
+        T = x.shape[1]
+        Bx = x.transpose(1, 2).contiguous()  # (B, C, T)
+        y = self.conv(Bx)
+        if y.shape[-1] != T:  # even kernel -> trim trailing position
+            y = y[..., :T]
+        return y.transpose(1, 2).contiguous()
+
+
+def _swap_shortconv_to_bidir(module: nn.Module, config: Lfm2Config) -> None:
+    """Recursively replace every upstream Lfm2ShortConv with Lfm2ShortConvBidir.
+
+    Walks the module tree top-down, identifies modules with the same attribute
+    surface as Lfm2ShortConv (has `.conv` (Conv1d) AND `.in_proj` (Linear)),
+    and rebinds the parent's attribute to a fresh Lfm2ShortConvBidir of the
+    same layer_idx.
+    """
+    Lfm2ShortConv = _lfm2_mod.Lfm2ShortConv
+    for parent_name, parent in list(module.named_modules()):
+        for child_name, child in list(parent.named_children()):
+            if isinstance(child, Lfm2ShortConv):
+                layer_idx = getattr(child, "layer_idx", 0)
+                new = Lfm2ShortConvBidir(config, layer_idx)
+                # transfer trained conv.weight if it exists on the source
+                if hasattr(child, "conv") and hasattr(child.conv, "weight"):
+                    with torch.no_grad():
+                        new.conv.weight.data.copy_(child.conv.weight.data.to(new.conv.weight.dtype))
+                setattr(parent, child_name, new)
+
+
+# --------------------------------------------------------------------------- #
+#  Bidirectional base model                                                   #
+# --------------------------------------------------------------------------- #
+
+
+def _install_bidirectional_patches_once():
+    """Idempotently install the bidirectional attention-mask replacement on
+    the upstream `transformers.models.lfm2.modeling_lfm2` module.
+
+    This runs at IMPORT TIME of this file. It is a no-op for trained ckpts
+    that already loaded with the same patch installed (e.g. via the
+    training-time `encoder_lfm.modeling.lfm2_bidirectional` module).
+    """
+    if getattr(_lfm2_mod, "_lfm2_bidir_inference_patch_installed", False):
+        return
+    _lfm2_mod.create_causal_mask = _bidirectional_mask
+    _lfm2_mod._lfm2_bidir_inference_patch_installed = True
+
+
+_install_bidirectional_patches_once()
+
+
+# --------------------------------------------------------------------------- #
+#  Lfm2 bidirectional MLM head                                                #
+# --------------------------------------------------------------------------- #
+
+
+class Lfm2BidirForMaskedLM(PreTrainedModel):
+    """Lfm2 encoder MLM, bidirectional attention + symmetric shortconv.
+
+    Tied lm_head -> embed_tokens.weight. No KV cache. Single forward pass.
+
+    For sentence-transformers compatibility, the base model is reachable via
+    `.lfm2` and can be passed to AutoModel-style consumers (mean-pool the
+    last hidden state).
+    """
+
+    config_class = Lfm2Config
+    base_model_prefix = "lfm2"
+    supports_gradient_checkpointing = False
+    _tied_weights_keys = {"lm_head.weight": "lfm2.embed_tokens.weight"}
+    # Defensive: stripped slim ckpts won't contain these. Tell HF to skip
+    # missing-key errors for them in case anyone re-ships a non-stripped variant
+    # later.
+    _keys_to_ignore_on_load_missing = [
+        r"lfm2\.layers\.\d+\.conv\.in_proj\.weight",
+        r"lfm2\.layers\.\d+\.conv\.out_proj\.weight",
+        r"lfm2\.layers\.\d+\.conv\.in_proj\.bias",
+        r"lfm2\.layers\.\d+\.conv\.out_proj\.bias",
+    ]
+    # And don't error on attempts to load the same in unexpected-keys mode.
+    _keys_to_ignore_on_load_unexpected = list(_keys_to_ignore_on_load_missing)
+
+    def __init__(self, config: Lfm2Config):
+        super().__init__(config)
+        # Ensure attention mask is bidirectional (no-op if installed)
+        _install_bidirectional_patches_once()
+
+        # MLM has no KV cache
+        config = type(config).from_dict({**config.to_dict(), "use_cache": False})
+        self.config = config
+
+        # Use upstream Lfm2Model body (with conv + full_attention hybrid blocks)
+        self.lfm2 = _lfm2_mod.Lfm2Model(config)
+
+        # Swap every Lfm2ShortConv -> Lfm2ShortConvBidir BEFORE state_dict load.
+        # This removes in_proj/out_proj from the module tree so missing keys in
+        # the slim safetensors aren't a problem.
+        _swap_shortconv_to_bidir(self.lfm2, config)
+
+        # MLM head: tied linear projection to vocab.
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+        self.post_init()
+        self.tie_weights()
+
+    # ----- HF wiring --------------------------------------------------------
+
+    def tie_weights(self, *a, **k):
+        try:
+            super().tie_weights(*a, **k)
+        except TypeError:
+            super().tie_weights()
+        self.lm_head.weight = self.lfm2.embed_tokens.weight
+
+    def get_input_embeddings(self):
+        return self.lfm2.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.lfm2.embed_tokens = value
+        self.tie_weights()
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    # ----- forward ----------------------------------------------------------
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        output_hidden_states: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        **kwargs,
+    ) -> MaskedLMOutput:
+        return_dict = True if return_dict is None else return_dict
+        outputs = self.lfm2(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds,
+            use_cache=False,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=True,
+        )
+        hidden = outputs.last_hidden_state
+        logits = self.lm_head(hidden)
+
+        loss = None
+        if labels is not None:
+            loss = F.cross_entropy(
+                logits.view(-1, self.config.vocab_size),
+                labels.view(-1),
+                ignore_index=-100,
+            )
+
+        if not return_dict:
+            out = (logits,) + outputs[1:]
+            return ((loss,) + out) if loss is not None else out
+        return MaskedLMOutput(
+            loss=loss,
+            logits=logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+# --------------------------------------------------------------------------- #
+#  Optional "AutoModel" entry: returns the base Lfm2Model (for embeddings)    #
+# --------------------------------------------------------------------------- #
+
+
+class Lfm2BidirModel(PreTrainedModel):
+    """Lfm2 bidirectional encoder base — no MLM head.
+
+    Useful for sentence-transformers / embedding pipelines. Returns hidden
+    states; downstream can mean-pool, CLS-pool, etc.
+    """
+
+    config_class = Lfm2Config
+    base_model_prefix = "lfm2"
+
+    _keys_to_ignore_on_load_missing = Lfm2BidirForMaskedLM._keys_to_ignore_on_load_missing
+    _keys_to_ignore_on_load_unexpected = Lfm2BidirForMaskedLM._keys_to_ignore_on_load_unexpected
+
+    def __init__(self, config: Lfm2Config):
+        super().__init__(config)
+        _install_bidirectional_patches_once()
+        config = type(config).from_dict({**config.to_dict(), "use_cache": False})
+        self.config = config
+        self.lfm2 = _lfm2_mod.Lfm2Model(config)
+        _swap_shortconv_to_bidir(self.lfm2, config)
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        output_hidden_states: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        **kwargs,
+    ) -> BaseModelOutput:
+        return_dict = True if return_dict is None else return_dict
+        outputs = self.lfm2(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds,
+            use_cache=False,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=True,
+        )
+        if not return_dict:
+            return (outputs.last_hidden_state,) + tuple(
+                v for v in (outputs.hidden_states, outputs.attentions) if v is not None
+            )
+        return BaseModelOutput(
+            last_hidden_state=outputs.last_hidden_state,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+# ---- token-classification head (inlined; self-contained) ----
+from transformers.modeling_outputs import TokenClassifierOutput
+
+class Lfm2BidirForTokenClassification(PreTrainedModel):
+    config_class = Lfm2Config
+    base_model_prefix = "lfm2"
+    supports_gradient_checkpointing = True
+    # slim ckpt lacks the stripped conv projections; the ckpt's lm_head is unused here
+    _keys_to_ignore_on_load_missing = list(Lfm2BidirForMaskedLM._keys_to_ignore_on_load_missing)
+    _keys_to_ignore_on_load_unexpected = list(
+        Lfm2BidirForMaskedLM._keys_to_ignore_on_load_missing
+    ) + [r"lm_head\.weight"]
+
+    def __init__(self, config: Lfm2Config):
+        super().__init__(config)
+        _install_bidirectional_patches_once()
+
+        # no KV cache for an encoder
+        config = type(config).from_dict({**config.to_dict(), "use_cache": False})
+        self.config = config
+        self.num_labels = config.num_labels
+
+        self.lfm2 = _lfm2_mod.Lfm2Model(config)
+        # swap shortconv -> symmetric bidirectional BEFORE state_dict load so the
+        # module tree matches the slim safetensors (conv.weight only)
+        _swap_shortconv_to_bidir(self.lfm2, config)
+
+        drop = (getattr(config, "classifier_dropout", None)
+                or getattr(config, "hidden_dropout", None) or 0.1)
+        self.dropout = nn.Dropout(drop)
+        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
+        self.label_smoothing = float(getattr(config, "label_smoothing", 0.0))
+
+        # inverse-frequency class weights; set by the trainer, defaults to ones
+        self.register_buffer("class_weights", torch.ones(config.num_labels), persistent=False)
+        self.post_init()
+
+    def set_class_weights(self, weights: torch.Tensor) -> None:
+        if weights.shape != (self.num_labels,):
+            raise ValueError(f"class_weights must be ({self.num_labels},), got {tuple(weights.shape)}")
+        self.class_weights = weights.to(device=self.class_weights.device, dtype=torch.float32)
+
+    def get_input_embeddings(self):
+        return self.lfm2.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.lfm2.embed_tokens = value
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        output_hidden_states: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        **kwargs,
+    ) -> TokenClassifierOutput:
+        return_dict = True if return_dict is None else return_dict
+        outputs = self.lfm2(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds,
+            use_cache=False,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=True,
+        )
+        hidden = self.dropout(outputs.last_hidden_state)
+        logits = self.classifier(hidden)
+
+        loss = None
+        if labels is not None:
+            loss = F.cross_entropy(
+                logits.view(-1, self.num_labels).float(),
+                labels.view(-1),
+                weight=self.class_weights.float(),
+                ignore_index=-100,
+                label_smoothing=self.label_smoothing,
+            )
+
+        if not return_dict:
+            out = (logits,) + outputs[1:]
+            return ((loss,) + out) if loss is not None else out
+        return TokenClassifierOutput(
+            loss=loss,
+            logits=logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )

tokenizer_config.json

@@ -0,0 +1,23 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [],
+  "is_local": false,
+  "legacy": false,
+  "local_files_only": false,
+  "mask_token": "<|mask|>",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {},
+  "pad_token": "<|pad|>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "use_default_system_prompt": false,
+  "use_fast": true
+}