Upload DeBERTa-v3-large PII models, tokenizers, and code

README.md

@@ -0,0 +1,33 @@
+---
+license: mit
+tags:
+- deberta
+- pii-detection
+- token-classification
+- kaggle
+datasets:
+- pii-data-detection
+metrics:
+- f5-score
+---
+
+# DeBERTa-v3-large PII Data Detection (Final Model)
+
+This repository contains the fine-tuned DeBERTa-v3-large model checkpoint from the Kaggle PII Data Detection Competition.
+
+## Model Details
+- **Base Architecture**: microsoft/deberta-v3-large
+- **Task**: Token classification for Personally Identifiable Information (PII) detection
+- **Max Sequence Length**: 4096
+- **Training Setup**:
+  - Competition training data + moredata_dataset_fixed.json
+  - Optimizer: AdamW, lr=1e-5, weight_decay=0.0
+  - Batch size: 2
+  - Epochs: 5 (Fold 0)
+- **Validation Metric**: **0.9692** at Epoch 3
+
+## Checkpoints Included
+- `deberta_v3_large_w10_ado4_wd0_bs2_lr1e5_5epo_fold0_best.pth` (1.74 GB): Model weights with highest validation score (0.9692).
+- `deberta_v3_large_w10_ado4_wd0_bs2_lr1e5_5epo_fold0_last.pth` (5.21 GB): Full training checkpoint including optimizer, scheduler, and scaler states.
+- Tokenizer files (`spm.model`, `tokenizer.json`, `config.json`, etc.)
+- Training and inference code under `code/`.

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "[MASK]": 128000
+}

code/inference-single-model-no-pp-privatelb-0-963.ipynb

@@ -0,0 +1,1677 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "8020243e",
+   "metadata": {
+    "papermill": {
+     "duration": 0.003262,
+     "end_time": "2025-04-30T15:27:34.703550",
+     "exception": false,
+     "start_time": "2025-04-30T15:27:34.700288",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "source": [
+    "## Introduction\n",
+    "\n",
+    "Congratulations to everyone who persevered through this challenging competition!\n",
+    "\n",
+    "I've been eagerly looking forward to reading the top solutions, and now that the competition has finally concluded, I am thrilled.\n",
+    "\n",
+    "It has been a great learning journey for all of us. I am excited to share that I plan to publish a training notebook that demonstrates how to achieve a score of `Private Leaderboard 0.963` using a single model.\n",
+    "\n",
+    "[Training Notebook PrivateLB 0.963](https://www.kaggle.com/code/haqishen/training-single-model-to-get-privatelb-0-963/)\n",
+    "\n",
+    "And this notebook shows how to submit the model trained above.\n",
+    "\n",
+    "\n",
+    "## Acknowledge\n",
+    "\n",
+    "I was provided Z8G5 Fury workstation with dual RTX6000 ada GPU.\n",
+    "\n",
+    "For training a single model the above notebook shows, it takes less than 40min!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "4e44c5cf",
+   "metadata": {
+    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
+    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:27:34.710901Z",
+     "iopub.status.busy": "2025-04-30T15:27:34.710600Z",
+     "iopub.status.idle": "2025-04-30T15:27:56.410746Z",
+     "shell.execute_reply": "2025-04-30T15:27:56.410065Z"
+    },
+    "papermill": {
+     "duration": 21.706244,
+     "end_time": "2025-04-30T15:27:56.412829",
+     "exception": false,
+     "start_time": "2025-04-30T15:27:34.706585",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "2025-04-30 15:27:44.649168: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
+      "2025-04-30 15:27:44.649298: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
+      "2025-04-30 15:27:44.786214: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n"
+     ]
+    }
+   ],
+   "source": [
+    "import re\n",
+    "import os\n",
+    "import json\n",
+    "import torch\n",
+    "import argparse\n",
+    "from pathlib import Path\n",
+    "from itertools import chain\n",
+    "\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "from transformers import AutoTokenizer, AutoModelForTokenClassification, AutoModel, AutoConfig, DataCollatorForTokenClassification\n",
+    "from datasets import Dataset\n",
+    "from spacy.lang.en import English\n",
+    "\n",
+    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "d7fc24ad",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:27:56.421309Z",
+     "iopub.status.busy": "2025-04-30T15:27:56.420333Z",
+     "iopub.status.idle": "2025-04-30T15:27:57.444502Z",
+     "shell.execute_reply": "2025-04-30T15:27:57.443587Z"
+    },
+    "papermill": {
+     "duration": 1.030446,
+     "end_time": "2025-04-30T15:27:57.446580",
+     "exception": false,
+     "start_time": "2025-04-30T15:27:56.416134",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "pii-dbtv3-large-9688  pii-detection-removal-from-educational-data\r\n"
+     ]
+    }
+   ],
+   "source": [
+    "!ls ../input/"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "dea14c4c",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:27:57.454606Z",
+     "iopub.status.busy": "2025-04-30T15:27:57.454035Z",
+     "iopub.status.idle": "2025-04-30T15:27:57.459265Z",
+     "shell.execute_reply": "2025-04-30T15:27:57.458458Z"
+    },
+    "papermill": {
+     "duration": 0.011012,
+     "end_time": "2025-04-30T15:27:57.460840",
+     "exception": false,
+     "start_time": "2025-04-30T15:27:57.449828",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "backbone = 'microsoft/deberta-v3-large'\n",
+    "model_dir = '/kaggle/input/pii-dbtv3-large-9688'\n",
+    "th = 0.5\n",
+    "\n",
+    "id2label = {0: 'B-EMAIL', 1: 'B-ID_NUM', 2: 'B-NAME_STUDENT', 3: 'B-PHONE_NUM', 4: 'B-STREET_ADDRESS', 5: 'B-URL_PERSONAL', 6: 'B-USERNAME', 7: 'I-ID_NUM', 8: 'I-NAME_STUDENT', 9: 'I-PHONE_NUM', 10: 'I-STREET_ADDRESS', 11: 'I-URL_PERSONAL', 12: 'O'}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "e1da943a",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:27:57.468276Z",
+     "iopub.status.busy": "2025-04-30T15:27:57.467797Z",
+     "iopub.status.idle": "2025-04-30T15:27:57.473293Z",
+     "shell.execute_reply": "2025-04-30T15:27:57.472311Z"
+    },
+    "papermill": {
+     "duration": 0.011058,
+     "end_time": "2025-04-30T15:27:57.474961",
+     "exception": false,
+     "start_time": "2025-04-30T15:27:57.463903",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "def tokenize(example, tokenizer):\n",
+    "    text = []\n",
+    "    token_map = []\n",
+    "    \n",
+    "    idx = 0\n",
+    "    \n",
+    "    for t, ws in zip(example[\"tokens\"], example[\"trailing_whitespace\"]):\n",
+    "        \n",
+    "        text.append(t)\n",
+    "        token_map.extend([idx]*len(t))\n",
+    "        if ws:\n",
+    "            text.append(\" \")\n",
+    "            token_map.append(-1)\n",
+    "            \n",
+    "        idx += 1\n",
+    "        \n",
+    "        \n",
+    "    tokenized = tokenizer(\"\".join(text), return_offsets_mapping=True, truncation=False)\n",
+    "    \n",
+    "        \n",
+    "    return {\n",
+    "        **tokenized,\n",
+    "        \"token_map\": token_map,\n",
+    "    }"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "e9da25fc",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:27:57.482197Z",
+     "iopub.status.busy": "2025-04-30T15:27:57.481934Z",
+     "iopub.status.idle": "2025-04-30T15:27:58.693956Z",
+     "shell.execute_reply": "2025-04-30T15:27:58.693020Z"
+    },
+    "papermill": {
+     "duration": 1.218282,
+     "end_time": "2025-04-30T15:27:58.696470",
+     "exception": false,
+     "start_time": "2025-04-30T15:27:57.478188",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "    "
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "5951514148bd4f938c2559866f91be40",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "#0:   0%|          | 0/5 [00:00<?, ?ex/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "88430d270ddf4976af32c09d1afb0ad8",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "#1:   0%|          | 0/5 [00:00<?, ?ex/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "with open(\"/kaggle/input/pii-detection-removal-from-educational-data/test.json\", \"r\") as f:\n",
+    "    data = json.load(f)\n",
+    "\n",
+    "ds = Dataset.from_dict({\n",
+    "    \"full_text\": [x[\"full_text\"] for x in data],\n",
+    "    \"document\": [x[\"document\"] for x in data],\n",
+    "    \"tokens\": [x[\"tokens\"] for x in data],\n",
+    "    \"trailing_whitespace\": [x[\"trailing_whitespace\"] for x in data],\n",
+    "})\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained(os.path.join(model_dir, 'fold0'))\n",
+    "collator = DataCollatorForTokenClassification(tokenizer)\n",
+    "ds = ds.map(tokenize, fn_kwargs={\"tokenizer\": tokenizer}, num_proc=2)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "babea7d6",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:27:58.705593Z",
+     "iopub.status.busy": "2025-04-30T15:27:58.704818Z",
+     "iopub.status.idle": "2025-04-30T15:27:58.716765Z",
+     "shell.execute_reply": "2025-04-30T15:27:58.715954Z"
+    },
+    "papermill": {
+     "duration": 0.018147,
+     "end_time": "2025-04-30T15:27:58.718406",
+     "exception": false,
+     "start_time": "2025-04-30T15:27:58.700259",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "dict_keys(['full_text', 'document', 'tokens', 'trailing_whitespace', 'input_ids', 'token_type_ids', 'attention_mask', 'offset_mapping', 'token_map'])"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ds[0].keys()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "e8da3b76",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:27:58.727230Z",
+     "iopub.status.busy": "2025-04-30T15:27:58.726677Z",
+     "iopub.status.idle": "2025-04-30T15:28:24.372489Z",
+     "shell.execute_reply": "2025-04-30T15:28:24.371483Z"
+    },
+    "papermill": {
+     "duration": 25.652138,
+     "end_time": "2025-04-30T15:28:24.374509",
+     "exception": false,
+     "start_time": "2025-04-30T15:27:58.722371",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "DebertaV2ForTokenClassification(\n",
+       "  (deberta): DebertaV2Model(\n",
+       "    (embeddings): DebertaV2Embeddings(\n",
+       "      (word_embeddings): Embedding(128100, 1024, padding_idx=0)\n",
+       "      (LayerNorm): LayerNorm((1024,), eps=1e-07, elementwise_affine=True)\n",
+       "      (dropout): StableDropout()\n",
+       "    )\n",
+       "    (encoder): DebertaV2Encoder(\n",
+       "      (layer): ModuleList(\n",
+       "        (0-23): 24 x DebertaV2Layer(\n",
+       "          (attention): DebertaV2Attention(\n",
+       "            (self): DisentangledSelfAttention(\n",
+       "              (query_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
+       "              (key_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
+       "              (value_proj): Linear(in_features=1024, out_features=1024, bias=True)\n",
+       "              (pos_dropout): StableDropout()\n",
+       "              (dropout): StableDropout()\n",
+       "            )\n",
+       "            (output): DebertaV2SelfOutput(\n",
+       "              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n",
+       "              (LayerNorm): LayerNorm((1024,), eps=1e-07, elementwise_affine=True)\n",
+       "              (dropout): StableDropout()\n",
+       "            )\n",
+       "          )\n",
+       "          (intermediate): DebertaV2Intermediate(\n",
+       "            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n",
+       "            (intermediate_act_fn): GELUActivation()\n",
+       "          )\n",
+       "          (output): DebertaV2Output(\n",
+       "            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n",
+       "            (LayerNorm): LayerNorm((1024,), eps=1e-07, elementwise_affine=True)\n",
+       "            (dropout): StableDropout()\n",
+       "          )\n",
+       "        )\n",
+       "      )\n",
+       "      (rel_embeddings): Embedding(512, 1024)\n",
+       "      (LayerNorm): LayerNorm((1024,), eps=1e-07, elementwise_affine=True)\n",
+       "    )\n",
+       "  )\n",
+       "  (dropout): Dropout(p=0.1, inplace=False)\n",
+       "  (classifier): Linear(in_features=1024, out_features=13, bias=True)\n",
+       ")"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "def get_model(conf):\n",
+    "    return AutoModelForTokenClassification.from_config(\n",
+    "        conf\n",
+    "    )\n",
+    "\n",
+    "fold = 0\n",
+    "conf = AutoConfig.from_pretrained(os.path.join(model_dir, f'fold{fold}'))\n",
+    "model = get_model(conf)\n",
+    "model_file = os.path.join(model_dir, f'fold{fold}', f'genv1_noext1_ds3_ht1024_dbtv3_large_wu0.1_w10_ado4_wd0_bs2_lr1e5_5f_5epo_fold{fold}_best.pth')\n",
+    "sd = torch.load(model_file, map_location='cpu')\n",
+    "model.load_state_dict(sd, strict=True)\n",
+    "model.eval()\n",
+    "model.cuda()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "d6469695",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:28:24.383514Z",
+     "iopub.status.busy": "2025-04-30T15:28:24.382837Z",
+     "iopub.status.idle": "2025-04-30T15:28:25.508686Z",
+     "shell.execute_reply": "2025-04-30T15:28:25.507737Z"
+    },
+    "papermill": {
+     "duration": 1.132525,
+     "end_time": "2025-04-30T15:28:25.510805",
+     "exception": false,
+     "start_time": "2025-04-30T15:28:24.378280",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Wed Apr 30 15:28:25 2025       \r\n",
+      "+-----------------------------------------------------------------------------------------+\r\n",
+      "| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |\r\n",
+      "|-----------------------------------------+------------------------+----------------------+\r\n",
+      "| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |\r\n",
+      "| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |\r\n",
+      "|                                         |                        |               MIG M. |\r\n",
+      "|=========================================+========================+======================|\r\n",
+      "|   0  Tesla P100-PCIE-16GB           Off |   00000000:00:04.0 Off |                    0 |\r\n",
+      "| N/A   38C    P0             31W /  250W |    1933MiB /  16384MiB |      0%      Default |\r\n",
+      "|                                         |                        |                  N/A |\r\n",
+      "+-----------------------------------------+------------------------+----------------------+\r\n",
+      "                                                                                         \r\n",
+      "+-----------------------------------------------------------------------------------------+\r\n",
+      "| Processes:                                                                              |\r\n",
+      "|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |\r\n",
+      "|        ID   ID                                                               Usage      |\r\n",
+      "|=========================================================================================|\r\n",
+      "+-----------------------------------------------------------------------------------------+\r\n"
+     ]
+    }
+   ],
+   "source": [
+    "!nvidia-smi"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "22f8828a",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:28:25.520396Z",
+     "iopub.status.busy": "2025-04-30T15:28:25.519800Z",
+     "iopub.status.idle": "2025-04-30T15:28:28.012067Z",
+     "shell.execute_reply": "2025-04-30T15:28:28.011227Z"
+    },
+    "papermill": {
+     "duration": 2.499221,
+     "end_time": "2025-04-30T15:28:28.014108",
+     "exception": false,
+     "start_time": "2025-04-30T15:28:25.514887",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "preds_final = []\n",
+    "with torch.no_grad():\n",
+    "    for sample in ds:\n",
+    "        input_ids = torch.tensor([sample['input_ids']]).to(device)\n",
+    "        attention_mask = torch.tensor([sample['attention_mask']]).to(device)\n",
+    "\n",
+    "        preds = model(input_ids, attention_mask=attention_mask, labels=None)['logits'].softmax(2)\n",
+    "        preds[:, :, -1][preds[:, :, -1] < th] = 0\n",
+    "        preds = torch.argmax(preds, axis=2).cpu().numpy()\n",
+    "        preds_final.append(preds[0])\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "9c2a0aed",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:28:28.022790Z",
+     "iopub.status.busy": "2025-04-30T15:28:28.022520Z",
+     "iopub.status.idle": "2025-04-30T15:28:29.163583Z",
+     "shell.execute_reply": "2025-04-30T15:28:29.162604Z"
+    },
+    "papermill": {
+     "duration": 1.147856,
+     "end_time": "2025-04-30T15:28:29.165845",
+     "exception": false,
+     "start_time": "2025-04-30T15:28:28.017989",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Wed Apr 30 15:28:29 2025       \r\n",
+      "+-----------------------------------------------------------------------------------------+\r\n",
+      "| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |\r\n",
+      "|-----------------------------------------+------------------------+----------------------+\r\n",
+      "| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |\r\n",
+      "| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |\r\n",
+      "|                                         |                        |               MIG M. |\r\n",
+      "|=========================================+========================+======================|\r\n",
+      "|   0  Tesla P100-PCIE-16GB           Off |   00000000:00:04.0 Off |                    0 |\r\n",
+      "| N/A   39C    P0             36W /  250W |    4065MiB /  16384MiB |      0%      Default |\r\n",
+      "|                                         |                        |                  N/A |\r\n",
+      "+-----------------------------------------+------------------------+----------------------+\r\n",
+      "                                                                                         \r\n",
+      "+-----------------------------------------------------------------------------------------+\r\n",
+      "| Processes:                                                                              |\r\n",
+      "|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |\r\n",
+      "|        ID   ID                                                               Usage      |\r\n",
+      "|=========================================================================================|\r\n",
+      "+-----------------------------------------------------------------------------------------+\r\n"
+     ]
+    }
+   ],
+   "source": [
+    "!nvidia-smi"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "710d5fa1",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:28:29.175351Z",
+     "iopub.status.busy": "2025-04-30T15:28:29.174744Z",
+     "iopub.status.idle": "2025-04-30T15:28:29.229118Z",
+     "shell.execute_reply": "2025-04-30T15:28:29.228370Z"
+    },
+    "papermill": {
+     "duration": 0.061412,
+     "end_time": "2025-04-30T15:28:29.231214",
+     "exception": false,
+     "start_time": "2025-04-30T15:28:29.169802",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "pairs = []\n",
+    "processed = []\n",
+    "for p, token_map, offsets, tokens, doc in zip(preds_final, ds[\"token_map\"],\n",
+    "                                              ds[\"offset_mapping\"], ds[\"tokens\"],\n",
+    "                                              ds[\"document\"]):\n",
+    "    for token_pred, (start_idx, end_idx) in zip(p, offsets):\n",
+    "        label_pred = id2label[token_pred]\n",
+    "\n",
+    "        if start_idx + end_idx == 0:\n",
+    "            continue\n",
+    "\n",
+    "        if token_map[start_idx] == -1:\n",
+    "            start_idx += 1\n",
+    "\n",
+    "        # ignore \"\\n\\n\"\n",
+    "        while start_idx < len(token_map) and tokens[token_map[start_idx]].isspace():\n",
+    "            start_idx += 1\n",
+    "\n",
+    "        if start_idx >= len(token_map):\n",
+    "            break\n",
+    "\n",
+    "        token_id = token_map[start_idx]\n",
+    "\n",
+    "        # ignore \"O\" predictions and whitespace preds\n",
+    "        if label_pred not in (\"O\") and token_id != -1:\n",
+    "#         if label_pred not in (\"O\", \"B-EMAIL\", \"B-PHONE_NUM\", \"I-PHONE_NUM\") and token_id != -1:\n",
+    "            pair = (doc, token_id)\n",
+    "\n",
+    "            if pair not in pairs:\n",
+    "                processed.append({\"document\": doc, \"token\": token_id, \"label\": label_pred, \"token_str\": tokens[token_id]})\n",
+    "                pairs.append(pair)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "551a878b",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:28:29.241209Z",
+     "iopub.status.busy": "2025-04-30T15:28:29.240471Z",
+     "iopub.status.idle": "2025-04-30T15:28:29.262801Z",
+     "shell.execute_reply": "2025-04-30T15:28:29.261886Z"
+    },
+    "papermill": {
+     "duration": 0.0292,
+     "end_time": "2025-04-30T15:28:29.264560",
+     "exception": false,
+     "start_time": "2025-04-30T15:28:29.235360",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>document</th>\n",
+       "      <th>token</th>\n",
+       "      <th>label</th>\n",
+       "      <th>token_str</th>\n",
+       "      <th>row_id</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>7</td>\n",
+       "      <td>9</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Nathalie</td>\n",
+       "      <td>0</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>7</td>\n",
+       "      <td>10</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Sylla</td>\n",
+       "      <td>1</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>7</td>\n",
+       "      <td>482</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Nathalie</td>\n",
+       "      <td>2</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>7</td>\n",
+       "      <td>483</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Sylla</td>\n",
+       "      <td>3</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>7</td>\n",
+       "      <td>741</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Nathalie</td>\n",
+       "      <td>4</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>7</td>\n",
+       "      <td>742</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Sylla</td>\n",
+       "      <td>5</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>10</td>\n",
+       "      <td>0</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Diego</td>\n",
+       "      <td>6</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>7</th>\n",
+       "      <td>10</td>\n",
+       "      <td>1</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Estrada</td>\n",
+       "      <td>7</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>8</th>\n",
+       "      <td>10</td>\n",
+       "      <td>464</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Diego</td>\n",
+       "      <td>8</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>9</th>\n",
+       "      <td>10</td>\n",
+       "      <td>465</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Estrada</td>\n",
+       "      <td>9</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>10</th>\n",
+       "      <td>16</td>\n",
+       "      <td>4</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Gilberto</td>\n",
+       "      <td>10</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>11</th>\n",
+       "      <td>16</td>\n",
+       "      <td>5</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Gamboa</td>\n",
+       "      <td>11</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>12</th>\n",
+       "      <td>20</td>\n",
+       "      <td>5</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Sindy</td>\n",
+       "      <td>12</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>13</th>\n",
+       "      <td>20</td>\n",
+       "      <td>6</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Samaca</td>\n",
+       "      <td>13</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>14</th>\n",
+       "      <td>56</td>\n",
+       "      <td>12</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Nadine</td>\n",
+       "      <td>14</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>15</th>\n",
+       "      <td>56</td>\n",
+       "      <td>13</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Born</td>\n",
+       "      <td>15</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>16</th>\n",
+       "      <td>86</td>\n",
+       "      <td>6</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Eladio</td>\n",
+       "      <td>16</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>17</th>\n",
+       "      <td>86</td>\n",
+       "      <td>7</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Amaya</td>\n",
+       "      <td>17</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>18</th>\n",
+       "      <td>93</td>\n",
+       "      <td>0</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Silvia</td>\n",
+       "      <td>18</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>19</th>\n",
+       "      <td>93</td>\n",
+       "      <td>1</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Villalobos</td>\n",
+       "      <td>19</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>20</th>\n",
+       "      <td>104</td>\n",
+       "      <td>8</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Sakir</td>\n",
+       "      <td>20</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>21</th>\n",
+       "      <td>104</td>\n",
+       "      <td>9</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Ahmad</td>\n",
+       "      <td>21</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>22</th>\n",
+       "      <td>112</td>\n",
+       "      <td>5</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Francisco</td>\n",
+       "      <td>22</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>23</th>\n",
+       "      <td>112</td>\n",
+       "      <td>6</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Ferreira</td>\n",
+       "      <td>23</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>24</th>\n",
+       "      <td>123</td>\n",
+       "      <td>32</td>\n",
+       "      <td>B-NAME_STUDENT</td>\n",
+       "      <td>Stefano</td>\n",
+       "      <td>24</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>25</th>\n",
+       "      <td>123</td>\n",
+       "      <td>33</td>\n",
+       "      <td>I-NAME_STUDENT</td>\n",
+       "      <td>Lovato</td>\n",
+       "      <td>25</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "    document  token           label   token_str  row_id\n",
+       "0          7      9  B-NAME_STUDENT    Nathalie       0\n",
+       "1          7     10  I-NAME_STUDENT       Sylla       1\n",
+       "2          7    482  B-NAME_STUDENT    Nathalie       2\n",
+       "3          7    483  I-NAME_STUDENT       Sylla       3\n",
+       "4          7    741  B-NAME_STUDENT    Nathalie       4\n",
+       "5          7    742  I-NAME_STUDENT       Sylla       5\n",
+       "6         10      0  B-NAME_STUDENT       Diego       6\n",
+       "7         10      1  I-NAME_STUDENT     Estrada       7\n",
+       "8         10    464  B-NAME_STUDENT       Diego       8\n",
+       "9         10    465  I-NAME_STUDENT     Estrada       9\n",
+       "10        16      4  B-NAME_STUDENT    Gilberto      10\n",
+       "11        16      5  I-NAME_STUDENT      Gamboa      11\n",
+       "12        20      5  B-NAME_STUDENT       Sindy      12\n",
+       "13        20      6  I-NAME_STUDENT      Samaca      13\n",
+       "14        56     12  B-NAME_STUDENT      Nadine      14\n",
+       "15        56     13  I-NAME_STUDENT        Born      15\n",
+       "16        86      6  B-NAME_STUDENT      Eladio      16\n",
+       "17        86      7  I-NAME_STUDENT       Amaya      17\n",
+       "18        93      0  B-NAME_STUDENT      Silvia      18\n",
+       "19        93      1  I-NAME_STUDENT  Villalobos      19\n",
+       "20       104      8  B-NAME_STUDENT       Sakir      20\n",
+       "21       104      9  I-NAME_STUDENT       Ahmad      21\n",
+       "22       112      5  B-NAME_STUDENT   Francisco      22\n",
+       "23       112      6  I-NAME_STUDENT    Ferreira      23\n",
+       "24       123     32  B-NAME_STUDENT     Stefano      24\n",
+       "25       123     33  I-NAME_STUDENT      Lovato      25"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "df = pd.DataFrame(processed)\n",
+    "df[\"row_id\"] = list(range(len(df)))\n",
+    "display(df.head(100))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "e3c5b306",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-04-30T15:28:29.274581Z",
+     "iopub.status.busy": "2025-04-30T15:28:29.273853Z",
+     "iopub.status.idle": "2025-04-30T15:28:29.287091Z",
+     "shell.execute_reply": "2025-04-30T15:28:29.286356Z"
+    },
+    "papermill": {
+     "duration": 0.01995,
+     "end_time": "2025-04-30T15:28:29.288780",
+     "exception": false,
+     "start_time": "2025-04-30T15:28:29.268830",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "df[[\"row_id\", \"document\", \"token\", \"label\"]].to_csv(\"submission.csv\", index=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "493a9eb6",
+   "metadata": {
+    "papermill": {
+     "duration": 0.003789,
+     "end_time": "2025-04-30T15:28:29.297088",
+     "exception": false,
+     "start_time": "2025-04-30T15:28:29.293299",
+     "status": "completed"
+    },
+    "tags": []
+   },
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kaggle": {
+   "accelerator": "gpu",
+   "dataSources": [
+    {
+     "databundleVersionId": 7500999,
+     "sourceId": 66653,
+     "sourceType": "competition"
+    },
+    {
+     "datasetId": 4626838,
+     "sourceId": 7882793,
+     "sourceType": "datasetVersion"
+    }
+   ],
+   "dockerImageVersionId": 30665,
+   "isGpuEnabled": true,
+   "isInternetEnabled": false,
+   "language": "python",
+   "sourceType": "notebook"
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.13"
+  },
+  "papermill": {
+   "default_parameters": {},
+   "duration": 60.15036,
+   "end_time": "2025-04-30T15:28:32.295763",
+   "environment_variables": {},
+   "exception": null,
+   "input_path": "__notebook__.ipynb",
+   "output_path": "__notebook__.ipynb",
+   "parameters": {},
+   "start_time": "2025-04-30T15:27:32.145403",
+   "version": "2.5.0"
+  },
+  "widgets": {
+   "application/vnd.jupyter.widget-state+json": {
+    "state": {
+     "06a9c6b88ead4f49a96be340378209a1": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "ProgressStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "ProgressStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "StyleView",
+       "bar_color": null,
+       "description_width": ""
+      }
+     },
+     "131b74fe075f4174b6362f1d1d8d6c18": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "1.2.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "1.2.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "overflow_x": null,
+       "overflow_y": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "1a471d08224e42b095b0283f8c3dd9e9": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "1.2.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "1.2.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "overflow_x": null,
+       "overflow_y": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "2767366afecc4147be4dfcabbe18f673": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "1.2.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "1.2.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "overflow_x": null,
+       "overflow_y": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "281fad7ccc0949d9b376b9bc51a5a37c": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "ProgressStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "ProgressStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "StyleView",
+       "bar_color": null,
+       "description_width": ""
+      }
+     },
+     "35b69818cee842c1bcdb21fa19a0df48": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "FloatProgressModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "FloatProgressModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "1.5.0",
+       "_view_name": "ProgressView",
+       "bar_style": "success",
+       "description": "",
+       "description_tooltip": null,
+       "layout": "IPY_MODEL_2767366afecc4147be4dfcabbe18f673",
+       "max": 5.0,
+       "min": 0.0,
+       "orientation": "horizontal",
+       "style": "IPY_MODEL_06a9c6b88ead4f49a96be340378209a1",
+       "value": 5.0
+      }
+     },
+     "41df8caea2b9469dbf1adc1b5e056087": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "DescriptionStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "DescriptionStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "StyleView",
+       "description_width": ""
+      }
+     },
+     "5512699220b84dbe87b51b77be6ac346": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "1.2.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "1.2.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "overflow_x": null,
+       "overflow_y": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "5951514148bd4f938c2559866f91be40": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "HBoxModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "HBoxModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "1.5.0",
+       "_view_name": "HBoxView",
+       "box_style": "",
+       "children": [
+        "IPY_MODEL_680ee12021f24e88969f766421390554",
+        "IPY_MODEL_35b69818cee842c1bcdb21fa19a0df48",
+        "IPY_MODEL_dcf6b07ee9cb4f16b314dbe42a5e7abe"
+       ],
+       "layout": "IPY_MODEL_1a471d08224e42b095b0283f8c3dd9e9"
+      }
+     },
+     "59596944958e440695e4ba4f96570ada": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "1.5.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_tooltip": null,
+       "layout": "IPY_MODEL_74ff34caa9cb4410ad25cb99dbccb46f",
+       "placeholder": "​",
+       "style": "IPY_MODEL_41df8caea2b9469dbf1adc1b5e056087",
+       "value": "#1: 100%"
+      }
+     },
+     "6735ff3730f74fed8b055a7f19afd508": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "DescriptionStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "DescriptionStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "StyleView",
+       "description_width": ""
+      }
+     },
+     "680ee12021f24e88969f766421390554": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "1.5.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_tooltip": null,
+       "layout": "IPY_MODEL_ac3981001ad041e997dbc05dad677c06",
+       "placeholder": "​",
+       "style": "IPY_MODEL_d2c93cda9bed48ccb273911aaf36048a",
+       "value": "#0: 100%"
+      }
+     },
+     "6fe5d4d95b844ceba24521a27090e7ae": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "DescriptionStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "DescriptionStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "StyleView",
+       "description_width": ""
+      }
+     },
+     "74ff34caa9cb4410ad25cb99dbccb46f": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "1.2.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "1.2.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "overflow_x": null,
+       "overflow_y": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "87d695ef5aea48a882031c27f85a3752": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "FloatProgressModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "FloatProgressModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "1.5.0",
+       "_view_name": "ProgressView",
+       "bar_style": "success",
+       "description": "",
+       "description_tooltip": null,
+       "layout": "IPY_MODEL_131b74fe075f4174b6362f1d1d8d6c18",
+       "max": 5.0,
+       "min": 0.0,
+       "orientation": "horizontal",
+       "style": "IPY_MODEL_281fad7ccc0949d9b376b9bc51a5a37c",
+       "value": 5.0
+      }
+     },
+     "88430d270ddf4976af32c09d1afb0ad8": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "HBoxModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "HBoxModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "1.5.0",
+       "_view_name": "HBoxView",
+       "box_style": "",
+       "children": [
+        "IPY_MODEL_59596944958e440695e4ba4f96570ada",
+        "IPY_MODEL_87d695ef5aea48a882031c27f85a3752",
+        "IPY_MODEL_d79bec3cedfb4ded91d9d164a266ea3c"
+       ],
+       "layout": "IPY_MODEL_5512699220b84dbe87b51b77be6ac346"
+      }
+     },
+     "aa22d45919ae45e4852612397edf8e6b": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "1.2.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "1.2.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "overflow_x": null,
+       "overflow_y": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "ac3981001ad041e997dbc05dad677c06": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "1.2.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "1.2.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "overflow_x": null,
+       "overflow_y": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "b7789187719a4774a9d31cb1f5b82884": {
+      "model_module": "@jupyter-widgets/base",
+      "model_module_version": "1.2.0",
+      "model_name": "LayoutModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/base",
+       "_model_module_version": "1.2.0",
+       "_model_name": "LayoutModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "LayoutView",
+       "align_content": null,
+       "align_items": null,
+       "align_self": null,
+       "border": null,
+       "bottom": null,
+       "display": null,
+       "flex": null,
+       "flex_flow": null,
+       "grid_area": null,
+       "grid_auto_columns": null,
+       "grid_auto_flow": null,
+       "grid_auto_rows": null,
+       "grid_column": null,
+       "grid_gap": null,
+       "grid_row": null,
+       "grid_template_areas": null,
+       "grid_template_columns": null,
+       "grid_template_rows": null,
+       "height": null,
+       "justify_content": null,
+       "justify_items": null,
+       "left": null,
+       "margin": null,
+       "max_height": null,
+       "max_width": null,
+       "min_height": null,
+       "min_width": null,
+       "object_fit": null,
+       "object_position": null,
+       "order": null,
+       "overflow": null,
+       "overflow_x": null,
+       "overflow_y": null,
+       "padding": null,
+       "right": null,
+       "top": null,
+       "visibility": null,
+       "width": null
+      }
+     },
+     "d2c93cda9bed48ccb273911aaf36048a": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "DescriptionStyleModel",
+      "state": {
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "DescriptionStyleModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/base",
+       "_view_module_version": "1.2.0",
+       "_view_name": "StyleView",
+       "description_width": ""
+      }
+     },
+     "d79bec3cedfb4ded91d9d164a266ea3c": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "1.5.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_tooltip": null,
+       "layout": "IPY_MODEL_aa22d45919ae45e4852612397edf8e6b",
+       "placeholder": "​",
+       "style": "IPY_MODEL_6735ff3730f74fed8b055a7f19afd508",
+       "value": " 5/5 [00:00&lt;00:00, 82.52ex/s]"
+      }
+     },
+     "dcf6b07ee9cb4f16b314dbe42a5e7abe": {
+      "model_module": "@jupyter-widgets/controls",
+      "model_module_version": "1.5.0",
+      "model_name": "HTMLModel",
+      "state": {
+       "_dom_classes": [],
+       "_model_module": "@jupyter-widgets/controls",
+       "_model_module_version": "1.5.0",
+       "_model_name": "HTMLModel",
+       "_view_count": null,
+       "_view_module": "@jupyter-widgets/controls",
+       "_view_module_version": "1.5.0",
+       "_view_name": "HTMLView",
+       "description": "",
+       "description_tooltip": null,
+       "layout": "IPY_MODEL_b7789187719a4774a9d31cb1f5b82884",
+       "placeholder": "​",
+       "style": "IPY_MODEL_6fe5d4d95b844ceba24521a27090e7ae",
+       "value": " 5/5 [00:00&lt;00:00, 59.91ex/s]"
+      }
+     }
+    },
+    "version_major": 2,
+    "version_minor": 0
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

code/inference-single-model-no-pp-privatelb-0-963.py

@@ -0,0 +1,264 @@
+
+# %%
+import re
+import os
+import json
+import torch
+import argparse
+from tqdm import tqdm
+from pathlib import Path
+from itertools import chain
+
+import numpy as np
+import pandas as pd
+from transformers import AutoTokenizer, AutoModelForTokenClassification, AutoModel, AutoConfig, DataCollatorForTokenClassification
+from datasets import Dataset
+from spacy.lang.en import English
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+USE_SIMPLIFIED_LABELS_INFERENCE = True # <<< CHANGE THIS TO False TO USE ORIGINAL BIO LABELS FOR INFERENCE
+
+# %%
+backbone = 'microsoft/deberta-v3-large'
+
+
+model_dir_for_inference = '/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model/deberta_v3_large_w10_ado1_wd0_bs2_lr1e5_5epo_simplified_labels_fold0_best_hf'
+
+
+# th = 0.5 # Threshold for 'O' class - replaced by 'O' scaling
+o_class_scaler = 0.1 # Factor to scale down 'O' class probability
+
+# Define label sets based on the flag
+if USE_SIMPLIFIED_LABELS_INFERENCE:
+    pii_types_active = [
+        'EMAIL', 'ID_NUM', 'NAME_STUDENT', 'PHONE_NUM', 
+        'STREET_ADDRESS', 'URL_PERSONAL', 'USERNAME'
+    ]
+    all_labels_active_inf = sorted(pii_types_active + ['O'])
+else:
+    # Original BIO labels (ensure this matches the set used in training if USE_SIMPLIFIED_LABELS was False)
+    original_target_bio_inf = sorted([
+        'B-EMAIL', 'B-ID_NUM', 'B-NAME_STUDENT', 'B-PHONE_NUM', 
+        'B-STREET_ADDRESS', 'B-URL_PERSONAL', 'B-USERNAME', 'I-ID_NUM', 
+        'I-NAME_STUDENT', 'I-PHONE_NUM', 'I-STREET_ADDRESS', 'I-URL_PERSONAL',
+        'O' 
+    ])
+    all_labels_active_inf = original_target_bio_inf
+
+id2label_active_inf = {i: l for i, l in enumerate(all_labels_active_inf)}
+# Determine O_LABEL_INDEX for thresholding
+O_LABEL_STR = "O"
+O_LABEL_INDEX = -1
+for i, label_str_val in id2label_active_inf.items():
+    if label_str_val == O_LABEL_STR:
+        O_LABEL_INDEX = i
+        break
+if O_LABEL_INDEX == -1:
+    raise ValueError(f"Could not find '{O_LABEL_STR}' in id2label_active_inf: {id2label_active_inf}")
+
+print(f"Using {'simplified' if USE_SIMPLIFIED_LABELS_INFERENCE else 'original BIO'} id2label for inference: {id2label_active_inf}")
+print(f"Index for 'O' label is: {O_LABEL_INDEX}")
+
+
+# %%
+def tokenize(example, tokenizer):
+    text = []
+    token_map = []
+    
+    idx = 0
+    
+    for t, ws in zip(example["tokens"], example["trailing_whitespace"]):
+        
+        text.append(t)
+        token_map.extend([idx]*len(t))
+        if ws:
+            text.append(" ")
+            token_map.append(-1)
+            
+        idx += 1
+        
+    tokenized = tokenizer("".join(text), return_offsets_mapping=True, truncation=False) 
+    
+        
+    return {
+        **tokenized,
+        "token_map": token_map,
+    }
+
+# %%
+with open("/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/pii-data-detection-3rd-place-solution/datasets/test.json", "r") as f:
+    data = json.load(f)
+
+ds = Dataset.from_dict({
+    "full_text": [x["full_text"] for x in data],
+    "document": [x["document"] for x in data],
+    "tokens": [x["tokens"] for x in data],
+    "trailing_whitespace": [x["trailing_whitespace"] for x in data],
+})
+
+tokenizer = AutoTokenizer.from_pretrained(model_dir_for_inference)
+collator = DataCollatorForTokenClassification(tokenizer)
+ds = ds.map(tokenize, fn_kwargs={"tokenizer": tokenizer}, num_proc=2)
+
+# %%
+ds[0].keys()
+
+# %%
+
+print(f"Loading model from: {model_dir_for_inference}")
+model = AutoModelForTokenClassification.from_pretrained(model_dir_for_inference)
+model.eval()
+model.to(device)
+
+
+# %%
+preds_final_simplified_ids = []
+with torch.no_grad():
+    for sample in tqdm(ds, desc="Inferencing"):
+        input_ids = torch.tensor([sample['input_ids']]).to(device)
+        attention_mask = torch.tensor([sample['attention_mask']]).to(device)
+
+        outputs = model(input_ids, attention_mask=attention_mask)
+        logits = outputs.logits # Shape (1, seq_len, num_active_labels)
+        
+        all_probs = logits.softmax(dim=2) # Shape (1, seq_len, num_active_labels)
+
+        # scale down 'O' probability
+        all_probs_scaled = all_probs.clone()
+        if O_LABEL_INDEX != -1:
+            all_probs_scaled[:, :, O_LABEL_INDEX] *= o_class_scaler
+        else:
+            print("Warning: O_LABEL_INDEX not found, 'O' class scaling not applied.")
+
+        predicted_ids = torch.argmax(all_probs_scaled, dim=2).cpu().numpy()
+        preds_final_simplified_ids.append(predicted_ids[0]) # remove batch dim
+
+# %%
+# Store simplified predictions mapped to original tokens first
+intermediate_processed = []
+unique_token_preds = set() # To handle cases where multiple subwords map to the same original token
+
+for pred_ids_for_doc, token_map_for_doc, offsets_for_doc, tokens_for_doc, doc_id in zip(
+    preds_final_simplified_ids, ds["token_map"], ds["offset_mapping"], ds["tokens"], ds["document"]
+):
+    for subword_pred_id, (start_char_idx, end_char_idx) in zip(pred_ids_for_doc, offsets_for_doc):
+        active_label_pred_str = id2label_active_inf[subword_pred_id]
+
+        if start_char_idx == end_char_idx == 0:  # Skip special tokens like [CLS], [SEP] based on offset
+            continue
+        
+        # Find the original token_id using the start_char_idx from offset_mapping
+        current_char_idx = start_char_idx
+        if current_char_idx < len(token_map_for_doc) and token_map_for_doc[current_char_idx] == -1 and current_char_idx + 1 < len(token_map_for_doc):
+            current_char_idx +=1
+        
+        if current_char_idx >= len(token_map_for_doc):
+            continue
+            
+        original_token_id = token_map_for_doc[current_char_idx]
+
+        # Ignore "O" predictions and invalid token_ids
+        if active_label_pred_str != "O" and original_token_id != -1:
+            token_text_content = tokens_for_doc[original_token_id].strip()
+            # Only proceed if the token is not empty after stripping whitespace
+            if token_text_content: 
+                prediction_key = (doc_id, original_token_id)
+                if prediction_key not in unique_token_preds:
+                    intermediate_processed.append({
+                        "document": doc_id,
+                        "token": original_token_id,
+                        "label_pred": active_label_pred_str, # Store the predicted label
+                        "token_str": tokens_for_doc[original_token_id] # Store original token
+                    })
+                    unique_token_preds.add(prediction_key)
+
+# %%
+df_intermediate = pd.DataFrame(intermediate_processed)
+
+final_predictions_for_submission = []
+
+if not df_intermediate.empty:
+    if USE_SIMPLIFIED_LABELS_INFERENCE:
+        # NAME_STUDENT capitalization check
+        name_student_mask = df_intermediate['label_pred'] == 'NAME_STUDENT'
+        
+        def is_potentially_name_case(s):
+            if not s: return False
+            return (s[0].isalpha() and s[0].isupper()) or s.isupper()
+
+        not_name_case_mask = ~df_intermediate['token_str'].apply(is_potentially_name_case)
+        numeric_token_mask = df_intermediate['token_str'].apply(lambda x: x.isdigit())
+        rows_to_filter_out_indices = df_intermediate[name_student_mask & (not_name_case_mask | numeric_token_mask)].index
+        
+        if not rows_to_filter_out_indices.empty:
+            print(f"Applying NAME_STUDENT capitalization/numeric heuristic: Filtering out {len(rows_to_filter_out_indices)} rows.")
+            df_intermediate = df_intermediate.drop(rows_to_filter_out_indices)
+
+    if not df_intermediate.empty:
+        df_intermediate = df_intermediate.sort_values(by=["document", "token"])
+    
+        for doc_id, group in df_intermediate.groupby("document"):
+            sorted_group = group.sort_values("token")
+            
+            previous_token_id = -1
+            previous_pii_type_simplified = None
+            
+            for _, row in sorted_group.iterrows():
+                current_token_id = row["token"]
+                current_label_pred = row["label_pred"] #direct model output
+                
+                final_label_to_submit = ""
+
+                if USE_SIMPLIFIED_LABELS_INFERENCE:
+                    # Reconstruct BIO
+                    # A new entity starts if:
+                    # 1. It's the first PII token for the document.
+                    # 2. The current token is not contiguous with the previous PII token.
+                    # 3. The PII type changes.
+                    if (previous_pii_type_simplified is None) or \
+                       (current_token_id != previous_token_id + 1) or \
+                       (previous_pii_type_simplified != current_label_pred): # current_label_pred is simplified here
+                        bio_tag = "B-"
+                    else:
+                        bio_tag = "I-"
+                    final_label_to_submit = bio_tag + current_label_pred
+                    previous_pii_type_simplified = current_label_pred
+                else:
+                    if current_label_pred != "O":
+                        final_label_to_submit = current_label_pred
+                    else:
+                        continue 
+                
+                if final_label_to_submit:
+                    final_predictions_for_submission.append({
+                        "document": doc_id,
+                        "token": current_token_id,
+                        "label": final_label_to_submit,
+                        "token_str": row["token_str"]
+                    })
+                
+                previous_token_id = current_token_id
+else:
+    print("No PII predictions made after initial filtering.")
+
+
+# %%
+df_final_submission = pd.DataFrame(final_predictions_for_submission)
+if not df_final_submission.empty:
+    df_final_submission["row_id"] = list(range(len(df_final_submission)))
+    print(df_final_submission.head(20))
+    df_submission = df_final_submission[["row_id", "document", "token", "label"]]
+else:
+    df_submission = pd.DataFrame(columns=["row_id", "document", "token", "label"])
+    print("Final DataFrame is empty. No PII entities to submit.")
+
+
+# %%
+df_submission.to_csv("submission.csv", index=False)
+print("Submission file created.")
+
+# %%
+
+
+

code/inference_actual_orig.py

@@ -0,0 +1,172 @@
+# %% [markdown]
+# ## Introduction
+# 
+# Congratulations to everyone who persevered through this challenging competition!
+# 
+# I've been eagerly looking forward to reading the top solutions, and now that the competition has finally concluded, I am thrilled.
+# 
+# It has been a great learning journey for all of us. I am excited to share that I plan to publish a training notebook that demonstrates how to achieve a score of `Private Leaderboard 0.963` using a single model.
+# 
+# [Training Notebook PrivateLB 0.963](https://www.kaggle.com/code/haqishen/training-single-model-to-get-privatelb-0-963/)
+# 
+# And this notebook shows how to submit the model trained above.
+# 
+# 
+# ## Acknowledge
+# 
+# I was provided Z8G5 Fury workstation with dual RTX6000 ada GPU.
+# 
+# For training a single model the above notebook shows, it takes less than 40min!
+
+# %%
+import re
+import os
+import json
+import torch
+import argparse
+from pathlib import Path
+from itertools import chain
+
+import numpy as np
+import pandas as pd
+from transformers import AutoTokenizer, AutoModelForTokenClassification, AutoModel, AutoConfig, DataCollatorForTokenClassification
+from datasets import Dataset
+from spacy.lang.en import English
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+# %%
+!ls ../input/
+
+# %%
+backbone = 'microsoft/deberta-v3-large'
+model_dir = '/kaggle/input/pii-dbtv3-large-9688'
+th = 0.5
+
+id2label = {0: 'B-EMAIL', 1: 'B-ID_NUM', 2: 'B-NAME_STUDENT', 3: 'B-PHONE_NUM', 4: 'B-STREET_ADDRESS', 5: 'B-URL_PERSONAL', 6: 'B-USERNAME', 7: 'I-ID_NUM', 8: 'I-NAME_STUDENT', 9: 'I-PHONE_NUM', 10: 'I-STREET_ADDRESS', 11: 'I-URL_PERSONAL', 12: 'O'}
+
+# %%
+def tokenize(example, tokenizer):
+    text = []
+    token_map = []
+    
+    idx = 0
+    
+    for t, ws in zip(example["tokens"], example["trailing_whitespace"]):
+        
+        text.append(t)
+        token_map.extend([idx]*len(t))
+        if ws:
+            text.append(" ")
+            token_map.append(-1)
+            
+        idx += 1
+        
+        
+    tokenized = tokenizer("".join(text), return_offsets_mapping=True, truncation=False)
+    
+        
+    return {
+        **tokenized,
+        "token_map": token_map,
+    }
+
+# %%
+with open("/kaggle/input/pii-detection-removal-from-educational-data/test.json", "r") as f:
+    data = json.load(f)
+
+ds = Dataset.from_dict({
+    "full_text": [x["full_text"] for x in data],
+    "document": [x["document"] for x in data],
+    "tokens": [x["tokens"] for x in data],
+    "trailing_whitespace": [x["trailing_whitespace"] for x in data],
+})
+
+tokenizer = AutoTokenizer.from_pretrained(os.path.join(model_dir, 'fold0'))
+collator = DataCollatorForTokenClassification(tokenizer)
+ds = ds.map(tokenize, fn_kwargs={"tokenizer": tokenizer}, num_proc=2)
+
+# %%
+ds[0].keys()
+
+# %%
+def get_model(conf):
+    return AutoModelForTokenClassification.from_config(
+        conf
+    )
+
+fold = 0
+conf = AutoConfig.from_pretrained(os.path.join(model_dir, f'fold{fold}'))
+model = get_model(conf)
+# model_file = os.path.join(model_dir, f'fold{fold}', f'genv1_noext1_ds3_ht1024_dbtv3_large_wu0.1_w10_ado4_wd0_bs2_lr1e5_5f_5epo_fold{fold}_best.pth')
+model_file = "/kaggle/input/single_model_only_comp_data/pytorch/training_length_4096/1/deberta_v3_large_w10_ado4_wd0_bs2_lr1e5_5epo_fold0_best.pth"
+sd = torch.load(model_file, map_location='cpu')
+model.load_state_dict(sd, strict=True)
+model.eval()
+model.cuda()
+
+
+# %%
+!nvidia-smi
+
+# %%
+preds_final = []
+with torch.no_grad():
+    for sample in ds:
+        input_ids = torch.tensor([sample['input_ids']]).to(device)
+        attention_mask = torch.tensor([sample['attention_mask']]).to(device)
+
+        preds = model(input_ids, attention_mask=attention_mask, labels=None)['logits'].softmax(2)
+        preds[:, :, -1][preds[:, :, -1] < th] = 0
+        preds = torch.argmax(preds, axis=2).cpu().numpy()
+        preds_final.append(preds[0])
+
+
+# %%
+!nvidia-smi
+
+# %%
+pairs = []
+processed = []
+for p, token_map, offsets, tokens, doc in zip(preds_final, ds["token_map"],
+                                              ds["offset_mapping"], ds["tokens"],
+                                              ds["document"]):
+    for token_pred, (start_idx, end_idx) in zip(p, offsets):
+        label_pred = id2label[token_pred]
+
+        if start_idx + end_idx == 0:
+            continue
+
+        if token_map[start_idx] == -1:
+            start_idx += 1
+
+        # ignore "\n\n"
+        while start_idx < len(token_map) and tokens[token_map[start_idx]].isspace():
+            start_idx += 1
+
+        if start_idx >= len(token_map):
+            break
+
+        token_id = token_map[start_idx]
+
+        # ignore "O" predictions and whitespace preds
+        if label_pred not in ("O") and token_id != -1:
+#         if label_pred not in ("O", "B-EMAIL", "B-PHONE_NUM", "I-PHONE_NUM") and token_id != -1:
+            pair = (doc, token_id)
+
+            if pair not in pairs:
+                processed.append({"document": doc, "token": token_id, "label": label_pred, "token_str": tokens[token_id]})
+                pairs.append(pair)
+
+# %%
+df = pd.DataFrame(processed)
+df["row_id"] = list(range(len(df)))
+display(df.head(100))
+
+# %%
+df[["row_id", "document", "token", "label"]].to_csv("submission.csv", index=False)
+
+# %%
+
+
+

code/kaggle_upload.py

@@ -0,0 +1,21 @@
+import kagglehub
+
+kagglehub.login()
+
+# Replace with path to directory containing model files.
+LOCAL_MODEL_DIR = '/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model/deberta_v3_large_w10_ado1_wd0_bs2_lr1e5_5epo_simplified_labels_fold0_best_hf'
+
+MODEL_SLUG = 'Single_model_only_comp_data' # Replace with model slug.
+
+# Learn more about naming model variations at
+# https://www.kaggle.com/docs/models#name-model.
+VARIATION_SLUG = 'training_length_4096_8labels_moredata' # Replace with variation slug.
+
+kagglehub.model_upload(
+  handle = f"andreifrcanu/{MODEL_SLUG}/transformers/{VARIATION_SLUG}",
+  local_model_dir = LOCAL_MODEL_DIR,
+  version_notes = 'training_length_4096_8labels_moredata')
+# kagglehub.model_upload(
+#   handle = f"andreifrcanu/{MODEL_SLUG}/pyTorch/{VARIATION_SLUG}",
+#   local_model_dir = LOCAL_MODEL_DIR,
+#   version_notes = 'training_length_4096_origlabels_actualmoredata')

code/submission.csv

@@ -0,0 +1,36 @@
+row_id,document,token,label
+0,7,9,B-NAME_STUDENT
+1,7,10,I-NAME_STUDENT
+2,7,52,B-NAME_STUDENT
+3,7,482,B-NAME_STUDENT
+4,7,483,I-NAME_STUDENT
+5,7,741,B-NAME_STUDENT
+6,7,742,I-NAME_STUDENT
+7,10,0,B-NAME_STUDENT
+8,10,1,I-NAME_STUDENT
+9,10,465,B-NAME_STUDENT
+10,16,4,B-NAME_STUDENT
+11,16,5,I-NAME_STUDENT
+12,20,5,B-NAME_STUDENT
+13,20,6,I-NAME_STUDENT
+14,56,13,B-NAME_STUDENT
+15,86,6,B-NAME_STUDENT
+16,86,7,I-NAME_STUDENT
+17,93,0,B-NAME_STUDENT
+18,93,1,I-NAME_STUDENT
+19,104,8,B-NAME_STUDENT
+20,104,9,I-NAME_STUDENT
+21,112,6,B-NAME_STUDENT
+22,123,33,B-NAME_STUDENT
+23,123,35,B-ID_NUM
+24,123,38,B-NAME_STUDENT
+25,123,224,B-NAME_STUDENT
+26,123,741,B-URL_PERSONAL
+27,123,1509,B-URL_PERSONAL
+28,123,1549,B-URL_PERSONAL
+29,123,1550,I-URL_PERSONAL
+30,123,1575,B-URL_PERSONAL
+31,123,1591,B-URL_PERSONAL
+32,123,1648,B-URL_PERSONAL
+33,123,1690,B-URL_PERSONAL
+34,123,1691,I-URL_PERSONAL

code/training-actualorig-withotherdata.py

@@ -0,0 +1,446 @@
+# %% [markdown]
+# **Notice: You should run this notebook on a GPU with at least 24GB RAM for avoiding OOM**
+# 
+# # Introduction
+# 
+# Hi all,
+# 
+# In this notebook, I will share the comprehensive steps and strategies I employed to achieve `0.963` on the Private Leaderboard using a single model.
+# 
+# Throughout this competition, I explored various methodologies and refined my approaches to push the model to its limits. This notebook is structured to guide you through each phase of my process.
+# 
+# Whether you're a beginner eager to learn the ropes of competitive machine learning or an experienced practitioner looking for new insights, this notebook aims to provide valuable perspectives and techniques that can be adapted to your future projects.
+# 
+# Let's dive into the details and uncover the techniques that led to a successful submission!
+# 
+# [Submission Notebook PrivateLB 0.963](https://www.kaggle.com/code/haqishen/single-model-no-pp-privatelb-0-963)
+# 
+# # Summary
+# 
+# * With [2k external training samples](https://www.kaggle.com/code/valentinwerner/fix-punctuation-tokenization-external-dataset)
+# * `TRAINING_MAX_LENGTH = 1024`, which should be afforded by many people
+#     * When the text length exceeds 1024, only the first 512 and the last 512 characters of the text will be used for training.
+#     * For inference, no truncation.
+# * `downsample rate = 0.3`, randomly downsample negative samples (articles without any PII tokens) to 30%
+# * `attention_probs_dropout_prob = 0.4`, default value was `0.1`
+# * `weight_decay = 0.`, do not use weight decay for optimizer. default value was `0.01`
+# * `pos_weight = 10`, means for positive samples, weighted 10x for loss calculation.
+#     * As many people knows, for most public submission notebooks, they use predictions with `th > 0.9` to get higher score.
+#     * By adapting `pos_weight`, I was able to achieve best performance with `th around 0.5`
+# * cosine with warmup learning rate schedule -- almost became a standard setting.
+#     * warmup 0.5 epo
+#     * cosine 4.5 epo
+# 
+# # Acknowledge
+# 
+# I was provided Z8G5 Fury workstation with dual RTX6000 ada GPU.
+# 
+# For training a single model this notebook shows, it takes less than 40min!
+# 
+
+# %%
+import os
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+import warnings
+warnings.filterwarnings("ignore")
+DEBUG=False
+
+# %%
+# !pip install -q seqeval
+
+# %%
+import gc
+import json
+import copy
+import time
+import pickle
+import numpy as np
+import pandas as pd
+from itertools import chain
+from transformers import AutoTokenizer
+from transformers import AutoModelForTokenClassification
+from transformers import get_cosine_schedule_with_warmup
+from tqdm import tqdm
+
+from seqeval.metrics import recall_score, precision_score
+from seqeval.metrics import f1_score
+
+import torch
+from torch import nn
+from torch.utils.data import Dataset, DataLoader
+from torch.nn.utils.rnn import pad_sequence
+from torch.nn import CrossEntropyLoss
+from torch.optim import AdamW
+
+import matplotlib.pyplot as plt
+from pylab import rcParams
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+# %% [markdown]
+# # Config
+
+# %%
+exp_name = 'deberta_v3_large_w10_ado4_wd0_bs2_lr1e5_5epo'
+
+backbone = 'microsoft/deberta-v3-large'
+
+TRAINING_MAX_LENGTH = 4096
+batch_size = 2
+lr = 1e-5
+ds_rate = 0.3
+attention_probs_dropout_prob = 0.4
+hidden_dropout_prob = 0.1
+wd = 0.
+pos_weight = 10
+n_epochs = 5
+
+use_amp = True
+
+# %% [markdown]
+# # Data
+
+# %%
+from sklearn.model_selection import train_test_split
+
+data = json.load(open('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/pii-data-detection-3rd-place-solution/datasets/train.json'))
+for sample in data:
+    if len(set(sample['labels'])) > 1:
+        sample['is_pos'] = 1
+    else:
+        sample['is_pos'] = 0
+print("original datapoints: ", len(data))
+
+
+moredata = json.load(open('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/pii-data-detection-3rd-place-solution/datasets/moredata_dataset_fixed.json'))
+print("moredata datapoints: ", len(moredata))
+
+
+# %%
+from sklearn.model_selection import GroupKFold
+import random
+random.seed(0)
+kf = GroupKFold(n_splits=5)
+folds = []
+for i, (_, valid_index) in enumerate(kf.split(np.array(data), groups=[x['document'] for x in data])):
+    folds.append([data[i]['document'] for i in valid_index])
+
+# %%
+all_labels = sorted(list(set(chain(*[x["labels"] for x in data]))))
+label2id = {l: i for i,l in enumerate(all_labels)}
+id2label = {v:k for k,v in label2id.items()}
+
+target = [
+    'B-EMAIL', 'B-ID_NUM', 'B-NAME_STUDENT', 'B-PHONE_NUM', 
+    'B-STREET_ADDRESS', 'B-URL_PERSONAL', 'B-USERNAME', 'I-ID_NUM', 
+    'I-NAME_STUDENT', 'I-PHONE_NUM', 'I-STREET_ADDRESS', 'I-URL_PERSONAL'
+]
+
+print(len(all_labels), id2label)
+
+# %% [markdown]
+# # Dataset
+
+# %%
+def tokenize(example, tokenizer, label2id, max_length):
+
+    # rebuild text from tokens
+    text = []
+    labels = []
+
+    for t, l, ws in zip(
+        example["tokens"], example["provided_labels"], example["trailing_whitespace"]
+    ):
+        text.append(t)
+        labels.extend([l] * len(t))
+
+        if ws:
+            text.append(" ")
+            labels.append("O")
+
+    tokenized = tokenizer("".join(text), return_offsets_mapping=True, truncation=False)
+    if len(tokenized['input_ids']) > max_length:
+        for key in tokenized.keys():
+            tokenized[key] = tokenized[key][:max_length//2] + tokenized[key][-max_length//2:]
+
+    labels = np.array(labels)
+
+    text = "".join(text)
+    token_labels = []
+
+    for start_idx, end_idx in tokenized.offset_mapping:
+        # CLS token
+        if start_idx == 0 and end_idx == 0:
+            token_labels.append(label2id["O"])
+            continue
+
+        # case when token starts with whitespace
+        if text[start_idx].isspace():
+            start_idx += 1
+
+        token_labels.append(label2id[labels[start_idx]])
+
+    length = len(tokenized.input_ids)
+
+    return {**tokenized, "labels": token_labels, "length": length}
+
+# %%
+tokenizer = AutoTokenizer.from_pretrained(backbone)
+
+class PIIDataset(Dataset):
+    def __init__(self, tokenizer, data, label2id, max_length):
+        self.tokenizer = tokenizer
+        self.data = copy.deepcopy(data)
+        self.label2id = label2id
+        self.max_length = max_length
+
+        for i in tqdm(range(len(self.data))):
+            if self.data[i].get('input_ids') is None:
+                self.data[i]['provided_labels'] = self.data[i]['labels']
+                self.data[i] = tokenize(
+                    self.data[i],
+                    self.tokenizer,
+                    self.label2id,
+                    self.max_length
+                )
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        return self.data[idx]
+
+
+def collate_fn(batch):
+    input_ids = [torch.tensor(x['input_ids']) for x in batch]
+    attention_masks = [torch.tensor(x['attention_mask']) for x in batch]
+    labels = [torch.tensor(x['labels']) for x in batch]
+
+    input_ids_padded = pad_sequence(input_ids, batch_first=True, padding_value=0)
+    attention_masks_padded = pad_sequence(attention_masks, batch_first=True, padding_value=0)
+    labels_padded = pad_sequence(labels, batch_first=True, padding_value=-100)
+
+    return {
+        'input_ids': input_ids_padded,
+        'attention_mask': attention_masks_padded,
+        'labels': labels_padded,
+    }
+
+ds = PIIDataset(tokenizer, data[:20], label2id, TRAINING_MAX_LENGTH)
+len(ds[9]['input_ids'])
+
+# %% [markdown]
+# # Model
+
+# %%
+model = AutoModelForTokenClassification.from_pretrained(
+        backbone,
+        attention_probs_dropout_prob=attention_probs_dropout_prob,
+        hidden_dropout_prob=hidden_dropout_prob,
+        num_labels=len(all_labels),
+        id2label=id2label,
+        label2id=label2id,
+        ignore_mismatched_sizes=True
+    )
+
+# %% [markdown]
+# # Loss
+
+# %%
+weights = torch.ones(13) * pos_weight
+weights[12] = 1
+
+CE = nn.CrossEntropyLoss(weight=weights.to(device), ignore_index=-100)
+def criterion(logits, labels):
+    return CE(logits.view(-1, len(all_labels)), labels.view(-1))
+
+# %% [markdown]
+# # Metric
+
+# %%
+def compute_metrics(p, all_labels, th=0.5):
+    predictions, labels = p
+    true_predictions = []
+    true_labels = []
+    for tmp, batch_l in zip(predictions, labels):
+        batch_p = copy.deepcopy(tmp)
+        batch_p[:, :, -1][batch_p[:, :, -1] < th] = -9999
+        batch_p = np.argmax(batch_p, axis=2)
+        true_predictions.extend([all_labels[p] for (p, l) in zip(batch_p.reshape(-1), batch_l.reshape(-1)) if l != -100])
+        true_labels.extend([all_labels[l] for l in batch_l.reshape(-1) if l != -100])
+
+    recall = recall_score([true_labels], [true_predictions])
+    precision = precision_score([true_labels], [true_predictions])
+    f1_score = (1 + 5*5) * recall * precision / (5*5*precision + recall)
+
+    results = {
+        'recall': recall,
+        'precision': precision,
+        'f1': f1_score
+    }
+    return results
+
+
+# %% [markdown]
+# # Train & Valid Func
+
+# %%
+def train_one_epoch(model, loader, optimizer, scheduler, scaler=None):
+    model.train()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    for batch in bar:
+        optimizer.zero_grad()
+        scheduler.step()
+
+        input_ids = batch['input_ids'].to(device)
+        attention_mask = batch['attention_mask'].to(device)
+        labels = batch['labels'].to(device)
+
+        if scaler is not None:
+            with torch.cuda.amp.autocast():
+                outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+                loss = criterion(outputs['logits'], labels)
+            scaler.scale(loss).backward()
+            scaler.step(optimizer)
+            scaler.update()
+        else:
+            outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+            loss = criterion(outputs['logits'], labels)
+            loss.backward()
+            optimizer.step()
+        
+        running_loss.append(loss.item())
+        bar.set_description(f'train loss: {np.mean(running_loss[-20:]):.4f}')
+
+    return np.mean(running_loss)
+
+
+def valid_one_epoch(model, loader):
+    model.eval()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    predictions = []
+    labels = []
+
+    with torch.no_grad():
+        for batch in bar:
+            input_ids = batch['input_ids'].to(device)
+            attention_mask = batch['attention_mask'].to(device)
+            labels_ = batch['labels'].to(device)
+            
+            outputs = model(input_ids, attention_mask=attention_mask, labels=labels_)
+            loss = criterion(outputs['logits'], labels_)
+            
+            running_loss.append(loss.item())
+            predictions.append(outputs['logits'].detach().cpu().numpy())
+            labels.append(labels_.detach().cpu().numpy())
+
+            bar.set_description(f'valid loss: {np.mean(running_loss[-20:]):.4f}')
+
+    return np.mean(running_loss), compute_metrics((predictions, labels), all_labels)
+
+# %% [markdown]
+# # Scheduler
+
+# %%
+rcParams['figure.figsize'] = 20, 2
+
+def get_scheduler(optimizer, num_training_steps, warmup_steps_ratio=0.1):
+    return get_cosine_schedule_with_warmup(
+        optimizer,
+        int(num_training_steps * warmup_steps_ratio),
+        num_training_steps
+    )
+
+opt = AdamW(model.parameters(), lr=lr)
+sch = get_scheduler(opt, 100, 0.1)
+
+lrs = []
+for epoch in range(1, 100+1):
+    sch.step()
+    lrs.append(opt.param_groups[0]["lr"])
+
+plt.plot(range(len(lrs)), lrs)
+
+# %% [markdown]
+# # Run
+
+# %%
+def run(fold,model):
+
+    log_file = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model/logs', f'{exp_name}.txt')
+    model_file = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model', f'{exp_name}_fold{fold}_best.pth')
+
+    valid_doc_ids = folds[fold]
+    train_data = [x for x in data    if (x['document'] not in valid_doc_ids) and ((x['is_pos'] == 1) or (x['is_pos'] == 0 and random.random() < ds_rate))] + moredata
+    
+    valid_data = [x for x in data if x['document'] in valid_doc_ids]
+
+    dataset_train = PIIDataset(tokenizer, train_data, label2id, TRAINING_MAX_LENGTH)
+    dataset_valid = PIIDataset(tokenizer, valid_data, label2id, TRAINING_MAX_LENGTH)
+    print(len(dataset_train), len(dataset_valid))
+
+    loader_train = DataLoader(dataset_train, batch_size=batch_size, shuffle=True, num_workers=8, collate_fn=collate_fn)
+    loader_valid = DataLoader(dataset_valid, batch_size=batch_size, shuffle=False, num_workers=8, collate_fn=collate_fn)
+
+
+    # model = get_model()
+    model = model.to(device)
+
+    optimizer = AdamW(model.parameters(), lr=lr, weight_decay=wd)
+    scaler = torch.cuda.amp.GradScaler() if use_amp else None
+    from_epoch = 0
+    metric_best = 0
+
+    scheduler = get_scheduler(optimizer, n_epochs*len(loader_train), 0.1)
+
+    for epoch in range(1, n_epochs+1):
+        if epoch < from_epoch + 1:
+            print(logs[epoch-1])
+            continue
+
+        print(time.ctime(), 'Epoch:', epoch)
+
+        train_loss = train_one_epoch(model, loader_train, optimizer, scheduler, scaler)
+        valid_loss, result = valid_one_epoch(model, loader_valid)
+        metric = result["f1"]
+
+        content = time.ctime() + ' ' + \
+                f'Fold {fold}, Epoch {epoch}, lr: {optimizer.param_groups[0]["lr"]:.7f}, ' + \
+                f'train loss: {train_loss:.5f}, valid loss: {valid_loss:.5f}, ' + \
+                f'recall: {result["recall"]:.4f} precision: {result["precision"]:.4f} metric: {metric:.4f}.'
+        print(content)
+        with open(log_file, 'a') as appender:
+            appender.write(content + '\n')
+
+        if metric > metric_best:
+            print(f'metric_best ({metric_best:.6f} --> {metric:.6f}). Saving model ...')
+#             if not DEBUG:
+            torch.save(model.state_dict(), model_file)
+            metric_best = metric
+
+        # Save Last
+        if not DEBUG:
+            torch.save(
+                {
+                    'epoch': epoch,
+                    'model_state_dict': model.state_dict(),
+                    'optimizer_state_dict': optimizer.state_dict(),
+                    'scaler_state_dict': scaler.state_dict() if scaler else None,
+                    'metric_best': metric_best,
+                },
+                model_file.replace('_best', '_last')
+            )
+
+    del model
+    torch.cuda.empty_cache()
+    gc.collect()
+
+# %%
+run(0,model)
+
+# %%
+
+
+

code/training-single-model-to-get-privatelb-0-963_actualorig.py

@@ -0,0 +1,446 @@
+# %% [markdown]
+# **Notice: You should run this notebook on a GPU with at least 24GB RAM for avoiding OOM**
+# 
+# # Introduction
+# 
+# Hi all,
+# 
+# In this notebook, I will share the comprehensive steps and strategies I employed to achieve `0.963` on the Private Leaderboard using a single model.
+# 
+# Throughout this competition, I explored various methodologies and refined my approaches to push the model to its limits. This notebook is structured to guide you through each phase of my process.
+# 
+# Whether you're a beginner eager to learn the ropes of competitive machine learning or an experienced practitioner looking for new insights, this notebook aims to provide valuable perspectives and techniques that can be adapted to your future projects.
+# 
+# Let's dive into the details and uncover the techniques that led to a successful submission!
+# 
+# [Submission Notebook PrivateLB 0.963](https://www.kaggle.com/code/haqishen/single-model-no-pp-privatelb-0-963)
+# 
+# # Summary
+# 
+# * With [2k external training samples](https://www.kaggle.com/code/valentinwerner/fix-punctuation-tokenization-external-dataset)
+# * `TRAINING_MAX_LENGTH = 1024`, which should be afforded by many people
+#     * When the text length exceeds 1024, only the first 512 and the last 512 characters of the text will be used for training.
+#     * For inference, no truncation.
+# * `downsample rate = 0.3`, randomly downsample negative samples (articles without any PII tokens) to 30%
+# * `attention_probs_dropout_prob = 0.4`, default value was `0.1`
+# * `weight_decay = 0.`, do not use weight decay for optimizer. default value was `0.01`
+# * `pos_weight = 10`, means for positive samples, weighted 10x for loss calculation.
+#     * As many people knows, for most public submission notebooks, they use predictions with `th > 0.9` to get higher score.
+#     * By adapting `pos_weight`, I was able to achieve best performance with `th around 0.5`
+# * cosine with warmup learning rate schedule -- almost became a standard setting.
+#     * warmup 0.5 epo
+#     * cosine 4.5 epo
+# 
+# # Acknowledge
+# 
+# I was provided Z8G5 Fury workstation with dual RTX6000 ada GPU.
+# 
+# For training a single model this notebook shows, it takes less than 40min!
+# 
+
+# %%
+import os
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+import warnings
+warnings.filterwarnings("ignore")
+DEBUG=False
+
+# %%
+# !pip install -q seqeval
+
+# %%
+import gc
+import json
+import copy
+import time
+import pickle
+import numpy as np
+import pandas as pd
+from itertools import chain
+from transformers import AutoTokenizer
+from transformers import AutoModelForTokenClassification
+from transformers import get_cosine_schedule_with_warmup
+from tqdm import tqdm
+
+from seqeval.metrics import recall_score, precision_score
+from seqeval.metrics import f1_score
+
+import torch
+from torch import nn
+from torch.utils.data import Dataset, DataLoader
+from torch.nn.utils.rnn import pad_sequence
+from torch.nn import CrossEntropyLoss
+from torch.optim import AdamW
+
+import matplotlib.pyplot as plt
+from pylab import rcParams
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+# %% [markdown]
+# # Config
+
+# %%
+exp_name = 'deberta_v3_large_w10_ado4_wd0_bs2_lr1e5_5epo'
+
+backbone = 'microsoft/deberta-v3-large'
+
+TRAINING_MAX_LENGTH = 4096
+batch_size = 2
+lr = 1e-5
+ds_rate = 0.3
+attention_probs_dropout_prob = 0.4
+hidden_dropout_prob = 0.1
+wd = 0.
+pos_weight = 10
+n_epochs = 5
+
+use_amp = True
+
+# %% [markdown]
+# # Data
+
+# %%
+from sklearn.model_selection import train_test_split
+
+data = json.load(open('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/pii-data-detection-3rd-place-solution/datasets/train.json'))
+for sample in data:
+    if len(set(sample['labels'])) > 1:
+        sample['is_pos'] = 1
+    else:
+        sample['is_pos'] = 0
+print("original datapoints: ", len(data))
+
+
+# moredata = json.load(open('/kaggle/input/fix-punctuation-tokenization-external-dataset/moredata_dataset_fixed.json'))
+# print("moredata datapoints: ", len(moredata))
+
+
+# %%
+from sklearn.model_selection import GroupKFold
+import random
+random.seed(0)
+kf = GroupKFold(n_splits=5)
+folds = []
+for i, (_, valid_index) in enumerate(kf.split(np.array(data), groups=[x['document'] for x in data])):
+    folds.append([data[i]['document'] for i in valid_index])
+
+# %%
+all_labels = sorted(list(set(chain(*[x["labels"] for x in data]))))
+label2id = {l: i for i,l in enumerate(all_labels)}
+id2label = {v:k for k,v in label2id.items()}
+
+target = [
+    'B-EMAIL', 'B-ID_NUM', 'B-NAME_STUDENT', 'B-PHONE_NUM', 
+    'B-STREET_ADDRESS', 'B-URL_PERSONAL', 'B-USERNAME', 'I-ID_NUM', 
+    'I-NAME_STUDENT', 'I-PHONE_NUM', 'I-STREET_ADDRESS', 'I-URL_PERSONAL'
+]
+
+print(len(all_labels), id2label)
+
+# %% [markdown]
+# # Dataset
+
+# %%
+def tokenize(example, tokenizer, label2id, max_length):
+
+    # rebuild text from tokens
+    text = []
+    labels = []
+
+    for t, l, ws in zip(
+        example["tokens"], example["provided_labels"], example["trailing_whitespace"]
+    ):
+        text.append(t)
+        labels.extend([l] * len(t))
+
+        if ws:
+            text.append(" ")
+            labels.append("O")
+
+    tokenized = tokenizer("".join(text), return_offsets_mapping=True, truncation=False)
+    if len(tokenized['input_ids']) > max_length:
+        for key in tokenized.keys():
+            tokenized[key] = tokenized[key][:max_length//2] + tokenized[key][-max_length//2:]
+
+    labels = np.array(labels)
+
+    text = "".join(text)
+    token_labels = []
+
+    for start_idx, end_idx in tokenized.offset_mapping:
+        # CLS token
+        if start_idx == 0 and end_idx == 0:
+            token_labels.append(label2id["O"])
+            continue
+
+        # case when token starts with whitespace
+        if text[start_idx].isspace():
+            start_idx += 1
+
+        token_labels.append(label2id[labels[start_idx]])
+
+    length = len(tokenized.input_ids)
+
+    return {**tokenized, "labels": token_labels, "length": length}
+
+# %%
+tokenizer = AutoTokenizer.from_pretrained(backbone)
+
+class PIIDataset(Dataset):
+    def __init__(self, tokenizer, data, label2id, max_length):
+        self.tokenizer = tokenizer
+        self.data = copy.deepcopy(data)
+        self.label2id = label2id
+        self.max_length = max_length
+
+        for i in tqdm(range(len(self.data))):
+            if self.data[i].get('input_ids') is None:
+                self.data[i]['provided_labels'] = self.data[i]['labels']
+                self.data[i] = tokenize(
+                    self.data[i],
+                    self.tokenizer,
+                    self.label2id,
+                    self.max_length
+                )
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        return self.data[idx]
+
+
+def collate_fn(batch):
+    input_ids = [torch.tensor(x['input_ids']) for x in batch]
+    attention_masks = [torch.tensor(x['attention_mask']) for x in batch]
+    labels = [torch.tensor(x['labels']) for x in batch]
+
+    input_ids_padded = pad_sequence(input_ids, batch_first=True, padding_value=0)
+    attention_masks_padded = pad_sequence(attention_masks, batch_first=True, padding_value=0)
+    labels_padded = pad_sequence(labels, batch_first=True, padding_value=-100)
+
+    return {
+        'input_ids': input_ids_padded,
+        'attention_mask': attention_masks_padded,
+        'labels': labels_padded,
+    }
+
+ds = PIIDataset(tokenizer, data[:20], label2id, TRAINING_MAX_LENGTH)
+len(ds[9]['input_ids'])
+
+# %% [markdown]
+# # Model
+
+# %%
+model = AutoModelForTokenClassification.from_pretrained(
+        backbone,
+        attention_probs_dropout_prob=attention_probs_dropout_prob,
+        hidden_dropout_prob=hidden_dropout_prob,
+        num_labels=len(all_labels),
+        id2label=id2label,
+        label2id=label2id,
+        ignore_mismatched_sizes=True
+    )
+
+# %% [markdown]
+# # Loss
+
+# %%
+weights = torch.ones(13) * pos_weight
+weights[12] = 1
+
+CE = nn.CrossEntropyLoss(weight=weights.to(device), ignore_index=-100)
+def criterion(logits, labels):
+    return CE(logits.view(-1, len(all_labels)), labels.view(-1))
+
+# %% [markdown]
+# # Metric
+
+# %%
+def compute_metrics(p, all_labels, th=0.5):
+    predictions, labels = p
+    true_predictions = []
+    true_labels = []
+    for tmp, batch_l in zip(predictions, labels):
+        batch_p = copy.deepcopy(tmp)
+        batch_p[:, :, -1][batch_p[:, :, -1] < th] = -9999
+        batch_p = np.argmax(batch_p, axis=2)
+        true_predictions.extend([all_labels[p] for (p, l) in zip(batch_p.reshape(-1), batch_l.reshape(-1)) if l != -100])
+        true_labels.extend([all_labels[l] for l in batch_l.reshape(-1) if l != -100])
+
+    recall = recall_score([true_labels], [true_predictions])
+    precision = precision_score([true_labels], [true_predictions])
+    f1_score = (1 + 5*5) * recall * precision / (5*5*precision + recall)
+
+    results = {
+        'recall': recall,
+        'precision': precision,
+        'f1': f1_score
+    }
+    return results
+
+
+# %% [markdown]
+# # Train & Valid Func
+
+# %%
+def train_one_epoch(model, loader, optimizer, scheduler, scaler=None):
+    model.train()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    for batch in bar:
+        optimizer.zero_grad()
+        scheduler.step()
+
+        input_ids = batch['input_ids'].to(device)
+        attention_mask = batch['attention_mask'].to(device)
+        labels = batch['labels'].to(device)
+
+        if scaler is not None:
+            with torch.cuda.amp.autocast():
+                outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+                loss = criterion(outputs['logits'], labels)
+            scaler.scale(loss).backward()
+            scaler.step(optimizer)
+            scaler.update()
+        else:
+            outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+            loss = criterion(outputs['logits'], labels)
+            loss.backward()
+            optimizer.step()
+        
+        running_loss.append(loss.item())
+        bar.set_description(f'train loss: {np.mean(running_loss[-20:]):.4f}')
+
+    return np.mean(running_loss)
+
+
+def valid_one_epoch(model, loader):
+    model.eval()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    predictions = []
+    labels = []
+
+    with torch.no_grad():
+        for batch in bar:
+            input_ids = batch['input_ids'].to(device)
+            attention_mask = batch['attention_mask'].to(device)
+            labels_ = batch['labels'].to(device)
+            
+            outputs = model(input_ids, attention_mask=attention_mask, labels=labels_)
+            loss = criterion(outputs['logits'], labels_)
+            
+            running_loss.append(loss.item())
+            predictions.append(outputs['logits'].detach().cpu().numpy())
+            labels.append(labels_.detach().cpu().numpy())
+
+            bar.set_description(f'valid loss: {np.mean(running_loss[-20:]):.4f}')
+
+    return np.mean(running_loss), compute_metrics((predictions, labels), all_labels)
+
+# %% [markdown]
+# # Scheduler
+
+# %%
+rcParams['figure.figsize'] = 20, 2
+
+def get_scheduler(optimizer, num_training_steps, warmup_steps_ratio=0.1):
+    return get_cosine_schedule_with_warmup(
+        optimizer,
+        int(num_training_steps * warmup_steps_ratio),
+        num_training_steps
+    )
+
+opt = AdamW(model.parameters(), lr=lr)
+sch = get_scheduler(opt, 100, 0.1)
+
+lrs = []
+for epoch in range(1, 100+1):
+    sch.step()
+    lrs.append(opt.param_groups[0]["lr"])
+
+plt.plot(range(len(lrs)), lrs)
+
+# %% [markdown]
+# # Run
+
+# %%
+def run(fold,model):
+
+    log_file = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model/logs', f'{exp_name}.txt')
+    model_file = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model', f'{exp_name}_fold{fold}_best.pth')
+
+    valid_doc_ids = folds[fold]
+    # train_data = [x for x in data    if (x['document'] not in valid_doc_ids) and ((x['is_pos'] == 1) or (x['is_pos'] == 0 and random.random() < ds_rate))] + moredata
+    train_data = [x for x in data    if (x['document'] not in valid_doc_ids) and ((x['is_pos'] == 1) or (x['is_pos'] == 0 and random.random() < ds_rate))]
+    valid_data = [x for x in data if x['document'] in valid_doc_ids]
+
+    dataset_train = PIIDataset(tokenizer, train_data, label2id, TRAINING_MAX_LENGTH)
+    dataset_valid = PIIDataset(tokenizer, valid_data, label2id, TRAINING_MAX_LENGTH)
+    print(len(dataset_train), len(dataset_valid))
+
+    loader_train = DataLoader(dataset_train, batch_size=batch_size, shuffle=True, num_workers=8, collate_fn=collate_fn)
+    loader_valid = DataLoader(dataset_valid, batch_size=batch_size, shuffle=False, num_workers=8, collate_fn=collate_fn)
+
+
+    # model = get_model()
+    model = model.to(device)
+
+    optimizer = AdamW(model.parameters(), lr=lr, weight_decay=wd)
+    scaler = torch.cuda.amp.GradScaler() if use_amp else None
+    from_epoch = 0
+    metric_best = 0
+
+    scheduler = get_scheduler(optimizer, n_epochs*len(loader_train), 0.1)
+
+    for epoch in range(1, n_epochs+1):
+        if epoch < from_epoch + 1:
+            print(logs[epoch-1])
+            continue
+
+        print(time.ctime(), 'Epoch:', epoch)
+
+        train_loss = train_one_epoch(model, loader_train, optimizer, scheduler, scaler)
+        valid_loss, result = valid_one_epoch(model, loader_valid)
+        metric = result["f1"]
+
+        content = time.ctime() + ' ' + \
+                f'Fold {fold}, Epoch {epoch}, lr: {optimizer.param_groups[0]["lr"]:.7f}, ' + \
+                f'train loss: {train_loss:.5f}, valid loss: {valid_loss:.5f}, ' + \
+                f'recall: {result["recall"]:.4f} precision: {result["precision"]:.4f} metric: {metric:.4f}.'
+        print(content)
+        with open(log_file, 'a') as appender:
+            appender.write(content + '\n')
+
+        if metric > metric_best:
+            print(f'metric_best ({metric_best:.6f} --> {metric:.6f}). Saving model ...')
+#             if not DEBUG:
+            torch.save(model.state_dict(), model_file)
+            metric_best = metric
+
+        # Save Last
+        if not DEBUG:
+            torch.save(
+                {
+                    'epoch': epoch,
+                    'model_state_dict': model.state_dict(),
+                    'optimizer_state_dict': optimizer.state_dict(),
+                    'scaler_state_dict': scaler.state_dict() if scaler else None,
+                    'metric_best': metric_best,
+                },
+                model_file.replace('_best', '_last')
+            )
+
+    del model
+    torch.cuda.empty_cache()
+    gc.collect()
+
+# %%
+run(0,model)
+
+# %%
+
+
+

code/training-single-model-to-get-privatelb-0-963_orig.py

@@ -0,0 +1,594 @@
+# %% [markdown]
+# **Notice: You should run this notebook on a GPU with at least 24GB RAM for avoiding OOM**
+# 
+# # Introduction
+# 
+# Hi all,
+# 
+# In this notebook, I will share the comprehensive steps and strategies I employed to achieve `0.963` on the Private Leaderboard using a single model.
+# 
+# Throughout this competition, I explored various methodologies and refined my approaches to push the model to its limits. This notebook is structured to guide you through each phase of my process.
+# 
+# Whether you're a beginner eager to learn the ropes of competitive machine learning or an experienced practitioner looking for new insights, this notebook aims to provide valuable perspectives and techniques that can be adapted to your future projects.
+# 
+# Let's dive into the details and uncover the techniques that led to a successful submission!
+# 
+# [Submission Notebook PrivateLB 0.963](https://www.kaggle.com/code/haqishen/single-model-no-pp-privatelb-0-963)
+# 
+# # Summary
+# 
+# * With [2k external training samples](https://www.kaggle.com/code/valentinwerner/fix-punctuation-tokenization-external-dataset)
+# * `TRAINING_MAX_LENGTH = 1024`, which should be afforded by many people
+#     * When the text length exceeds 1024, only the first 512 and the last 512 characters of the text will be used for training.
+#     * For inference, no truncation.
+# * `downsample rate = 0.3`, randomly downsample negative samples (articles without any PII tokens) to 30%
+# * `attention_probs_dropout_prob = 0.4`, default value was `0.1`
+# * `weight_decay = 0.`, do not use weight decay for optimizer. default value was `0.01`
+# * `pos_weight = 10`, means for positive samples, weighted 10x for loss calculation.
+#     * As many people knows, for most public submission notebooks, they use predictions with `th > 0.9` to get higher score.
+#     * By adapting `pos_weight`, I was able to achieve best performance with `th around 0.5`
+# * cosine with warmup learning rate schedule -- almost became a standard setting.
+#     * warmup 0.5 epo
+#     * cosine 4.5 epo
+# 
+# # Acknowledge
+# 
+# I was provided Z8G5 Fury workstation with dual RTX6000 ada GPU.
+# 
+# For training a single model this notebook shows, it takes less than 40min!
+# 
+
+# %%
+import os
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+import warnings
+warnings.filterwarnings("ignore")
+DEBUG=False
+
+USE_SIMPLIFIED_LABELS = False # <<< CHANGE THIS TO False TO USE ORIGINAL BIO LABELS
+
+# %%
+# !pip install -q seqeval
+
+# %%
+import gc
+import json
+import copy
+import time
+import pickle
+import numpy as np
+import pandas as pd
+from itertools import chain
+from transformers import AutoTokenizer
+from transformers import AutoModelForTokenClassification
+from transformers import get_cosine_schedule_with_warmup
+from tqdm import tqdm
+
+from seqeval.metrics import recall_score, precision_score
+from seqeval.metrics import f1_score
+
+import torch
+from torch import nn
+from torch.utils.data import Dataset, DataLoader
+from torch.nn.utils.rnn import pad_sequence
+from torch.nn import CrossEntropyLoss
+from torch.optim import AdamW
+
+import matplotlib.pyplot as plt
+from pylab import rcParams
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+# %% [markdown]
+# # Config
+
+# %%
+exp_name_base = 'deberta_v3_large_w10_ado1_wd0_bs2_lr1e5_5epo' # Changed ado4 to ado1
+exp_name = f"{exp_name_base}_{'simplified_labels' if USE_SIMPLIFIED_LABELS else 'original_bio_labels'}"
+
+
+backbone = 'microsoft/deberta-v3-large'
+
+TRAINING_MAX_LENGTH = 4096
+batch_size = 2
+lr = 1e-5
+ds_rate = 0.3
+attention_probs_dropout_prob = 0.1 # Changed from 0.4 to 0.1 to match config_29th.json
+hidden_dropout_prob = 0.1
+wd = 0.
+pos_weight = 10
+n_epochs = 5
+
+use_amp = True
+
+# %% [markdown]
+# # Data
+
+# %%
+from sklearn.model_selection import train_test_split
+
+data = json.load(open('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/pii-data-detection-3rd-place-solution/datasets/train.json'))
+for sample in data:
+    if len(set(sample['labels'])) > 1:
+        sample['is_pos'] = 1
+    else:
+        sample['is_pos'] = 0
+print("original datapoints: ", len(data))
+
+
+# moredata = json.load(open('/kaggle/input/fix-punctuation-tokenization-external-dataset/moredata_dataset_fixed.json'))
+# print("moredata datapoints: ", len(moredata))
+
+
+# %%
+from sklearn.model_selection import GroupKFold
+import random
+random.seed(0)
+kf = GroupKFold(n_splits=5)
+folds = []
+for i, (_, valid_index) in enumerate(kf.split(np.array(data), groups=[x['document'] for x in data])):
+    folds.append([data[i]['document'] for i in valid_index])
+
+# %%
+# Define simplified labels
+pii_types = [
+    'EMAIL', 'ID_NUM', 'NAME_STUDENT', 'PHONE_NUM', 
+    'STREET_ADDRESS', 'URL_PERSONAL', 'USERNAME'
+]
+all_labels_simplified = sorted(pii_types + ['O'])
+label2id_simplified = {l: i for i, l in enumerate(all_labels_simplified)}
+id2label_simplified = {v: k for k, v in label2id_simplified.items()}
+
+# Original target BIO labels
+original_target_bio = sorted([
+    'B-EMAIL', 'B-ID_NUM', 'B-NAME_STUDENT', 'B-PHONE_NUM', 
+    'B-STREET_ADDRESS', 'B-URL_PERSONAL', 'B-USERNAME', 'I-ID_NUM', 
+    'I-NAME_STUDENT', 'I-PHONE_NUM', 'I-STREET_ADDRESS', 'I-URL_PERSONAL',
+    'O' # Ensure 'O' is part of the original set as well
+])
+# Add I-EMAIL, I-USERNAME if they are possible, though not in the provided list.
+# For now, sticking to the provided original_target_bio and adding 'O'.
+# Ensure 'O' is consistently handled.
+if 'O' not in original_target_bio:
+    original_target_bio_with_o = sorted(list(set(original_target_bio + ['O'])))
+else:
+    original_target_bio_with_o = sorted(list(set(original_target_bio)))
+
+label2id_original_bio = {l: i for i, l in enumerate(original_target_bio_with_o)}
+id2label_original_bio = {v: k for k, v in label2id_original_bio.items()}
+
+
+if USE_SIMPLIFIED_LABELS:
+    all_labels = all_labels_simplified
+    label2id = label2id_simplified
+    id2label = id2label_simplified
+    print(f"Using SIMPLIFIED labels ({len(all_labels)}): {id2label}")
+else:
+    all_labels = original_target_bio_with_o
+    label2id = label2id_original_bio
+    id2label = id2label_original_bio
+    print(f"Using ORIGINAL BIO labels ({len(all_labels)}): {id2label}")
+
+
+# %% [markdown]
+# # Dataset
+
+# %%
+def tokenize(example, tokenizer, current_label2id, max_length, use_simplified): # Pass current_label2id and use_simplified flag
+
+    # rebuild text from tokens
+    text = []
+    char_labels = [] 
+
+    for t, l, ws in zip(
+        example["tokens"], example["provided_labels"], example["trailing_whitespace"]
+    ):
+        text.append(t)
+        current_label = l
+        if use_simplified and l != "O":
+            current_label = l[2:] # Strip "B-" or "I-"
+        
+        # Ensure current_label is in the target label set (either simplified or original BIO)
+        # This is a safeguard. If a label from data isn't in the defined 'all_labels', map it to 'O'.
+        if current_label not in current_label2id:
+            # This case should ideally not happen if data labels are consistent with chosen scheme.
+            # For BIO, 'l' itself should be in label2id. For simplified, l[2:] should be.
+            # print(f"Warning: Label \'{current_label}\' (from original \'{l}\') not in target label set. Mapping to 'O'.")
+            current_label = "O"
+            
+        char_labels.extend([current_label] * len(t))
+
+        if ws:
+            text.append(" ")
+            char_labels.append("O") # Whitespace is always "O"
+
+    tokenized = tokenizer("".join(text), return_offsets_mapping=True, truncation=False)
+    # Truncation strategy
+    if len(tokenized['input_ids']) > max_length:
+        # Implement the "first N/2 + last N/2" truncation strategy
+        first_half_len = max_length // 2
+        second_half_len = max_length - first_half_len # Ensures total length is max_length
+
+        keys_to_truncate = ['input_ids', 'attention_mask', 'offset_mapping']
+        for key in keys_to_truncate:
+            if key in tokenized:
+                original_list = tokenized[key]
+                # This slicing takes the first 'first_half_len' elements and the last 'second_half_len' elements
+                tokenized[key] = original_list[:first_half_len] + original_list[-second_half_len:]
+            # Consider adding a warning if a key is expected but not found, though unlikely for these standard keys.
+
+    char_labels_arr = np.array(char_labels) # This remains after truncation, using the original full char_labels
+
+    text_str = "".join(text) # Renamed from 'text' to avoid conflict
+    token_labels = []
+
+    # Access offset_mapping using dictionary style access as tokenized might be a dict
+    current_offset_mapping = tokenized["offset_mapping"] 
+
+    for start_idx, end_idx in current_offset_mapping:
+        # Handle special tokens like [CLS], [SEP] which have (0,0) offset_mapping
+        # or any other token that doesn't map to actual characters in the original text.
+        if start_idx == end_idx:
+            token_labels.append(current_label2id["O"]) # Assign 'O' to special/empty tokens
+            continue
+
+        # Ensure indices are within the bounds of char_labels_arr and form a valid span
+        # end_idx is exclusive for slicing, so it can be == len(char_labels_arr)
+        if not (start_idx < end_idx and start_idx < len(char_labels_arr) and end_idx <= len(char_labels_arr)):
+            token_labels.append(current_label2id["O"]) # Assign 'O' if token is outside char_labels range or invalid span
+            continue
+        
+        char_span_labels = char_labels_arr[start_idx:end_idx]
+        
+        # If char_span_labels is somehow empty (e.g. if previous check was less strict)
+        if not char_span_labels.size: 
+            token_labels.append(current_label2id["O"])
+            continue
+
+        unique_labels, counts = np.unique(char_span_labels, return_counts=True)
+        
+        pii_labels_present_in_token = []
+        pii_counts_present_in_token = []
+
+        for lbl, count in zip(unique_labels, counts): # lbl is already simplified or original based on earlier logic
+            if lbl != "O": 
+                pii_labels_present_in_token.append(lbl)
+                pii_counts_present_in_token.append(count)
+        
+        if pii_labels_present_in_token:
+            # If PII labels are present, choose the most frequent PII label
+            most_frequent_pii_idx = np.argmax(pii_counts_present_in_token)
+            most_frequent_pii_label = pii_labels_present_in_token[most_frequent_pii_idx]
+            token_labels.append(current_label2id[most_frequent_pii_label])
+        else:
+            # If only \'O\' is present or no PII labels found in the span
+            token_labels.append(current_label2id["O"])
+
+    length = len(tokenized["input_ids"]) # Use dictionary-style access
+
+    return {**tokenized, "labels": token_labels, "length": length}
+
+# %%
+tokenizer = AutoTokenizer.from_pretrained(backbone)
+
+class PIIDataset(Dataset):
+    def __init__(self, tokenizer, data, current_label2id, max_length, use_simplified): # Pass current_label2id and use_simplified
+        self.tokenizer = tokenizer
+        self.data = copy.deepcopy(data)
+        self.current_label2id = current_label2id # Store the active label2id
+        self.max_length = max_length
+        self.use_simplified = use_simplified
+
+        for i in tqdm(range(len(self.data))):
+            if self.data[i].get('input_ids') is None:
+                self.data[i]['provided_labels'] = self.data[i]['labels'] 
+                self.data[i] = tokenize(
+                    self.data[i],
+                    self.tokenizer,
+                    self.current_label2id, # Pass the active label2id
+                    self.max_length,
+                    self.use_simplified # Pass the flag
+                )
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        return self.data[idx]
+
+
+def collate_fn(batch):
+    input_ids = [torch.tensor(x['input_ids']) for x in batch]
+    attention_masks = [torch.tensor(x['attention_mask']) for x in batch]
+    labels = [torch.tensor(x['labels']) for x in batch]
+
+    input_ids_padded = pad_sequence(input_ids, batch_first=True, padding_value=tokenizer.pad_token_id if tokenizer.pad_token_id is not None else 0)
+    attention_masks_padded = pad_sequence(attention_masks, batch_first=True, padding_value=0)
+    labels_padded = pad_sequence(labels, batch_first=True, padding_value=-100)
+
+    return {
+        'input_ids': input_ids_padded,
+        'attention_mask': attention_masks_padded,
+        'labels': labels_padded,
+    }
+
+# Test with simplified labels
+# The global label2id and USE_SIMPLIFIED_LABELS will be used here by PIIDataset
+ds = PIIDataset(tokenizer, data[:20], label2id, TRAINING_MAX_LENGTH, USE_SIMPLIFIED_LABELS)
+print(f"Example tokenized input IDs length: {len(ds[9]['input_ids'])}")
+print(f"Example tokenized labels: {ds[9]['labels']}")
+
+
+# %% [markdown]
+# # Model
+
+# %%
+model = AutoModelForTokenClassification.from_pretrained(
+        backbone,
+        attention_probs_dropout_prob=attention_probs_dropout_prob,
+        hidden_dropout_prob=hidden_dropout_prob,
+        num_labels=len(all_labels), # Use length of active labels
+        id2label=id2label,         # Use active id2label
+        label2id=label2id,         # Use active label2id
+        ignore_mismatched_sizes=True      
+    )
+
+# %% [markdown]
+# # Loss
+
+# %%
+# Adjust weights for the active label set
+weights = torch.ones(len(all_labels)) * pos_weight 
+if 'O' in label2id:
+    weights[label2id['O']] = 1
+else:
+    print(f"Warning: 'O' label not found in active label2id for weight assignment: {label2id}. Assuming last index for 'O'.")
+    weights[-1] = 1
+
+
+CE = nn.CrossEntropyLoss(weight=weights.to(device), ignore_index=-100) # Use general CE
+def criterion(logits, labels):
+    # logits shape: (batch_size, seq_len, num_active_labels)
+    # labels shape: (batch_size, seq_len)
+    return CE(logits.view(-1, len(all_labels)), labels.view(-1)) # Use len(all_labels)
+
+# %% [markdown]
+# # Metric
+
+# %%
+def compute_metrics(p, current_all_labels_list, th=0.5): # Pass the active list of all label strings
+    predictions, labels = p 
+    
+    o_label_str = "O" 
+    o_label_idx = -1
+    # current_all_labels_list is the list of string names for labels, e.g., all_labels
+    for i, lbl_str in enumerate(current_all_labels_list): 
+        if lbl_str == o_label_str:
+            o_label_idx = i
+            break
+    if o_label_idx == -1:
+        print(f"Warning: Could not find 'O' label in {current_all_labels_list} for metric thresholding. Using last index.")
+        o_label_idx = len(current_all_labels_list) - 1
+
+
+    true_predictions = []
+    true_labels = []
+    for tmp, batch_l in zip(predictions, labels): 
+        batch_p_probs = torch.softmax(torch.from_numpy(tmp), dim=-1).numpy() 
+        
+        o_column_probs_slice = batch_p_probs[:, :, o_label_idx] 
+
+        condition_mask = o_column_probs_slice < th 
+
+        o_column_probs_slice[condition_mask] = -9999 
+
+        batch_p_final_ids = np.argmax(batch_p_probs, axis=-1) 
+
+        true_predictions.extend([current_all_labels_list[p_id] for (p_id, l_id) in zip(batch_p_final_ids.reshape(-1), batch_l.reshape(-1)) if l_id != -100])
+        true_labels.extend([current_all_labels_list[l_id] for l_id in batch_l.reshape(-1) if l_id != -100])
+
+    # Using seqeval for P, R, F1 (F5)
+    # seqeval expects lists of lists of strings.
+    # Here true_labels and true_predictions are flat lists.
+    # For proper seqeval, we'd need to group them back by sequence.
+    # However, the original script's compute_metrics calculates overall P, R, F5 on flat lists.
+    # Let's keep that structure for now, as it's what the script used for its internal F5.
+    # Note: This F5 is on simplified labels, not BIO.
+    
+    recall = recall_score([true_labels], [true_predictions], average='micro') # Use micro for overall
+    precision = precision_score([true_labels], [true_predictions], average='micro')
+    # Calculate F5 score manually based on P and R
+    beta = 5
+    f5_score = (1 + beta**2) * precision * recall / ((beta**2 * precision) + recall + 1e-100)
+    if precision + recall == 0: # Handle case where P and R are both zero
+        f5_score = 0.0
+
+    results = {
+        'recall': recall,
+        'precision': precision,
+        'f1': f5_score # Reporting F5 as 'f1' to match original key, or add 'f5'
+    }
+    return results
+
+
+# %% [markdown]
+# # Train & Valid Func
+
+# %%
+def train_one_epoch(model, loader, optimizer, scheduler, scaler=None):
+    model.train()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    for batch in bar:
+        optimizer.zero_grad()
+        
+        input_ids = batch['input_ids'].to(device)
+        attention_mask = batch['attention_mask'].to(device)
+        labels = batch['labels'].to(device)
+
+        if scaler is not None:
+            with torch.cuda.amp.autocast():
+                outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+                loss = criterion(outputs['logits'], labels) # criterion uses simplified label count
+            scaler.scale(loss).backward()
+            scaler.step(optimizer)
+            scaler.update()
+        else:
+            outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+            loss = criterion(outputs['logits'], labels) # criterion uses simplified label count
+            loss.backward()
+            optimizer.step()
+        
+        scheduler.step() # Moved scheduler step here, common practice per batch
+        running_loss.append(loss.item())
+        bar.set_description(f'train loss: {np.mean(running_loss[-20:]):.4f}')
+
+    return np.mean(running_loss)
+
+
+def valid_one_epoch(model, loader):
+    model.eval()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    predictions_logits = [] 
+    true_labels_list = [] 
+
+    with torch.no_grad():
+        for batch in bar:
+            input_ids = batch['input_ids'].to(device)
+            attention_mask = batch['attention_mask'].to(device)
+            labels_batch = batch['labels'].to(device) # Renamed to avoid conflict
+
+            outputs = model(input_ids, attention_mask=attention_mask)
+            loss = criterion(outputs.logits, labels_batch) # Use labels_batch
+            running_loss.append(loss.item())
+            
+            # Store logits and true labels for metric calculation
+            predictions_logits.append(outputs.logits.cpu().numpy())
+            true_labels_list.append(labels_batch.cpu().numpy()) # Use labels_batch
+            
+            bar.set_description(f'valid loss: {np.mean(running_loss[-20:]):.4f}')
+    
+    # Pass the active all_labels (list of strings) to compute_metrics
+    metrics_results = compute_metrics((predictions_logits, true_labels_list), all_labels)
+    return np.mean(running_loss), metrics_results
+
+# %% [markdown]
+# # Scheduler
+
+# %%
+rcParams['figure.figsize'] = 20, 2
+
+def get_scheduler(optimizer, num_training_steps, warmup_steps_ratio=0.1):
+    return get_cosine_schedule_with_warmup(
+        optimizer,
+        int(num_training_steps * warmup_steps_ratio),
+        num_training_steps
+    )
+
+opt = AdamW(model.parameters(), lr=lr)
+sch = get_scheduler(opt, 100, 0.1)
+
+lrs = []
+for epoch in range(1, 100+1):
+    sch.step()
+    lrs.append(opt.param_groups[0]["lr"])
+
+plt.plot(range(len(lrs)), lrs)
+
+# %% [markdown]
+# # Run
+
+# %%
+def run(fold,model_to_train): 
+
+    log_file = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model/logs', f'{exp_name}.txt') # exp_name is now dynamic
+    # Define a directory for saving Hugging Face model (config, weights, tokenizer_config etc.)
+    model_save_dir_best = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model', f'{exp_name}_fold{fold}_best_hf') # exp_name is now dynamic
+    model_save_dir_last = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model', f'{exp_name}_fold{fold}_last_hf') # exp_name is now dynamic
+    
+    # Create directories if they don't exist
+    os.makedirs(model_save_dir_best, exist_ok=True)
+    os.makedirs(model_save_dir_last, exist_ok=True)
+    # Path for optimizer and other states (still using torch.save for these with custom loop)
+    optimizer_etc_file_last = os.path.join(model_save_dir_last, 'optimizer_etc_states.pth')
+
+
+    valid_doc_ids = folds[fold]
+    # train_data = [x for x in data    if (x['document'] not in valid_doc_ids) and ((x['is_pos'] == 1) or (x['is_pos'] == 0 and random.random() < ds_rate))] + moredata
+    train_data = [x for x in data    if (x['document'] not in valid_doc_ids) and ((x['is_pos'] == 1) or (x['is_pos'] == 0 and random.random() < ds_rate))]
+    valid_data = [x for x in data if x['document'] in valid_doc_ids]
+
+    # Use active label2id and USE_SIMPLIFIED_LABELS flag
+    dataset_train = PIIDataset(tokenizer, train_data, label2id, TRAINING_MAX_LENGTH, USE_SIMPLIFIED_LABELS)
+    dataset_valid = PIIDataset(tokenizer, valid_data, label2id, TRAINING_MAX_LENGTH, USE_SIMPLIFIED_LABELS)
+    print(len(dataset_train), len(dataset_valid))
+
+    loader_train = DataLoader(dataset_train, batch_size=batch_size, shuffle=True, num_workers=2, collate_fn=collate_fn) # Reduced num_workers for potential local issues
+    loader_valid = DataLoader(dataset_valid, batch_size=batch_size, shuffle=False, num_workers=2, collate_fn=collate_fn)
+
+
+    # model = get_model() # Original script had this, but model is passed now
+    current_model = model_to_train.to(device) # Use the passed model
+
+    optimizer = AdamW(current_model.parameters(), lr=lr, weight_decay=wd)
+    scaler = torch.cuda.amp.GradScaler() if use_amp else None
+    from_epoch = 0 # Assuming fresh train, not resuming
+    metric_best = 0
+
+    scheduler = get_scheduler(optimizer, n_epochs*len(loader_train), 0.1) # warmup_steps_ratio is 0.1
+
+    for epoch in range(1, n_epochs+1):
+        # if epoch < from_epoch + 1:
+        #     print(logs[epoch-1]) # Assuming logs is not used for fresh train
+        #     continue
+
+        print(time.ctime(), 'Epoch:', epoch)
+
+        train_loss = train_one_epoch(current_model, loader_train, optimizer, scheduler, scaler)
+        valid_loss, result = valid_one_epoch(current_model, loader_valid)
+        metric = result["f1"] # This is F5 score on simplified labels
+
+        content = time.ctime() + ' ' + \
+                f'Fold {fold}, Epoch {epoch}, lr: {optimizer.param_groups[0]["lr"]:.7f}, ' + \
+                f'train loss: {train_loss:.5f}, valid loss: {valid_loss:.5f}, ' + \
+                f'recall: {result["recall"]:.4f} precision: {result["precision"]:.4f} metric (F5 on simplified): {metric:.4f}.'
+        print(content)
+        with open(log_file, 'a') as appender:
+            appender.write(content + '\n')
+
+        if metric > metric_best:
+            print(f'metric_best ({metric_best:.6f} --> {metric:.6f}). Saving model to {model_save_dir_best} ...')
+            if not DEBUG:
+                current_model.save_pretrained(model_save_dir_best, safe_serialization=True)
+                tokenizer.save_pretrained(model_save_dir_best) # Save tokenizer too
+            metric_best = metric
+
+        # Save Last states (model, optimizer, scheduler)
+        if not DEBUG:
+            current_model.save_pretrained(model_save_dir_last, safe_serialization=True)
+            tokenizer.save_pretrained(model_save_dir_last)
+            torch.save(
+                {
+                    'epoch': epoch,
+                    # model_state_dict is not needed if using save_pretrained for the model itself
+                    'optimizer_state_dict': optimizer.state_dict(),
+                    'scheduler_state_dict': scheduler.state_dict(), # Save scheduler state
+                    'scaler_state_dict': scaler.state_dict() if scaler else None,
+                    'metric_best': metric_best,
+                },
+                optimizer_etc_file_last # Save optimizer and other states separately
+            )
+    
+    del current_model # Use current_model
+    torch.cuda.empty_cache()
+    gc.collect()
+
+# %%
+# model is defined globally from "Model" cell. Pass it to run.
+run(0, model)
+
+# %%
+
+
+

code/training-single-model-v8.py

@@ -0,0 +1,572 @@
+# %% [markdown]
+# **Notice: You should run this notebook on a GPU with at least 24GB RAM for avoiding OOM**
+# 
+# # Introduction
+# 
+# Hi all,
+# 
+# In this notebook, I will share the comprehensive steps and strategies I employed to achieve `0.963` on the Private Leaderboard using a single model.
+# 
+# Throughout this competition, I explored various methodologies and refined my approaches to push the model to its limits. This notebook is structured to guide you through each phase of my process.
+# 
+# Whether you're a beginner eager to learn the ropes of competitive machine learning or an experienced practitioner looking for new insights, this notebook aims to provide valuable perspectives and techniques that can be adapted to your future projects.
+# 
+# Let's dive into the details and uncover the techniques that led to a successful submission!
+# 
+# [Submission Notebook PrivateLB 0.963](https://www.kaggle.com/code/haqishen/single-model-no-pp-privatelb-0-963)
+# 
+# # Summary
+# 
+# * With [2k external training samples](https://www.kaggle.com/code/valentinwerner/fix-punctuation-tokenization-external-dataset)
+# * `TRAINING_MAX_LENGTH = 1024`, which should be afforded by many people
+#     * When the text length exceeds 1024, only the first 512 and the last 512 characters of the text will be used for training.
+#     * For inference, no truncation.
+# * `downsample rate = 0.3`, randomly downsample negative samples (articles without any PII tokens) to 30%
+# * `attention_probs_dropout_prob = 0.4`, default value was `0.1`
+# * `weight_decay = 0.`, do not use weight decay for optimizer. default value was `0.01`
+# * `pos_weight = 10`, means for positive samples, weighted 10x for loss calculation.
+#     * As many people knows, for most public submission notebooks, they use predictions with `th > 0.9` to get higher score.
+#     * By adapting `pos_weight`, I was able to achieve best performance with `th around 0.5`
+# * cosine with warmup learning rate schedule -- almost became a standard setting.
+#     * warmup 0.5 epo
+#     * cosine 4.5 epo
+# 
+# # Acknowledge
+# 
+# I was provided Z8G5 Fury workstation with dual RTX6000 ada GPU.
+# 
+# For training a single model this notebook shows, it takes less than 40min!
+# 
+
+# %%
+import os
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+import warnings
+warnings.filterwarnings("ignore")
+DEBUG=False
+
+# %%
+# !pip install -q seqeval
+
+# %%
+import gc
+import json
+import copy
+import time
+import pickle
+import numpy as np
+import pandas as pd
+from itertools import chain
+from transformers import AutoTokenizer
+from transformers import AutoModelForTokenClassification
+from transformers import get_cosine_schedule_with_warmup
+from tqdm import tqdm
+
+from seqeval.metrics import recall_score, precision_score
+from seqeval.metrics import f1_score
+
+import torch
+from torch import nn
+from torch.utils.data import Dataset, DataLoader
+from torch.nn.utils.rnn import pad_sequence
+from torch.nn import CrossEntropyLoss
+from torch.optim import AdamW
+
+import matplotlib.pyplot as plt
+from pylab import rcParams
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+# %% [markdown]
+# # Config
+
+# %%
+exp_name = 'deberta_v3_large_w10_ado4_wd0_bs2_lr1e5_5epo_simplified_labels' # Updated exp_name
+
+backbone = 'microsoft/deberta-v3-large'
+
+TRAINING_MAX_LENGTH = 4096
+batch_size = 2
+lr = 1e-5
+ds_rate = 0.3
+attention_probs_dropout_prob = 0.4
+hidden_dropout_prob = 0.1
+wd = 0.
+pos_weight = 10
+n_epochs = 5
+
+use_amp = True
+
+# %% [markdown]
+# # Data
+
+# %%
+from sklearn.model_selection import train_test_split
+
+data = json.load(open('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/pii-data-detection-3rd-place-solution/datasets/train.json'))
+for sample in data:
+    if len(set(sample['labels'])) > 1:
+        sample['is_pos'] = 1
+    else:
+        sample['is_pos'] = 0
+print("original datapoints: ", len(data))
+
+
+# moredata = json.load(open('/kaggle/input/fix-punctuation-tokenization-external-dataset/moredata_dataset_fixed.json'))
+# print("moredata datapoints: ", len(moredata))
+
+
+# %%
+from sklearn.model_selection import GroupKFold
+import random
+random.seed(0)
+kf = GroupKFold(n_splits=5)
+folds = []
+for i, (_, valid_index) in enumerate(kf.split(np.array(data), groups=[x['document'] for x in data])):
+    folds.append([data[i]['document'] for i in valid_index])
+
+# %%
+# Define simplified labels
+pii_types = [
+    'EMAIL', 'ID_NUM', 'NAME_STUDENT', 'PHONE_NUM', 
+    'STREET_ADDRESS', 'URL_PERSONAL', 'USERNAME'
+]
+# all_labels_simplified will be pii_types + ['O']
+all_labels_simplified = sorted(pii_types + ['O'])
+label2id_simplified = {l: i for i, l in enumerate(all_labels_simplified)}
+id2label_simplified = {v: k for k, v in label2id_simplified.items()}
+
+# Original target for reference or other logic if needed, but not for model head
+original_target_bio = [
+    'B-EMAIL', 'B-ID_NUM', 'B-NAME_STUDENT', 'B-PHONE_NUM', 
+    'B-STREET_ADDRESS', 'B-URL_PERSONAL', 'B-USERNAME', 'I-ID_NUM', 
+    'I-NAME_STUDENT', 'I-PHONE_NUM', 'I-STREET_ADDRESS', 'I-URL_PERSONAL'
+]
+
+print(f"Simplified labels ({len(all_labels_simplified)}): {id2label_simplified}")
+
+# %% [markdown]
+# # Dataset
+
+# %%
+def tokenize(example, tokenizer, label2id, max_length): # label2id here will be label2id_simplified
+
+    # rebuild text from tokens
+    text = []
+    char_labels = [] # Renamed from 'labels' to avoid confusion with example['provided_labels']
+
+    for t, l, ws in zip(
+        example["tokens"], example["provided_labels"], example["trailing_whitespace"]
+    ):
+        text.append(t)
+        # Simplify label: strip B-/I- prefix if it's a PII label
+        current_label = l
+        if l != "O":
+            current_label = l[2:] # Strip "B-" or "I-"
+        char_labels.extend([current_label] * len(t))
+
+        if ws:
+            text.append(" ")
+            char_labels.append("O") # Whitespace is always "O"
+
+    tokenized = tokenizer("".join(text), return_offsets_mapping=True, truncation=False)
+    # Truncation strategy
+    if len(tokenized['input_ids']) > max_length:
+        # Truncate by taking first max_length//2 and last max_length//2 tokens
+        # This strategy is mentioned in the original notebook's markdown for inputs longer than max_length
+        # For 'offset_mapping', we need to be careful. If we truncate input_ids, we must truncate offset_mapping too.
+        # The original code did this:
+        # for key in tokenized.keys():
+        #     tokenized[key] = tokenized[key][:max_length//2] + tokenized[key][-max_length//2:]
+        # This is complex for offset_mapping as the second half's offsets are relative to the original string.
+        # A simpler truncation for now, or ensure max_length is sufficient.
+        # For simplicity, let's use Hugging Face's truncation, but it needs to be enabled in tokenizer call if not already.
+        # The original code had truncation=False then a manual truncation.
+        # Let's stick to the original manual truncation for input_ids and attention_mask,
+        # and apply it carefully to offset_mapping.
+        # However, the provided code snippet for tokenize in the original script does:
+        # tokenized = tokenizer("".join(text), return_offsets_mapping=True, truncation=False)
+        # if len(tokenized['input_ids']) > max_length:
+        #     for key in tokenized.keys():
+        #         tokenized[key] = tokenized[key][:max_length//2] + tokenized[key][-max_length//2:]
+        # This manual truncation is applied *after* tokenization.
+        # Let's re-enable truncation in the tokenizer call for safety, though the manual one might override.
+        # For this modification, we'll assume the existing truncation logic is what the user wants to keep.
+        # The key change is how labels are derived.
+        
+        # Re-applying the original truncation logic if input_ids are too long
+        if len(tokenized['input_ids']) > max_length:
+            first_half_len = max_length // 2
+            second_half_start = len(tokenized['input_ids']) - (max_length - first_half_len) # Ensure total is max_length
+
+            new_tokenized = {}
+            for key in tokenized.keys():
+                if key == 'offset_mapping': # Special handling for offset_mapping if needed, or just truncate
+                     new_tokenized[key] = tokenized[key][:first_half_len] + tokenized[key][second_half_start:]
+                else:
+                    new_tokenized[key] = tokenized[key][:first_half_len] + tokenized[key][second_half_start:]
+            tokenized = new_tokenized
+
+
+    char_labels_arr = np.array(char_labels) # Use the simplified char_labels
+
+    text_str = "".join(text) # Renamed from 'text' to avoid conflict
+    token_labels = []
+
+    # Access offset_mapping using dictionary style access as tokenized might be a dict
+    current_offset_mapping = tokenized["offset_mapping"] 
+
+    for start_idx, end_idx in current_offset_mapping:
+        # CLS token
+        if start_idx == 0 and end_idx == 0:
+            token_labels.append(label2id["O"]) # Use the passed simplified label2id
+            continue
+
+        # case when token starts with whitespace
+        # Ensure start_idx is within bounds of text_str
+        if start_idx < len(text_str) and text_str[start_idx].isspace():
+            start_idx += 1
+        
+        # Ensure start_idx is within bounds of char_labels_arr after potential increment
+        # And handle cases where start_idx might become >= len(char_labels_arr) due to offsets near end
+        if start_idx >= len(char_labels_arr):
+             # This can happen if offset_mapping points beyond the original text due to truncation or special tokens
+             # A safe fallback is 'O' or to ensure char_labels_arr is long enough / handle offsets carefully
+             # Given the original code's `while start_idx >= len(labels): start_idx -= 1`, let's adapt that.
+             current_char_label_idx = min(start_idx, len(char_labels_arr) - 1)
+        else:
+            current_char_label_idx = start_idx
+
+        token_labels.append(label2id[char_labels_arr[current_char_label_idx]])
+
+    length = len(tokenized["input_ids"]) # Use dictionary-style access
+
+    return {**tokenized, "labels": token_labels, "length": length}
+
+# %%
+tokenizer = AutoTokenizer.from_pretrained(backbone)
+
+class PIIDataset(Dataset):
+    def __init__(self, tokenizer, data, label2id, max_length): # label2id will be simplified
+        self.tokenizer = tokenizer
+        self.data = copy.deepcopy(data)
+        self.label2id = label2id
+        self.max_length = max_length
+
+        for i in tqdm(range(len(self.data))):
+            if self.data[i].get('input_ids') is None:
+                self.data[i]['provided_labels'] = self.data[i]['labels'] # Keep original BIO labels here for tokenize
+                self.data[i] = tokenize(
+                    self.data[i],
+                    self.tokenizer,
+                    self.label2id, # Pass simplified label2id
+                    self.max_length
+                )
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        return self.data[idx]
+
+
+def collate_fn(batch):
+    input_ids = [torch.tensor(x['input_ids']) for x in batch]
+    attention_masks = [torch.tensor(x['attention_mask']) for x in batch]
+    labels = [torch.tensor(x['labels']) for x in batch]
+
+    input_ids_padded = pad_sequence(input_ids, batch_first=True, padding_value=tokenizer.pad_token_id if tokenizer.pad_token_id is not None else 0)
+    attention_masks_padded = pad_sequence(attention_masks, batch_first=True, padding_value=0)
+    labels_padded = pad_sequence(labels, batch_first=True, padding_value=-100)
+
+    return {
+        'input_ids': input_ids_padded,
+        'attention_mask': attention_masks_padded,
+        'labels': labels_padded,
+    }
+
+# Test with simplified labels
+ds = PIIDataset(tokenizer, data[:20], label2id_simplified, TRAINING_MAX_LENGTH)
+print(f"Example tokenized input IDs length: {len(ds[9]['input_ids'])}")
+print(f"Example tokenized labels: {ds[9]['labels']}")
+
+
+# %% [markdown]
+# # Model
+
+# %%
+model = AutoModelForTokenClassification.from_pretrained(
+        backbone,
+        attention_probs_dropout_prob=attention_probs_dropout_prob,
+        hidden_dropout_prob=hidden_dropout_prob,
+        num_labels=len(all_labels_simplified), # Use length of simplified labels (8)
+        id2label=id2label_simplified,         # Use simplified id2label
+        label2id=label2id_simplified,         # Use simplified label2id
+        ignore_mismatched_sizes=True      # Good practice if changing head
+    )
+
+# %% [markdown]
+# # Loss
+
+# %%
+# Adjust weights for simplified labels
+weights_simplified = torch.ones(len(all_labels_simplified)) * pos_weight
+# Assuming 'O' is the label for non-PII and its ID is known via label2id_simplified
+if 'O' in label2id_simplified:
+    weights_simplified[label2id_simplified['O']] = 1
+else:
+    # Fallback if 'O' is not named exactly like that, though it should be.
+    # This assumes 'O' is the last class if not found by name.
+    print("Warning: 'O' label not found in label2id_simplified for weight assignment. Assuming last index for 'O'.")
+    weights_simplified[-1] = 1
+
+
+CE_simplified = nn.CrossEntropyLoss(weight=weights_simplified.to(device), ignore_index=-100)
+def criterion(logits, labels):
+    # logits shape: (batch_size, seq_len, num_simplified_labels)
+    # labels shape: (batch_size, seq_len)
+    return CE_simplified(logits.view(-1, len(all_labels_simplified)), labels.view(-1))
+
+# %% [markdown]
+# # Metric
+
+# %%
+def compute_metrics(p, current_all_labels, th=0.5): # Pass simplified all_labels
+    predictions, labels = p # predictions are logits
+    
+    # Determine index of 'O' label in the current_all_labels (simplified)
+    o_label_str = "O" # Assuming 'O' is always named this way
+    o_label_idx = -1
+    for i, lbl_str in enumerate(current_all_labels): # current_all_labels is a list of strings like ['EMAIL', ..., 'O']
+        if lbl_str == o_label_str:
+            o_label_idx = i
+            break
+    if o_label_idx == -1:
+        print(f"Warning: Could not find 'O' label in {current_all_labels} for metric thresholding. Using last index.")
+        o_label_idx = len(current_all_labels) - 1
+
+
+    true_predictions = []
+    true_labels = []
+    for tmp, batch_l in zip(predictions, labels): # tmp contains logits for a batch item
+        batch_p_probs = torch.softmax(torch.from_numpy(tmp), dim=-1).numpy() # Convert logits to probs
+        
+        # Get a view of the 'O' column probabilities
+        # batch_p_probs shape is (batch_size, seq_len, num_simplified_labels)
+        o_column_probs_slice = batch_p_probs[:, :, o_label_idx] # Shape (batch_size, seq_len)
+
+        # Create the boolean mask based on this slice
+        condition_mask = o_column_probs_slice < th # Shape (batch_size, seq_len)
+
+        # Assign to the 'O' column slice using this mask. This modifies batch_p_probs in place.
+        o_column_probs_slice[condition_mask] = -9999 
+
+        batch_p_final_ids = np.argmax(batch_p_probs, axis=-1) # Get final predicted IDs
+
+        true_predictions.extend([current_all_labels[p_id] for (p_id, l_id) in zip(batch_p_final_ids.reshape(-1), batch_l.reshape(-1)) if l_id != -100])
+        true_labels.extend([current_all_labels[l_id] for l_id in batch_l.reshape(-1) if l_id != -100])
+
+    # Using seqeval for P, R, F1 (F5)
+    # seqeval expects lists of lists of strings.
+    # Here true_labels and true_predictions are flat lists.
+    # For proper seqeval, we'd need to group them back by sequence.
+    # However, the original script's compute_metrics calculates overall P, R, F5 on flat lists.
+    # Let's keep that structure for now, as it's what the script used for its internal F5.
+    # Note: This F5 is on simplified labels, not BIO.
+    
+    recall = recall_score([true_labels], [true_predictions], average='micro') # Use micro for overall
+    precision = precision_score([true_labels], [true_predictions], average='micro')
+    # Calculate F5 score manually based on P and R
+    beta = 5
+    f5_score = (1 + beta**2) * precision * recall / ((beta**2 * precision) + recall + 1e-100)
+    if precision + recall == 0: # Handle case where P and R are both zero
+        f5_score = 0.0
+
+    results = {
+        'recall': recall,
+        'precision': precision,
+        'f1': f5_score # Reporting F5 as 'f1' to match original key, or add 'f5'
+    }
+    return results
+
+
+# %% [markdown]
+# # Train & Valid Func
+
+# %%
+def train_one_epoch(model, loader, optimizer, scheduler, scaler=None):
+    model.train()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    for batch in bar:
+        optimizer.zero_grad()
+        
+        input_ids = batch['input_ids'].to(device)
+        attention_mask = batch['attention_mask'].to(device)
+        labels = batch['labels'].to(device)
+
+        if scaler is not None:
+            with torch.cuda.amp.autocast():
+                outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+                loss = criterion(outputs['logits'], labels) # criterion uses simplified label count
+            scaler.scale(loss).backward()
+            scaler.step(optimizer)
+            scaler.update()
+        else:
+            outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+            loss = criterion(outputs['logits'], labels) # criterion uses simplified label count
+            loss.backward()
+            optimizer.step()
+        
+        scheduler.step() # Moved scheduler step here, common practice per batch
+        running_loss.append(loss.item())
+        bar.set_description(f'train loss: {np.mean(running_loss[-20:]):.4f}')
+
+    return np.mean(running_loss)
+
+
+def valid_one_epoch(model, loader):
+    model.eval()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    predictions_logits = [] # Store logits
+    true_labels_list = [] # Store true labels
+
+    with torch.no_grad():
+        for batch in bar:
+            input_ids = batch['input_ids'].to(device)
+            attention_mask = batch['attention_mask'].to(device)
+            labels_ = batch['labels'].to(device)
+            
+            outputs = model(input_ids, attention_mask=attention_mask, labels=labels_)
+            loss = criterion(outputs['logits'], labels_) # criterion uses simplified label count
+            
+            running_loss.append(loss.item())
+            predictions_logits.append(outputs['logits'].detach().cpu().numpy())
+            true_labels_list.append(labels_.detach().cpu().numpy())
+
+            bar.set_description(f'valid loss: {np.mean(running_loss[-20:]):.4f}')
+    
+    # Pass simplified all_labels_simplified to compute_metrics
+    metrics_results = compute_metrics((predictions_logits, true_labels_list), all_labels_simplified)
+    return np.mean(running_loss), metrics_results
+
+# %% [markdown]
+# # Scheduler
+
+# %%
+rcParams['figure.figsize'] = 20, 2
+
+def get_scheduler(optimizer, num_training_steps, warmup_steps_ratio=0.1):
+    return get_cosine_schedule_with_warmup(
+        optimizer,
+        int(num_training_steps * warmup_steps_ratio),
+        num_training_steps
+    )
+
+opt = AdamW(model.parameters(), lr=lr)
+sch = get_scheduler(opt, 100, 0.1)
+
+lrs = []
+for epoch in range(1, 100+1):
+    sch.step()
+    lrs.append(opt.param_groups[0]["lr"])
+
+plt.plot(range(len(lrs)), lrs)
+
+# %% [markdown]
+# # Run
+
+# %%
+def run(fold,model_to_train): # Renamed model to model_to_train to avoid conflict
+
+    log_file = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model/logs', f'{exp_name}.txt')
+    # Define a directory for saving Hugging Face model (config, weights, tokenizer_config etc.)
+    model_save_dir_best = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model', f'{exp_name}_fold{fold}_best_hf')
+    model_save_dir_last = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model', f'{exp_name}_fold{fold}_last_hf')
+    
+    # Create directories if they don't exist
+    os.makedirs(model_save_dir_best, exist_ok=True)
+    os.makedirs(model_save_dir_last, exist_ok=True)
+    # Path for optimizer and other states (still using torch.save for these with custom loop)
+    optimizer_etc_file_last = os.path.join(model_save_dir_last, 'optimizer_etc_states.pth')
+
+
+    valid_doc_ids = folds[fold]
+    # train_data = [x for x in data    if (x['document'] not in valid_doc_ids) and ((x['is_pos'] == 1) or (x['is_pos'] == 0 and random.random() < ds_rate))] + moredata
+    train_data = [x for x in data    if (x['document'] not in valid_doc_ids) and ((x['is_pos'] == 1) or (x['is_pos'] == 0 and random.random() < ds_rate))]
+    valid_data = [x for x in data if x['document'] in valid_doc_ids]
+
+    # Use simplified label2id
+    dataset_train = PIIDataset(tokenizer, train_data, label2id_simplified, TRAINING_MAX_LENGTH)
+    dataset_valid = PIIDataset(tokenizer, valid_data, label2id_simplified, TRAINING_MAX_LENGTH)
+    print(len(dataset_train), len(dataset_valid))
+
+    loader_train = DataLoader(dataset_train, batch_size=batch_size, shuffle=True, num_workers=2, collate_fn=collate_fn) # Reduced num_workers for potential local issues
+    loader_valid = DataLoader(dataset_valid, batch_size=batch_size, shuffle=False, num_workers=2, collate_fn=collate_fn)
+
+
+    # model = get_model() # Original script had this, but model is passed now
+    current_model = model_to_train.to(device) # Use the passed model
+
+    optimizer = AdamW(current_model.parameters(), lr=lr, weight_decay=wd)
+    scaler = torch.cuda.amp.GradScaler() if use_amp else None
+    from_epoch = 0 # Assuming fresh train, not resuming
+    metric_best = 0
+
+    scheduler = get_scheduler(optimizer, n_epochs*len(loader_train), 0.1) # warmup_steps_ratio is 0.1
+
+    for epoch in range(1, n_epochs+1):
+        # if epoch < from_epoch + 1:
+        #     print(logs[epoch-1]) # Assuming logs is not used for fresh train
+        #     continue
+
+        print(time.ctime(), 'Epoch:', epoch)
+
+        train_loss = train_one_epoch(current_model, loader_train, optimizer, scheduler, scaler)
+        valid_loss, result = valid_one_epoch(current_model, loader_valid)
+        metric = result["f1"] # This is F5 score on simplified labels
+
+        content = time.ctime() + ' ' + \
+                f'Fold {fold}, Epoch {epoch}, lr: {optimizer.param_groups[0]["lr"]:.7f}, ' + \
+                f'train loss: {train_loss:.5f}, valid loss: {valid_loss:.5f}, ' + \
+                f'recall: {result["recall"]:.4f} precision: {result["precision"]:.4f} metric (F5 on simplified): {metric:.4f}.'
+        print(content)
+        with open(log_file, 'a') as appender:
+            appender.write(content + '\n')
+
+        if metric > metric_best:
+            print(f'metric_best ({metric_best:.6f} --> {metric:.6f}). Saving model to {model_save_dir_best} ...')
+            if not DEBUG:
+                current_model.save_pretrained(model_save_dir_best, safe_serialization=True)
+                tokenizer.save_pretrained(model_save_dir_best) # Save tokenizer too
+            metric_best = metric
+
+        # Save Last states (model, optimizer, scheduler)
+        if not DEBUG:
+            current_model.save_pretrained(model_save_dir_last, safe_serialization=True)
+            tokenizer.save_pretrained(model_save_dir_last)
+            torch.save(
+                {
+                    'epoch': epoch,
+                    # model_state_dict is not needed if using save_pretrained for the model itself
+                    'optimizer_state_dict': optimizer.state_dict(),
+                    'scheduler_state_dict': scheduler.state_dict(), # Save scheduler state
+                    'scaler_state_dict': scaler.state_dict() if scaler else None,
+                    'metric_best': metric_best,
+                },
+                optimizer_etc_file_last # Save optimizer and other states separately
+            )
+    
+    del current_model # Use current_model
+    torch.cuda.empty_cache()
+    gc.collect()
+
+# %%
+# model is defined globally from "Model" cell. Pass it to run.
+run(0, model)
+
+# %%
+
+
+

code/training-single-model-v9.py

@@ -0,0 +1,564 @@
+# %% [markdown]
+# **Notice: You should run this notebook on a GPU with at least 24GB RAM for avoiding OOM**
+# 
+# # Introduction
+# 
+# Hi all,
+# 
+# In this notebook, I will share the comprehensive steps and strategies I employed to achieve `0.963` on the Private Leaderboard using a single model.
+# 
+# Throughout this competition, I explored various methodologies and refined my approaches to push the model to its limits. This notebook is structured to guide you through each phase of my process.
+# 
+# Whether you're a beginner eager to learn the ropes of competitive machine learning or an experienced practitioner looking for new insights, this notebook aims to provide valuable perspectives and techniques that can be adapted to your future projects.
+# 
+# Let's dive into the details and uncover the techniques that led to a successful submission!
+# 
+# [Submission Notebook PrivateLB 0.963](https://www.kaggle.com/code/haqishen/single-model-no-pp-privatelb-0-963)
+# 
+# # Summary
+# 
+# * With [2k external training samples](https://www.kaggle.com/code/valentinwerner/fix-punctuation-tokenization-external-dataset)
+# * `TRAINING_MAX_LENGTH = 1024`, which should be afforded by many people
+#     * When the text length exceeds 1024, only the first 512 and the last 512 characters of the text will be used for training.
+#     * For inference, no truncation.
+# * `downsample rate = 0.3`, randomly downsample negative samples (articles without any PII tokens) to 30%
+# * `attention_probs_dropout_prob = 0.4`, default value was `0.1`
+# * `weight_decay = 0.`, do not use weight decay for optimizer. default value was `0.01`
+# * `pos_weight = 10`, means for positive samples, weighted 10x for loss calculation.
+#     * As many people knows, for most public submission notebooks, they use predictions with `th > 0.9` to get higher score.
+#     * By adapting `pos_weight`, I was able to achieve best performance with `th around 0.5`
+# * cosine with warmup learning rate schedule -- almost became a standard setting.
+#     * warmup 0.5 epo
+#     * cosine 4.5 epo
+# 
+# # Acknowledge
+# 
+# I was provided Z8G5 Fury workstation with dual RTX6000 ada GPU.
+# 
+# For training a single model this notebook shows, it takes less than 40min!
+# 
+
+# %%
+import os
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+import warnings
+warnings.filterwarnings("ignore")
+DEBUG=False
+
+# %%
+# !pip install -q seqeval
+
+# %%
+import gc
+import json
+import copy
+import time
+import pickle
+import numpy as np
+import pandas as pd
+from itertools import chain
+from transformers import AutoTokenizer
+from transformers import AutoModelForTokenClassification
+from transformers import get_cosine_schedule_with_warmup
+from tqdm import tqdm
+
+from seqeval.metrics import recall_score, precision_score
+from seqeval.metrics import f1_score
+
+import torch
+from torch import nn
+from torch.utils.data import Dataset, DataLoader
+from torch.nn.utils.rnn import pad_sequence
+from torch.nn import CrossEntropyLoss
+from torch.optim import AdamW
+
+import matplotlib.pyplot as plt
+from pylab import rcParams
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+# %% [markdown]
+# # Config
+
+# %%
+exp_name = 'deberta_v3_large_w10_ado4_wd0_bs2_lr1e5_5epo_simplified_labels' # Updated exp_name
+
+backbone = 'microsoft/deberta-v3-large'
+
+TRAINING_MAX_LENGTH = 4096
+batch_size = 2
+lr = 1e-5
+ds_rate = 0.3
+attention_probs_dropout_prob = 0.4
+hidden_dropout_prob = 0.1
+wd = 0.
+pos_weight = 10
+n_epochs = 5
+
+use_amp = True
+
+# %% [markdown]
+# # Data
+
+# %%
+from sklearn.model_selection import train_test_split
+
+data = json.load(open('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/pii-data-detection-3rd-place-solution/datasets/train.json'))
+for sample in data:
+    if len(set(sample['labels'])) > 1:
+        sample['is_pos'] = 1
+    else:
+        sample['is_pos'] = 0
+print("original datapoints: ", len(data))
+
+
+# moredata = json.load(open('/kaggle/input/fix-punctuation-tokenization-external-dataset/moredata_dataset_fixed.json'))
+# print("moredata datapoints: ", len(moredata))
+
+
+# %%
+from sklearn.model_selection import GroupKFold
+import random
+random.seed(0)
+kf = GroupKFold(n_splits=5)
+folds = []
+for i, (_, valid_index) in enumerate(kf.split(np.array(data), groups=[x['document'] for x in data])):
+    folds.append([data[i]['document'] for i in valid_index])
+
+# %%
+# Define simplified labels
+pii_types = [
+    'EMAIL', 'ID_NUM', 'NAME_STUDENT', 'PHONE_NUM', 
+    'STREET_ADDRESS', 'URL_PERSONAL', 'USERNAME'
+]
+# all_labels_simplified will be pii_types + ['O']
+all_labels_simplified = sorted(pii_types + ['O'])
+label2id_simplified = {l: i for i, l in enumerate(all_labels_simplified)}
+id2label_simplified = {v: k for k, v in label2id_simplified.items()}
+
+# Original target for reference or other logic if needed, but not for model head
+original_target_bio = [
+    'B-EMAIL', 'B-ID_NUM', 'B-NAME_STUDENT', 'B-PHONE_NUM', 
+    'B-STREET_ADDRESS', 'B-URL_PERSONAL', 'B-USERNAME', 'I-ID_NUM', 
+    'I-NAME_STUDENT', 'I-PHONE_NUM', 'I-STREET_ADDRESS', 'I-URL_PERSONAL'
+]
+
+print(f"Simplified labels ({len(all_labels_simplified)}): {id2label_simplified}")
+
+# %% [markdown]
+# # Dataset
+
+# %%
+def tokenize(example, tokenizer, label2id, max_length): # label2id here will be label2id_simplified
+
+    # rebuild text from tokens
+    text = []
+    char_labels = [] # Renamed from 'labels' to avoid confusion with example['provided_labels']
+
+    for t, l, ws in zip(
+        example["tokens"], example["provided_labels"], example["trailing_whitespace"]
+    ):
+        text.append(t)
+        # Simplify label: strip B-/I- prefix if it's a PII label
+        current_label = l
+        if l != "O":
+            current_label = l[2:] # Strip "B-" or "I-"
+        char_labels.extend([current_label] * len(t))
+
+        if ws:
+            text.append(" ")
+            char_labels.append("O") # Whitespace is always "O"
+
+    tokenized = tokenizer("".join(text), return_offsets_mapping=True, truncation=False)
+    # Truncation strategy
+    if len(tokenized['input_ids']) > max_length:
+        # Implement the "first N/2 + last N/2" truncation strategy
+        first_half_len = max_length // 2
+        second_half_len = max_length - first_half_len # Ensures total length is max_length
+
+        keys_to_truncate = ['input_ids', 'attention_mask', 'offset_mapping']
+        for key in keys_to_truncate:
+            if key in tokenized:
+                original_list = tokenized[key]
+                # This slicing takes the first 'first_half_len' elements and the last 'second_half_len' elements
+                tokenized[key] = original_list[:first_half_len] + original_list[-second_half_len:]
+            # Consider adding a warning if a key is expected but not found, though unlikely for these standard keys.
+
+    char_labels_arr = np.array(char_labels) # This remains after truncation, using the original full char_labels
+
+    text_str = "".join(text) # Renamed from 'text' to avoid conflict
+    token_labels = []
+
+    # Access offset_mapping using dictionary style access as tokenized might be a dict
+    current_offset_mapping = tokenized["offset_mapping"] 
+
+    for start_idx, end_idx in current_offset_mapping:
+        # Handle special tokens like [CLS], [SEP] which have (0,0) offset_mapping
+        # or any other token that doesn't map to actual characters in the original text.
+        if start_idx == end_idx:
+            token_labels.append(label2id["O"]) # Assign 'O' to special/empty tokens
+            continue
+
+        # Ensure indices are within the bounds of char_labels_arr and form a valid span
+        # end_idx is exclusive for slicing, so it can be == len(char_labels_arr)
+        if not (start_idx < end_idx and start_idx < len(char_labels_arr) and end_idx <= len(char_labels_arr)):
+            token_labels.append(label2id["O"]) # Assign 'O' if token is outside char_labels range or invalid span
+            continue
+        
+        char_span_labels = char_labels_arr[start_idx:end_idx]
+        
+        # If char_span_labels is somehow empty (e.g. if previous check was less strict)
+        if not char_span_labels.size: 
+            token_labels.append(label2id["O"])
+            continue
+
+        unique_labels, counts = np.unique(char_span_labels, return_counts=True)
+        
+        pii_labels_present_in_token = []
+        pii_counts_present_in_token = []
+
+        for label, count in zip(unique_labels, counts):
+            if label != "O": # Assuming "O" is the string for non-PII
+                pii_labels_present_in_token.append(label)
+                pii_counts_present_in_token.append(count)
+        
+        if pii_labels_present_in_token:
+            # If PII labels are present, choose the most frequent PII label
+            most_frequent_pii_idx = np.argmax(pii_counts_present_in_token)
+            most_frequent_pii_label = pii_labels_present_in_token[most_frequent_pii_idx]
+            token_labels.append(label2id[most_frequent_pii_label])
+        else:
+            # If only 'O' is present or no PII labels found in the span
+            token_labels.append(label2id["O"])
+
+    length = len(tokenized["input_ids"]) # Use dictionary-style access
+
+    return {**tokenized, "labels": token_labels, "length": length}
+
+# %%
+tokenizer = AutoTokenizer.from_pretrained(backbone)
+
+class PIIDataset(Dataset):
+    def __init__(self, tokenizer, data, label2id, max_length): # label2id will be simplified
+        self.tokenizer = tokenizer
+        self.data = copy.deepcopy(data)
+        self.label2id = label2id
+        self.max_length = max_length
+
+        for i in tqdm(range(len(self.data))):
+            if self.data[i].get('input_ids') is None:
+                self.data[i]['provided_labels'] = self.data[i]['labels'] # Keep original BIO labels here for tokenize
+                self.data[i] = tokenize(
+                    self.data[i],
+                    self.tokenizer,
+                    self.label2id, # Pass simplified label2id
+                    self.max_length
+                )
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        return self.data[idx]
+
+
+def collate_fn(batch):
+    input_ids = [torch.tensor(x['input_ids']) for x in batch]
+    attention_masks = [torch.tensor(x['attention_mask']) for x in batch]
+    labels = [torch.tensor(x['labels']) for x in batch]
+
+    input_ids_padded = pad_sequence(input_ids, batch_first=True, padding_value=tokenizer.pad_token_id if tokenizer.pad_token_id is not None else 0)
+    attention_masks_padded = pad_sequence(attention_masks, batch_first=True, padding_value=0)
+    labels_padded = pad_sequence(labels, batch_first=True, padding_value=-100)
+
+    return {
+        'input_ids': input_ids_padded,
+        'attention_mask': attention_masks_padded,
+        'labels': labels_padded,
+    }
+
+# Test with simplified labels
+ds = PIIDataset(tokenizer, data[:20], label2id_simplified, TRAINING_MAX_LENGTH)
+print(f"Example tokenized input IDs length: {len(ds[9]['input_ids'])}")
+print(f"Example tokenized labels: {ds[9]['labels']}")
+
+
+# %% [markdown]
+# # Model
+
+# %%
+model = AutoModelForTokenClassification.from_pretrained(
+        backbone,
+        attention_probs_dropout_prob=attention_probs_dropout_prob,
+        hidden_dropout_prob=hidden_dropout_prob,
+        num_labels=len(all_labels_simplified), # Use length of simplified labels (8)
+        id2label=id2label_simplified,         # Use simplified id2label
+        label2id=label2id_simplified,         # Use simplified label2id
+        ignore_mismatched_sizes=True      # Good practice if changing head
+    )
+
+# %% [markdown]
+# # Loss
+
+# %%
+# Adjust weights for simplified labels
+weights_simplified = torch.ones(len(all_labels_simplified)) * pos_weight
+# Assuming 'O' is the label for non-PII and its ID is known via label2id_simplified
+if 'O' in label2id_simplified:
+    weights_simplified[label2id_simplified['O']] = 1
+else:
+    # Fallback if 'O' is not named exactly like that, though it should be.
+    # This assumes 'O' is the last class if not found by name.
+    print("Warning: 'O' label not found in label2id_simplified for weight assignment. Assuming last index for 'O'.")
+    weights_simplified[-1] = 1
+
+
+CE_simplified = nn.CrossEntropyLoss(weight=weights_simplified.to(device), ignore_index=-100)
+def criterion(logits, labels):
+    # logits shape: (batch_size, seq_len, num_simplified_labels)
+    # labels shape: (batch_size, seq_len)
+    return CE_simplified(logits.view(-1, len(all_labels_simplified)), labels.view(-1))
+
+# %% [markdown]
+# # Metric
+
+# %%
+def compute_metrics(p, current_all_labels, th=0.5): # Pass simplified all_labels
+    predictions, labels = p # predictions are logits
+    
+    # Determine index of 'O' label in the current_all_labels (simplified)
+    o_label_str = "O" # Assuming 'O' is always named this way
+    o_label_idx = -1
+    for i, lbl_str in enumerate(current_all_labels): # current_all_labels is a list of strings like ['EMAIL', ..., 'O']
+        if lbl_str == o_label_str:
+            o_label_idx = i
+            break
+    if o_label_idx == -1:
+        print(f"Warning: Could not find 'O' label in {current_all_labels} for metric thresholding. Using last index.")
+        o_label_idx = len(current_all_labels) - 1
+
+
+    true_predictions = []
+    true_labels = []
+    for tmp, batch_l in zip(predictions, labels): # tmp contains logits for a batch item
+        batch_p_probs = torch.softmax(torch.from_numpy(tmp), dim=-1).numpy() # Convert logits to probs
+        
+        # Get a view of the 'O' column probabilities
+        # batch_p_probs shape is (batch_size, seq_len, num_simplified_labels)
+        o_column_probs_slice = batch_p_probs[:, :, o_label_idx] # Shape (batch_size, seq_len)
+
+        # Create the boolean mask based on this slice
+        condition_mask = o_column_probs_slice < th # Shape (batch_size, seq_len)
+
+        # Assign to the 'O' column slice using this mask. This modifies batch_p_probs in place.
+        o_column_probs_slice[condition_mask] = -9999 
+
+        batch_p_final_ids = np.argmax(batch_p_probs, axis=-1) # Get final predicted IDs
+
+        true_predictions.extend([current_all_labels[p_id] for (p_id, l_id) in zip(batch_p_final_ids.reshape(-1), batch_l.reshape(-1)) if l_id != -100])
+        true_labels.extend([current_all_labels[l_id] for l_id in batch_l.reshape(-1) if l_id != -100])
+
+    # Using seqeval for P, R, F1 (F5)
+    # seqeval expects lists of lists of strings.
+    # Here true_labels and true_predictions are flat lists.
+    # For proper seqeval, we'd need to group them back by sequence.
+    # However, the original script's compute_metrics calculates overall P, R, F5 on flat lists.
+    # Let's keep that structure for now, as it's what the script used for its internal F5.
+    # Note: This F5 is on simplified labels, not BIO.
+    
+    recall = recall_score([true_labels], [true_predictions], average='micro') # Use micro for overall
+    precision = precision_score([true_labels], [true_predictions], average='micro')
+    # Calculate F5 score manually based on P and R
+    beta = 5
+    f5_score = (1 + beta**2) * precision * recall / ((beta**2 * precision) + recall + 1e-100)
+    if precision + recall == 0: # Handle case where P and R are both zero
+        f5_score = 0.0
+
+    results = {
+        'recall': recall,
+        'precision': precision,
+        'f1': f5_score # Reporting F5 as 'f1' to match original key, or add 'f5'
+    }
+    return results
+
+
+# %% [markdown]
+# # Train & Valid Func
+
+# %%
+def train_one_epoch(model, loader, optimizer, scheduler, scaler=None):
+    model.train()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    for batch in bar:
+        optimizer.zero_grad()
+        
+        input_ids = batch['input_ids'].to(device)
+        attention_mask = batch['attention_mask'].to(device)
+        labels = batch['labels'].to(device)
+
+        if scaler is not None:
+            with torch.cuda.amp.autocast():
+                outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+                loss = criterion(outputs['logits'], labels) # criterion uses simplified label count
+            scaler.scale(loss).backward()
+            scaler.step(optimizer)
+            scaler.update()
+        else:
+            outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+            loss = criterion(outputs['logits'], labels) # criterion uses simplified label count
+            loss.backward()
+            optimizer.step()
+        
+        scheduler.step() # Moved scheduler step here, common practice per batch
+        running_loss.append(loss.item())
+        bar.set_description(f'train loss: {np.mean(running_loss[-20:]):.4f}')
+
+    return np.mean(running_loss)
+
+
+def valid_one_epoch(model, loader):
+    model.eval()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    predictions_logits = [] # Store logits
+    true_labels_list = [] # Store true labels
+
+    with torch.no_grad():
+        for batch in bar:
+            input_ids = batch['input_ids'].to(device)
+            attention_mask = batch['attention_mask'].to(device)
+            labels_ = batch['labels'].to(device)
+            
+            outputs = model(input_ids, attention_mask=attention_mask, labels=labels_)
+            loss = criterion(outputs['logits'], labels_) # criterion uses simplified label count
+            
+            running_loss.append(loss.item())
+            predictions_logits.append(outputs['logits'].detach().cpu().numpy())
+            true_labels_list.append(labels_.detach().cpu().numpy())
+
+            bar.set_description(f'valid loss: {np.mean(running_loss[-20:]):.4f}')
+    
+    # Pass simplified all_labels_simplified to compute_metrics
+    metrics_results = compute_metrics((predictions_logits, true_labels_list), all_labels_simplified)
+    return np.mean(running_loss), metrics_results
+
+# %% [markdown]
+# # Scheduler
+
+# %%
+rcParams['figure.figsize'] = 20, 2
+
+def get_scheduler(optimizer, num_training_steps, warmup_steps_ratio=0.1):
+    return get_cosine_schedule_with_warmup(
+        optimizer,
+        int(num_training_steps * warmup_steps_ratio),
+        num_training_steps
+    )
+
+opt = AdamW(model.parameters(), lr=lr)
+sch = get_scheduler(opt, 100, 0.1)
+
+lrs = []
+for epoch in range(1, 100+1):
+    sch.step()
+    lrs.append(opt.param_groups[0]["lr"])
+
+plt.plot(range(len(lrs)), lrs)
+
+# %% [markdown]
+# # Run
+
+# %%
+def run(fold,model_to_train): # Renamed model to model_to_train to avoid conflict
+
+    log_file = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model/logs', f'{exp_name}.txt')
+    # Define a directory for saving Hugging Face model (config, weights, tokenizer_config etc.)
+    model_save_dir_best = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model', f'{exp_name}_fold{fold}_best_hf')
+    model_save_dir_last = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model', f'{exp_name}_fold{fold}_last_hf')
+    
+    # Create directories if they don't exist
+    os.makedirs(model_save_dir_best, exist_ok=True)
+    os.makedirs(model_save_dir_last, exist_ok=True)
+    # Path for optimizer and other states (still using torch.save for these with custom loop)
+    optimizer_etc_file_last = os.path.join(model_save_dir_last, 'optimizer_etc_states.pth')
+
+
+    valid_doc_ids = folds[fold]
+    # train_data = [x for x in data    if (x['document'] not in valid_doc_ids) and ((x['is_pos'] == 1) or (x['is_pos'] == 0 and random.random() < ds_rate))] + moredata
+    train_data = [x for x in data    if (x['document'] not in valid_doc_ids) and ((x['is_pos'] == 1) or (x['is_pos'] == 0 and random.random() < ds_rate))]
+    valid_data = [x for x in data if x['document'] in valid_doc_ids]
+
+    # Use simplified label2id
+    dataset_train = PIIDataset(tokenizer, train_data, label2id_simplified, TRAINING_MAX_LENGTH)
+    dataset_valid = PIIDataset(tokenizer, valid_data, label2id_simplified, TRAINING_MAX_LENGTH)
+    print(len(dataset_train), len(dataset_valid))
+
+    loader_train = DataLoader(dataset_train, batch_size=batch_size, shuffle=True, num_workers=2, collate_fn=collate_fn) # Reduced num_workers for potential local issues
+    loader_valid = DataLoader(dataset_valid, batch_size=batch_size, shuffle=False, num_workers=2, collate_fn=collate_fn)
+
+
+    # model = get_model() # Original script had this, but model is passed now
+    current_model = model_to_train.to(device) # Use the passed model
+
+    optimizer = AdamW(current_model.parameters(), lr=lr, weight_decay=wd)
+    scaler = torch.cuda.amp.GradScaler() if use_amp else None
+    from_epoch = 0 # Assuming fresh train, not resuming
+    metric_best = 0
+
+    scheduler = get_scheduler(optimizer, n_epochs*len(loader_train), 0.1) # warmup_steps_ratio is 0.1
+
+    for epoch in range(1, n_epochs+1):
+        # if epoch < from_epoch + 1:
+        #     print(logs[epoch-1]) # Assuming logs is not used for fresh train
+        #     continue
+
+        print(time.ctime(), 'Epoch:', epoch)
+
+        train_loss = train_one_epoch(current_model, loader_train, optimizer, scheduler, scaler)
+        valid_loss, result = valid_one_epoch(current_model, loader_valid)
+        metric = result["f1"] # This is F5 score on simplified labels
+
+        content = time.ctime() + ' ' + \
+                f'Fold {fold}, Epoch {epoch}, lr: {optimizer.param_groups[0]["lr"]:.7f}, ' + \
+                f'train loss: {train_loss:.5f}, valid loss: {valid_loss:.5f}, ' + \
+                f'recall: {result["recall"]:.4f} precision: {result["precision"]:.4f} metric (F5 on simplified): {metric:.4f}.'
+        print(content)
+        with open(log_file, 'a') as appender:
+            appender.write(content + '\n')
+
+        if metric > metric_best:
+            print(f'metric_best ({metric_best:.6f} --> {metric:.6f}). Saving model to {model_save_dir_best} ...')
+            if not DEBUG:
+                current_model.save_pretrained(model_save_dir_best, safe_serialization=True)
+                tokenizer.save_pretrained(model_save_dir_best) # Save tokenizer too
+            metric_best = metric
+
+        # Save Last states (model, optimizer, scheduler)
+        if not DEBUG:
+            current_model.save_pretrained(model_save_dir_last, safe_serialization=True)
+            tokenizer.save_pretrained(model_save_dir_last)
+            torch.save(
+                {
+                    'epoch': epoch,
+                    # model_state_dict is not needed if using save_pretrained for the model itself
+                    'optimizer_state_dict': optimizer.state_dict(),
+                    'scheduler_state_dict': scheduler.state_dict(), # Save scheduler state
+                    'scaler_state_dict': scaler.state_dict() if scaler else None,
+                    'metric_best': metric_best,
+                },
+                optimizer_etc_file_last # Save optimizer and other states separately
+            )
+    
+    del current_model # Use current_model
+    torch.cuda.empty_cache()
+    gc.collect()
+
+# %%
+# model is defined globally from "Model" cell. Pass it to run.
+run(0, model)
+
+# %%
+
+
+

code/training-subm-moredata.py

@@ -0,0 +1,531 @@
+
+# %%
+import os
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+import warnings
+warnings.filterwarnings("ignore")
+DEBUG=False
+
+USE_SIMPLIFIED_LABELS = True # <<< CHANGE THIS TO False TO USE ORIGINAL BIO LABELS
+
+# %%
+# !pip install -q seqeval
+
+# %%
+import gc
+import json
+import copy
+import time
+import pickle
+import numpy as np
+import pandas as pd
+from itertools import chain
+from transformers import AutoTokenizer
+from transformers import AutoModelForTokenClassification
+from transformers import get_cosine_schedule_with_warmup
+from tqdm import tqdm
+
+from seqeval.metrics import recall_score, precision_score
+from seqeval.metrics import f1_score
+
+import torch
+from torch import nn
+from torch.utils.data import Dataset, DataLoader
+from torch.nn.utils.rnn import pad_sequence
+from torch.nn import CrossEntropyLoss
+from torch.optim import AdamW
+
+import matplotlib.pyplot as plt
+from pylab import rcParams
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+# %% [markdown]
+# # Config
+
+# %%
+exp_name_base = 'deberta_v3_large_w10_ado1_wd0_bs2_lr1e5_5epo' # Changed ado4 to ado1
+exp_name = f"{exp_name_base}_{'simplified_labels' if USE_SIMPLIFIED_LABELS else 'original_bio_labels'}"
+
+
+backbone = 'microsoft/deberta-v3-large'
+
+TRAINING_MAX_LENGTH = 4096
+batch_size = 2
+lr = 1e-5
+ds_rate = 0.3
+attention_probs_dropout_prob = 0.4
+hidden_dropout_prob = 0.1
+wd = 0.
+pos_weight = 10
+n_epochs = 5
+
+use_amp = True
+
+# %% [markdown]
+# # Data
+
+# %%
+from sklearn.model_selection import train_test_split
+
+data = json.load(open('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/pii-data-detection-3rd-place-solution/datasets/train.json'))
+for sample in data:
+    if len(set(sample['labels'])) > 1:
+        sample['is_pos'] = 1
+    else:
+        sample['is_pos'] = 0
+print("original datapoints: ", len(data))
+
+
+moredata = json.load(open('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/pii-data-detection-3rd-place-solution/datasets/moredata_dataset_fixed.json'))
+print("moredata datapoints: ", len(moredata))
+
+
+# %%
+from sklearn.model_selection import GroupKFold
+import random
+random.seed(0)
+kf = GroupKFold(n_splits=5)
+folds = []
+for i, (_, valid_index) in enumerate(kf.split(np.array(data), groups=[x['document'] for x in data])):
+    folds.append([data[i]['document'] for i in valid_index])
+
+# %%
+# Define simplified labels
+pii_types = [
+    'EMAIL', 'ID_NUM', 'NAME_STUDENT', 'PHONE_NUM', 
+    'STREET_ADDRESS', 'URL_PERSONAL', 'USERNAME'
+]
+all_labels_simplified = sorted(pii_types + ['O'])
+label2id_simplified = {l: i for i, l in enumerate(all_labels_simplified)}
+id2label_simplified = {v: k for k, v in label2id_simplified.items()}
+
+# Original target BIO labels
+original_target_bio = sorted([
+    'B-EMAIL', 'B-ID_NUM', 'B-NAME_STUDENT', 'B-PHONE_NUM', 
+    'B-STREET_ADDRESS', 'B-URL_PERSONAL', 'B-USERNAME', 'I-ID_NUM', 
+    'I-NAME_STUDENT', 'I-PHONE_NUM', 'I-STREET_ADDRESS', 'I-URL_PERSONAL',
+    'O' 
+])
+
+if 'O' not in original_target_bio:
+    original_target_bio_with_o = sorted(list(set(original_target_bio + ['O'])))
+else:
+    original_target_bio_with_o = sorted(list(set(original_target_bio)))
+
+label2id_original_bio = {l: i for i, l in enumerate(original_target_bio_with_o)}
+id2label_original_bio = {v: k for k, v in label2id_original_bio.items()}
+
+
+if USE_SIMPLIFIED_LABELS:
+    all_labels = all_labels_simplified
+    label2id = label2id_simplified
+    id2label = id2label_simplified
+    print(f"Using SIMPLIFIED labels ({len(all_labels)}): {id2label}")
+else:
+    all_labels = original_target_bio_with_o
+    label2id = label2id_original_bio
+    id2label = id2label_original_bio
+    print(f"Using ORIGINAL BIO labels ({len(all_labels)}): {id2label}")
+
+
+# %% [markdown]
+# # Dataset
+
+# %%
+def tokenize(example, tokenizer, current_label2id, max_length, use_simplified): # Pass current_label2id and use_simplified flag
+
+    # rebuild text from tokens
+    text = []
+    char_labels = [] 
+
+    for t, l, ws in zip(
+        example["tokens"], example["provided_labels"], example["trailing_whitespace"]
+    ):
+        text.append(t)
+        current_label = l
+        if use_simplified and l != "O":
+            current_label = l[2:] # Strip "B-" or "I-"
+        
+        # Ensure current_label is in the target label set (either simplified or original BIO)
+        # If a label from data isn't in the defined 'all_labels', map it to 'O'.
+        if current_label not in current_label2id:
+            current_label = "O"
+            
+        char_labels.extend([current_label] * len(t))
+
+        if ws:
+            text.append(" ")
+            char_labels.append("O") # Whitespace is always "O"
+
+    tokenized = tokenizer("".join(text), return_offsets_mapping=True, truncation=False)
+
+    if len(tokenized['input_ids']) > max_length:
+        # "first N/2 + last N/2" truncation strategy
+        first_half_len = max_length // 2
+        second_half_len = max_length - first_half_len 
+
+        keys_to_truncate = ['input_ids', 'attention_mask', 'offset_mapping']
+        for key in keys_to_truncate:
+            if key in tokenized:
+                original_list = tokenized[key]
+                # takes the first 'first_half_len' elements and the last 'second_half_len' elements
+                tokenized[key] = original_list[:first_half_len] + original_list[-second_half_len:]
+            
+
+    char_labels_arr = np.array(char_labels) 
+
+    text_str = "".join(text)
+    token_labels = []
+
+
+    current_offset_mapping = tokenized["offset_mapping"] 
+
+    for start_idx, end_idx in current_offset_mapping:
+        # Handle special tokens like [CLS], [SEP] which have (0,0) offset_mapping
+
+        if start_idx == end_idx:
+            token_labels.append(current_label2id["O"]) # Assign 'O' to special/empty tokens
+            continue
+
+        # Ensure indices are within the bounds of char_labels_arr and form a valid span
+        # end_idx is exclusive for slicing, so it can be == len(char_labels_arr)
+        if not (start_idx < end_idx and start_idx < len(char_labels_arr) and end_idx <= len(char_labels_arr)):
+            token_labels.append(current_label2id["O"]) # Assign 'O' if token is outside char_labels range or invalid span
+            continue
+        
+        char_span_labels = char_labels_arr[start_idx:end_idx]
+        
+
+        if not char_span_labels.size: 
+            token_labels.append(current_label2id["O"])
+            continue
+
+        unique_labels, counts = np.unique(char_span_labels, return_counts=True)
+        
+        pii_labels_present_in_token = []
+        pii_counts_present_in_token = []
+
+        for lbl, count in zip(unique_labels, counts): # lbl is already simplified or original based on earlier logic
+            if lbl != "O": 
+                pii_labels_present_in_token.append(lbl)
+                pii_counts_present_in_token.append(count)
+        
+        if pii_labels_present_in_token:
+            # If PII labels are present, choose the most frequent PII label
+            most_frequent_pii_idx = np.argmax(pii_counts_present_in_token)
+            most_frequent_pii_label = pii_labels_present_in_token[most_frequent_pii_idx]
+            token_labels.append(current_label2id[most_frequent_pii_label])
+        else:
+            # If only \'O\' is present or no PII labels found in the span
+            token_labels.append(current_label2id["O"])
+
+    length = len(tokenized["input_ids"]) # Use dictionary-style access
+
+    return {**tokenized, "labels": token_labels, "length": length}
+
+# %%
+tokenizer = AutoTokenizer.from_pretrained(backbone)
+
+class PIIDataset(Dataset):
+    def __init__(self, tokenizer, data, current_label2id, max_length, use_simplified): # Pass current_label2id and use_simplified
+        self.tokenizer = tokenizer
+        self.data = copy.deepcopy(data)
+        self.current_label2id = current_label2id # Store the active label2id
+        self.max_length = max_length
+        self.use_simplified = use_simplified
+
+        for i in tqdm(range(len(self.data))):
+            if self.data[i].get('input_ids') is None:
+                self.data[i]['provided_labels'] = self.data[i]['labels'] 
+                self.data[i] = tokenize(
+                    self.data[i],
+                    self.tokenizer,
+                    self.current_label2id, # Pass the active label2id
+                    self.max_length,
+                    self.use_simplified # Pass the flag
+                )
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        return self.data[idx]
+
+
+def collate_fn(batch):
+    input_ids = [torch.tensor(x['input_ids']) for x in batch]
+    attention_masks = [torch.tensor(x['attention_mask']) for x in batch]
+    labels = [torch.tensor(x['labels']) for x in batch]
+
+    input_ids_padded = pad_sequence(input_ids, batch_first=True, padding_value=tokenizer.pad_token_id if tokenizer.pad_token_id is not None else 0)
+    attention_masks_padded = pad_sequence(attention_masks, batch_first=True, padding_value=0)
+    labels_padded = pad_sequence(labels, batch_first=True, padding_value=-100)
+
+    return {
+        'input_ids': input_ids_padded,
+        'attention_mask': attention_masks_padded,
+        'labels': labels_padded,
+    }
+
+# Test with simplified labels
+
+ds = PIIDataset(tokenizer, data[:20], label2id, TRAINING_MAX_LENGTH, USE_SIMPLIFIED_LABELS)
+print(f"Example tokenized input IDs length: {len(ds[9]['input_ids'])}")
+print(f"Example tokenized labels: {ds[9]['labels']}")
+
+
+# %% [markdown]
+# # Model
+
+# %%
+model = AutoModelForTokenClassification.from_pretrained(
+        backbone,
+        attention_probs_dropout_prob=attention_probs_dropout_prob,
+        hidden_dropout_prob=hidden_dropout_prob,
+        num_labels=len(all_labels), # Use length of active labels
+        id2label=id2label,         # Use active id2label
+        label2id=label2id,         # Use active label2id
+        ignore_mismatched_sizes=True      
+    )
+
+# %% [markdown]
+# # Loss
+
+# %%
+
+weights = torch.ones(len(all_labels)) * pos_weight 
+if 'O' in label2id:
+    weights[label2id['O']] = 1
+else:
+    print(f"Warning: 'O' label not found in active label2id for weight assignment: {label2id}. Assuming last index for 'O'.")
+    weights[-1] = 1
+
+
+CE = nn.CrossEntropyLoss(weight=weights.to(device), ignore_index=-100) 
+def criterion(logits, labels):
+
+
+    return CE(logits.view(-1, len(all_labels)), labels.view(-1))
+
+# %% [markdown]
+# # Metric
+
+# %%
+def compute_metrics(p, current_all_labels_list, th=0.5):
+    predictions, labels = p 
+    
+    o_label_str = "O" 
+    o_label_idx = -1
+
+    for i, lbl_str in enumerate(current_all_labels_list): 
+        if lbl_str == o_label_str:
+            o_label_idx = i
+            break
+    if o_label_idx == -1:
+        print(f"Warning: Could not find 'O' label in {current_all_labels_list} for metric thresholding. Using last index.")
+        o_label_idx = len(current_all_labels_list) - 1
+
+
+    true_predictions = []
+    true_labels = []
+    for tmp, batch_l in zip(predictions, labels): 
+        batch_p_probs = torch.softmax(torch.from_numpy(tmp), dim=-1).numpy() 
+        
+        o_column_probs_slice = batch_p_probs[:, :, o_label_idx] 
+
+        condition_mask = o_column_probs_slice < th 
+
+        o_column_probs_slice[condition_mask] = -9999 
+
+        batch_p_final_ids = np.argmax(batch_p_probs, axis=-1) 
+
+        true_predictions.extend([current_all_labels_list[p_id] for (p_id, l_id) in zip(batch_p_final_ids.reshape(-1), batch_l.reshape(-1)) if l_id != -100])
+        true_labels.extend([current_all_labels_list[l_id] for l_id in batch_l.reshape(-1) if l_id != -100])
+    
+    recall = recall_score([true_labels], [true_predictions], average='micro') # Use micro for overall
+    precision = precision_score([true_labels], [true_predictions], average='micro')
+
+    beta = 5
+    f5_score = (1 + beta**2) * precision * recall / ((beta**2 * precision) + recall + 1e-100)
+    if precision + recall == 0: # Handle case where P and R are both zero
+        f5_score = 0.0
+
+    results = {
+        'recall': recall,
+        'precision': precision,
+        'f1': f5_score
+    }
+    return results
+
+
+# %% [markdown]
+# # Train & Valid
+
+# %%
+def train_one_epoch(model, loader, optimizer, scheduler, scaler=None):
+    model.train()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    for batch in bar:
+        optimizer.zero_grad()
+        
+        input_ids = batch['input_ids'].to(device)
+        attention_mask = batch['attention_mask'].to(device)
+        labels = batch['labels'].to(device)
+
+        if scaler is not None:
+            with torch.cuda.amp.autocast():
+                outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+                loss = criterion(outputs['logits'], labels) 
+            scaler.scale(loss).backward()
+            scaler.step(optimizer)
+            scaler.update()
+        else:
+            outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+            loss = criterion(outputs['logits'], labels) 
+            loss.backward()
+            optimizer.step()
+        
+        scheduler.step() 
+        running_loss.append(loss.item())
+        bar.set_description(f'train loss: {np.mean(running_loss[-20:]):.4f}')
+
+    return np.mean(running_loss)
+
+
+def valid_one_epoch(model, loader):
+    model.eval()
+    running_loss = []
+    bar = tqdm(loader, total=len(loader))
+    predictions_logits = [] 
+    true_labels_list = [] 
+
+    with torch.no_grad():
+        for batch in bar:
+            input_ids = batch['input_ids'].to(device)
+            attention_mask = batch['attention_mask'].to(device)
+            labels_batch = batch['labels'].to(device) 
+
+            outputs = model(input_ids, attention_mask=attention_mask)
+            loss = criterion(outputs.logits, labels_batch) 
+            running_loss.append(loss.item())
+            
+            predictions_logits.append(outputs.logits.cpu().numpy())
+            true_labels_list.append(labels_batch.cpu().numpy()) 
+            
+            bar.set_description(f'valid loss: {np.mean(running_loss[-20:]):.4f}')
+    
+    # Pass the active all_labels (list of strings) to compute_metrics
+    metrics_results = compute_metrics((predictions_logits, true_labels_list), all_labels)
+    return np.mean(running_loss), metrics_results
+
+# %% [markdown]
+# # Scheduler
+
+# %%
+rcParams['figure.figsize'] = 20, 2
+
+def get_scheduler(optimizer, num_training_steps, warmup_steps_ratio=0.1):
+    return get_cosine_schedule_with_warmup(
+        optimizer,
+        int(num_training_steps * warmup_steps_ratio),
+        num_training_steps
+    )
+
+opt = AdamW(model.parameters(), lr=lr)
+sch = get_scheduler(opt, 100, 0.1)
+
+lrs = []
+for epoch in range(1, 100+1):
+    sch.step()
+    lrs.append(opt.param_groups[0]["lr"])
+
+plt.plot(range(len(lrs)), lrs)
+
+# %% [markdown]
+# # Run
+
+# %%
+def run(fold,model_to_train): 
+
+    log_file = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model/logs', f'{exp_name}.txt') 
+    
+    model_save_dir_best = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model', f'{exp_name}_fold{fold}_best_hf') 
+    model_save_dir_last = os.path.join('/export/home/acs/stud/t/tudor.farcasanu/SSL_Assignment/single_model/model', f'{exp_name}_fold{fold}_last_hf') 
+    
+    
+    os.makedirs(model_save_dir_best, exist_ok=True)
+    os.makedirs(model_save_dir_last, exist_ok=True)
+    
+    optimizer_etc_file_last = os.path.join(model_save_dir_last, 'optimizer_etc_states.pth')
+
+
+    valid_doc_ids = folds[fold]
+    train_data = [x for x in data    if (x['document'] not in valid_doc_ids) and ((x['is_pos'] == 1) or (x['is_pos'] == 0 and random.random() < ds_rate))] + moredata
+    
+    valid_data = [x for x in data if x['document'] in valid_doc_ids]
+
+    dataset_train = PIIDataset(tokenizer, train_data, label2id, TRAINING_MAX_LENGTH, USE_SIMPLIFIED_LABELS)
+    dataset_valid = PIIDataset(tokenizer, valid_data, label2id, TRAINING_MAX_LENGTH, USE_SIMPLIFIED_LABELS)
+    print(len(dataset_train), len(dataset_valid))
+
+    loader_train = DataLoader(dataset_train, batch_size=batch_size, shuffle=True, num_workers=2, collate_fn=collate_fn)
+    loader_valid = DataLoader(dataset_valid, batch_size=batch_size, shuffle=False, num_workers=2, collate_fn=collate_fn)
+
+    current_model = model_to_train.to(device)
+
+    optimizer = AdamW(current_model.parameters(), lr=lr, weight_decay=wd)
+    scaler = torch.cuda.amp.GradScaler() if use_amp else None
+    from_epoch = 0
+    metric_best = 0
+
+    scheduler = get_scheduler(optimizer, n_epochs*len(loader_train), 0.1)
+
+    for epoch in range(1, n_epochs+1):
+        print(time.ctime(), 'Epoch:', epoch)
+
+        train_loss = train_one_epoch(current_model, loader_train, optimizer, scheduler, scaler)
+        valid_loss, result = valid_one_epoch(current_model, loader_valid)
+        metric = result["f1"] # F5 score on simplified labels
+
+        content = time.ctime() + ' ' + \
+                f'Fold {fold}, Epoch {epoch}, lr: {optimizer.param_groups[0]["lr"]:.7f}, ' + \
+                f'train loss: {train_loss:.5f}, valid loss: {valid_loss:.5f}, ' + \
+                f'recall: {result["recall"]:.4f} precision: {result["precision"]:.4f} metric (F5 on simplified): {metric:.4f}.'
+        print(content)
+        with open(log_file, 'a') as appender:
+            appender.write(content + '\n')
+
+        if metric > metric_best:
+            print(f'metric_best ({metric_best:.6f} --> {metric:.6f}). Saving model to {model_save_dir_best} ...')
+            if not DEBUG:
+                current_model.save_pretrained(model_save_dir_best, safe_serialization=True)
+                tokenizer.save_pretrained(model_save_dir_best)
+            metric_best = metric
+
+        if not DEBUG:
+            current_model.save_pretrained(model_save_dir_last, safe_serialization=True)
+            tokenizer.save_pretrained(model_save_dir_last)
+            torch.save(
+                {
+                    'epoch': epoch,
+                    'optimizer_state_dict': optimizer.state_dict(),
+                    'scheduler_state_dict': scheduler.state_dict(),
+                    'scaler_state_dict': scaler.state_dict() if scaler else None,
+                    'metric_best': metric_best,
+                },
+                optimizer_etc_file_last
+            )
+    
+    del current_model
+    torch.cuda.empty_cache()
+    gc.collect()
+
+# %%
+run(0, model)
+
+# %%
+
+
+

config.json

@@ -0,0 +1,62 @@
+{
+  "_name_or_path": "microsoft/deberta-v3-large",
+  "attention_probs_dropout_prob": 0.1,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2",
+    "3": "LABEL_3",
+    "4": "LABEL_4",
+    "5": "LABEL_5",
+    "6": "LABEL_6",
+    "7": "LABEL_7",
+    "8": "LABEL_8",
+    "9": "LABEL_9",
+    "10": "LABEL_10",
+    "11": "LABEL_11",
+    "12": "LABEL_12"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_10": 10,
+    "LABEL_11": 11,
+    "LABEL_12": 12,
+    "LABEL_2": 2,
+    "LABEL_3": 3,
+    "LABEL_4": 4,
+    "LABEL_5": 5,
+    "LABEL_6": 6,
+    "LABEL_7": 7,
+    "LABEL_8": 8,
+    "LABEL_9": 9
+  },
+  "layer_norm_eps": 1e-07,
+  "max_position_embeddings": 512,
+  "max_relative_positions": -1,
+  "model_type": "deberta-v2",
+  "norm_rel_ebd": "layer_norm",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 0,
+  "pooler_dropout": 0,
+  "pooler_hidden_act": "gelu",
+  "pooler_hidden_size": 1024,
+  "pos_att_type": [
+    "p2c",
+    "c2p"
+  ],
+  "position_biased_input": false,
+  "position_buckets": 256,
+  "relative_attention": true,
+  "share_att_key": true,
+  "target_modules": null,
+  "transformers_version": "4.38.2",
+  "type_vocab_size": 0,
+  "vocab_size": 128100
+}

deberta_v3_large_w10_ado4_wd0_bs2_lr1e5_5epo_fold0_best.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49842d0c1521ba11d9d9d0c86a4ecb5c1b519f72b7a15a6a173be165b72a706f
+size 1736286050

deberta_v3_large_w10_ado4_wd0_bs2_lr1e5_5epo_fold0_last.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c0b7f029d3421c9078ef5f341ce237ff8bc3395beeb625c484343a69204ec13
+size 5208909490

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "[CLS]",
+  "cls_token": "[CLS]",
+  "eos_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd
+size 2464616

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128000": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "[CLS]",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "eos_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "sp_model_kwargs": {},
+  "split_by_punct": false,
+  "tokenizer_class": "DebertaV2Tokenizer",
+  "unk_token": "[UNK]",
+  "vocab_type": "spm"
+}