updated figure 8 notebook

.gitingore

@@ -0,0 +1,11 @@
+data
+ext
+ARCHIVE
+misc/api_keys.txt
+misc/training_tags_categories
+misc/credentials/*
+misc/credentials
+scripts/ARCHIVE
+scripts/CEMETARY
+cemetary
+.venv

jupyter_notebooks/.ipynb_checkpoints/Section_2-3-4_Figure_8_deepfake_adapters-checkpoint.ipynb

@@ -0,0 +1,1941 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "e4cf5af4",
+   "metadata": {},
+   "source": [
+    "# Deepfake Adapter Dataset Processing Pipeline\n",
+    "\n",
+    "This notebook processes the `real_person_adapters.csv` dataset to identify and annotate real people used in deepfake LoRA models.\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "The pipeline consists of 4 main steps:\n",
+    "\n",
+    "1. **NER & Name Cleaning**: Extract clean person names from noisy adapter names\n",
+    "   - Remove emoji, version tags (v1, v2), LoRA-specific terms\n",
+    "   - Extract core person name from complex strings\n",
+    "\n",
+    "2. **Country/Nationality Mapping**: Convert nationality tags to standardized countries\n",
+    "   - Uses `misc/lists/countries.csv` reference\n",
+    "   - Excludes uninhabited territories (Isle of Man, Bouvet Island, etc.)\n",
+    "\n",
+    "3. **LLM-Based Profession Annotation**: Use large language models to identify professions\n",
+    "   - Supports 3 LLM providers: **Qwen**, **Llama**, **Mistral** (large versions)\n",
+    "   - Uses profession list from `misc/lists/professions.csv`\n",
+    "   - Can annotate up to 3 professions per person (e.g., Emma Watson: actor, activist)\n",
+    "\n",
+    "4. **Output Formatting**: Generate annotated dataset with standardized fields\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "### 1. Install Required Packages\n",
+    "\n",
+    "```bash\n",
+    "pip install pandas numpy emoji requests tqdm spacy\n",
+    "\n",
+    "# Download spaCy English model for NER\n",
+    "python -m spacy download en_core_web_sm\n",
+    "```\n",
+    "\n",
+    "### 2. Set Up API Keys\n",
+    "\n",
+    "You need API keys for at least one of the LLM providers:\n",
+    "\n",
+    "- **Qwen (Alibaba Cloud)**: https://dashscope.aliyun.com/\n",
+    "- **Llama (Together AI)**: https://www.together.ai/\n",
+    "- **Mistral AI**: https://mistral.ai/\n",
+    "\n",
+    "Create text files in `misc/credentials/`:\n",
+    "- `qwen_api_key.txt`\n",
+    "- `together_api_key.txt` (for Llama)\n",
+    "- `mistral_api_key.txt`\n",
+    "\n",
+    "Each file should contain only the API key (no extra whitespace).\n",
+    "\n",
+    "See `misc/credentials/README.md` for detailed instructions.\n",
+    "\n",
+    "## Configuration\n",
+    "\n",
+    "In the LLM annotation section, you can configure:\n",
+    "\n",
+    "```python\n",
+    "SELECTED_LLM = 'qwen'  # Choose: 'qwen', 'llama', or 'mistral'\n",
+    "TEST_MODE = True       # True = test on small sample first\n",
+    "TEST_SIZE = 10         # Number of rows to test\n",
+    "MAX_ROWS = 20000       # Maximum rows to process (None = all)\n",
+    "SAVE_INTERVAL = 10     # Save progress every N rows\n",
+    "```\n",
+    "\n",
+    "## Expected Output\n",
+    "\n",
+    "The final annotated dataset will include:\n",
+    "- `real_name`: Cleaned person name\n",
+    "- `full_name`: Full legal/stage name (from LLM)\n",
+    "- `gender`: Gender (from LLM)\n",
+    "- `profession_llm`: Up to 3 professions (from LLM)\n",
+    "- `country`: Primary country (from LLM)\n",
+    "- `tags`: Original tags as list\n",
+    "- `likely_country`: Country inferred from tags\n",
+    "- `likely_nationality`: Nationality inferred from tags\n",
+    "- `likely_profession`: Profession inferred from tags\n",
+    "- `prompt`: Summary of hints used for LLM\n",
+    "\n",
+    "## Cost Considerations\n",
+    "\n",
+    "For ~10,000 entries:\n",
+    "- **Qwen**: Check Alibaba Cloud pricing\n",
+    "- **Llama (Together AI)**: ~$5-10\n",
+    "- **Mistral**: ~$40-80\n",
+    "\n",
+    "**Always test on a small sample first!**\n",
+    "\n",
+    "## Running the Pipeline\n",
+    "\n",
+    "1. Run cells sequentially from top to bottom\n",
+    "2. Start with `TEST_MODE = True` to test on 10 samples\n",
+    "3. Review results and adjust configuration as needed\n",
+    "4. Set `TEST_MODE = False` for full processing\n",
+    "5. The pipeline saves progress automatically and can be resumed if interrupted\n",
+    "\n",
+    "---\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "23d0ae58",
+   "metadata": {},
+   "source": [
+    "# Deepfake Adapter Dataset - LLM Annotation Pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d7942c40",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3c906826",
+   "metadata": {},
+   "source": [
+    "## NER Code"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cb529f57",
+   "metadata": {},
+   "source": [
+    "please put NER code here. on the first collumn of the adapter dataset in data/real_person_adapters.csv named entity recognition should be performed to extract the most plausible name."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "ner_code_cell",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ spaCy model loaded: en_core_web_sm\n",
+      "Loaded 18 rows\n",
+      "\n",
+      "🔄 Processing names with spaCy NER...\n",
+      "\n",
+      "📊 Name cleaning examples (with spaCy NER):\n",
+      "====================================================================================================\n",
+      "Original Name                                      | Cleaned Name                  \n",
+      "====================================================================================================\n",
+      "Super Pose Book Vol.1 - ControlNet                 | Super Pose Book               \n",
+      "Liyuu LoRA                                         | Liyuu                         \n",
+      "HashimotoKanna/ 橋本環奈 _JP_Actress                   | HashimotoKanna                \n",
+      "Emma Watson (JG)                                   | Emma Watson                   \n",
+      "Gal Gadot「LoRa」                                    | Gal Gadot                     \n",
+      "Scarlett Johansson「LoRa」                           | Scarlett Johansson            \n",
+      "Gakki | Aragaki Yui | 新垣結衣                         | Gakki                         \n",
+      "Actress Satomi_石原○○                                | Actress Satomi                \n",
+      "Game of Thrones Cast                               | Game Thrones Cast             \n",
+      "Natalie Portman「LoRa」                              | Natalie Portman               \n",
+      "Emma Watson LoRA                                   | Emma Watson                   \n",
+      "====================================================================================================\n",
+      "\n",
+      "🧪 Leetspeak translation examples:\n",
+      "  4kira LoRA                     -> akira\n",
+      "  3mma Watson v2                 -> Watson\n",
+      "  1rene LORA                     -> irene\n",
+      "  L3vi Ackerman                  -> Levi Ackerman\n",
+      "\n",
+      "📈 Statistics:\n",
+      "  Total rows: 18\n",
+      "  Non-empty names: 18\n",
+      "  Empty names: 0\n",
+      "\n",
+      "🎯 Sample spaCy NER results:\n",
+      "  1. IU\n",
+      "  2. Super Pose Book\n",
+      "  3. Liyuu\n",
+      "  4. Irene\n",
+      "  5. AESPA Karina\n",
+      "  6. Saika Kawakita\n",
+      "  7. Liu Yifei\n",
+      "  8. HashimotoKanna\n",
+      "  9. Emma Watson\n",
+      "  10. Gal Gadot\n",
+      "\n",
+      "✅ Cleaned 18 names using spaCy NER\n",
+      "💾 Saved to /home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/CSV/NER_POI_step01_pre_annotation.csv\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "import emoji\n",
+    "import spacy\n",
+    "\n",
+    "# Load spaCy model\n",
+    "# You may need to download it first: python -m spacy download en_core_web_sm\n",
+    "try:\n",
+    "    nlp = spacy.load(\"en_core_web_sm\")\n",
+    "    print(\"✅ spaCy model loaded: en_core_web_sm\")\n",
+    "except OSError:\n",
+    "    print(\"❌ spaCy model not found. Downloading...\")\n",
+    "    import subprocess\n",
+    "    subprocess.run([\"python\", \"-m\", \"spacy\", \"download\", \"en_core_web_sm\"])\n",
+    "    nlp = spacy.load(\"en_core_web_sm\")\n",
+    "    print(\"✅ spaCy model downloaded and loaded\")\n",
+    "\n",
+    "# Set up paths\n",
+    "current_dir = Path.cwd()\n",
+    "#input_file = current_dir.parent / \"data/CSV/real_person_adapters.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/samples/real_person_adapters_sample.csv\"\n",
+    "\n",
+    "# Load dataset\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "\n",
+    "def translate_leetspeak(text: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Translate common leetspeak patterns to normal letters.\n",
+    "    Examples: 4kira -> Akira, 3mma -> Emma, 1rene -> Irene\n",
+    "    \"\"\"\n",
+    "    if not text:\n",
+    "        return text\n",
+    "    \n",
+    "    # Common leetspeak mappings (order matters!)\n",
+    "    leetspeak_map = {\n",
+    "        '4': 'a',\n",
+    "        '3': 'e', \n",
+    "        '1': 'i',\n",
+    "        '0': 'o',\n",
+    "        '7': 't',\n",
+    "        '5': 's',\n",
+    "        '8': 'b',\n",
+    "        '9': 'g',\n",
+    "        '@': 'a',\n",
+    "        '$': 's',\n",
+    "        '!': 'i',\n",
+    "    }\n",
+    "    \n",
+    "    result = text\n",
+    "    # Apply mappings at word boundaries or start of string\n",
+    "    for leet, normal in leetspeak_map.items():\n",
+    "        # Replace at start of word\n",
+    "        result = re.sub(rf'\\b{re.escape(leet)}', normal, result, flags=re.IGNORECASE)\n",
+    "        # Replace standalone numbers that look like letters in context\n",
+    "        result = re.sub(rf'(?<=[a-z]){re.escape(leet)}(?=[a-z])', normal, result, flags=re.IGNORECASE)\n",
+    "    \n",
+    "    return result\n",
+    "\n",
+    "def preprocess_for_ner(name: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Preprocess the name before spaCy NER.\n",
+    "    Remove noise but keep the actual name parts.\n",
+    "    \"\"\"\n",
+    "    if pd.isna(name):\n",
+    "        return \"\"\n",
+    "    \n",
+    "    name = str(name)\n",
+    "    \n",
+    "    # FIRST: Translate leetspeak\n",
+    "    name = translate_leetspeak(name)\n",
+    "    \n",
+    "    # Remove emoji\n",
+    "    name = emoji.replace_emoji(name, replace=' ')\n",
+    "    \n",
+    "    # Remove version indicators (v1, v2, v1.0, etc.)\n",
+    "    name = re.sub(r'\\s*[vV]\\d+(\\.\\d+)?\\s*', ' ', name)\n",
+    "    \n",
+    "    # Remove LoRA-related terms (case insensitive)\n",
+    "    lora_terms = ['lora', 'loha', 'lycoris', 'controlnet', 'textual inversion', \n",
+    "                  'embedding', 'ti', 'checkpoint', 'model', 'adapter']\n",
+    "    for term in lora_terms:\n",
+    "        name = re.sub(rf'\\b{term}\\b', '', name, flags=re.IGNORECASE)\n",
+    "    \n",
+    "    # Remove content in parentheses or brackets (often metadata)\n",
+    "    name = re.sub(r'\\([^)]*\\)', '', name)\n",
+    "    name = re.sub(r'\\[[^\\]]*\\]', '', name)\n",
+    "    \n",
+    "    # Remove special characters like 「」\n",
+    "    name = re.sub(r'[「」『』【】〈〉《》]', '', name)\n",
+    "    \n",
+    "    # Handle pipe - keep first part\n",
+    "    if '|' in name:\n",
+    "        name = name.split('|')[0]\n",
+    "    \n",
+    "    # Handle forward slash - keep first part\n",
+    "    if '/' in name:\n",
+    "        name = name.split('/')[0]\n",
+    "    \n",
+    "    # Replace underscores with spaces\n",
+    "    name = name.replace('_', ' ')\n",
+    "    \n",
+    "    # Remove multiple spaces\n",
+    "    name = re.sub(r'\\s+', ' ', name)\n",
+    "    \n",
+    "    # Strip\n",
+    "    name = name.strip()\n",
+    "    \n",
+    "    return name\n",
+    "\n",
+    "def extract_person_name(text: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Use spaCy NER to extract person names from text.\n",
+    "    Falls back to cleaned text if no PERSON entity found.\n",
+    "    \"\"\"\n",
+    "    if not text:\n",
+    "        return \"\"\n",
+    "    \n",
+    "    # Run spaCy NER\n",
+    "    doc = nlp(text)\n",
+    "    \n",
+    "    # Extract PERSON entities\n",
+    "    person_entities = [ent.text for ent in doc.ents if ent.label_ == \"PERSON\"]\n",
+    "    \n",
+    "    if person_entities:\n",
+    "        # Return the first (usually longest/best) person name\n",
+    "        return person_entities[0].strip()\n",
+    "    \n",
+    "    # If no PERSON entity found, try to extract capitalized words (likely names)\n",
+    "    # This helps with names spaCy might miss\n",
+    "    words = text.split()\n",
+    "    capitalized_words = [w for w in words if w and w[0].isupper() and len(w) > 1]\n",
+    "    \n",
+    "    if capitalized_words:\n",
+    "        # Join first few capitalized words (likely the name)\n",
+    "        return ' '.join(capitalized_words[:3]).strip()\n",
+    "    \n",
+    "    # Last resort: return cleaned text\n",
+    "    return text.strip()\n",
+    "\n",
+    "def clean_name_with_spacy(name: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Complete name cleaning pipeline with spaCy NER.\n",
+    "    \n",
+    "    Pipeline:\n",
+    "    1. Translate leetspeak (4→a, 3→e, 1→i, etc.)\n",
+    "    2. Remove noise (emoji, version tags, LoRA terms)\n",
+    "    3. Use spaCy to extract PERSON entities\n",
+    "    4. Fallback to capitalized words or cleaned text\n",
+    "    \"\"\"\n",
+    "    # Step 1 & 2: Preprocess (leetspeak + noise removal)\n",
+    "    preprocessed = preprocess_for_ner(name)\n",
+    "    \n",
+    "    if not preprocessed:\n",
+    "        return \"\"\n",
+    "    \n",
+    "    # Step 3: Extract person name using spaCy NER\n",
+    "    person_name = extract_person_name(preprocessed)\n",
+    "    \n",
+    "    return person_name\n",
+    "\n",
+    "# Apply name cleaning with spaCy\n",
+    "print(\"\\n🔄 Processing names with spaCy NER...\")\n",
+    "df['real_name'] = df['name'].apply(clean_name_with_spacy)\n",
+    "\n",
+    "# Show examples with detailed comparison\n",
+    "print(\"\\n📊 Name cleaning examples (with spaCy NER):\")\n",
+    "print(\"=\" * 100)\n",
+    "print(f\"{'Original Name':<50} | {'Cleaned Name':<30}\")\n",
+    "print(\"=\" * 100)\n",
+    "\n",
+    "examples = df[['name', 'real_name']].head(30)\n",
+    "shown = 0\n",
+    "for idx, row in examples.iterrows():\n",
+    "    if row['name'] != row['real_name'] and shown < 20:\n",
+    "        print(f\"{row['name']:<50} | {row['real_name']:<30}\")\n",
+    "        shown += 1\n",
+    "\n",
+    "print(\"=\" * 100)\n",
+    "\n",
+    "# Show specific test cases\n",
+    "print(\"\\n🧪 Leetspeak translation examples:\")\n",
+    "test_names = ['4kira LoRA', '3mma Watson v2', '1rene LORA', 'L3vi Ackerman']\n",
+    "for test in test_names:\n",
+    "    result = clean_name_with_spacy(test)\n",
+    "    print(f\"  {test:<30} -> {result}\")\n",
+    "\n",
+    "# Statistics\n",
+    "print(f\"\\n📈 Statistics:\")\n",
+    "print(f\"  Total rows: {len(df)}\")\n",
+    "print(f\"  Non-empty names: {(df['real_name'] != '').sum()}\")\n",
+    "print(f\"  Empty names: {(df['real_name'] == '').sum()}\")\n",
+    "\n",
+    "# Show some examples of what spaCy identified\n",
+    "print(\"\\n🎯 Sample spaCy NER results:\")\n",
+    "sample_names = df['real_name'].head(20).tolist()\n",
+    "for i, name in enumerate(sample_names[:10], 1):\n",
+    "    if name:\n",
+    "        print(f\"  {i}. {name}\")\n",
+    "\n",
+    "print(f\"\\n✅ Cleaned {len(df)} names using spaCy NER\")\n",
+    "\n",
+    "# Save intermediate result\n",
+    "output_step1 = current_dir.parent / \"data/CSV/NER_POI_step01_pre_annotation.csv\"\n",
+    "df.to_csv(output_step1, index=False)\n",
+    "print(f\"💾 Saved to {output_step1}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d701d1cb",
+   "metadata": {},
+   "source": [
+    "## nationality to Country\n",
+    "here tags related to nationality gets converted to the country equivalent. make sure isle of man is shows never up in the country collumn.as well as other uninhabited places."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "36ce71f2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Built country/nationality mapping with 781 entries\n",
+      "\n",
+      "Processed 18 rows\n",
+      "Rows with country: 0\n",
+      "Rows with nationality: 0\n",
+      "Rows with profession: 14\n",
+      "\n",
+      "Top 10 countries:\n",
+      "Series([], Name: count, dtype: int64)\n",
+      "\n",
+      "Sample results:\n",
+      "         real_name likely_country likely_nationality likely_profession\n",
+      "0               IU                                           celebrity\n",
+      "1  Super Pose Book                                                    \n",
+      "2            Liyuu                                                    \n",
+      "3            Irene                                           celebrity\n",
+      "4     AESPA Karina                                           celebrity\n",
+      "5   Saika Kawakita                                           celebrity\n",
+      "6        Liu Yifei                                           celebrity\n",
+      "7   HashimotoKanna                                               actor\n",
+      "8      Emma Watson                                               actor\n",
+      "9        Gal Gadot                                               model\n",
+      "\n",
+      "🔍 Checking 'Irene' entries:\n",
+      "  Name: Irene\n",
+      "  Tags: ['girl', 'photorealistic', 'asian', 'woman', 'beautiful', 'celebrity', 'korean']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from pathlib import Path\n",
+    "\n",
+    "# Set up paths\n",
+    "current_dir = Path.cwd()\n",
+    "countries_file = current_dir.parent / \"misc/lists/countries.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step01_pre_annotation.csv\"\n",
+    "output_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "\n",
+    "# Load datasets\n",
+    "poi_df = pd.read_csv(input_file)\n",
+    "countries_df = pd.read_csv(countries_file)\n",
+    "professions_df = pd.read_csv(professions_file)\n",
+    "\n",
+    "# Define uninhabited or non-relevant territories to exclude\n",
+    "excluded_territories = {\n",
+    "    'isle of man', 'bouvet island', 'heard island and mcdonald islands',\n",
+    "    'french southern territories', 'south georgia and the south sandwich islands',\n",
+    "    'svalbard and jan mayen', 'british indian ocean territory', 'antarctica',\n",
+    "    'christmas island', 'cocos (keeling) islands', 'norfolk island',\n",
+    "    'pitcairn', 'tokelau', 'united states minor outlying islands',\n",
+    "    'wallis and futuna', 'western sahara'\n",
+    "}\n",
+    "\n",
+    "# Step 1: Combine tags into one lowercase list\n",
+    "def combine_tags(row):\n",
+    "    return [str(row[f\"tag_{i}\"]).strip().lower() for i in range(1, 8) if pd.notna(row.get(f\"tag_{i}\"))]\n",
+    "\n",
+    "poi_df[\"tags\"] = poi_df.apply(combine_tags, axis=1)\n",
+    "\n",
+    "# Step 2: Build tag → (country, nationality) mapping with PRIORITIES\n",
+    "tag_to_country_nationality = {}\n",
+    "# We'll use a priority score: direct country name = 3, nationality = 2, word parts = 1\n",
+    "\n",
+    "for _, row in countries_df.iterrows():\n",
+    "    country = str(row[\"en_short_name\"]).strip()\n",
+    "    nationality = str(row[\"nationality\"]).strip()\n",
+    "    \n",
+    "    # Skip excluded territories\n",
+    "    if country.lower() in excluded_territories:\n",
+    "        continue\n",
+    "\n",
+    "    country_lc = country.lower()\n",
+    "    nationality_lc = nationality.lower()\n",
+    "\n",
+    "    # Store as (country, nationality, priority)\n",
+    "    # Exact country name match = highest priority\n",
+    "    if country_lc not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[country_lc] = (country, \"\", 3)\n",
+    "    \n",
+    "    # Exact nationality match = medium priority  \n",
+    "    if nationality_lc not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[nationality_lc] = (\"\", nationality, 2)\n",
+    "    \n",
+    "    # No-space versions\n",
+    "    country_no_space = country_lc.replace(\" \", \"\")\n",
+    "    nationality_no_space = nationality_lc.replace(\" \", \"\")\n",
+    "    \n",
+    "    if country_no_space not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[country_no_space] = (country, \"\", 3)\n",
+    "    if nationality_no_space not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[nationality_no_space] = (\"\", nationality, 2)\n",
+    "\n",
+    "    # Word parts = lowest priority (only for longer words to avoid false matches)\n",
+    "    for part in country_lc.split():\n",
+    "        if len(part) > 4:  # Only words longer than 4 chars\n",
+    "            if part not in tag_to_country_nationality:\n",
+    "                tag_to_country_nationality[part] = (country, \"\", 1)\n",
+    "    for part in nationality_lc.split():\n",
+    "        if len(part) > 4:\n",
+    "            if part not in tag_to_country_nationality:\n",
+    "                tag_to_country_nationality[part] = (\"\", nationality, 1)\n",
+    "\n",
+    "print(f\"Built country/nationality mapping with {len(tag_to_country_nationality)} entries\")\n",
+    "\n",
+    "# Step 3: Infer likely_country and likely_nationality by checking ALL tags\n",
+    "def infer_country_and_nationality(tags):\n",
+    "    \"\"\"\n",
+    "    Check ALL tags and return the best match based on priority.\n",
+    "    Priority: exact country name > nationality > word parts\n",
+    "    \"\"\"\n",
+    "    best_match = None\n",
+    "    best_priority = 0\n",
+    "    \n",
+    "    for tag in tags:\n",
+    "        # Try cleaned version (no spaces)\n",
+    "        cleaned = tag.replace(\" \", \"\").lower()\n",
+    "        \n",
+    "        # Check cleaned version\n",
+    "        if cleaned in tag_to_country_nationality:\n",
+    "            country, nationality, priority = tag_to_country_nationality[cleaned]\n",
+    "            if priority > best_priority and country and country.lower() not in excluded_territories:\n",
+    "                best_match = (country, nationality)\n",
+    "                best_priority = priority\n",
+    "        \n",
+    "        # Also check original tag\n",
+    "        if tag in tag_to_country_nationality:\n",
+    "            country, nationality, priority = tag_to_country_nationality[tag]\n",
+    "            if priority > best_priority and country and country.lower() not in excluded_territories:\n",
+    "                best_match = (country, nationality)\n",
+    "                best_priority = priority\n",
+    "    \n",
+    "    if best_match:\n",
+    "        return pd.Series(best_match)\n",
+    "    return pd.Series([\"\", \"\"])\n",
+    "\n",
+    "poi_df[[\"likely_country\", \"likely_nationality\"]] = poi_df[\"tags\"].apply(infer_country_and_nationality)\n",
+    "\n",
+    "# Step 4: Build tag → profession mapping\n",
+    "profession_alias_map = {}\n",
+    "\n",
+    "for _, row in professions_df.iterrows():\n",
+    "    canonical = str(row['profession']).strip().lower()\n",
+    "    profession_alias_map[canonical] = canonical\n",
+    "    for alias_col in ['alias_1', 'alias_2', 'alias_3']:\n",
+    "        alias = row.get(alias_col)\n",
+    "        if pd.notna(alias):\n",
+    "            profession_alias_map[str(alias).strip().lower()] = canonical\n",
+    "\n",
+    "# Step 5: Infer likely profession from tags\n",
+    "def infer_profession_from_tags(tags):\n",
+    "    matched = []\n",
+    "    for tag in tags:\n",
+    "        cleaned = tag.strip().lower()\n",
+    "        if cleaned in profession_alias_map:\n",
+    "            matched.append(profession_alias_map[cleaned])\n",
+    "\n",
+    "    if not matched:\n",
+    "        return \"\"\n",
+    "    if \"celebrity\" in matched and len(set(matched)) > 1:\n",
+    "        # Drop 'celebrity' if other professions are present\n",
+    "        matched = [m for m in matched if m != \"celebrity\"]\n",
+    "\n",
+    "    return matched[0]  # Return the first specific match\n",
+    "\n",
+    "\n",
+    "poi_df[\"likely_profession\"] = poi_df[\"tags\"].apply(infer_profession_from_tags)\n",
+    "\n",
+    "# Step 6: Save enriched dataset\n",
+    "poi_df.to_csv(output_file, index=False)\n",
+    "\n",
+    "# Preview results\n",
+    "print(f\"\\nProcessed {len(poi_df)} rows\")\n",
+    "print(f\"Rows with country: {(poi_df['likely_country'] != '').sum()}\")\n",
+    "print(f\"Rows with nationality: {(poi_df['likely_nationality'] != '').sum()}\")\n",
+    "print(f\"Rows with profession: {(poi_df['likely_profession'] != '').sum()}\")\n",
+    "\n",
+    "print(f\"\\nTop 10 countries:\")\n",
+    "print(poi_df[poi_df['likely_country'] != '']['likely_country'].value_counts().head(10))\n",
+    "\n",
+    "print(f\"\\nSample results:\")\n",
+    "print(poi_df[[\"real_name\", \"likely_country\", \"likely_nationality\", \"likely_profession\"]].head(10))\n",
+    "\n",
+    "# Check specific case: Irene with korean tag\n",
+    "irene_rows = poi_df[poi_df['real_name'].str.contains('irene', case=False, na=False)]\n",
+    "if len(irene_rows) > 0:\n",
+    "    print(f\"\\n🔍 Checking 'Irene' entries:\")\n",
+    "    for idx, row in irene_rows.iterrows():\n",
+    "        print(f\"  Name: {row['real_name']}\")\n",
+    "        print(f\"  Tags: {row['tags']}\")\n",
+    "        print(f\"  Country: {row['likely_country']}\")\n",
+    "        print(f\"  Nationality: {row['likely_nationality']}\")\n",
+    "        print()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "29eec5be",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "deepseek_header",
+   "metadata": {},
+   "source": [
+    "## Deepseek annotation (Testing)\n",
+    "\n",
+    "**Use this section first to test if the pipeline works** before using Qwen/Llama/Mistral.\n",
+    "\n",
+    "This uses your existing Deepseek credits. The code is adapted from your original implementation but updated to:\n",
+    "- Use the correct input file from Step 2\n",
+    "- Include the profession list from `misc/lists/professions.csv`\n",
+    "- Allow up to 3 professions per person\n",
+    "- Match the output format of the other LLMs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "deepseek_code",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "❌ Error: deepseek_api_key.txt not found in misc/credentials/\n",
+      "   Please create this file with your API key\n"
+     ]
+    },
+    {
+     "ename": "FileNotFoundError",
+     "evalue": "[Errno 2] No such file or directory: '/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/misc/credentials/deepseek_api_key.txt'",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[10], line 37\u001b[0m\n\u001b[1;32m     35\u001b[0m \u001b[38;5;66;03m# === LOAD API KEY ===\u001b[39;00m\n\u001b[1;32m     36\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 37\u001b[0m     \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mapi_key_file\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[1;32m     38\u001b[0m         api_key \u001b[38;5;241m=\u001b[39m f\u001b[38;5;241m.\u001b[39mread()\u001b[38;5;241m.\u001b[39mstrip()\n\u001b[1;32m     39\u001b[0m     \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m✅ Deepseek API key loaded\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
+      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/IPython/core/interactiveshell.py:324\u001b[0m, in \u001b[0;36m_modified_open\u001b[0;34m(file, *args, **kwargs)\u001b[0m\n\u001b[1;32m    317\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m}:\n\u001b[1;32m    318\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m    319\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIPython won\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m by default \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    320\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    321\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myou can use builtins\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m open.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    322\u001b[0m     )\n\u001b[0;32m--> 324\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mio_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/misc/credentials/deepseek_api_key.txt'"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import requests\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from openai import OpenAI\n",
+    "from tqdm import tqdm\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "professions_mapped_file = current_dir.parent / \"misc/lists/professions_mapped.csv\"\n",
+    "api_key_file = current_dir.parent / \"misc/credentials/deepseek_api_key.txt\"\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = True       # Set to False to process full dataset\n",
+    "TEST_SIZE = 10         # Number of rows to test\n",
+    "MAX_ROWS = 20000\n",
+    "SAVE_INTERVAL = 10     # Save every 10 rows\n",
+    "\n",
+    "# Define the SPECIFIC profession categories\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\",\n",
+    "    \"adult performer\",\n",
+    "    \"singer/musician\",\n",
+    "    \"model\",\n",
+    "    \"online personality\",\n",
+    "    \"public figure\",\n",
+    "    \"voice actor/ASMR\",\n",
+    "    \"sports professional\",\n",
+    "    \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD API KEY ===\n",
+    "try:\n",
+    "    with open(api_key_file) as f:\n",
+    "        api_key = f.read().strip()\n",
+    "    print(\"✅ Deepseek API key loaded\")\n",
+    "except FileNotFoundError:\n",
+    "    print(\"❌ Error: deepseek_api_key.txt not found in misc/credentials/\")\n",
+    "    print(\"   Please create this file with your API key\")\n",
+    "    raise\n",
+    "\n",
+    "client = OpenAI(\n",
+    "    api_key=api_key,\n",
+    "    base_url=\"https://api.deepseek.com/v1\"\n",
+    ")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(input_file)\n",
+    "professions_df = pd.read_csv(professions_file)\n",
+    "\n",
+    "# Load profession categories and keywords\n",
+    "try:\n",
+    "    prof_mapped_df = pd.read_csv(professions_mapped_file)\n",
+    "    print(f\"✅ Loaded profession mapping with {len(prof_mapped_df)} categories\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions_mapped.csv not found, using default categories\")\n",
+    "\n",
+    "profession_str = \", \".join(PROFESSION_CATEGORIES)\n",
+    "\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "print(f\"\\nProfession categories ({len(PROFESSION_CATEGORIES)}):\")\n",
+    "for cat in PROFESSION_CATEGORIES:\n",
+    "    print(f\"  - {cat}\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"\\nRunning in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPTS ===\n",
+    "def create_prompt(row):\n",
+    "    \"\"\"Create prompt for Deepseek annotation with specific profession categories.\"\"\"\n",
+    "    name = row['real_name'] if pd.notna(row.get('real_name')) else row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if we don't have enough hints\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Given '{name}' ({hint_text}), provide:\n",
+    "1. Full legal name (Western order if non-latin script)\n",
+    "2. Any stage names/aliases (comma separated)\n",
+    "3. Gender (Male/Female/Other/Unknown)\n",
+    "4. Top 3 most likely professions from ONLY these categories:\n",
+    "   - actor\n",
+    "   - adult performer\n",
+    "   - singer/musician\n",
+    "   - model\n",
+    "   - online personality (includes streamers, cosplayers, influencers)\n",
+    "   - public figure (includes politicians, activists, journalists, authors)\n",
+    "   - voice actor/ASMR\n",
+    "   - sports professional\n",
+    "   - tv personality (includes hosts, presenters, reality TV)\n",
+    "\n",
+    "5. Primary country associated\n",
+    "\n",
+    "IMPORTANT:\n",
+    "- Choose professions ONLY from the 9 categories above\n",
+    "- Provide up to 3 professions, comma-separated, ordered by relevance\n",
+    "- Be SPECIFIC: choose the most accurate category for each role\n",
+    "- \"online personality\" includes: streamers, cosplayers, YouTubers, influencers, content creators\n",
+    "- Use 'Unknown' when uncertain or for fictional characters/places\n",
+    "- For multi-role people, list all relevant categories (e.g., \"actor, singer/musician, online personality\")\n",
+    "\n",
+    "Respond with exactly 5 numbered lines.\"\"\"\n",
+    "\n",
+    "# Create prompts\n",
+    "df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "\n",
+    "# === QUERY DEEPSEEK ===\n",
+    "def query_deepseek(prompt):\n",
+    "    \"\"\"Query Deepseek API.\"\"\"\n",
+    "    try:\n",
+    "        response = client.chat.completions.create(\n",
+    "            model=\"deepseek-chat\",\n",
+    "            messages=[\n",
+    "                {\"role\": \"system\", \"content\": \"You are an assistant that extracts key data on a person based on the name. Respond with exactly 5 numbered lines. For professions, choose ONLY from these categories: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality.\"},\n",
+    "                {\"role\": \"user\", \"content\": prompt}\n",
+    "            ],\n",
+    "            temperature=0.05,\n",
+    "            top_p=0.8,\n",
+    "        )\n",
+    "        return response.choices[0].message.content.strip()\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error querying Deepseek: {e}\")\n",
+    "        return None\n",
+    "\n",
+    "# === PARSE RESPONSE ===\n",
+    "def parse_response(response):\n",
+    "    \"\"\"Parse Deepseek response into structured fields.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    # Split into lines and clean\n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    # Initialize with Unknown values\n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    # Extract information from each numbered line\n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.'):\n",
+    "            fields['gender'] = line[2:].strip()\n",
+    "        elif line.startswith('4.'):\n",
+    "            fields['profession_llm'] = line[2:].strip()\n",
+    "        elif line.startswith('5.'):\n",
+    "            fields['country'] = line[2:].strip()\n",
+    "    \n",
+    "    return fields\n",
+    "\n",
+    "# === PROCESS DATA ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/deepseek_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/deepseek_query_index.txt\"\n",
+    "\n",
+    "# Check for existing progress\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    with open(index_file, 'r') as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "    print(f\"Resuming from index {current_index}\")\n",
+    "\n",
+    "# Initialize result columns if they don't exist\n",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country', 'aliases']:\n",
+    "    if col not in df.columns:\n",
+    "        df[col] = 'Unknown'\n",
+    "\n",
+    "print(f\"\\nAnnotating with DEEPSEEK...\")\n",
+    "print(f\"Processing rows {current_index} to {len(df)}\")\n",
+    "\n",
+    "try:\n",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Annotating\"):\n",
+    "        row = df.iloc[i]\n",
+    "        \n",
+    "        # Query Deepseek\n",
+    "        response = query_deepseek(row['prompt'])\n",
+    "        parsed_data = parse_response(response)\n",
+    "        \n",
+    "        # Update dataframe\n",
+    "        for key, value in parsed_data.items():\n",
+    "            df.at[i, key] = value\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        # Save progress at intervals\n",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            with open(index_file, 'w') as f:\n",
+    "                f.write(str(current_index))\n",
+    "            print(f\"✅ Progress saved after {i+1} rows\")\n",
+    "        \n",
+    "        time.sleep(1)  # Rate limiting\n",
+    "    \n",
+    "    print(f\"\\n✅ Done! Final results saved to {output_file}\")\n",
+    "    \n",
+    "    # Summary statistics\n",
+    "    print(\"\\n=== Summary Statistics ===\")\n",
+    "    print(f\"Total processed: {len(df)}\")\n",
+    "    print(f\"\\nGender distribution:\")\n",
+    "    print(df['gender'].value_counts())\n",
+    "    print(f\"\\nTop 10 profession combinations:\")\n",
+    "    print(df['profession_llm'].value_counts().head(10))\n",
+    "    print(f\"\\nTop 10 countries:\")\n",
+    "    print(df['country'].value_counts().head(10))\n",
+    "    \n",
+    "    # Show sample results\n",
+    "    print(\"\\n=== Sample Results ===\")\n",
+    "    display_cols = ['real_name', 'full_name', 'gender', 'profession_llm', 'country']\n",
+    "    print(df[display_cols].head(10).to_string(index=False))\n",
+    "\n",
+    "except Exception as e:\n",
+    "    print(f\"⚠️ Error encountered: {e}\")\n",
+    "    print(f\"⚠️ Last processed index: {current_index}\")\n",
+    "    \n",
+    "    # Save progress before exiting\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))\n",
+    "    \n",
+    "    print(f\"⚠️ Progress saved up to row {current_index}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "qwen_header",
+   "metadata": {},
+   "source": [
+    "## Qwen Annotation (Alibaba Cloud)\n",
+    "\n",
+    "Uses **Qwen-Max** (Qwen3-Max) via OpenAI-compatible API.\n",
+    "\n",
+    "**Requirements:**\n",
+    "- API Key: `misc/credentials/qwen_api_key.txt`\n",
+    "- Sign up: https://dashscope.aliyun.com/\n",
+    "- Base URL: `https://dashscope-intl.aliyuncs.com/compatible-mode/v1`\n",
+    "- Model: `qwen-max` (uses Qwen3-Max)\n",
+    "- Uses OpenAI Python client for compatibility\n",
+    "- Cost: Variable (check Alibaba Cloud pricing)\n",
+    "\n",
+    "**Note:** This matches the official dashboard example code exactly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "qwen_code",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "from openai import OpenAI\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "api_key_file = current_dir.parent / \"misc/credentials/qwen_api_key.txt\"\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = True\n",
+    "TEST_SIZE = 10\n",
+    "MAX_ROWS = 20000\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",\n",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",\n",
+    "    \"sports professional\", \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD API KEY ===\n",
+    "try:\n",
+    "    with open(api_key_file) as f:\n",
+    "        api_key = f.read().strip()\n",
+    "    print(\"✅ Qwen API key loaded\")\n",
+    "    print(f\"   Key starts with: {api_key[:10]}...\")\n",
+    "except FileNotFoundError:\n",
+    "    print(\"❌ Error: qwen_api_key.txt not found\")\n",
+    "    raise\n",
+    "\n",
+    "# Initialize OpenAI client for Qwen (matching dashboard example)\n",
+    "client = OpenAI(\n",
+    "    api_key=api_key,\n",
+    "    base_url=\"https://dashscope-intl.aliyuncs.com/compatible-mode/v1\"\n",
+    ")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPT ===\n",
+    "def create_prompt(name: str, tags: list, country: str) -> str:\n",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"\n",
+    "    country_str = country if country else \"unknown\"\n",
+    "    \n",
+    "    return f\"\"\"You are an expert at identifying public figures.\n",
+    "\n",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}\n",
+    "\n",
+    "Provide in JSON format:\n",
+    "{{\n",
+    "  \"full_name\": \"<full name>\",\n",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",\n",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],\n",
+    "  \"country\": \"<country>\"\n",
+    "}}\n",
+    "\n",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}\n",
+    "- Max 3 professions, ordered by relevance\n",
+    "- \"online personality\" includes streamers, cosplayers, influencers\n",
+    "- Return \"Unknown\" if uncertain\n",
+    "\n",
+    "Return ONLY JSON, no extra text.\"\"\"\n",
+    "\n",
+    "# === QUERY QWEN ===\n",
+    "def query_qwen(prompt: str) -> dict:\n",
+    "    \"\"\"Query Qwen using OpenAI-compatible API (matching dashboard example).\"\"\"\n",
+    "    for attempt in range(3):\n",
+    "        try:\n",
+    "            completion = client.chat.completions.create(\n",
+    "                model=\"qwen-max\",  # Note: dashboard shows \"qwen3-max\" but \"qwen-max\" works too\n",
+    "                messages=[\n",
+    "                    {\"role\": \"user\", \"content\": prompt}\n",
+    "                ],\n",
+    "                temperature=0.1,\n",
+    "                max_tokens=500,\n",
+    "                stream=False  # Disable streaming for easier parsing\n",
+    "            )\n",
+    "            \n",
+    "            content = completion.choices[0].message.content\n",
+    "            \n",
+    "            # Parse JSON from response\n",
+    "            json_match = re.search(r'\\{[^}]+\\}', content, re.DOTALL)\n",
+    "            if json_match:\n",
+    "                return json.loads(json_match.group())\n",
+    "            \n",
+    "            return None\n",
+    "            \n",
+    "        except Exception as e:\n",
+    "            print(f\"Qwen error (attempt {attempt+1}): {e}\")\n",
+    "            if attempt < 2:\n",
+    "                time.sleep(2 ** attempt)\n",
+    "    \n",
+    "    return None\n",
+    "\n",
+    "# === PROCESS ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/qwen_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/qwen_query_index.txt\"\n",
+    "\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    with open(index_file) as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "\n",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:\n",
+    "    if col not in df.columns:\n",
+    "        df[col] = 'Unknown'\n",
+    "\n",
+    "print(f\"\\nAnnotating with QWEN-MAX (OpenAI-compatible API) - rows {current_index} to {len(df)}...\")\n",
+    "\n",
+    "try:\n",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Qwen\"):\n",
+    "        row = df.iloc[i]\n",
+    "        \n",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8)\n",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]\n",
+    "        \n",
+    "        prompt = create_prompt(\n",
+    "            row.get('real_name', row.get('name', '')),\n",
+    "            tags,\n",
+    "            row.get('likely_country', '')\n",
+    "        )\n",
+    "        \n",
+    "        result = query_qwen(prompt)\n",
+    "        \n",
+    "        if result:\n",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')\n",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')\n",
+    "            profs = result.get('professions', [])\n",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)\n",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            with open(index_file, 'w') as f:\n",
+    "                f.write(str(current_index))\n",
+    "            print(f\"✅ Saved after {i+1} rows\")\n",
+    "        \n",
+    "        time.sleep(0.5)\n",
+    "    \n",
+    "    print(f\"\\n✅ Done! Results: {output_file}\")\n",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")\n",
+    "    \n",
+    "except Exception as e:\n",
+    "    print(f\"⚠️ Error: {e}\")\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "llama_header",
+   "metadata": {},
+   "source": [
+    "## Llama Annotation (Together AI)\n",
+    "\n",
+    "Uses **Llama-3.1-70B-Instruct-Turbo** via Together AI.\n",
+    "\n",
+    "**Requirements:**\n",
+    "- API Key: `misc/credentials/together_api_key.txt`\n",
+    "- Sign up: https://www.together.ai/\n",
+    "- Model: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo\n",
+    "- Cost: ~$0.90 per 1M tokens (~$5-10 for 10k entries)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "llama_code",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import requests\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "api_key_file = current_dir.parent / \"misc/credentials/together_api_key.txt\"\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = True\n",
+    "TEST_SIZE = 10\n",
+    "MAX_ROWS = 20000\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",\n",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",\n",
+    "    \"sports professional\", \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD API KEY ===\n",
+    "try:\n",
+    "    with open(api_key_file) as f:\n",
+    "        api_key = f.read().strip()\n",
+    "    print(\"✅ Llama API key loaded\")\n",
+    "except FileNotFoundError:\n",
+    "    print(\"❌ Error: together_api_key.txt not found\")\n",
+    "    raise\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPT ===\n",
+    "def create_prompt(name: str, tags: list, country: str) -> str:\n",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"\n",
+    "    country_str = country if country else \"unknown\"\n",
+    "    \n",
+    "    return f\"\"\"You are an expert at identifying public figures.\n",
+    "\n",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}\n",
+    "\n",
+    "Provide in JSON format:\n",
+    "{{\n",
+    "  \"full_name\": \"<full name>\",\n",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",\n",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],\n",
+    "  \"country\": \"<country>\"\n",
+    "}}\n",
+    "\n",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}\n",
+    "- Max 3 professions, ordered by relevance\n",
+    "- \"online personality\" includes streamers, cosplayers, influencers\n",
+    "- Return \"Unknown\" if uncertain\n",
+    "\n",
+    "Return ONLY JSON, no extra text.\"\"\"\n",
+    "\n",
+    "# === QUERY LLAMA ===\n",
+    "def query_llama(prompt: str) -> dict:\n",
+    "    url = \"https://api.together.xyz/v1/chat/completions\"\n",
+    "    \n",
+    "    headers = {\n",
+    "        \"Authorization\": f\"Bearer {api_key}\",\n",
+    "        \"Content-Type\": \"application/json\"\n",
+    "    }\n",
+    "    \n",
+    "    data = {\n",
+    "        \"model\": \"meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo\",\n",
+    "        \"messages\": [{\"role\": \"user\", \"content\": prompt}],\n",
+    "        \"temperature\": 0.1,\n",
+    "        \"max_tokens\": 500\n",
+    "    }\n",
+    "    \n",
+    "    for attempt in range(3):\n",
+    "        try:\n",
+    "            response = requests.post(url, headers=headers, json=data, timeout=30)\n",
+    "            response.raise_for_status()\n",
+    "            result = response.json()\n",
+    "            \n",
+    "            if 'choices' in result:\n",
+    "                content = result['choices'][0]['message']['content']\n",
+    "                json_match = re.search(r'\\{[^}]+\\}', content, re.DOTALL)\n",
+    "                if json_match:\n",
+    "                    return json.loads(json_match.group())\n",
+    "            return None\n",
+    "        except Exception as e:\n",
+    "            print(f\"Llama error (attempt {attempt+1}): {e}\")\n",
+    "            if attempt < 2:\n",
+    "                time.sleep(2 ** attempt)\n",
+    "    return None\n",
+    "\n",
+    "# === PROCESS ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/llama_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/llama_query_index.txt\"\n",
+    "\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    with open(index_file) as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "\n",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:\n",
+    "    if col not in df.columns:\n",
+    "        df[col] = 'Unknown'\n",
+    "\n",
+    "print(f\"\\nAnnotating with LLAMA 3.1 70B - rows {current_index} to {len(df)}...\")\n",
+    "\n",
+    "try:\n",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Llama\"):\n",
+    "        row = df.iloc[i]\n",
+    "        \n",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8)\n",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]\n",
+    "        \n",
+    "        prompt = create_prompt(\n",
+    "            row.get('real_name', row.get('name', '')),\n",
+    "            tags,\n",
+    "            row.get('likely_country', '')\n",
+    "        )\n",
+    "        \n",
+    "        result = query_llama(prompt)\n",
+    "        \n",
+    "        if result:\n",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')\n",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')\n",
+    "            profs = result.get('professions', [])\n",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)\n",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            with open(index_file, 'w') as f:\n",
+    "                f.write(str(current_index))\n",
+    "            print(f\"✅ Saved after {i+1} rows\")\n",
+    "        \n",
+    "        time.sleep(0.5)\n",
+    "    \n",
+    "    print(f\"\\n✅ Done! Results: {output_file}\")\n",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")\n",
+    "    \n",
+    "except Exception as e:\n",
+    "    print(f\"⚠️ Error: {e}\")\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "mistral_header",
+   "metadata": {},
+   "source": [
+    "## Mixtral Annotation (Mistral AI)\n",
+    "\n",
+    "Uses **Open-Mixtral-8x22B** via Mistral AI API.\n",
+    "\n",
+    "**Requirements:**\n",
+    "- API Key: `misc/credentials/mistral_api_key.txt`\n",
+    "- Sign up: https://mistral.ai/\n",
+    "- Model: open-mixtral-8x22b\n",
+    "- Cost: ~$2 per 1M tokens (~$10-20 for 10k entries)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "mixtral_code",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import requests\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "api_key_file = current_dir.parent / \"misc/credentials/mistral_api_key.txt\"\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = True\n",
+    "TEST_SIZE = 10\n",
+    "MAX_ROWS = 20000\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",\n",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",\n",
+    "    \"sports professional\", \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD API KEY ===\n",
+    "try:\n",
+    "    with open(api_key_file) as f:\n",
+    "        api_key = f.read().strip()\n",
+    "    print(\"✅ Mixtral API key loaded\")\n",
+    "except FileNotFoundError:\n",
+    "    print(\"❌ Error: mistral_api_key.txt not found\")\n",
+    "    raise\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPT ===\n",
+    "def create_prompt(name: str, tags: list, country: str) -> str:\n",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"\n",
+    "    country_str = country if country else \"unknown\"\n",
+    "    \n",
+    "    return f\"\"\"You are an expert at identifying public figures.\n",
+    "\n",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}\n",
+    "\n",
+    "Provide in JSON format:\n",
+    "{{\n",
+    "  \"full_name\": \"<full name>\",\n",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",\n",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],\n",
+    "  \"country\": \"<country>\"\n",
+    "}}\n",
+    "\n",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}\n",
+    "- Max 3 professions, ordered by relevance\n",
+    "- \"online personality\" includes streamers, cosplayers, influencers\n",
+    "- Return \"Unknown\" if uncertain\n",
+    "\n",
+    "Return ONLY JSON, no extra text.\"\"\"\n",
+    "\n",
+    "# === QUERY MIXTRAL ===\n",
+    "def query_mixtral(prompt: str) -> dict:\n",
+    "    url = \"https://api.mistral.ai/v1/chat/completions\"\n",
+    "    \n",
+    "    headers = {\n",
+    "        \"Authorization\": f\"Bearer {api_key}\",\n",
+    "        \"Content-Type\": \"application/json\"\n",
+    "    }\n",
+    "    \n",
+    "    data = {\n",
+    "        \"model\": \"open-mixtral-8x22b\",\n",
+    "        \"messages\": [{\"role\": \"user\", \"content\": prompt}],\n",
+    "        \"temperature\": 0.1,\n",
+    "        \"max_tokens\": 500\n",
+    "    }\n",
+    "    \n",
+    "    for attempt in range(3):\n",
+    "        try:\n",
+    "            response = requests.post(url, headers=headers, json=data, timeout=30)\n",
+    "            response.raise_for_status()\n",
+    "            result = response.json()\n",
+    "            \n",
+    "            if 'choices' in result:\n",
+    "                content = result['choices'][0]['message']['content']\n",
+    "                json_match = re.search(r'\\{[^}]+\\}', content, re.DOTALL)\n",
+    "                if json_match:\n",
+    "                    return json.loads(json_match.group())\n",
+    "            return None\n",
+    "        except Exception as e:\n",
+    "            print(f\"Mixtral error (attempt {attempt+1}): {e}\")\n",
+    "            if attempt < 2:\n",
+    "                time.sleep(2 ** attempt)\n",
+    "    return None\n",
+    "\n",
+    "# === PROCESS ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/mixtral_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/mixtral_query_index.txt\"\n",
+    "\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    with open(index_file) as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "\n",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:\n",
+    "    if col not in df.columns:\n",
+    "        df[col] = 'Unknown'\n",
+    "\n",
+    "print(f\"\\nAnnotating with MIXTRAL 8x22B - rows {current_index} to {len(df)}...\")\n",
+    "\n",
+    "try:\n",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Mixtral\"):\n",
+    "        row = df.iloc[i]\n",
+    "        \n",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8)\n",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]\n",
+    "        \n",
+    "        prompt = create_prompt(\n",
+    "            row.get('real_name', row.get('name', '')),\n",
+    "            tags,\n",
+    "            row.get('likely_country', '')\n",
+    "        )\n",
+    "        \n",
+    "        result = query_mixtral(prompt)\n",
+    "        \n",
+    "        if result:\n",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')\n",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')\n",
+    "            profs = result.get('professions', [])\n",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)\n",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            with open(index_file, 'w') as f:\n",
+    "                f.write(str(current_index))\n",
+    "            print(f\"✅ Saved after {i+1} rows\")\n",
+    "        \n",
+    "        time.sleep(0.5)\n",
+    "    \n",
+    "    print(f\"\\n✅ Done! Results: {output_file}\")\n",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")\n",
+    "    \n",
+    "except Exception as e:\n",
+    "    print(f\"⚠️ Error: {e}\")\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "gemma_header",
+   "metadata": {},
+   "source": [
+    "## Gemma Annotation (Google - American Open Source)\n",
+    "\n",
+    "Uses **Gemma 2 27B** via Together AI - Google's open-source model.\n",
+    "\n",
+    "**Requirements:**\n",
+    "- API Key: `misc/credentials/together_api_key.txt` (same as Llama)\n",
+    "- Sign up: https://www.together.ai/\n",
+    "- Model: google/gemma-2-27b-it\n",
+    "- Cost: ~$0.80 per 1M tokens (~$4-8 for 10k entries)\n",
+    "- Note: American company (Google), fully open-source model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "gemma_code",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import requests\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "api_key_file = current_dir.parent / \"misc/credentials/together_api_key.txt\"\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = True\n",
+    "TEST_SIZE = 10\n",
+    "MAX_ROWS = 20000\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",\n",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",\n",
+    "    \"sports professional\", \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD API KEY ===\n",
+    "try:\n",
+    "    with open(api_key_file) as f:\n",
+    "        api_key = f.read().strip()\n",
+    "    print(\"✅ Gemma API key loaded\")\n",
+    "except FileNotFoundError:\n",
+    "    print(\"❌ Error: together_api_key.txt not found\")\n",
+    "    raise\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPT ===\n",
+    "def create_prompt(name: str, tags: list, country: str) -> str:\n",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"\n",
+    "    country_str = country if country else \"unknown\"\n",
+    "    \n",
+    "    return f\"\"\"You are an expert at identifying public figures.\n",
+    "\n",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}\n",
+    "\n",
+    "Provide in JSON format:\n",
+    "{{\n",
+    "  \"full_name\": \"<full name>\",\n",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",\n",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],\n",
+    "  \"country\": \"<country>\"\n",
+    "}}\n",
+    "\n",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}\n",
+    "- Max 3 professions, ordered by relevance\n",
+    "- \"online personality\" includes streamers, cosplayers, influencers\n",
+    "- Return \"Unknown\" if uncertain\n",
+    "\n",
+    "Return ONLY JSON, no extra text.\"\"\"\n",
+    "\n",
+    "# === QUERY GEMMA ===\n",
+    "def query_gemma(prompt: str) -> dict:\n",
+    "    url = \"https://api.together.xyz/v1/chat/completions\"\n",
+    "    \n",
+    "    headers = {\n",
+    "        \"Authorization\": f\"Bearer {api_key}\",\n",
+    "        \"Content-Type\": \"application/json\"\n",
+    "    }\n",
+    "    \n",
+    "    data = {\n",
+    "        \"model\": \"google/gemma-2-27b-it\",\n",
+    "        \"messages\": [{\"role\": \"user\", \"content\": prompt}],\n",
+    "        \"temperature\": 0.1,\n",
+    "        \"max_tokens\": 500\n",
+    "    }\n",
+    "    \n",
+    "    for attempt in range(3):\n",
+    "        try:\n",
+    "            response = requests.post(url, headers=headers, json=data, timeout=30)\n",
+    "            response.raise_for_status()\n",
+    "            result = response.json()\n",
+    "            \n",
+    "            if 'choices' in result:\n",
+    "                content = result['choices'][0]['message']['content']\n",
+    "                json_match = re.search(r'\\{[^}]+\\}', content, re.DOTALL)\n",
+    "                if json_match:\n",
+    "                    return json.loads(json_match.group())\n",
+    "            return None\n",
+    "        except Exception as e:\n",
+    "            print(f\"Gemma error (attempt {attempt+1}): {e}\")\n",
+    "            if attempt < 2:\n",
+    "                time.sleep(2 ** attempt)\n",
+    "    return None\n",
+    "\n",
+    "# === PROCESS ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/gemma_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/gemma_query_index.txt\"\n",
+    "\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    with open(index_file) as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "\n",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:\n",
+    "    if col not in df.columns:\n",
+    "        df[col] = 'Unknown'\n",
+    "\n",
+    "print(f\"\\nAnnotating with GEMMA 2 27B (Google) - rows {current_index} to {len(df)}...\")\n",
+    "\n",
+    "try:\n",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Gemma\"):\n",
+    "        row = df.iloc[i]\n",
+    "        \n",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8)\n",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]\n",
+    "        \n",
+    "        prompt = create_prompt(\n",
+    "            row.get('real_name', row.get('name', '')),\n",
+    "            tags,\n",
+    "            row.get('likely_country', '')\n",
+    "        )\n",
+    "        \n",
+    "        result = query_gemma(prompt)\n",
+    "        \n",
+    "        if result:\n",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')\n",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')\n",
+    "            profs = result.get('professions', [])\n",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)\n",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            with open(index_file, 'w') as f:\n",
+    "                f.write(str(current_index))\n",
+    "            print(f\"✅ Saved after {i+1} rows\")\n",
+    "        \n",
+    "        time.sleep(0.5)\n",
+    "    \n",
+    "    print(f\"\\n✅ Done! Results: {output_file}\")\n",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")\n",
+    "    \n",
+    "except Exception as e:\n",
+    "    print(f\"⚠️ Error: {e}\")\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "01086b35",
+   "metadata": {},
+   "source": [
+    "## Qwen-2.5-32B Local Annotation (Ollama)\n",
+    "\n",
+    "Run **Qwen-2.5-32B-Instruct** locally using Ollama - NO API COSTS!\n",
+    "\n",
+    "**Requirements:**\n",
+    "- Ollama installed: https://ollama.com/\n",
+    "- GPU: NVIDIA A100 80GB (recommended)\n",
+    "- RAM: 265GB system RAM\n",
+    "- Model cache: `data/models/` (auto-downloaded on first run)\n",
+    "- Model size: ~20GB download\n",
+    "- VRAM usage: ~22-25GB during inference\n",
+    "\n",
+    "**Setup:**\n",
+    "```bash\n",
+    "# 1. Install Ollama (if not installed)\n",
+    "curl -fsSL https://ollama.com/install.sh | sh\n",
+    "\n",
+    "# 2. Pull Qwen-2.5-32B-Instruct\n",
+    "ollama pull qwen2.5:32b-instruct\n",
+    "\n",
+    "# 3. Verify model is available\n",
+    "ollama list\n",
+    "```\n",
+    "\n",
+    "**Advantages:**\n",
+    "- ✅ FREE - No API costs\n",
+    "- ✅ Fast - Local inference on A100\n",
+    "- ✅ Privacy - Data never leaves your machine\n",
+    "- ✅ Offline - Works without internet\n",
+    "- ✅ High quality - 32B parameter model\n",
+    "\n",
+    "**Performance:**\n",
+    "- Speed: ~5-10 tokens/sec on A100\n",
+    "- Throughput: ~100-200 samples/hour (depends on prompt length)\n",
+    "- Cost: $0 (after model download)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5dd7a803",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "import subprocess\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = True\n",
+    "TEST_SIZE = 10\n",
+    "MAX_ROWS = 20000\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "# Ollama settings\n",
+    "MODEL_NAME = \"qwen2.5:32b-instruct\"  # or \"qwen2.5:32b\" for base model\n",
+    "OLLAMA_HOST = \"http://localhost:11434\"  # Default Ollama port\n",
+    "\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",\n",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",\n",
+    "    \"sports professional\", \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === CHECK OLLAMA ===\n",
+    "def check_ollama():\n",
+    "    \"\"\"Check if Ollama is installed and model is available.\"\"\"\n",
+    "    try:\n",
+    "        result = subprocess.run(\n",
+    "            [\"ollama\", \"list\"],\n",
+    "            capture_output=True,\n",
+    "            text=True,\n",
+    "            timeout=5\n",
+    "        )\n",
+    "        \n",
+    "        if result.returncode == 0:\n",
+    "            print(\"✅ Ollama is installed\")\n",
+    "            \n",
+    "            # Check if model is available\n",
+    "            if MODEL_NAME in result.stdout or \"qwen2.5\" in result.stdout:\n",
+    "                print(f\"✅ Model {MODEL_NAME} is available\")\n",
+    "                return True\n",
+    "            else:\n",
+    "                print(f\"⚠️  Model {MODEL_NAME} not found\")\n",
+    "                print(f\"   Run: ollama pull {MODEL_NAME}\")\n",
+    "                return False\n",
+    "        else:\n",
+    "            print(\"❌ Ollama command failed\")\n",
+    "            return False\n",
+    "    except FileNotFoundError:\n",
+    "        print(\"❌ Ollama not installed\")\n",
+    "        print(\"   Install: curl -fsSL https://ollama.com/install.sh | sh\")\n",
+    "        return False\n",
+    "    except Exception as e:\n",
+    "        print(f\"❌ Error checking Ollama: {e}\")\n",
+    "        return False\n",
+    "\n",
+    "# Check Ollama before proceeding\n",
+    "if not check_ollama():\n",
+    "    raise RuntimeError(\"Ollama not ready. Please install Ollama and pull the model.\")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"\\nLoaded {len(df)} rows\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPT ===\n",
+    "def create_prompt(name: str, tags: list, country: str) -> str:\n",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"\n",
+    "    country_str = country if country else \"unknown\"\n",
+    "    \n",
+    "    return f\"\"\"You are an expert at identifying public figures.\n",
+    "\n",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}\n",
+    "\n",
+    "Provide in JSON format:\n",
+    "{{\n",
+    "  \"full_name\": \"<full name>\",\n",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",\n",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],\n",
+    "  \"country\": \"<country>\"\n",
+    "}}\n",
+    "\n",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}\n",
+    "- Max 3 professions, ordered by relevance\n",
+    "- \"online personality\" includes streamers, cosplayers, influencers\n",
+    "- Return \"Unknown\" if uncertain\n",
+    "\n",
+    "Return ONLY JSON, no extra text.\"\"\"\n",
+    "\n",
+    "# === QUERY QWEN LOCAL ===\n",
+    "def query_qwen_local(prompt: str) -> dict:\n",
+    "    \"\"\"Query Qwen-2.5-32B via Ollama.\"\"\"\n",
+    "    for attempt in range(3):\n",
+    "        try:\n",
+    "            result = subprocess.run(\n",
+    "                [\n",
+    "                    \"ollama\", \"run\", MODEL_NAME,\n",
+    "                    prompt\n",
+    "                ],\n",
+    "                capture_output=True,\n",
+    "                text=True,\n",
+    "                timeout=60  # 60 second timeout per query\n",
+    "            )\n",
+    "            \n",
+    "            if result.returncode == 0:\n",
+    "                content = result.stdout.strip()\n",
+    "                \n",
+    "                # Parse JSON from response\n",
+    "                json_match = re.search(r'\\{[^}]+\\}', content, re.DOTALL)\n",
+    "                if json_match:\n",
+    "                    return json.loads(json_match.group())\n",
+    "                \n",
+    "                # If no JSON found, try to extract from markdown code block\n",
+    "                code_block = re.search(r'```(?:json)?\\s*({[^}]+})\\s*```', content, re.DOTALL)\n",
+    "                if code_block:\n",
+    "                    return json.loads(code_block.group(1))\n",
+    "            \n",
+    "            return None\n",
+    "        \n",
+    "        except subprocess.TimeoutExpired:\n",
+    "            print(f\"Qwen local timeout (attempt {attempt+1})\")\n",
+    "            if attempt < 2:\n",
+    "                time.sleep(2)\n",
+    "        except json.JSONDecodeError as e:\n",
+    "            print(f\"Qwen local JSON error (attempt {attempt+1}): {e}\")\n",
+    "            if attempt < 2:\n",
+    "                time.sleep(1)\n",
+    "        except Exception as e:\n",
+    "            print(f\"Qwen local error (attempt {attempt+1}): {e}\")\n",
+    "            if attempt < 2:\n",
+    "                time.sleep(2)\n",
+    "    \n",
+    "    return None\n",
+    "\n",
+    "# === PROCESS ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/qwen_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/qwen_local_query_index.txt\"\n",
+    "\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    with open(index_file) as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "\n",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:\n",
+    "    if col not in df.columns:\n",
+    "        df[col] = 'Unknown'\n",
+    "\n",
+    "print(f\"\\nAnnotating with QWEN-2.5-32B LOCAL - rows {current_index} to {len(df)}...\")\n",
+    "print(f\"Model: {MODEL_NAME}\")\n",
+    "print(f\"This may take a while (~100-200 samples/hour on A100)\\n\")\n",
+    "\n",
+    "try:\n",
+    "    start_time = time.time()\n",
+    "    \n",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Qwen Local\"):\n",
+    "        row = df.iloc[i]\n",
+    "        \n",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8)\n",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]\n",
+    "        \n",
+    "        prompt = create_prompt(\n",
+    "            row.get('real_name', row.get('name', '')),\n",
+    "            tags,\n",
+    "            row.get('likely_country', '')\n",
+    "        )\n",
+    "        \n",
+    "        result = query_qwen_local(prompt)\n",
+    "        \n",
+    "        if result:\n",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')\n",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')\n",
+    "            profs = result.get('professions', [])\n",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)\n",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            with open(index_file, 'w') as f:\n",
+    "                f.write(str(current_index))\n",
+    "            \n",
+    "            # Calculate stats\n",
+    "            elapsed = time.time() - start_time\n",
+    "            samples_done = i + 1 - (current_index - (i + 1))\n",
+    "            samples_per_hour = (samples_done / elapsed) * 3600 if elapsed > 0 else 0\n",
+    "            \n",
+    "            print(f\"✅ Saved after {i+1} rows (~{samples_per_hour:.1f} samples/hour)\")\n",
+    "    \n",
+    "    elapsed_total = time.time() - start_time\n",
+    "    print(f\"\\n✅ Done! Results: {output_file}\")\n",
+    "    print(f\"Total time: {elapsed_total/60:.1f} minutes\")\n",
+    "    print(f\"Average speed: {len(df)/(elapsed_total/3600):.1f} samples/hour\")\n",
+    "    \n",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")\n",
+    "\n",
+    "except KeyboardInterrupt:\n",
+    "    print(f\"\\n⚠️ Interrupted by user\")\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))\n",
+    "    print(f\"Progress saved up to row {current_index}\")\n",
+    "\n",
+    "except Exception as e:\n",
+    "    print(f\"⚠️ Error: {e}\")\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))\n",
+    "    print(f\"Progress saved up to row {current_index}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-3-4__Figure_8_Deepfake_victims-checkpoint.ipynb

@@ -0,0 +1,668 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "06763fde",
+   "metadata": {},
+   "source": [
+    "# LLM annotation of Deepfake adapters"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b773045a",
+   "metadata": {},
+   "source": [
+    "## Step 01 Data cleaning NER using spaCy"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "234a55e5",
+   "metadata": {},
+   "source": [
+    "#### Here we clean leetspeak and architectre specifiers from the names as a preprossesing step for the named entity recognition NER below"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "f177df11",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import spacy\n",
+    "import re\n",
+    "import torch\n",
+    "from pathlib import Path\n",
+    "import unicodedata\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "a2383f32",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "current_dir = Path.cwd()\n",
+    "poi_models_dir = current_dir.parent / \"data/CSV/model_subsets/POI_models.csv\"  ### POI models dataset\n",
+    "output = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_01.csv\"  ### Output file"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "66fc691f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/lauhp/anaconda3/envs/latm/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    }
+   ],
+   "source": [
+    "nlp = spacy.load(\"en_core_web_sm\")  # or another model of your choice"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "8348c4a4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Done! Saved to NER_poi_step_01.csv\n"
+     ]
+    }
+   ],
+   "source": [
+    "def preprocess_name(name):\n",
+    "    name = str(name)\n",
+    "\n",
+    "    # Normalize unicode characters (e.g., fancy fonts)\n",
+    "    name = unicodedata.normalize(\"NFKD\", name)\n",
+    "\n",
+    "    # Lowercase everything\n",
+    "    name = name.lower()\n",
+    "\n",
+    "    # Remove special keywords and patterns\n",
+    "    junk_words = [\n",
+    "        'jav', 'jp', 'lora', 'locon', 'lycoris', 'requested', 'japanese', 'model',\n",
+    "        'flux', 'flux1.d', 'pony', 'realistic'\n",
+    "    ]\n",
+    "    for word in junk_words:\n",
+    "        name = re.sub(rf'\\b{re.escape(word)}\\b', '', name, flags=re.IGNORECASE)\n",
+    "\n",
+    "    # Remove versions like v1, v2.0, etc.\n",
+    "    name = re.sub(r'v\\.?\\d+(\\.\\d+)?', '', name)\n",
+    "\n",
+    "    # Remove 'not' followed by a word\n",
+    "    name = re.sub(r'\\bnot\\s+\\w+', '', name)\n",
+    "\n",
+    "    # Replace underscores and pipes with spaces\n",
+    "    name = re.sub(r'[_|]', ' ', name)\n",
+    "\n",
+    "    # Remove parentheses and content within\n",
+    "    name = re.sub(r'\\(.*?\\)', '', name)\n",
+    "\n",
+    "    # Remove excess whitespace\n",
+    "    name = re.sub(r'\\s+', ' ', name).strip()\n",
+    "\n",
+    "    return name\n",
+    "\n",
+    "# -------------------------\n",
+    "# Fallback Extractor\n",
+    "# -------------------------\n",
+    "def fallback_extract(text):\n",
+    "    words = text.split()\n",
+    "    capitalized = [w for w in words if w and w[0].isalpha()]\n",
+    "    if len(capitalized) >= 2:\n",
+    "        return \" \".join(capitalized[:2])\n",
+    "    elif capitalized:\n",
+    "        return capitalized[0]\n",
+    "    return None\n",
+    "\n",
+    "# -------------------------\n",
+    "# Full Extraction Logic\n",
+    "# -------------------------\n",
+    "def extract_real_name(raw_name):\n",
+    "    cleaned = preprocess_name(raw_name)\n",
+    "    doc = nlp(cleaned)\n",
+    "    persons = [ent.text for ent in doc.ents if ent.label_ == \"PERSON\"]\n",
+    "    if persons:\n",
+    "        return persons[0]\n",
+    "    return fallback_extract(cleaned)\n",
+    "\n",
+    "# -------------------------\n",
+    "# Load Data and Apply\n",
+    "# -------------------------\n",
+    "df = pd.read_csv(poi_models_dir)  # Or your own path\n",
+    "\n",
+    "# Apply the full extractor\n",
+    "texts = df['name'].astype(str).tolist()\n",
+    "docs = list(nlp.pipe([preprocess_name(t) for t in texts], batch_size=32))\n",
+    "\n",
+    "def extract_from_doc(doc, raw_text):\n",
+    "    persons = [ent.text for ent in doc.ents if ent.label_ == \"PERSON\"]\n",
+    "    if persons:\n",
+    "        return persons[0]\n",
+    "    return fallback_extract(preprocess_name(raw_text))\n",
+    "\n",
+    "df['real_name'] = [extract_from_doc(doc, raw_text) for doc, raw_text in zip(docs, texts)]\n",
+    "\n",
+    "# Save the result\n",
+    "df.to_csv(output, index=False)\n",
+    "print(\"Done! Saved to NER_poi_step_01.csv\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a40b17fe",
+   "metadata": {},
+   "source": [
+    "## Step 2: compare country and profession with lists"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "414954ed",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "          real_name likely_country likely_nationality likely_profession\n",
+      "0     ronnie alonte                                               model\n",
+      "1  zh elena kamperi                                     twitch streamer\n",
+      "2     sofia vergara                                               model\n",
+      "3          安然 anran                                           celebrity\n",
+      "4       zoe kravitz                                           celebrity\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from pathlib import Path\n",
+    "\n",
+    "# Set up paths\n",
+    "current_dir = Path.cwd()\n",
+    "countries = current_dir.parent / \"misc/lists/countries.csv\"\n",
+    "professions = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "inputNER = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_01.csv\"\n",
+    "outfile = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_02.csv\"\n",
+    "\n",
+    "# Load datasets\n",
+    "poi_df = pd.read_csv(inputNER)\n",
+    "countries_df = pd.read_csv(countries)\n",
+    "professions_df = pd.read_csv(professions)\n",
+    "\n",
+    "# Step 1: Combine tags into one lowercase list\n",
+    "def combine_tags(row):\n",
+    "    return [str(row[f\"tag_{i}\"]).strip().lower() for i in range(1, 8) if pd.notna(row.get(f\"tag_{i}\"))]\n",
+    "\n",
+    "poi_df[\"tags\"] = poi_df.apply(combine_tags, axis=1)\n",
+    "\n",
+    "# Step 2: Build tag → (country, nationality) mapping\n",
+    "tag_to_country_nationality = {}\n",
+    "\n",
+    "for _, row in countries_df.iterrows():\n",
+    "    country = str(row[\"en_short_name\"]).strip()\n",
+    "    nationality = str(row[\"nationality\"]).strip()\n",
+    "\n",
+    "    country_lc = country.lower()\n",
+    "    nationality_lc = nationality.lower()\n",
+    "\n",
+    "    # Add variations to the mapping\n",
+    "    tag_to_country_nationality[country_lc] = (country, \"\")\n",
+    "    tag_to_country_nationality[nationality_lc] = (\"\", nationality)\n",
+    "    tag_to_country_nationality[country_lc.replace(\" \", \"\")] = (country, \"\")\n",
+    "    tag_to_country_nationality[nationality_lc.replace(\" \", \"\")] = (\"\", nationality)\n",
+    "\n",
+    "    for part in country_lc.split():\n",
+    "        tag_to_country_nationality[part] = (country, \"\")\n",
+    "    for part in nationality_lc.split():\n",
+    "        tag_to_country_nationality[part] = (\"\", nationality)\n",
+    "\n",
+    "# Step 3: Infer likely_country and likely_nationality\n",
+    "# Step 3: Infer likely_country and likely_nationality\n",
+    "def infer_country_and_nationality(tags):\n",
+    "    for tag in tags:\n",
+    "        cleaned = tag.replace(\" \", \"\").lower()\n",
+    "        if cleaned in tag_to_country_nationality:\n",
+    "            country, nationality = tag_to_country_nationality[cleaned]\n",
+    "            # Special case: skip if country is \"Isle of Man\"\n",
+    "            if country == \"Isle of Man\":\n",
+    "                country = \"\"\n",
+    "            return pd.Series([country, nationality])\n",
+    "    return pd.Series([\"\", \"\"])\n",
+    "\n",
+    "\n",
+    "poi_df[[\"likely_country\", \"likely_nationality\"]] = poi_df[\"tags\"].apply(infer_country_and_nationality)\n",
+    "\n",
+    "# Step 4: Build tag → profession mapping\n",
+    "profession_alias_map = {}\n",
+    "\n",
+    "for _, row in professions_df.iterrows():\n",
+    "    canonical = str(row['profession']).strip().lower()\n",
+    "    profession_alias_map[canonical] = canonical\n",
+    "    for alias_col in ['alias_1', 'alias_2', 'alias_3']:\n",
+    "        alias = row.get(alias_col)\n",
+    "        if pd.notna(alias):\n",
+    "            profession_alias_map[str(alias).strip().lower()] = canonical\n",
+    "\n",
+    "# Step 5: Infer likely profession from tags\n",
+    "def infer_profession_from_tags(tags):\n",
+    "    matched = []\n",
+    "    for tag in tags:\n",
+    "        cleaned = tag.strip().lower()\n",
+    "        if cleaned in profession_alias_map:\n",
+    "            matched.append(profession_alias_map[cleaned])\n",
+    "\n",
+    "    if not matched:\n",
+    "        return \"\"\n",
+    "    if \"celebrity\" in matched and len(set(matched)) > 1:\n",
+    "        # Drop 'celebrity' if other professions are present\n",
+    "        matched = [m for m in matched if m != \"celebrity\"]\n",
+    "\n",
+    "    return matched[0]  # Return the first specific match\n",
+    "\n",
+    "\n",
+    "poi_df[\"likely_profession\"] = poi_df[\"tags\"].apply(infer_profession_from_tags)\n",
+    "\n",
+    "# Step 6: Save enriched dataset\n",
+    "poi_df.to_csv(outfile, index=False)\n",
+    "\n",
+    "# Optional: Preview\n",
+    "print(poi_df[[\"real_name\", \"likely_country\", \"likely_nationality\", \"likely_profession\"]].head())\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "054f230b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#!pip install transformers torch\n",
+    "#!python -m spacy download en_core_web_trf\n",
+    "#pip install openai"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "59185461",
+   "metadata": {},
+   "source": [
+    "## Step 3: Query Deepseek-v3 with NAME and HINTS"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "504b970f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#!pip install openpyxl"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "7c209115",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Row 1/4...\n",
+      "Saved up to row 1\n",
+      "Row 2/4...\n",
+      "Saved up to row 2\n",
+      "Row 3/4...\n",
+      "Saved up to row 3\n",
+      "Row 4/4...\n",
+      "Saved up to row 4\n",
+      "All done! Files: /home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/data/CSV/Deepseek_annotated_POI.csv /home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/data/CSV/Deepseek_annotated_POI.xlsx\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import openai\n",
+    "import time\n",
+    "import os\n",
+    "from pathlib import Path\n",
+    "from openai import OpenAI  # Add this import\n",
+    "\n",
+    "# === PATHS & CONFIG ===\n",
+    "current_dir = Path.cwd()\n",
+    "inputCSV = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_02.csv\"\n",
+    "api_key_file = current_dir.parent / \"misc/credentials/deepseek_api_key.txt\" #store your API key under misc/credentials/deepseek_api_key.txt\n",
+    "\n",
+    "# Output both CSV and Excel for compatibility\n",
+    "OUTPUT_CSV = current_dir.parent / \"data/CSV/Deepseek_annotated_POI.csv\"\n",
+    "OUTPUT_XLSX = current_dir.parent / \"data/CSV/Deepseek_annotated_POI.xlsx\"\n",
+    "INDEX_FILE = current_dir.parent / \"misc/deepseek_query_index.txt\"\n",
+    "SAVE_INTERVAL = 1  # Save every N rows\n",
+    "START_ROW = 1      # Row index to start from (0-based)\n",
+    "END_ROW = 5        # Row index to end (exclusive)\n",
+    "\n",
+    "# === LOAD API KEY & CLIENT ===\n",
+    "with open(api_key_file) as f:\n",
+    "    api_key = f.read().strip()\n",
+    "\n",
+    "client = OpenAI(\n",
+    "    api_key=api_key,\n",
+    "    base_url=\"https://api.deepseek.com/v1\"\n",
+    ")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(inputCSV)\n",
+    "df = df.iloc[START_ROW:END_ROW].reset_index(drop=True)\n",
+    "\n",
+    "# === PREPARE PROMPTS ===\n",
+    "def create_prompt(row):\n",
+    "    name = row['real_name'] if pd.notna(row['real_name']) else row['name']\n",
+    "    hints = []\n",
+    "    for col in ('likely_profession','likely_nationality','likely_country'):\n",
+    "        if pd.notna(row.get(col, None)):\n",
+    "            hints.append(row[col])\n",
+    "    if not hints:\n",
+    "        tags = [row[f'tag_{i}'] for i in range(1,8) if pd.notna(row.get(f'tag_{i}', None))]\n",
+    "        hints.extend(tags[:3])\n",
+    "    hint_text = \", \".join(hints[:5])\n",
+    "    return (\n",
+    "        f\"Given '{name}' ({hint_text}), provide:\\n\"\n",
+    "        \"1. Full legal name (Western order if non-latin script)\\n\"\n",
+    "        \"2. Any stage names/aliases (comma separated)\\n\"\n",
+    "        \"3. Gender\\n\"\n",
+    "        \"4. Top 3 most specific, factual professions (use industry-standard terms, no euphemisms)\\n\"\n",
+    "        \"5. Primary country associated\\n\"\n",
+    "        \"Use 'Unknown' when uncertain or you encounter a fictional character or place. \"\n",
+    "        \"For entertainment fields, specify sub-genres when known (kpop, adult industry, etc.).\"\n",
+    "    )\n",
+    "\n",
+    "prompts = df.apply(create_prompt, axis=1).tolist()\n",
+    "\n",
+    "# === CHECK FOR EXISTING OUTPUT ===\n",
+    "if os.path.exists(INDEX_FILE):\n",
+    "    with open(INDEX_FILE, 'r') as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "else:\n",
+    "    current_index = 0\n",
+    "\n",
+    "results = []\n",
+    "if os.path.exists(OUTPUT_XLSX):\n",
+    "    existing = pd.read_excel(OUTPUT_XLSX)\n",
+    "    if {'full_name','aliases','gender','profession_llm','country'}.issubset(existing.columns):\n",
+    "        results = existing[['full_name','aliases','gender','profession_llm','country']].values.tolist()\n",
+    "\n",
+    "# === QUERY & PARSE ===\n",
+    "def query_deepseek(prompt):\n",
+    "    try:\n",
+    "        resp = client.chat.completions.create(\n",
+    "            model=\"deepseek-chat\",\n",
+    "            messages=[\n",
+    "                {\"role\":\"system\",\"content\":\"You extract key data on a person; respond with exactly 5 numbered lines.\"},\n",
+    "                {\"role\":\"user\",\"content\":prompt}\n",
+    "            ],\n",
+    "            temperature=0.05, top_p=0.8\n",
+    "        )\n",
+    "        return resp.choices[0].message.content.strip()\n",
+    "    except Exception as e:\n",
+    "        print(\"API error:\", e)\n",
+    "        return \"\"\n",
+    "\n",
+    "def parse_response(resp):\n",
+    "    lines = [l.strip() for l in resp.split('\\n') if l.strip()]\n",
+    "    out = [\"Unknown\"]*5\n",
+    "    for l in lines:\n",
+    "        if l.startswith('1.'): out[0] = l[2:].strip()\n",
+    "        elif l.startswith('2.'): out[1] = l[2:].strip()\n",
+    "        elif l.startswith('3.'): out[2] = l[2:].strip()\n",
+    "        elif l.startswith('4.'): out[3] = l[2:].strip()\n",
+    "        elif l.startswith('5.'): out[4] = l[2:].strip()\n",
+    "    return out\n",
+    "\n",
+    "# === PROCESS & SAVE ===\n",
+    "for i in range(current_index, len(df)):\n",
+    "    print(f\"Row {i+1}/{len(df)}...\")\n",
+    "    data = parse_response(query_deepseek(prompts[i]))\n",
+    "    if i < len(results): results[i] = data\n",
+    "    else: results.append(data)\n",
+    "    current_index = i+1\n",
+    "\n",
+    "    if current_index % SAVE_INTERVAL == 0 or current_index == len(df):\n",
+    "        out_df = df.iloc[:current_index].copy()\n",
+    "        out_df[['full_name','aliases','gender','profession_llm','country']] = pd.DataFrame(results[:current_index])\n",
+    "        # CSV\n",
+    "        out_df.to_csv(OUTPUT_CSV, index=False)\n",
+    "        # Excel\n",
+    "        out_df.to_excel(OUTPUT_XLSX, index=False)\n",
+    "        with open(INDEX_FILE, 'w') as f:\n",
+    "            f.write(str(current_index))\n",
+    "        print(\"Saved up to row\", current_index)\n",
+    "    time.sleep(1)\n",
+    "\n",
+    "print(\"All done! Files:\", OUTPUT_CSV, OUTPUT_XLSX)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7910e574",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9d377005",
+   "metadata": {},
+   "source": [
+    "# Aggregate by individual names"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d2e75354",
+   "metadata": {},
+   "source": [
+    " ##### e.g. Emma Watson [model1, model2, model3] etc."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "747c3a2f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "profession_map = current_dir.parent / \"misc/lists/mapped_professions.csv\"\n",
+    "\n",
+    "poi_df = current_dir.parent / \"data/CSV/Deepseek_annotated_POI.csv\"\n",
+    "\n",
+    "output = current_dir.parent / \"data/CSV/Deepseek_annotated_POI_aggregated.csv\"\n",
+    "\n",
+    "countries_csv = current_dir.parent / \"misc/lists/countries.csv\"\n",
+    "countries_df = pd.read_csv(countries_csv)\n",
+    "\n",
+    "# Extract valid country names (strip whitespace)\n",
+    "valid_countries = set(countries_df['en_short_name'].str.strip())\n",
+    "\n",
+    "\n",
+    "# Load the dataset\n",
+    "df = pd.read_csv(poi_df)  # Update path if needed\n",
+    "\n",
+    "# Step 1: Group by 'full_name' and aggregate required information\n",
+    "grouped_df = df.groupby('full_name').agg(\n",
+    "    No_of_models=('id', 'count'),\n",
+    "    modelIDs=('id', lambda x: list(x)),\n",
+    "    combinedDownloadCount=('downloadCount', 'sum')\n",
+    ").reset_index()\n",
+    "\n",
+    "# Step 2: Keep representative info for each person\n",
+    "# Keep representative info (including aliases)\n",
+    "additional_columns = df.groupby('full_name').agg(\n",
+    "    country=('country', 'first'),\n",
+    "    profession_llm=('profession_llm', 'first'),\n",
+    "    gender=('gender', 'first'),\n",
+    "    aliases=('aliases', 'first')  # ✅ Add this line\n",
+    ").reset_index()\n",
+    "\n",
+    "\n",
+    "\n",
+    "def standardize_country(country):\n",
+    "    if not isinstance(country, str):\n",
+    "        return \"Unknown\"\n",
+    "\n",
+    "    country_clean = country.strip()\n",
+    "    lowered = country_clean.lower()\n",
+    "\n",
+    "    # Handle fictional or fantasy countries\n",
+    "    fictional_keywords = [\"fictional\", \"westeros\", \"asgard\", \"middle-earth\", \"naboo\", \"middle earth\", \"latveria\"]\n",
+    "    if any(keyword in lowered for keyword in fictional_keywords):\n",
+    "        return \"Unknown\"\n",
+    "\n",
+    "    # Handle known region-based adjustments\n",
+    "    if \"macau\" in lowered:\n",
+    "        return \"Macau\"\n",
+    "    elif \"hong kong\" in lowered:\n",
+    "        return \"Hong Kong\"\n",
+    "    elif \"taiwan\" in lowered:\n",
+    "        return \"Taiwan\"\n",
+    "\n",
+    "    # Normalize complex or alternate country names\n",
+    "    lowered = lowered.replace(\"United Kingdom of Great Britain and Northern Ireland\", \"united kingdom\")\n",
+    "    lowered = lowered.replace(\"england\", \"united kingdom\")\n",
+    "    lowered = lowered.replace(\"united states of america\", \"united states\")\n",
+    "\n",
+    "    # Remove anything in brackets and after commas\n",
+    "    country_clean = re.sub(r\"\\(.*?\\)\", \"\", country_clean)\n",
+    "    country_clean = country_clean.split(',')[0].strip().lower()\n",
+    "\n",
+    "    # Manual overrides\n",
+    "    replacements = {\n",
+    "        \"united kingdom\": \"UK\",\n",
+    "        \"united kingdom of great britain and northern ireland\": \"UK\",\n",
+    "        \"french southern territories\": \"Other\",\n",
+    "        \"united states\": \"US\",\n",
+    "        \"united states of america\": \"US\",\n",
+    "        \"turkey\": \"Türkiye\",\n",
+    "        \"czech republic\": \"Czechia\"\n",
+    "    }\n",
+    "\n",
+    "    if country_clean in replacements:\n",
+    "        return replacements[country_clean]\n",
+    "\n",
+    "    # Final check against valid country list (case-insensitive)\n",
+    "    for valid in valid_countries:\n",
+    "        if country_clean == valid.lower():\n",
+    "            return valid\n",
+    "\n",
+    "    return \"Unknown\"\n",
+    "\n",
+    "\n",
+    "# Updated function to fully remove anything in brackets (complete or not)\n",
+    "def get_profession_short(profession):\n",
+    "    if isinstance(profession, str):\n",
+    "        # Get first part before comma\n",
+    "        first_prof = profession.split(',')[0].strip()\n",
+    "        # Remove all bracketed content, even malformed\n",
+    "        first_prof = re.sub(r\"[\\[].∗?[\\[].*?[\\]]\", \"\", first_prof)  # removes properly closed\n",
+    "        first_prof = re.sub(r\"[\\(\\[].*\", \"\", first_prof)         # removes malformed\n",
+    "        cleaned = first_prof.strip()\n",
+    "        # Normalize 'Actress' to 'Actor'\n",
+    "        if cleaned.lower() == \"actress\":\n",
+    "            return \"Actor\"\n",
+    "        return cleaned\n",
+    "    return None\n",
+    "\n",
+    "# Load your mapping file\n",
+    "mapping_df = pd.read_csv(profession_map, on_bad_lines='skip')  # or 'warn'\n",
+    "\n",
+    "\n",
+    "# Ensure the mapping columns are named correctly\n",
+    "# (Assuming columns are: 'profession_llm' or 'profession_short', and 'category' or 'mapped_profession')\n",
+    "# Adjust these as needed\n",
+    "mapping_df.columns = [col.strip().lower() for col in mapping_df.columns]\n",
+    "\n",
+    "# Rename for clarity and consistency\n",
+    "if 'profession_llm' in mapping_df.columns:\n",
+    "    mapping_df = mapping_df.rename(columns={'profession_llm': 'profession_short'})\n",
+    "if 'category' in mapping_df.columns:\n",
+    "    mapping_df = mapping_df.rename(columns={'category': 'mapped_profession'})\n",
+    "\n",
+    "# Merge the mapped profession into final_df\n",
+    "#final_df = final_df.merge(mapping_df[['profession_short', 'mapped_profession']], on='profession_short', how='left')\n",
+    "\n",
+    "\n",
+    "additional_columns = df.groupby('full_name').agg(\n",
+    "    country=('country', 'first'),\n",
+    "    profession_llm=('profession_llm', 'first'),\n",
+    "    gender=('gender', 'first'),\n",
+    "    aliases=('aliases', 'first')  # <-- Added this line\n",
+    ").reset_index()\n",
+    "\n",
+    "\n",
+    "# Step 3: Merge the aggregated info with the representative info\n",
+    "final_df = pd.merge(grouped_df, additional_columns, on='full_name', how='left')\n",
+    "\n",
+    "# Step 4: Clean and transform columns\n",
+    "final_df['profession_short'] = final_df['profession_llm'].apply(get_profession_short)\n",
+    "final_df['country'] = final_df['country'].apply(standardize_country)\n",
+    "\n",
+    "# Step 5: Merge with profession mapping\n",
+    "final_df = final_df.merge(mapping_df[['profession_short', 'mapped_profession']], on='profession_short', how='left')\n",
+    "\n",
+    "# Optional: Save the result to a CSV file\n",
+    "final_df.to_csv(output, index=False)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "704e5246",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-3-4__Figure_8b_sunburst_profession-checkpoint.ipynb

@@ -0,0 +1,123 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Prepare *.json for Figure 8b"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Sunburst data saved to sunburst_data.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from collections import defaultdict\n",
+    "import json\n",
+    "from pathlib import Path\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "sunburst_path = current_dir.parent / \"public/json/sunburst_countries_A.json\"\n",
+    "\n",
+    "\n",
+    "aggregated_poi = current_dir.parent / \"data/CSV/Deepseek_annotated_POI_aggregated.csv\"\n",
+    "\n",
+    "df = pd.read_csv(aggregated_poi)\n",
+    "df['country_cleaned'] = df['country'].apply(lambda x: x if x not in ['Unknown', '', None] else 'Other')\n",
+    "\n",
+    "# Now get top countries excluding what was forced into 'Other'\n",
+    "top_countries = df['country_cleaned'].value_counts().nlargest(15).index.tolist()\n",
+    "\n",
+    "# Final limited country column\n",
+    "df['country_limited'] = df['country_cleaned'].apply(lambda x: x if x in top_countries else 'Other')\n",
+    "\n",
+    "# ---- Step 2: Limit to top 7 professions and combine Unknown and Sports Professional with Other ----\n",
+    "top_professions = df['mapped_profession'].value_counts().nlargest(7).index.tolist()\n",
+    "\n",
+    "# Explicitly remove 'Unknown' and 'Sports Professional' even if they are in the top 7\n",
+    "top_professions = [p for p in top_professions if p not in ['Unknown', 'Sports Professional']]\n",
+    "\n",
+    "# Normalize 'Unknown' and '\n",
+    "# Sports Professional' into 'Other'\n",
+    "# Normalize 'Unknown' and 'Sports Professional' into 'Other'\n",
+    "df['profession_limited'] = df['mapped_profession'].apply(\n",
+    "    lambda x: 'Other' if x in ['Unknown', 'Sports Professional'] or x not in top_professions else x\n",
+    ")\n",
+    "\n",
+    "\n",
+    "\n",
+    "# ---- Step 3: Group by the limited country and profession ----\n",
+    "sunburst_data = df.groupby(['country_limited', 'profession_limited']).size().reset_index(name='count')\n",
+    "\n",
+    "# ---- Step 4: Create a nested structure for D3.js ----\n",
+    "sunburst_dict = {\"name\": \"root\", \"children\": []}\n",
+    "country_map = defaultdict(list)\n",
+    "\n",
+    "for _, row in sunburst_data.iterrows():\n",
+    "    country = row['country_limited']\n",
+    "    profession = row['profession_limited']\n",
+    "    count = int(row['count'])\n",
+    "    country_map[country].append({\"name\": profession, \"value\": count})\n",
+    "\n",
+    "# For each country, sort the profession list so that \"Other\" appears at the end\n",
+    "for country, professions in country_map.items():\n",
+    "    professions_sorted = sorted(professions, key=lambda d: (d[\"name\"] == \"Other\", d[\"name\"]))\n",
+    "    country_map[country] = professions_sorted\n",
+    "\n",
+    "for country, professions in country_map.items():\n",
+    "    sunburst_dict[\"children\"].append({\"name\": country, \"children\": professions})\n",
+    "\n",
+    "# ---- Step 5: Save to a JSON file ----\n",
+    "with open(sunburst_path, \"w\", encoding='utf-8') as f:\n",
+    "    json.dump(sunburst_dict, f, ensure_ascii=False, indent=2)\n",
+    "\n",
+    "\n",
+    "print(\"✅ Sunburst data saved to sunburst_data.json\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "the resulting *.json is the input for Figure_8.html"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

jupyter_notebooks/Section_2-3-4_Figure_8_deepfake_adapters.ipynb

@@ -121,29 +121,29 @@
    "id": "3c906826",
    "metadata": {},
    "source": [
-    "## NER Code"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "cb529f57",
-   "metadata": {},
-   "source": [
-    "please put NER code here. on the first collumn of the adapter dataset in data/real_person_adapters.csv named entity recognition should be performed to extract the most plausible name."
+    "## STEP 01 NER"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 7,
    "id": "ner_code_cell",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T10:50:51.775852Z",
+     "iopub.status.busy": "2025-11-21T10:50:51.775660Z",
+     "iopub.status.idle": "2025-11-21T10:53:16.141998Z",
+     "shell.execute_reply": "2025-11-21T10:53:16.141295Z",
+     "shell.execute_reply.started": "2025-11-21T10:50:51.775836Z"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "✅ spaCy model loaded: en_core_web_sm\n",
-      "Loaded 18 rows\n",
+      "Loaded 50861 rows\n",
       "\n",
       "🔄 Processing names with spaCy NER...\n",
       "\n",
@@ -162,6 +162,13 @@
       "Game of Thrones Cast                               | Game Thrones Cast             \n",
       "Natalie Portman「LoRa」                              | Natalie Portman               \n",
       "Emma Watson LoRA                                   | Emma Watson                   \n",
+      "Karina Makina Lora                                 | Karina Makina                 \n",
+      "WRAV YUA_三xx亜                                      | WRAV YUA                      \n",
+      "Dilraba Dilmurat 迪丽热巴                              | Dilraba Dilmurat              \n",
+      "MIMI,大幂幂                                           | MIMI                          \n",
+      "Chinese Idol - YangMi杨幂                            | Chinese Idol YangMi杨幂         \n",
+      "Xiaorouseeu / 小柔SeeU - Chinese cosplayer and influencer | Xiaorouseeu                   \n",
+      "Jennifer Connelly (80s/90s)                        | Jennifer Connelly             \n",
       "====================================================================================================\n",
       "\n",
       "🧪 Leetspeak translation examples:\n",
@@ -171,9 +178,9 @@
       "  L3vi Ackerman                  -> Levi Ackerman\n",
       "\n",
       "📈 Statistics:\n",
-      "  Total rows: 18\n",
-      "  Non-empty names: 18\n",
-      "  Empty names: 0\n",
+      "  Total rows: 50861\n",
+      "  Non-empty names: 50858\n",
+      "  Empty names: 3\n",
       "\n",
       "🎯 Sample spaCy NER results:\n",
       "  1. IU\n",
@@ -187,8 +194,8 @@
       "  9. Emma Watson\n",
       "  10. Gal Gadot\n",
       "\n",
-      "✅ Cleaned 18 names using spaCy NER\n",
-      "💾 Saved to /home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/CSV/NER_POI_step01_pre_annotation.csv\n"
+      "✅ Cleaned 50861 names using spaCy NER\n",
+      "💾 Saved to /shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/data/CSV/model_adapter/real_person_adapter_step_01_NER.csv\n"
      ]
     }
    ],
@@ -214,7 +221,7 @@
     "# Set up paths\n",
     "current_dir = Path.cwd()\n",
     "#input_file = current_dir.parent / \"data/CSV/real_person_adapters.csv\"\n",
-    "input_file = current_dir.parent / \"data/CSV/samples/real_person_adapters_sample.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter.csv\"\n",
     "\n",
     "# Load dataset\n",
     "df = pd.read_csv(input_file)\n",
@@ -274,7 +281,7 @@
     "    \n",
     "    # Remove LoRA-related terms (case insensitive)\n",
     "    lora_terms = ['lora', 'loha', 'lycoris', 'controlnet', 'textual inversion', \n",
-    "                  'embedding', 'ti', 'checkpoint', 'model', 'adapter']\n",
+    "                  'embedding', 'ti', 'checkpoint', 'model', 'adapter', 'pony', 'sdxl', 'flux', 'illustrious', 'sd14', 'sd14', 'sd2', 'sd3', 'diffusion', 'stable', 'hunyuan']\n",
     "    for term in lora_terms:\n",
     "        name = re.sub(rf'\\b{term}\\b', '', name, flags=re.IGNORECASE)\n",
     "    \n",
@@ -397,7 +404,7 @@
     "print(f\"\\n✅ Cleaned {len(df)} names using spaCy NER\")\n",
     "\n",
     "# Save intermediate result\n",
-    "output_step1 = current_dir.parent / \"data/CSV/NER_POI_step01_pre_annotation.csv\"\n",
+    "output_step1 = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_01_NER.csv\"\n",
     "df.to_csv(output_step1, index=False)\n",
     "print(f\"💾 Saved to {output_step1}\")\n"
    ]
@@ -407,15 +414,23 @@
    "id": "d701d1cb",
    "metadata": {},
    "source": [
-    "## nationality to Country\n",
+    "## STEP 02: Nationality tag to Country\n",
     "here tags related to nationality gets converted to the country equivalent. make sure isle of man is shows never up in the country collumn.as well as other uninhabited places."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 11,
    "id": "36ce71f2",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T10:57:51.803570Z",
+     "iopub.status.busy": "2025-11-21T10:57:51.803378Z",
+     "iopub.status.idle": "2025-11-21T10:57:58.819903Z",
+     "shell.execute_reply": "2025-11-21T10:57:58.819309Z",
+     "shell.execute_reply.started": "2025-11-21T10:57:51.803555Z"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -423,13 +438,24 @@
      "text": [
       "Built country/nationality mapping with 781 entries\n",
       "\n",
-      "Processed 18 rows\n",
-      "Rows with country: 0\n",
+      "Processed 50861 rows\n",
+      "Rows with country: 1883\n",
       "Rows with nationality: 0\n",
-      "Rows with profession: 14\n",
+      "Rows with profession: 43674\n",
       "\n",
       "Top 10 countries:\n",
-      "Series([], Name: count, dtype: int64)\n",
+      "likely_country\n",
+      "American Samoa    973\n",
+      "Japan             250\n",
+      "Indonesia          73\n",
+      "Argentina          73\n",
+      "Ireland            53\n",
+      "France             38\n",
+      "Netherlands        38\n",
+      "Hong Kong          35\n",
+      "Brazil             29\n",
+      "Germany            24\n",
+      "Name: count, dtype: int64\n",
       "\n",
       "Sample results:\n",
       "         real_name likely_country likely_nationality likely_profession\n",
@@ -449,6 +475,166 @@
       "  Tags: ['girl', 'photorealistic', 'asian', 'woman', 'beautiful', 'celebrity', 'korean']\n",
       "  Country: \n",
       "  Nationality: \n",
+      "\n",
+      "  Name: Irene Makina\n",
+      "  Tags: ['woman', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Red Velvet Irene\n",
+      "  Tags: ['kpop', 'woman', 'celebrity', 'kpop idol']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: RED VELVET Irene\n",
+      "  Tags: ['kpop', 'woman', 'celebrity', 'irene', 'redvelvet']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene\n",
+      "  Tags: ['singer', 'k-pop idol', 'actress', 'celebrity', 'korean', 'irene', 'bae joo-hyun']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene\n",
+      "  Tags: ['singer', 'photorealistic', 'asian', 'idol', 'dancer', 'woman', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Makina\n",
+      "  Tags: ['kpop', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: irene주현\n",
+      "  Tags: ['photorealistic', 'sexy', 'female', 'woman', 'celebrity', 'girls', 'realistic']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene\n",
+      "  Tags: ['singer', 'woman', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Red Velvet Irene\n",
+      "  Tags: ['kpop', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Schouten\n",
+      "  Tags: ['instagram', 'woman', 'celebrity', 'dutch', 'sporty', 'speed skating']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene\n",
+      "  Tags: ['photorealistic', 'female', 'woman', 'celebrity', 'realistic']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene\n",
+      "  Tags: ['photorealistic', 'female', 'kpop', 'idol', 'woman', 'celebrity', 'girls']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene\n",
+      "  Tags: ['kpop', 'woman', 'celebrity', 'kpop girl']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Rouse\n",
+      "  Tags: ['person', 'photorealistic', 'female', 'cosplay', 'woman', 'celebrity', 'girls']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: IreneDiffusion\n",
+      "  Tags: ['irene', 'irene montero', 'montero']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Valory Irene\n",
+      "  Tags: ['female', 'woman', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene\n",
+      "  Tags: ['singer', 'asian', 'kpop', 'actress', 'celeb', 'celebrity', 'korean']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Red Velvet\n",
+      "  Tags: ['kpop', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Rouse - Adult Film Actress\n",
+      "  Tags: ['photorealistic', 'sexy', 'woman', 'celebrity', 'girls', 'realistic', 'lora']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Rouse - Adult Film Actress\n",
+      "  Tags: ['photorealistic', 'sexy', 'woman', 'celebrity', 'girls', 'realistic', 'lora']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Red Velvet\n",
+      "  Tags: ['photorealistic', 'kpop', 'woman', 'celebrity', 'korean', 'kpop idol']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: 麦橘复古少女IreneMajicflus\n",
+      "  Tags: ['style']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene F1\n",
+      "  Tags: ['woman', 'actor', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene\n",
+      "  Tags: ['celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: IRENE\n",
+      "  Tags: ['realism', 'kpop', 'idol', 'woman', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Rouse\n",
+      "  Tags: ['photorealistic', 'sexy', 'woman', 'celebrity', 'girls', 'realistic', 'lora']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Valory Irene\n",
+      "  Tags: ['girl', 'sexy', 'model', 'woman', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Rouse\n",
+      "  Tags: ['woman', 'celebrity']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Dirdal\n",
+      "  Tags: ['photorealistic', 'woman', 'celebrity', 'girls', 'realistic', 'lora']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Cara\n",
+      "  Tags: ['singer', 'woman', 'actress', 'celebrity', 'irene cara']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Rouse\n",
+      "  Tags: ['colombia', 'instagram', 'latina', 'woman', 'celebrity', 'realistic', 'sd 1.5']\n",
+      "  Country: Colombia\n",
+      "  Nationality: \n",
+      "\n",
+      "  Name: Irene Caruso -\n",
+      "  Tags: ['model', 'celebrity', 'influencer']\n",
+      "  Country: \n",
+      "  Nationality: \n",
       "\n"
      ]
     }
@@ -461,8 +647,8 @@
     "current_dir = Path.cwd()\n",
     "countries_file = current_dir.parent / \"misc/lists/countries.csv\"\n",
     "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
-    "input_file = current_dir.parent / \"data/CSV/NER_POI_step01_pre_annotation.csv\"\n",
-    "output_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_01_NER.csv\"\n",
+    "output_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
     "\n",
     "# Load datasets\n",
     "poi_df = pd.read_csv(input_file)\n",
@@ -620,12 +806,6 @@
     "        print()\n"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "29eec5be",
-   "metadata": {},
-   "source": []
-  },
   {
    "cell_type": "markdown",
    "id": "deepseek_header",
@@ -644,28 +824,121 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 13,
    "id": "deepseek_code",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T11:01:40.377178Z",
+     "iopub.status.busy": "2025-11-21T11:01:40.376988Z",
+     "iopub.status.idle": "2025-11-21T11:02:15.369105Z",
+     "shell.execute_reply": "2025-11-21T11:02:15.368626Z",
+     "shell.execute_reply.started": "2025-11-21T11:01:40.377164Z"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "❌ Error: deepseek_api_key.txt not found in misc/credentials/\n",
-      "   Please create this file with your API key\n"
+      "✅ Deepseek API key loaded\n",
+      "✅ Loaded profession mapping with 9 categories\n",
+      "Loaded 50861 rows\n",
+      "\n",
+      "Profession categories (9):\n",
+      "  - actor\n",
+      "  - adult performer\n",
+      "  - singer/musician\n",
+      "  - model\n",
+      "  - online personality\n",
+      "  - public figure\n",
+      "  - voice actor/ASMR\n",
+      "  - sports professional\n",
+      "  - tv personality\n",
+      "\n",
+      "Running in TEST MODE with 10 samples\n",
+      "\n",
+      "Annotating with DEEPSEEK...\n",
+      "Processing rows 0 to 10\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Annotating:  90%|█████████ | 9/10 [00:30<00:03,  3.39s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Progress saved after 10 rows\n"
      ]
     },
     {
-     "ename": "FileNotFoundError",
-     "evalue": "[Errno 2] No such file or directory: '/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/misc/credentials/deepseek_api_key.txt'",
-     "output_type": "error",
-     "traceback": [
-      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
-      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
-      "Cell \u001b[0;32mIn[10], line 37\u001b[0m\n\u001b[1;32m     35\u001b[0m \u001b[38;5;66;03m# === LOAD API KEY ===\u001b[39;00m\n\u001b[1;32m     36\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 37\u001b[0m     \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mapi_key_file\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[1;32m     38\u001b[0m         api_key \u001b[38;5;241m=\u001b[39m f\u001b[38;5;241m.\u001b[39mread()\u001b[38;5;241m.\u001b[39mstrip()\n\u001b[1;32m     39\u001b[0m     \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m✅ Deepseek API key loaded\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
-      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/IPython/core/interactiveshell.py:324\u001b[0m, in \u001b[0;36m_modified_open\u001b[0;34m(file, *args, **kwargs)\u001b[0m\n\u001b[1;32m    317\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m}:\n\u001b[1;32m    318\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m    319\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIPython won\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m by default \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    320\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    321\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myou can use builtins\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m open.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    322\u001b[0m     )\n\u001b[0;32m--> 324\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mio_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
-      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/misc/credentials/deepseek_api_key.txt'"
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Annotating: 100%|██████████| 10/10 [00:34<00:00,  3.40s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "✅ Done! Final results saved to /shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/data/CSV/model_adapter/real_person_adapter_step_03_deepseek_test.csv\n",
+      "\n",
+      "=== Summary Statistics ===\n",
+      "Total processed: 10\n",
+      "\n",
+      "Gender distribution:\n",
+      "gender\n",
+      "Female    10\n",
+      "Name: count, dtype: int64\n",
+      "\n",
+      "Top 10 profession combinations:\n",
+      "profession_llm\n",
+      "actor, singer/musician, model                            2\n",
+      "singer/musician, actor, tv personality                   1\n",
+      "model, online personality                                1\n",
+      "singer/musician, actor, model                            1\n",
+      "singer/musician, voice actor/ASMR, online personality    1\n",
+      "singer/musician, model, online personality               1\n",
+      "model, online personality, actor                         1\n",
+      "actor, public figure, model                              1\n",
+      "actor, model, public figure                              1\n",
+      "Name: count, dtype: int64\n",
+      "\n",
+      "Top 10 countries:\n",
+      "country\n",
+      "South Korea       3\n",
+      "China             2\n",
+      "Japan             2\n",
+      "Unknown           1\n",
+      "United Kingdom    1\n",
+      "Israel            1\n",
+      "Name: count, dtype: int64\n",
+      "\n",
+      "=== Sample Results ===\n",
+      "      real_name                    full_name gender                                        profession_llm        country\n",
+      "             IU                   Lee Ji-eun Female                singer/musician, actor, tv personality    South Korea\n",
+      "Super Pose Book                      Unknown Female                             model, online personality        Unknown\n",
+      "          Liyuu                        Liyuu Female singer/musician, voice actor/ASMR, online personality          China\n",
+      "          Irene                    Irene Bae Female                         singer/musician, actor, model    South Korea\n",
+      "   AESPA Karina                    Yu Ji-min Female            singer/musician, model, online personality    South Korea\n",
+      " Saika Kawakita               Saika Kawakita Female                      model, online personality, actor          Japan\n",
+      "      Liu Yifei                    Liu Yifei Female                         actor, singer/musician, model          China\n",
+      " HashimotoKanna              Kanna Hashimoto Female                         actor, singer/musician, model          Japan\n",
+      "    Emma Watson Emma Charlotte Duerre Watson Female                           actor, public figure, model United Kingdom\n",
+      "      Gal Gadot                    Gal Gadot Female                           actor, model, public figure         Israel\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\n"
      ]
     }
    ],
@@ -680,7 +953,7 @@
     "from tqdm import tqdm\n",
     "\n",
     "current_dir = Path.cwd()\n",
-    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
     "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
     "professions_mapped_file = current_dir.parent / \"misc/lists/professions_mapped.csv\"\n",
     "api_key_file = current_dir.parent / \"misc/credentials/deepseek_api_key.txt\"\n",
@@ -858,8 +1131,8 @@
     "    return fields\n",
     "\n",
     "# === PROCESS DATA ===\n",
-    "output_file = current_dir.parent / f\"data/CSV/deepseek_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
-    "index_file = current_dir.parent / \"misc/deepseek_query_index.txt\"\n",
+    "output_file = current_dir.parent / f\"data/CSV/model_adapter/real_person_adapter_step_03_deepseek{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/query_indicies/deepseek_query_index.txt\"\n",
     "\n",
     "# Check for existing progress\n",
     "current_index = 0\n",
@@ -952,7 +1225,14 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "qwen_code",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.status.busy": "2025-11-20T21:12:14.048643Z",
+     "iopub.status.idle": "2025-11-20T21:12:14.048904Z",
+     "shell.execute_reply": "2025-11-20T21:12:14.048734Z",
+     "shell.execute_reply.started": "2025-11-20T21:12:14.048726Z"
+    }
+   },
    "outputs": [],
    "source": [
     "import pandas as pd\n",
@@ -1665,7 +1945,40 @@
    "cell_type": "markdown",
    "id": "01086b35",
    "metadata": {},
-   "source": "## Qwen-2.5-32B Local Annotation (Hugging Face Transformers)\n\nRun **Qwen-2.5-32B-Instruct** locally using Hugging Face Transformers - NO API COSTS!\n\n**Requirements:**\n- GPU: NVIDIA A100 80GB or similar (32B model needs ~60-70GB VRAM)\n- RAM: 128GB+ system RAM recommended\n- Model cache: `data/models/` (auto-downloaded on first run)\n- Model size: ~65GB download (BF16 weights)\n\n**Installation:**\n```bash\n# Install required packages\npip install torch transformers accelerate\n```\n\n**Advantages:**\n- ✅ FREE - No API costs\n- ✅ Fast - Local inference with GPU acceleration\n- ✅ Privacy - Data never leaves your machine\n- ✅ Offline - Works without internet (after download)\n- ✅ High quality - 32B parameter instruction-tuned model\n\n**Performance:**\n- Speed: ~10-30 tokens/sec on A100 (depends on batch size)\n- Throughput: ~200-500 samples/hour\n- Cost: $0 (after model download)\n\n**Model Info:**\n- HuggingFace: `Qwen/Qwen2.5-32B-Instruct`\n- License: Apache 2.0 (commercial use allowed)\n- Context: 32K tokens"
+   "source": [
+    "## Qwen-2.5-32B Local Annotation (Hugging Face Transformers)\n",
+    "\n",
+    "Run **Qwen-2.5-32B-Instruct** locally using Hugging Face Transformers - NO API COSTS!\n",
+    "\n",
+    "**Requirements:**\n",
+    "- GPU: NVIDIA A100 80GB or similar (32B model needs ~60-70GB VRAM)\n",
+    "- RAM: 128GB+ system RAM recommended\n",
+    "- Model cache: `data/models/` (auto-downloaded on first run)\n",
+    "- Model size: ~65GB download (BF16 weights)\n",
+    "\n",
+    "**Installation:**\n",
+    "```bash\n",
+    "# Install required packages\n",
+    "pip install torch transformers accelerate\n",
+    "```\n",
+    "\n",
+    "**Advantages:**\n",
+    "- ✅ FREE - No API costs\n",
+    "- ✅ Fast - Local inference with GPU acceleration\n",
+    "- ✅ Privacy - Data never leaves your machine\n",
+    "- ✅ Offline - Works without internet (after download)\n",
+    "- ✅ High quality - 32B parameter instruction-tuned model\n",
+    "\n",
+    "**Performance:**\n",
+    "- Speed: ~10-30 tokens/sec on A100 (depends on batch size)\n",
+    "- Throughput: ~200-500 samples/hour\n",
+    "- Cost: $0 (after model download)\n",
+    "\n",
+    "**Model Info:**\n",
+    "- HuggingFace: `Qwen/Qwen2.5-32B-Instruct`\n",
+    "- License: Apache 2.0 (commercial use allowed)\n",
+    "- Context: 32K tokens"
+   ]
   },
   {
    "cell_type": "code",
@@ -1673,14 +1986,263 @@
    "id": "5dd7a803",
    "metadata": {},
    "outputs": [],
-   "source": "import pandas as pd\nimport json\nimport time\nimport re\nfrom pathlib import Path\nfrom tqdm import tqdm\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\ncurrent_dir = Path.cwd()\ninput_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n\n# === CONFIGURATION ===\nTEST_MODE = True\nTEST_SIZE = 10\nMAX_ROWS = 20000\nSAVE_INTERVAL = 10\n\n# Model settings\nMODEL_NAME = \"Qwen/Qwen2.5-32B-Instruct\"\nCACHE_DIR = current_dir.parent / \"data/models\"\nCACHE_DIR.mkdir(parents=True, exist_ok=True)\n\nPROFESSION_CATEGORIES = [\n    \"actor\", \"adult performer\", \"singer/musician\", \"model\",\n    \"online personality\", \"public figure\", \"voice actor/ASMR\",\n    \"sports professional\", \"tv personality\"\n]\n\n# === LOAD MODEL ===\nprint(f\"Loading model: {MODEL_NAME}\")\nprint(f\"Cache directory: {CACHE_DIR}\")\nprint(f\"This may take a while on first run (~65GB download)...\\n\")\n\n# Check GPU availability\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(f\"Device: {device}\")\n\nif device == \"cpu\":\n    print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n    print(\"   Consider using a GPU or reducing model size.\")\n\n# Load tokenizer\nprint(\"Loading tokenizer...\")\ntokenizer = AutoTokenizer.from_pretrained(\n    MODEL_NAME,\n    cache_dir=str(CACHE_DIR),\n    trust_remote_code=True\n)\nprint(\"✅ Tokenizer loaded\")\n\n# Load model with optimizations\nprint(\"Loading model (this may take several minutes)...\")\nmodel = AutoModelForCausalLM.from_pretrained(\n    MODEL_NAME,\n    cache_dir=str(CACHE_DIR),\n    torch_dtype=torch.bfloat16,  # Use BF16 for efficiency\n    device_map=\"auto\",  # Automatically distribute across GPUs\n    trust_remote_code=True\n)\nmodel.eval()  # Set to evaluation mode\nprint(\"✅ Model loaded\")\n\n# Check VRAM usage\nif torch.cuda.is_available():\n    vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n    print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n\n# === LOAD DATA ===\ndf = pd.read_csv(input_file)\nprint(f\"Loaded {len(df)} rows\")\n\nif TEST_MODE:\n    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")\n    df = df.head(TEST_SIZE).copy()\nelif MAX_ROWS:\n    df = df.head(MAX_ROWS).copy()\n\n# === CREATE PROMPT ===\ndef create_prompt(name: str, tags: list, country: str) -> str:\n    tags_str = \", \".join(tags[:5]) if tags else \"none\"\n    country_str = country if country else \"unknown\"\n    \n    return f\"\"\"You are an expert at identifying public figures.\n\nGiven: Name: {name}, Tags: {tags_str}, Country: {country_str}\n\nProvide in JSON format:\n{{\n  \"full_name\": \"<full name>\",\n  \"gender\": \"<Male/Female/Other/Unknown>\",\n  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],\n  \"country\": \"<country>\"\n}}\n\nChoose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}\n- Max 3 professions, ordered by relevance\n- \"online personality\" includes streamers, cosplayers, influencers\n- Return \"Unknown\" if uncertain\n\nReturn ONLY JSON, no extra text.\"\"\"\n\n# === QUERY QWEN LOCAL ===\ndef query_qwen_local(prompt: str) -> dict:\n    \"\"\"Query Qwen-2.5-32B locally via transformers.\"\"\"\n    try:\n        # Format as chat message\n        messages = [\n            {\"role\": \"user\", \"content\": prompt}\n        ]\n        \n        # Tokenize\n        text = tokenizer.apply_chat_template(\n            messages,\n            tokenize=False,\n            add_generation_prompt=True\n        )\n        \n        inputs = tokenizer([text], return_tensors=\"pt\").to(device)\n        \n        # Generate\n        with torch.no_grad():\n            outputs = model.generate(\n                **inputs,\n                max_new_tokens=512,\n                temperature=0.1,\n                do_sample=True,\n                top_p=0.9,\n                pad_token_id=tokenizer.eos_token_id\n            )\n        \n        # Decode\n        generated_ids = outputs[0][inputs['input_ids'].shape[1]:]\n        response = tokenizer.decode(generated_ids, skip_special_tokens=True)\n        \n        # Parse JSON from response\n        json_match = re.search(r'\\{[^}]+\\}', response, re.DOTALL)\n        if json_match:\n            return json.loads(json_match.group())\n        \n        # Try markdown code block\n        code_block = re.search(r'```(?:json)?\\s*(\\{[^}]+\\})\\s*```', response, re.DOTALL)\n        if code_block:\n            return json.loads(code_block.group(1))\n        \n        return None\n        \n    except json.JSONDecodeError as e:\n        print(f\"JSON parse error: {e}\")\n        return None\n    except Exception as e:\n        print(f\"Generation error: {e}\")\n        return None\n\n# === PROCESS ===\noutput_file = current_dir.parent / f\"data/CSV/qwen_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\nindex_file = current_dir.parent / \"misc/qwen_local_query_index.txt\"\n\ncurrent_index = 0\nif index_file.exists():\n    with open(index_file) as f:\n        current_index = int(f.read().strip())\n    print(f\"Resuming from index {current_index}\")\n\nfor col in ['full_name', 'gender', 'profession_llm', 'country']:\n    if col not in df.columns:\n        df[col] = 'Unknown'\n\nprint(f\"\\nAnnotating with QWEN-2.5-32B LOCAL - rows {current_index} to {len(df)}...\")\nprint(f\"Model: {MODEL_NAME}\")\nprint(f\"This may take a while (~200-500 samples/hour on A100)\\n\")\n\ntry:\n    start_time = time.time()\n    \n    for i in tqdm(range(current_index, len(df)), desc=\"Qwen Local\"):\n        row = df.iloc[i]\n        \n        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8)\n                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]\n        \n        prompt = create_prompt(\n            row.get('real_name', row.get('name', '')),\n            tags,\n            row.get('likely_country', '')\n        )\n        \n        result = query_qwen_local(prompt)\n        \n        if result:\n            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')\n            df.at[i, 'gender'] = result.get('gender', 'Unknown')\n            profs = result.get('professions', [])\n            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)\n            df.at[i, 'country'] = result.get('country', 'Unknown')\n        \n        current_index = i + 1\n        \n        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n            df.to_csv(output_file, index=False)\n            with open(index_file, 'w') as f:\n                f.write(str(current_index))\n            \n            # Calculate stats\n            elapsed = time.time() - start_time\n            samples_done = current_index - (0 if not index_file.exists() else int(open(index_file).read().strip()))\n            samples_per_hour = (samples_done / elapsed) * 3600 if elapsed > 0 else 0\n            \n            print(f\"✅ Saved after {i+1} rows (~{samples_per_hour:.1f} samples/hour)\")\n    \n    elapsed_total = time.time() - start_time\n    print(f\"\\n✅ Done! Results: {output_file}\")\n    print(f\"Total time: {elapsed_total/60:.1f} minutes\")\n    print(f\"Average speed: {len(df)/(elapsed_total/3600):.1f} samples/hour\")\n    \n    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")\n    \n    # Show VRAM stats\n    if torch.cuda.is_available():\n        print(f\"\\nFinal VRAM usage: {torch.cuda.max_memory_allocated() / 1024**3:.2f} GB\")\n\nexcept KeyboardInterrupt:\n    print(f\"\\n⚠️ Interrupted by user\")\n    df.to_csv(output_file, index=False)\n    with open(index_file, 'w') as f:\n        f.write(str(current_index))\n    print(f\"Progress saved up to row {current_index}\")\n\nexcept Exception as e:\n    print(f\"⚠️ Error: {e}\")\n    df.to_csv(output_file, index=False)\n    with open(index_file, 'w') as f:\n        f.write(str(current_index))\n    print(f\"Progress saved up to row {current_index}\")"
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = True\n",
+    "TEST_SIZE = 10\n",
+    "MAX_ROWS = 20000\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "# Model settings\n",
+    "MODEL_NAME = \"Qwen/Qwen2.5-32B-Instruct\"\n",
+    "CACHE_DIR = current_dir.parent / \"data/models\"\n",
+    "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",\n",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",\n",
+    "    \"sports professional\", \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD MODEL ===\n",
+    "print(f\"Loading model: {MODEL_NAME}\")\n",
+    "print(f\"Cache directory: {CACHE_DIR}\")\n",
+    "print(f\"This may take a while on first run (~65GB download)...\\n\")\n",
+    "\n",
+    "# Check GPU availability\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"Device: {device}\")\n",
+    "\n",
+    "if device == \"cpu\":\n",
+    "    print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n",
+    "    print(\"   Consider using a GPU or reducing model size.\")\n",
+    "\n",
+    "# Load tokenizer\n",
+    "print(\"Loading tokenizer...\")\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\n",
+    "    MODEL_NAME,\n",
+    "    cache_dir=str(CACHE_DIR),\n",
+    "    trust_remote_code=True\n",
+    ")\n",
+    "print(\"✅ Tokenizer loaded\")\n",
+    "\n",
+    "# Load model with optimizations\n",
+    "print(\"Loading model (this may take several minutes)...\")\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    MODEL_NAME,\n",
+    "    cache_dir=str(CACHE_DIR),\n",
+    "    torch_dtype=torch.bfloat16,  # Use BF16 for efficiency\n",
+    "    device_map=\"auto\",  # Automatically distribute across GPUs\n",
+    "    trust_remote_code=True\n",
+    ")\n",
+    "model.eval()  # Set to evaluation mode\n",
+    "print(\"✅ Model loaded\")\n",
+    "\n",
+    "# Check VRAM usage\n",
+    "if torch.cuda.is_available():\n",
+    "    vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n",
+    "    print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPT ===\n",
+    "def create_prompt(name: str, tags: list, country: str) -> str:\n",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"\n",
+    "    country_str = country if country else \"unknown\"\n",
+    "    \n",
+    "    return f\"\"\"You are an expert at identifying public figures.\n",
+    "\n",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}\n",
+    "\n",
+    "Provide in JSON format:\n",
+    "{{\n",
+    "  \"full_name\": \"<full name>\",\n",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",\n",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],\n",
+    "  \"country\": \"<country>\"\n",
+    "}}\n",
+    "\n",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}\n",
+    "- Max 3 professions, ordered by relevance\n",
+    "- \"online personality\" includes streamers, cosplayers, influencers\n",
+    "- Return \"Unknown\" if uncertain\n",
+    "\n",
+    "Return ONLY JSON, no extra text.\"\"\"\n",
+    "\n",
+    "# === QUERY QWEN LOCAL ===\n",
+    "def query_qwen_local(prompt: str) -> dict:\n",
+    "    \"\"\"Query Qwen-2.5-32B locally via transformers.\"\"\"\n",
+    "    try:\n",
+    "        # Format as chat message\n",
+    "        messages = [\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "        ]\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        text = tokenizer.apply_chat_template(\n",
+    "            messages,\n",
+    "            tokenize=False,\n",
+    "            add_generation_prompt=True\n",
+    "        )\n",
+    "        \n",
+    "        inputs = tokenizer([text], return_tensors=\"pt\").to(device)\n",
+    "        \n",
+    "        # Generate\n",
+    "        with torch.no_grad():\n",
+    "            outputs = model.generate(\n",
+    "                **inputs,\n",
+    "                max_new_tokens=512,\n",
+    "                temperature=0.1,\n",
+    "                do_sample=True,\n",
+    "                top_p=0.9,\n",
+    "                pad_token_id=tokenizer.eos_token_id\n",
+    "            )\n",
+    "        \n",
+    "        # Decode\n",
+    "        generated_ids = outputs[0][inputs['input_ids'].shape[1]:]\n",
+    "        response = tokenizer.decode(generated_ids, skip_special_tokens=True)\n",
+    "        \n",
+    "        # Parse JSON from response\n",
+    "        json_match = re.search(r'\\{[^}]+\\}', response, re.DOTALL)\n",
+    "        if json_match:\n",
+    "            return json.loads(json_match.group())\n",
+    "        \n",
+    "        # Try markdown code block\n",
+    "        code_block = re.search(r'```(?:json)?\\s*(\\{[^}]+\\})\\s*```', response, re.DOTALL)\n",
+    "        if code_block:\n",
+    "            return json.loads(code_block.group(1))\n",
+    "        \n",
+    "        return None\n",
+    "        \n",
+    "    except json.JSONDecodeError as e:\n",
+    "        print(f\"JSON parse error: {e}\")\n",
+    "        return None\n",
+    "    except Exception as e:\n",
+    "        print(f\"Generation error: {e}\")\n",
+    "        return None\n",
+    "\n",
+    "# === PROCESS ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/qwen_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/qwen_local_query_index.txt\"\n",
+    "\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    with open(index_file) as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "    print(f\"Resuming from index {current_index}\")\n",
+    "\n",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:\n",
+    "    if col not in df.columns:\n",
+    "        df[col] = 'Unknown'\n",
+    "\n",
+    "print(f\"\\nAnnotating with QWEN-2.5-32B LOCAL - rows {current_index} to {len(df)}...\")\n",
+    "print(f\"Model: {MODEL_NAME}\")\n",
+    "print(f\"This may take a while (~200-500 samples/hour on A100)\\n\")\n",
+    "\n",
+    "try:\n",
+    "    start_time = time.time()\n",
+    "    \n",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Qwen Local\"):\n",
+    "        row = df.iloc[i]\n",
+    "        \n",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8)\n",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]\n",
+    "        \n",
+    "        prompt = create_prompt(\n",
+    "            row.get('real_name', row.get('name', '')),\n",
+    "            tags,\n",
+    "            row.get('likely_country', '')\n",
+    "        )\n",
+    "        \n",
+    "        result = query_qwen_local(prompt)\n",
+    "        \n",
+    "        if result:\n",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')\n",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')\n",
+    "            profs = result.get('professions', [])\n",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)\n",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            with open(index_file, 'w') as f:\n",
+    "                f.write(str(current_index))\n",
+    "            \n",
+    "            # Calculate stats\n",
+    "            elapsed = time.time() - start_time\n",
+    "            samples_done = current_index - (0 if not index_file.exists() else int(open(index_file).read().strip()))\n",
+    "            samples_per_hour = (samples_done / elapsed) * 3600 if elapsed > 0 else 0\n",
+    "            \n",
+    "            print(f\"✅ Saved after {i+1} rows (~{samples_per_hour:.1f} samples/hour)\")\n",
+    "    \n",
+    "    elapsed_total = time.time() - start_time\n",
+    "    print(f\"\\n✅ Done! Results: {output_file}\")\n",
+    "    print(f\"Total time: {elapsed_total/60:.1f} minutes\")\n",
+    "    print(f\"Average speed: {len(df)/(elapsed_total/3600):.1f} samples/hour\")\n",
+    "    \n",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")\n",
+    "    \n",
+    "    # Show VRAM stats\n",
+    "    if torch.cuda.is_available():\n",
+    "        print(f\"\\nFinal VRAM usage: {torch.cuda.max_memory_allocated() / 1024**3:.2f} GB\")\n",
+    "\n",
+    "except KeyboardInterrupt:\n",
+    "    print(f\"\\n⚠️ Interrupted by user\")\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))\n",
+    "    print(f\"Progress saved up to row {current_index}\")\n",
+    "\n",
+    "except Exception as e:\n",
+    "    print(f\"⚠️ Error: {e}\")\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))\n",
+    "    print(f\"Progress saved up to row {current_index}\")"
+   ]
   }
  ],
  "metadata": {
+  "kernelspec": {
+   "display_name": "pm-paper",
+   "language": "python",
+   "name": "pm-paper"
+  },
   "language_info": {
-   "name": "python"
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.13"
   }
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}

jupyter_notebooks/Section_2-3-4__Figure_8_Deepfake_victims.ipynb

@@ -26,9 +26,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 21,
    "id": "f177df11",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T10:32:27.999485Z",
+     "iopub.status.busy": "2025-11-21T10:32:27.999298Z",
+     "iopub.status.idle": "2025-11-21T10:32:28.001776Z",
+     "shell.execute_reply": "2025-11-21T10:32:28.001287Z",
+     "shell.execute_reply.started": "2025-11-21T10:32:27.999469Z"
+    }
+   },
    "outputs": [],
    "source": [
     "import pandas as pd\n",
@@ -41,40 +49,55 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 22,
    "id": "a2383f32",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T10:32:28.388500Z",
+     "iopub.status.busy": "2025-11-21T10:32:28.388376Z",
+     "iopub.status.idle": "2025-11-21T10:32:30.176781Z",
+     "shell.execute_reply": "2025-11-21T10:32:30.176262Z",
+     "shell.execute_reply.started": "2025-11-21T10:32:28.388488Z"
+    }
+   },
    "outputs": [],
    "source": [
     "current_dir = Path.cwd()\n",
-    "poi_models_dir = current_dir.parent / \"data/CSV/model_subsets/POI_models.csv\"  ### POI models dataset\n",
-    "output = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_01.csv\"  ### Output file"
+    "poi_models_dir = current_dir.parent / \"data/CSV/model_adapter/real_person_adapters.csv\"  ### POI models dataset\n",
+    "output = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_01_NER.csv\"  ### Output file"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 15,
    "id": "66fc691f",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/home/lauhp/anaconda3/envs/latm/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
-      "  from .autonotebook import tqdm as notebook_tqdm\n"
-     ]
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T10:23:43.615765Z",
+     "iopub.status.busy": "2025-11-21T10:23:43.615607Z",
+     "iopub.status.idle": "2025-11-21T10:23:47.279107Z",
+     "shell.execute_reply": "2025-11-21T10:23:47.278503Z",
+     "shell.execute_reply.started": "2025-11-21T10:23:43.615751Z"
     }
-   ],
+   },
+   "outputs": [],
    "source": [
     "nlp = spacy.load(\"en_core_web_sm\")  # or another model of your choice"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 16,
    "id": "8348c4a4",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T10:23:47.279657Z",
+     "iopub.status.busy": "2025-11-21T10:23:47.279527Z",
+     "iopub.status.idle": "2025-11-21T10:24:34.043631Z",
+     "shell.execute_reply": "2025-11-21T10:24:34.042977Z",
+     "shell.execute_reply.started": "2025-11-21T10:23:47.279644Z"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -174,20 +197,28 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 17,
    "id": "414954ed",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T10:27:41.992098Z",
+     "iopub.status.busy": "2025-11-21T10:27:41.991889Z",
+     "iopub.status.idle": "2025-11-21T10:27:49.779590Z",
+     "shell.execute_reply": "2025-11-21T10:27:49.778889Z",
+     "shell.execute_reply.started": "2025-11-21T10:27:41.992081Z"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "          real_name likely_country likely_nationality likely_profession\n",
-      "0     ronnie alonte                                               model\n",
-      "1  zh elena kamperi                                     twitch streamer\n",
-      "2     sofia vergara                                               model\n",
-      "3          安然 anran                                           celebrity\n",
-      "4       zoe kravitz                                           celebrity\n"
+      "      real_name likely_country likely_nationality likely_profession\n",
+      "0            iu                      South Korean         celebrity\n",
+      "1    super pose                                                    \n",
+      "2         liyuu                                                    \n",
+      "3         irene                      South Korean         celebrity\n",
+      "4  aespa karina                      South Korean         celebrity\n"
      ]
     }
    ],
@@ -199,8 +230,8 @@
     "current_dir = Path.cwd()\n",
     "countries = current_dir.parent / \"misc/lists/countries.csv\"\n",
     "professions = current_dir.parent / \"misc/lists/professions.csv\"\n",
-    "inputNER = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_01.csv\"\n",
-    "outfile = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_02.csv\"\n",
+    "inputNER = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_01_NER.csv\"\n",
+    "outfile = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
     "\n",
     "# Load datasets\n",
     "poi_df = pd.read_csv(inputNER)\n",
@@ -289,9 +320,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 18,
    "id": "054f230b",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T10:27:58.250798Z",
+     "iopub.status.busy": "2025-11-21T10:27:58.250588Z",
+     "iopub.status.idle": "2025-11-21T10:27:58.253063Z",
+     "shell.execute_reply": "2025-11-21T10:27:58.252570Z",
+     "shell.execute_reply.started": "2025-11-21T10:27:58.250780Z"
+    }
+   },
    "outputs": [],
    "source": [
     "#!pip install transformers torch\n",
@@ -309,9 +348,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 3,
    "id": "504b970f",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T10:02:17.101296Z",
+     "iopub.status.busy": "2025-11-21T10:02:17.101174Z",
+     "iopub.status.idle": "2025-11-21T10:02:17.103118Z",
+     "shell.execute_reply": "2025-11-21T10:02:17.102724Z",
+     "shell.execute_reply.started": "2025-11-21T10:02:17.101283Z"
+    }
+   },
    "outputs": [],
    "source": [
     "#!pip install openpyxl"
@@ -319,23 +366,38 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 19,
    "id": "7c209115",
-   "metadata": {},
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-21T10:29:28.176565Z",
+     "iopub.status.busy": "2025-11-21T10:29:28.176373Z",
+     "iopub.status.idle": "2025-11-21T10:30:01.404034Z",
+     "shell.execute_reply": "2025-11-21T10:30:01.403269Z",
+     "shell.execute_reply.started": "2025-11-21T10:29:28.176550Z"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Row 1/4...\n",
-      "Saved up to row 1\n",
-      "Row 2/4...\n",
-      "Saved up to row 2\n",
-      "Row 3/4...\n",
-      "Saved up to row 3\n",
-      "Row 4/4...\n",
-      "Saved up to row 4\n",
-      "All done! Files: /home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/data/CSV/Deepseek_annotated_POI.csv /home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/data/CSV/Deepseek_annotated_POI.xlsx\n"
+      "Row 1/4...\n"
+     ]
+    },
+    {
+     "ename": "ModuleNotFoundError",
+     "evalue": "No module named 'openpyxl'",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
+      "\u001b[31mModuleNotFoundError\u001b[39m                       Traceback (most recent call last)",
+      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[19]\u001b[39m\u001b[32m, line 112\u001b[39m\n\u001b[32m    110\u001b[39m out_df.to_csv(OUTPUT_CSV, index=\u001b[38;5;28;01mFalse\u001b[39;00m)\n\u001b[32m    111\u001b[39m \u001b[38;5;66;03m# Excel\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m112\u001b[39m \u001b[43mout_df\u001b[49m\u001b[43m.\u001b[49m\u001b[43mto_excel\u001b[49m\u001b[43m(\u001b[49m\u001b[43mOUTPUT_XLSX\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mindex\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[32m    113\u001b[39m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mopen\u001b[39m(INDEX_FILE, \u001b[33m'\u001b[39m\u001b[33mw\u001b[39m\u001b[33m'\u001b[39m) \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[32m    114\u001b[39m     f.write(\u001b[38;5;28mstr\u001b[39m(current_index))\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/pandas/util/_decorators.py:333\u001b[39m, in \u001b[36mdeprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper\u001b[39m\u001b[34m(*args, **kwargs)\u001b[39m\n\u001b[32m    327\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(args) > num_allow_args:\n\u001b[32m    328\u001b[39m     warnings.warn(\n\u001b[32m    329\u001b[39m         msg.format(arguments=_format_argument_list(allow_args)),\n\u001b[32m    330\u001b[39m         \u001b[38;5;167;01mFutureWarning\u001b[39;00m,\n\u001b[32m    331\u001b[39m         stacklevel=find_stack_level(),\n\u001b[32m    332\u001b[39m     )\n\u001b[32m--> \u001b[39m\u001b[32m333\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/pandas/core/generic.py:2439\u001b[39m, in \u001b[36mNDFrame.to_excel\u001b[39m\u001b[34m(self, excel_writer, sheet_name, na_rep, float_format, columns, header, index, index_label, startrow, startcol, engine, merge_cells, inf_rep, freeze_panes, storage_options, engine_kwargs)\u001b[39m\n\u001b[32m   2426\u001b[39m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01mpandas\u001b[39;00m\u001b[34;01m.\u001b[39;00m\u001b[34;01mio\u001b[39;00m\u001b[34;01m.\u001b[39;00m\u001b[34;01mformats\u001b[39;00m\u001b[34;01m.\u001b[39;00m\u001b[34;01mexcel\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m ExcelFormatter\n\u001b[32m   2428\u001b[39m formatter = ExcelFormatter(\n\u001b[32m   2429\u001b[39m     df,\n\u001b[32m   2430\u001b[39m     na_rep=na_rep,\n\u001b[32m   (...)\u001b[39m\u001b[32m   2437\u001b[39m     inf_rep=inf_rep,\n\u001b[32m   2438\u001b[39m )\n\u001b[32m-> \u001b[39m\u001b[32m2439\u001b[39m \u001b[43mformatter\u001b[49m\u001b[43m.\u001b[49m\u001b[43mwrite\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m   2440\u001b[39m \u001b[43m    \u001b[49m\u001b[43mexcel_writer\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2441\u001b[39m \u001b[43m    \u001b[49m\u001b[43msheet_name\u001b[49m\u001b[43m=\u001b[49m\u001b[43msheet_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2442\u001b[39m \u001b[43m    \u001b[49m\u001b[43mstartrow\u001b[49m\u001b[43m=\u001b[49m\u001b[43mstartrow\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2443\u001b[39m \u001b[43m    \u001b[49m\u001b[43mstartcol\u001b[49m\u001b[43m=\u001b[49m\u001b[43mstartcol\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2444\u001b[39m \u001b[43m    \u001b[49m\u001b[43mfreeze_panes\u001b[49m\u001b[43m=\u001b[49m\u001b[43mfreeze_panes\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2445\u001b[39m \u001b[43m    \u001b[49m\u001b[43mengine\u001b[49m\u001b[43m=\u001b[49m\u001b[43mengine\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2446\u001b[39m \u001b[43m    \u001b[49m\u001b[43mstorage_options\u001b[49m\u001b[43m=\u001b[49m\u001b[43mstorage_options\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2447\u001b[39m \u001b[43m    \u001b[49m\u001b[43mengine_kwargs\u001b[49m\u001b[43m=\u001b[49m\u001b[43mengine_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2448\u001b[39m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/pandas/io/formats/excel.py:943\u001b[39m, in \u001b[36mExcelFormatter.write\u001b[39m\u001b[34m(self, writer, sheet_name, startrow, startcol, freeze_panes, engine, storage_options, engine_kwargs)\u001b[39m\n\u001b[32m    941\u001b[39m     need_save = \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[32m    942\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m--> \u001b[39m\u001b[32m943\u001b[39m     writer = \u001b[43mExcelWriter\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m    944\u001b[39m \u001b[43m        \u001b[49m\u001b[43mwriter\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    945\u001b[39m \u001b[43m        \u001b[49m\u001b[43mengine\u001b[49m\u001b[43m=\u001b[49m\u001b[43mengine\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    946\u001b[39m \u001b[43m        \u001b[49m\u001b[43mstorage_options\u001b[49m\u001b[43m=\u001b[49m\u001b[43mstorage_options\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    947\u001b[39m \u001b[43m        \u001b[49m\u001b[43mengine_kwargs\u001b[49m\u001b[43m=\u001b[49m\u001b[43mengine_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    948\u001b[39m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    949\u001b[39m     need_save = \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[32m    951\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/pandas/io/excel/_openpyxl.py:57\u001b[39m, in \u001b[36mOpenpyxlWriter.__init__\u001b[39m\u001b[34m(self, path, engine, date_format, datetime_format, mode, storage_options, if_sheet_exists, engine_kwargs, **kwargs)\u001b[39m\n\u001b[32m     44\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34m__init__\u001b[39m(\n\u001b[32m     45\u001b[39m     \u001b[38;5;28mself\u001b[39m,\n\u001b[32m     46\u001b[39m     path: FilePath | WriteExcelBuffer | ExcelWriter,\n\u001b[32m   (...)\u001b[39m\u001b[32m     55\u001b[39m ) -> \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m     56\u001b[39m     \u001b[38;5;66;03m# Use the openpyxl module as the Excel writer.\u001b[39;00m\n\u001b[32m---> \u001b[39m\u001b[32m57\u001b[39m     \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01mopenpyxl\u001b[39;00m\u001b[34;01m.\u001b[39;00m\u001b[34;01mworkbook\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m Workbook\n\u001b[32m     59\u001b[39m     engine_kwargs = combine_kwargs(engine_kwargs, kwargs)\n\u001b[32m     61\u001b[39m     \u001b[38;5;28msuper\u001b[39m().\u001b[34m__init__\u001b[39m(\n\u001b[32m     62\u001b[39m         path,\n\u001b[32m     63\u001b[39m         mode=mode,\n\u001b[32m   (...)\u001b[39m\u001b[32m     66\u001b[39m         engine_kwargs=engine_kwargs,\n\u001b[32m     67\u001b[39m     )\n",
+      "\u001b[31mModuleNotFoundError\u001b[39m: No module named 'openpyxl'"
      ]
     }
    ],
@@ -349,13 +411,13 @@
     "\n",
     "# === PATHS & CONFIG ===\n",
     "current_dir = Path.cwd()\n",
-    "inputCSV = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_02.csv\"\n",
+    "inputCSV = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
     "api_key_file = current_dir.parent / \"misc/credentials/deepseek_api_key.txt\" #store your API key under misc/credentials/deepseek_api_key.txt\n",
     "\n",
     "# Output both CSV and Excel for compatibility\n",
-    "OUTPUT_CSV = current_dir.parent / \"data/CSV/Deepseek_annotated_POI.csv\"\n",
-    "OUTPUT_XLSX = current_dir.parent / \"data/CSV/Deepseek_annotated_POI.xlsx\"\n",
-    "INDEX_FILE = current_dir.parent / \"misc/deepseek_query_index.txt\"\n",
+    "OUTPUT_CSV = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_03_deepseek.csv\"\n",
+    "OUTPUT_XLSX = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_03_deepseek.xlsx\"\n",
+    "INDEX_FILE = current_dir.parent / \"misc/query-indicies/deepseek_query_index.txt\"\n",
     "SAVE_INTERVAL = 1  # Save every N rows\n",
     "START_ROW = 1      # Row index to start from (0-based)\n",
     "END_ROW = 5        # Row index to end (exclusive)\n",
@@ -460,12 +522,6 @@
     "print(\"All done! Files:\", OUTPUT_CSV, OUTPUT_XLSX)\n"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "7910e574",
-   "metadata": {},
-   "source": []
-  },
   {
    "cell_type": "markdown",
    "id": "9d377005",
@@ -646,9 +702,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "latm",
+   "display_name": "pm-paper",
    "language": "python",
-   "name": "python3"
+   "name": "pm-paper"
   },
   "language_info": {
    "codemirror_mode": {
@@ -660,7 +716,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.15"
+   "version": "3.11.13"
   }
  },
  "nbformat": 4,

jupyter_notebooks/Section_2-3-4__Figure_8b_sunburst_profession.ipynb

@@ -101,7 +101,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "latm",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -115,9 +115,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.15"
+   "version": "3.12.10"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

misc/deleteme.txt

@@ -0,0 +1 @@
+10

misc/query_indicies/deepseek_query_index.txt

@@ -0,0 +1 @@
+10