added llm annotation code

jupyter_notebooks/DEEPFAKE_PIPELINE_GUIDE.md

@@ -0,0 +1,210 @@
+# Deepfake Adapter Dataset Processing - Quick Start Guide
+
+## Overview
+
+This pipeline processes the `real_person_adapters.csv` dataset to identify and annotate real people used in deepfake LoRA models using three LLM options: **Qwen**, **Llama**, and **Mistral**.
+
+## Quick Start
+
+### 1. Prerequisites
+
+```bash
+# Install required packages
+pip install pandas numpy emoji requests tqdm spacy
+
+# Download spaCy English model (for NER)
+python -m spacy download en_core_web_sm
+```
+
+**Note**: The spaCy model will be automatically downloaded when you run the notebook if not already installed.
+
+### 2. Set Up API Keys
+
+Choose at least ONE LLM provider and get an API key:
+
+| Provider | Model | Sign Up Link | Est. Cost (10k entries) |
+|----------|-------|--------------|-------------------------|
+| **Qwen** | Qwen-Max | https://dashscope.aliyun.com/ | Varies |
+| **Llama** | Llama-3.1-70B | https://www.together.ai/ | ~$5-10 |
+| **Mistral** | Mistral Large | https://mistral.ai/ | ~$40-80 |
+
+Create your API key file in `misc/credentials/`:
+
+```bash
+# For Qwen
+echo "your-api-key-here" > misc/credentials/qwen_api_key.txt
+
+# For Llama (via Together AI)
+echo "your-api-key-here" > misc/credentials/together_api_key.txt
+
+# For Mistral
+echo "your-api-key-here" > misc/credentials/mistral_api_key.txt
+```
+
+### 3. Run the Notebook
+
+Open `Section_2-3-4_Figure_8_deepfake_adapters.ipynb` and:
+
+1. **Run all cells sequentially** from top to bottom
+2. The default configuration uses Qwen in test mode (10 samples)
+3. Review the test results
+4. To process the full dataset, change in the LLM annotation cell:
+   ```python
+   TEST_MODE = False
+   ```
+
+## Pipeline Stages
+
+### Stage 1: NER & Name Cleaning
+- **Input**: `data/CSV/real_person_adapters.csv`
+- **Output**: `data/CSV/NER_POI_step01_pre_annotation.csv`
+- **Function**: Cleans adapter names to extract real person names
+  - Removes: emoji, "lora", "v1", special characters
+  - Example: "IU LoRA v2 🎤" → "IU"
+
+### Stage 2: Country/Nationality Mapping
+- **Input**: Step 1 output + `misc/lists/countries.csv`
+- **Output**: `data/CSV/NER_POI_step02_annotated.csv`
+- **Function**: Maps tags to standardized countries
+  - Example: "korean" → "South Korea"
+  - Excludes uninhabited territories
+
+### Stage 3: LLM Profession Annotation
+- **Input**: Step 2 output + `misc/lists/professions.csv`
+- **Output**: `data/CSV/{llm}_annotated_POI_test.csv` (test) or `{llm}_annotated_POI.csv` (full)
+- **Function**: Uses LLM to identify:
+  - Full name
+  - Gender
+  - Up to 3 professions (from profession list)
+  - Country
+- **Progress**: Automatically saves every 10 rows
+- **Resumable**: Can continue from last saved progress if interrupted
+
+## Configuration Options
+
+In the LLM annotation cell, you can configure:
+
+```python
+# Choose LLM provider
+SELECTED_LLM = 'qwen'  # Options: 'qwen', 'llama', 'mistral'
+
+# Test mode (recommended for first run)
+TEST_MODE = True       # True = test on small sample
+TEST_SIZE = 10         # Number of rows for testing
+
+# Processing limits
+MAX_ROWS = 20000      # Maximum rows to process (None = all)
+SAVE_INTERVAL = 10    # Save progress every N rows
+```
+
+## Expected Output Format
+
+The final dataset will include all original columns plus:
+
+| Column | Description | Example |
+|--------|-------------|---------|
+| `real_name` | Cleaned name | "IU" |
+| `full_name` | Full name from LLM | "Lee Ji-eun (IU)" |
+| `gender` | Gender from LLM | "Female" |
+| `profession_llm` | Up to 3 professions | "singer, actor, celebrity" |
+| `country` | Country from LLM | "South Korea" |
+| `likely_country` | Country from tags | "South Korea" |
+| `likely_nationality` | Nationality from tags | "South Korean" |
+| `tags` | Combined tags | "['korean', 'celebrity', 'singer']" |
+
+## Troubleshooting
+
+### API Key Errors
+```
+Warning: No API key for qwen
+```
+**Solution**: Ensure your API key file exists and contains only the key (no extra whitespace)
+
+### Rate Limiting
+```
+Qwen API error (attempt 1/3): 429 Too Many Requests
+```
+**Solution**: The code automatically retries with exponential backoff. You can also:
+- Increase `time.sleep(0.5)` to a higher value
+- Process in smaller batches
+
+### Progress Lost
+**Solution**: The pipeline saves progress automatically. Check:
+- `data/CSV/{llm}_annotated_POI_test.csv` - your partial results
+- `misc/{llm}_query_index.txt` - last processed index
+- Just re-run the cell and it will resume from the last saved progress
+
+### JSON Parse Errors from LLM
+```
+Qwen API error: JSONDecodeError
+```
+**Solution**: This is usually temporary. The code:
+- Returns "Unknown" for failed queries
+- Continues processing
+- You can manually review/reprocess failed entries later
+
+## Cost Management
+
+### Estimate Costs Before Processing
+
+For a dataset with N entries:
+- **Qwen**: Contact Alibaba Cloud for pricing
+- **Llama**: ~N × $0.0005 = ~$5 per 10k entries
+- **Mistral**: ~N × $0.004 = ~$40 per 10k entries
+
+### Best Practices
+
+1. **Always test first**: Run with `TEST_MODE = True` on 10 samples
+2. **Monitor API usage**: Check your API provider's dashboard
+3. **Use cheaper models first**: Try Llama before Mistral
+4. **Process in batches**: Set `MAX_ROWS` to process incrementally
+5. **Save intermediate results**: The automatic saving feature helps prevent data loss
+
+## Comparing Multiple LLMs
+
+To compare results from different LLMs:
+
+1. Run the pipeline with `SELECTED_LLM = 'qwen'`
+2. Change to `SELECTED_LLM = 'llama'` and run again
+3. Change to `SELECTED_LLM = 'mistral'` and run again
+4. Compare the three output files:
+   - `qwen_annotated_POI.csv`
+   - `llama_annotated_POI.csv`
+   - `mistral_annotated_POI.csv`
+
+## Files Created
+
+The pipeline creates these files:
+
+```
+data/CSV/
+├── NER_POI_step01_pre_annotation.csv     # After name cleaning
+├── NER_POI_step02_annotated.csv          # After country mapping
+├── qwen_annotated_POI_test.csv           # Test results (Qwen)
+├── qwen_annotated_POI.csv                # Full results (Qwen)
+├── llama_annotated_POI.csv               # Full results (Llama)
+└── mistral_annotated_POI.csv             # Full results (Mistral)
+
+misc/
+├── qwen_query_index.txt                  # Progress tracking
+├── llama_query_index.txt                 # Progress tracking
+└── mistral_query_index.txt               # Progress tracking
+```
+
+## Support
+
+For issues or questions:
+1. Check this guide for common problems
+2. Review `misc/credentials/README.md` for API setup
+3. Read the notebook documentation (first cell)
+4. Check API provider documentation for service-specific issues
+
+## Ethical Considerations
+
+This research documents ethical problems with AI deepfake models. The dataset and analysis help:
+- Understand the scope of unauthorized person likeness usage
+- Document professions/demographics most affected
+- Inform policy and technical solutions
+- Raise awareness about deepfake technology misuse
+
+Use this data responsibly and respect individual privacy and consent.

jupyter_notebooks/LLM_MODELS_COMPARISON.md

@@ -0,0 +1,326 @@
+# LLM Models for Deepfake Annotation
+
+## Overview
+
+The pipeline now includes **6 LLM options** in individual cells for easy comparison:
+
+1. **Deepseek** - Testing (use first!)
+2. **Qwen (API)** - Chinese (Alibaba Cloud)
+3. **Llama** - American (Meta)
+4. **Mixtral** - French (Mistral AI)
+5. **Gemma** - American Open Source (Google)
+6. **Qwen-2.5-32B Local** - FREE local inference (NEW!)
+
+## The 6 LLMs
+
+### 1. Deepseek (Testing)
+**Cell 10**
+
+- **Model**: deepseek-chat
+- **Provider**: DeepSeek
+- **API**: https://platform.deepseek.com/
+- **Cost**: ~$0.14-0.28 per 1M tokens (~$1-2 for 10k entries)
+- **Use case**: **Test this first!** Cheapest option to verify pipeline works
+- **API Key**: `misc/credentials/deepseek_api_key.txt`
+
+---
+
+### 2. Qwen API (Chinese)
+**Cells 11-12**
+
+- **Model**: qwen-max (automatically uses Qwen3-Max)
+- **Provider**: Alibaba Cloud DashScope
+- **API**: https://dashscope.aliyun.com/
+- **Cost**: Variable (check Alibaba pricing)
+- **Use case**: Chinese company, strong multilingual support
+- **API Key**: `misc/credentials/qwen_api_key.txt`
+- **Note**: Uses latest Qwen3-Max when you specify `qwen-max`
+
+---
+
+### 6. Qwen-2.5-32B Local (FREE!)
+**Cells 19-20** (NEW!)
+
+- **Model**: qwen2.5:32b-instruct
+- **Provider**: Ollama (local inference)
+- **Setup**: https://ollama.com/
+- **Cost**: **$0** (FREE - no API costs!)
+- **Requirements**:
+  - A100 80GB GPU (or similar)
+  - ~25GB VRAM during inference
+  - ~20GB storage for model download
+  - Ollama installed
+- **Speed**: 5-10 tokens/sec on A100 (~100-200 samples/hour)
+- **Use case**:
+  - ✅ Large datasets (>1000 samples) where cost matters
+  - ✅ Privacy-sensitive research data
+  - ✅ Offline processing
+  - ✅ Strong multilingual support
+- **Setup guide**: See `QWEN_LOCAL_SETUP.md`
+
+---
+
+### 3. Llama (American)
+**Cells 13-14**
+
+- **Model**: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
+- **Provider**: Together AI (hosting Meta's model)
+- **Developer**: Meta (American)
+- **API**: https://www.together.ai/
+- **Cost**: ~$0.90 per 1M tokens (~$5-10 for 10k entries)
+- **Use case**: Open-source American model, good quality
+- **API Key**: `misc/credentials/together_api_key.txt`
+
+---
+
+### 4. Mixtral (French)
+**Cells 15-16**
+
+- **Model**: open-mixtral-8x22b
+- **Provider**: Mistral AI
+- **Developer**: Mistral AI (French)
+- **API**: https://mistral.ai/
+- **Cost**: ~$2 per 1M tokens (~$10-20 for 10k entries)
+- **Use case**: European alternative, Mixture-of-Experts architecture
+- **API Key**: `misc/credentials/mistral_api_key.txt`
+- **Note**: Using open-mixtral-8x22b (cheaper than mistral-large)
+
+---
+
+### 5. Gemma (American Open Source)
+**Cells 17-18**
+
+- **Model**: google/gemma-2-27b-it
+- **Provider**: Together AI (hosting Google's model)
+- **Developer**: Google (American)
+- **API**: https://www.together.ai/ (same as Llama)
+- **Cost**: ~$0.80 per 1M tokens (~$4-8 for 10k entries)
+- **Use case**: American open-source alternative, competitive quality
+- **API Key**: `misc/credentials/together_api_key.txt` (same as Llama)
+- **Note**: Fully open-source, can be self-hosted
+
+---
+
+## Cost Comparison (10,000 entries)
+
+| Model | Provider | Cost | Time | Origin |
+|-------|----------|------|------|--------|
+| **Qwen-2.5-32B Local** | Ollama (local) | **$0** | ~50-100 hrs | 🇨🇳 Chinese |
+| **Deepseek** | DeepSeek | ~$1-2 | ~5-10 hrs | 🇨🇳 Chinese |
+| **Gemma 2** | Together AI | ~$4-8 | ~5-10 hrs | 🇺🇸 American (open) |
+| **Llama 3.1** | Together AI | ~$5-10 | ~5-10 hrs | 🇺🇸 American (open) |
+| **Mixtral** | Mistral AI | ~$10-20 | ~5-10 hrs | 🇫🇷 French (open) |
+| **Qwen API** | Alibaba | Variable | ~5-10 hrs | 🇨🇳 Chinese |
+
+**Note**: Local inference is FREE but slower. Good for large datasets where cost matters more than time.
+
+## Recommended Testing Order
+
+### 1. Start with Deepseek
+```python
+# Cell 10
+TEST_MODE = True
+TEST_SIZE = 10
+```
+- **Why**: Cheapest, verify pipeline works
+- **Cost**: Pennies for 10 samples
+
+### 2. Compare on Small Sample
+Pick 2-3 models and run on same 100 samples:
+```python
+# In each cell:
+TEST_MODE = True
+TEST_SIZE = 100
+```
+
+**Good combinations:**
+- Budget: Deepseek + Gemma
+- Quality: Llama + Mixtral
+- Geographic: Qwen + Llama + Mixtral
+
+### 3. Production Run
+Choose best model from testing and run full dataset:
+```python
+TEST_MODE = False
+MAX_ROWS = None  # or 20000
+```
+
+## API Key Setup
+
+### For Deepseek & Qwen (separate keys):
+```bash
+echo "your-deepseek-key" > misc/credentials/deepseek_api_key.txt
+echo "your-qwen-key" > misc/credentials/qwen_api_key.txt
+```
+
+### For Llama & Gemma (same Together AI key):
+```bash
+echo "your-together-key" > misc/credentials/together_api_key.txt
+```
+Both Llama and Gemma use the same Together AI key!
+
+### For Mixtral:
+```bash
+echo "your-mistral-key" > misc/credentials/mistral_api_key.txt
+```
+
+## Output Files
+
+Each LLM saves to a separate file:
+
+```
+data/CSV/
+├── deepseek_annotated_POI_test.csv       # Deepseek test
+├── deepseek_annotated_POI.csv            # Deepseek full
+├── qwen_annotated_POI_test.csv           # Qwen API test
+├── qwen_annotated_POI.csv                # Qwen API full
+├── qwen_local_annotated_POI_test.csv     # Qwen Local test (NEW!)
+├── qwen_local_annotated_POI.csv          # Qwen Local full (NEW!)
+├── llama_annotated_POI_test.csv          # Llama test
+├── llama_annotated_POI.csv               # Llama full
+├── mixtral_annotated_POI_test.csv        # Mixtral test
+├── mixtral_annotated_POI.csv             # Mixtral full
+├── gemma_annotated_POI_test.csv          # Gemma test
+└── gemma_annotated_POI.csv               # Gemma full
+```
+
+## Comparing Results
+
+After running multiple LLMs, compare results:
+
+```python
+import pandas as pd
+
+# Load results from different models
+deepseek_df = pd.read_csv('data/CSV/deepseek_annotated_POI_test.csv')
+qwen_df = pd.read_csv('data/CSV/qwen_annotated_POI_test.csv')
+qwen_local_df = pd.read_csv('data/CSV/qwen_local_annotated_POI_test.csv')  # NEW!
+llama_df = pd.read_csv('data/CSV/llama_annotated_POI_test.csv')
+mixtral_df = pd.read_csv('data/CSV/mixtral_annotated_POI_test.csv')
+gemma_df = pd.read_csv('data/CSV/gemma_annotated_POI_test.csv')
+
+# Compare profession distributions
+print("Deepseek professions:", deepseek_df['profession_llm'].value_counts().head())
+print("Qwen API professions:", qwen_df['profession_llm'].value_counts().head())
+print("Qwen Local professions:", qwen_local_df['profession_llm'].value_counts().head())  # NEW!
+print("Llama professions:", llama_df['profession_llm'].value_counts().head())
+print("Mixtral professions:", mixtral_df['profession_llm'].value_counts().head())
+print("Gemma professions:", gemma_df['profession_llm'].value_counts().head())
+
+# Compare specific cases
+print("\nIrene identification:")
+print("Deepseek:", deepseek_df[deepseek_df['real_name'] == 'Irene']['full_name'].values)
+print("Qwen API:", qwen_df[qwen_df['real_name'] == 'Irene']['full_name'].values)
+print("Qwen Local:", qwen_local_df[qwen_local_df['real_name'] == 'Irene']['full_name'].values)
+print("Llama:", llama_df[llama_df['real_name'] == 'Irene']['full_name'].values)
+print("Mixtral:", mixtral_df[mixtral_df['real_name'] == 'Irene']['full_name'].values)
+print("Gemma:", gemma_df[gemma_df['real_name'] == 'Irene']['full_name'].values)
+```
+
+## Model Characteristics
+
+### Deepseek
+- ✅ Very cheap
+- ✅ Good for testing
+- ⚠️ Less documentation
+- 🇨🇳 Chinese company
+
+### Qwen (Qwen3-Max)
+- ✅ Latest version automatically used
+- ✅ Strong multilingual
+- ✅ Good Asian name recognition
+- 💰 Variable cost
+- 🇨🇳 Chinese company (Alibaba)
+
+### Llama 3.1 70B
+- ✅ Open-source
+- ✅ Strong overall performance
+- ✅ Well-documented
+- ✅ American (Meta)
+- 💰 Mid-range cost
+
+### Mixtral 8x22B
+- ✅ Open-source
+- ✅ MoE architecture (efficient)
+- ✅ European alternative
+- 💰 Mid-range cost
+- 🇫🇷 French company
+
+### Gemma 2 27B
+- ✅ Fully open-source
+- ✅ Can self-host
+- ✅ American (Google)
+- ✅ Cheap via API
+- ✅ Good quality for size
+
+### Qwen-2.5-32B Local (NEW!)
+- ✅ **FREE** - $0 cost (no API fees)
+- ✅ **FAST** - Local inference on A100 (5-10 tokens/sec)
+- ✅ **PRIVATE** - Data never leaves your machine
+- ✅ **OFFLINE** - Works without internet
+- ✅ **HIGH QUALITY** - 32B parameter model
+- ✅ Strong multilingual support
+- ⚠️ Requires: A100 80GB GPU, ~25GB VRAM, Ollama installed
+- 🇨🇳 Chinese company (Alibaba)
+- 📦 Model size: ~20GB download
+
+## Decision Matrix
+
+### If you prioritize...
+
+**FREE / Zero Cost**: Use **Qwen-2.5-32B Local** (no API fees!)
+
+**Cost** (with API): Use **Deepseek** or **Gemma**
+
+**Quality**: Use **Qwen-2.5-32B Local**, **Llama**, or **Mixtral**
+
+**Privacy**: Use **Qwen-2.5-32B Local** (data stays on your machine)
+
+**American/Open Source**: Use **Gemma** or **Llama**
+
+**Asian Names**: Use **Qwen** (API or Local - strong multilingual)
+
+**European Provider**: Use **Mixtral**
+
+**Testing**: Use **Deepseek** first, always!
+
+## Running Multiple Models
+
+You can run all 6 models in sequence:
+
+```python
+# 1. Run Cell 10 (Deepseek) - verify works (~$1-2 for 10k)
+# 2. Run Cell 12 (Qwen API) - Chinese perspective (~variable cost)
+# 3. Run Cell 14 (Llama) - American perspective (~$5-10 for 10k)
+# 4. Run Cell 16 (Mixtral) - European perspective (~$10-20 for 10k)
+# 5. Run Cell 18 (Gemma) - Open source perspective (~$4-8 for 10k)
+# 6. Run Cell 20 (Qwen-2.5-32B Local) - FREE local inference ($0!)
+```
+
+Each saves to its own file, so you can compare results!
+
+## Notes
+
+- **Llama and Gemma use the same API key** (Together AI)
+- All models use the **same 9 profession categories**
+- All models have **automatic retries** with exponential backoff
+- All models **save progress** every 10 rows
+- All models are **resumable** if interrupted
+
+## Summary
+
+You now have **6 LLM options** to choose from:
+
+1. 🧪 **Deepseek** - Test first (cheapest API)
+2. ����🇳 **Qwen3-Max API** - Chinese, strong multilingual
+3. 🇺🇸 **Llama 3.1 70B** - American, open-source
+4. 🇫🇷 **Mixtral 8x22B** - French, open-source MoE
+5. 🇺🇸 **Gemma 2 27B** - American open-source (Google)
+6. 💰 **Qwen-2.5-32B Local** - FREE local inference (NEW!)
+
+Each in its own cell, easy to run and compare! 🎉
+
+**Recommended workflow**:
+1. Test with Deepseek (Cell 10) - verify pipeline works
+2. For small datasets (<1000): Use API (Deepseek/Gemma/Llama)
+3. For large datasets (>1000): Use Qwen-2.5-32B Local (Cell 20) - FREE!

jupyter_notebooks/QUICK_START_LOCAL.md

@@ -0,0 +1,171 @@
+# Quick Start: Running Qwen-2.5-32B Locally
+
+This is a quick guide to get you started with FREE local LLM inference using your A100 GPU.
+
+## Why Local?
+
+✅ **$0 cost** - No API fees
+✅ **Privacy** - Data stays on your machine
+✅ **Quality** - 32B parameter model with strong performance
+
+## Setup (One-time)
+
+### 1. Pull the Model (~10-30 minutes)
+
+```bash
+# Pull Qwen-2.5-32B-Instruct
+ollama pull qwen2.5:32b-instruct
+
+# Wait for download to complete (~20GB)
+# Model will be cached at: ~/.ollama/models/
+```
+
+### 2. Verify Model is Ready
+
+```bash
+# List installed models
+ollama list
+
+# Should show: qwen2.5:32b-instruct
+
+# Test it
+ollama run qwen2.5:32b-instruct "Hello, who are you?"
+```
+
+If you see a response, you're ready! ✅
+
+## Running the Notebook
+
+### Open the Notebook
+
+```bash
+cd jupyter_notebooks
+jupyter notebook Section_2-3-4_Figure_8_deepfake_adapters.ipynb
+```
+
+### Run the Cells
+
+1. **Cell 5**: NER & Name Cleaning (processes names)
+2. **Cell 7**: Country/Nationality Mapping
+3. **Cell 20**: Qwen-2.5-32B Local Annotation 👈 **This is the new one!**
+
+### Configure Cell 20
+
+```python
+# Start with test mode
+TEST_MODE = True
+TEST_SIZE = 10
+
+# Then run full dataset
+TEST_MODE = False
+MAX_ROWS = 20000  # or None for all
+```
+
+### Run Cell 20
+
+Just click "Run" or press Shift+Enter. The cell will:
+1. Check if Ollama is installed ✅
+2. Check if model is available ✅
+3. Start annotating
+4. Save progress every 10 rows
+5. Show completion stats
+
+### Monitor Progress
+
+```
+Qwen Local: 100%|██████████| 10/10 [02:30<00:00, 15.0s/it]
+✅ Saved after 10 rows (~24.0 samples/hour)
+
+✅ Done! Results: data/CSV/qwen_local_annotated_POI_test.csv
+Total time: 2.5 minutes
+Average speed: 240.0 samples/hour
+```
+
+## Performance
+
+On your A100 80GB:
+- **Speed**: ~5-10 tokens/second
+- **Throughput**: ~100-200 samples/hour
+- **Memory**: ~22-25GB VRAM
+- **Cost**: $0
+
+### Time Estimates
+
+| Dataset Size | Time |
+|-------------|------|
+| 10 samples (test) | ~2-3 minutes |
+| 100 samples | ~20-30 minutes |
+| 1,000 samples | ~5-10 hours |
+| 10,000 samples | ~50-100 hours |
+
+**Tip**: Run overnight or over the weekend for large datasets!
+
+## Troubleshooting
+
+### "Model not found"
+
+```bash
+ollama pull qwen2.5:32b-instruct
+```
+
+### "Ollama not running"
+
+```bash
+ollama serve
+```
+
+### Out of Memory
+
+Your A100 has 80GB VRAM - this should NOT happen with the 32B model (~25GB VRAM).
+
+If it does, try the quantized version:
+```bash
+ollama pull qwen2.5:32b-instruct-q4_0  # Only ~12GB VRAM
+```
+
+## Output
+
+Results saved to:
+- Test: `data/CSV/qwen_local_annotated_POI_test.csv`
+- Full: `data/CSV/qwen_local_annotated_POI.csv`
+
+Same format as API results - easy to compare!
+
+## Custom Model Cache Location
+
+To store models in `data/models/`:
+
+```bash
+export OLLAMA_MODELS="/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/models"
+ollama pull qwen2.5:32b-instruct
+```
+
+## Comparing API vs Local
+
+After running both:
+
+```python
+import pandas as pd
+
+qwen_api = pd.read_csv('data/CSV/qwen_annotated_POI_test.csv')
+qwen_local = pd.read_csv('data/CSV/qwen_local_annotated_POI_test.csv')
+
+# Check agreement
+agreement = (qwen_api['profession_llm'] == qwen_local['profession_llm']).mean()
+print(f"Agreement: {agreement*100:.1f}%")
+```
+
+## Full Documentation
+
+For more details, see:
+- `QWEN_LOCAL_SETUP.md` - Complete setup guide
+- `LLM_MODELS_COMPARISON.md` - All 6 LLM options compared
+
+## Summary
+
+✅ Ollama already installed
+✅ A100 80GB GPU - perfect for Qwen-2.5-32B
+✅ FREE inference - no API costs
+✅ Privacy - data stays local
+
+**Next step**: Run Cell 20 in the notebook! 🚀

jupyter_notebooks/QWEN_LOCAL_SETUP.md

@@ -0,0 +1,321 @@
+# Running Qwen-2.5-32B Locally with Ollama
+
+This guide explains how to run Qwen-2.5-32B-Instruct locally on your A100 GPU using Ollama.
+
+## Why Run Locally?
+
+✅ **FREE** - No API costs ($0 per query)
+✅ **FAST** - Local inference on A100 (5-10 tokens/sec)
+✅ **PRIVATE** - Data never leaves your machine
+✅ **OFFLINE** - Works without internet (after model download)
+✅ **HIGH QUALITY** - 32B parameter model with strong multilingual support
+
+## System Requirements
+
+### Minimum Specs
+- **GPU**: NVIDIA A100 80GB (or similar high-end GPU)
+- **VRAM**: 22-25GB during inference
+- **RAM**: 32GB system RAM (you have 265GB - more than enough!)
+- **Storage**: ~20GB for model download
+- **OS**: Linux (you're on Ubuntu)
+
+### Your Setup
+✅ NVIDIA A100 80GB - Perfect for Qwen-2.5-32B
+✅ 265GB RAM - Excellent
+✅ Linux (Ubuntu) - Supported
+✅ Ollama already installed at `/usr/local/bin/ollama`
+
+## Installation Steps
+
+### 1. Verify Ollama Installation
+
+```bash
+# Check if Ollama is installed
+which ollama
+# Should output: /usr/local/bin/ollama
+
+# Check Ollama version
+ollama --version
+```
+
+If not installed, install with:
+```bash
+curl -fsSL https://ollama.com/install.sh | sh
+```
+
+### 2. Pull Qwen-2.5-32B-Instruct Model
+
+```bash
+# This will download ~20GB
+ollama pull qwen2.5:32b-instruct
+
+# Alternative: Use the base model (not instruct-tuned)
+# ollama pull qwen2.5:32b
+```
+
+**Download time**: ~10-30 minutes depending on your internet speed.
+
+**Model cache location**: By default, models are cached at:
+- Linux: `~/.ollama/models/`
+
+To use custom cache location (e.g., `data/models/`):
+```bash
+export OLLAMA_MODELS="/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/models"
+ollama pull qwen2.5:32b-instruct
+```
+
+### 3. Verify Model is Ready
+
+```bash
+# List all installed models
+ollama list
+
+# Test the model
+ollama run qwen2.5:32b-instruct "Hello, who are you?"
+```
+
+You should see a response from Qwen!
+
+### 4. Start Ollama Server (if needed)
+
+Ollama runs as a background service by default. If you need to start it manually:
+
+```bash
+# Start Ollama server
+ollama serve
+
+# Or run in background
+nohup ollama serve > /dev/null 2>&1 &
+```
+
+## Using Qwen-2.5-32B in the Notebook
+
+### Cell 20: Qwen-2.5-32B Local Annotation
+
+The notebook cell handles everything automatically:
+
+1. **Checks Ollama installation**
+2. **Verifies model availability**
+3. **Runs inference locally**
+4. **Saves progress every 10 rows**
+
+### Configuration
+
+```python
+# In Cell 20
+TEST_MODE = True        # Start with small test
+TEST_SIZE = 10          # Test on 10 samples first
+MAX_ROWS = 20000        # Full dataset size
+SAVE_INTERVAL = 10      # Save every 10 rows
+
+MODEL_NAME = "qwen2.5:32b-instruct"  # Model to use
+OLLAMA_HOST = "http://localhost:11434"  # Default Ollama port
+```
+
+### Running the Pipeline
+
+1. **Test run first** (recommended):
+   ```python
+   TEST_MODE = True
+   TEST_SIZE = 10
+   ```
+   Run Cell 20 to test on 10 samples (~1-2 minutes)
+
+2. **Check results**:
+   ```python
+   # Output saved to:
+   data/CSV/qwen_local_annotated_POI_test.csv
+   ```
+
+3. **Full run**:
+   ```python
+   TEST_MODE = False
+   MAX_ROWS = 20000  # or None for all rows
+   ```
+   Run Cell 20 for full dataset (~2-3 hours for 10k samples on A100)
+
+### Performance Expectations
+
+On NVIDIA A100 80GB:
+- **Speed**: 5-10 tokens/second
+- **Throughput**: 100-200 samples/hour (depends on prompt length)
+- **Memory**: ~22-25GB VRAM during inference
+- **Time for 10k samples**: ~50-100 hours (can run overnight/over weekend)
+
+### Monitoring
+
+The cell shows progress updates:
+```
+Qwen Local: 100%|██████████| 10/10 [02:30<00:00, 15.0s/it]
+✅ Saved after 10 rows (~24.0 samples/hour)
+
+✅ Done! Results: data/CSV/qwen_local_annotated_POI_test.csv
+Total time: 2.5 minutes
+Average speed: 240.0 samples/hour
+```
+
+## Troubleshooting
+
+### Model Not Found
+
+```bash
+# Check if model is installed
+ollama list
+
+# If not listed, pull it
+ollama pull qwen2.5:32b-instruct
+```
+
+### Ollama Server Not Running
+
+```bash
+# Check if Ollama is running
+ps aux | grep ollama
+
+# If not running, start it
+ollama serve
+```
+
+### GPU Not Detected
+
+```bash
+# Check NVIDIA GPU
+nvidia-smi
+
+# Check CUDA
+nvcc --version
+
+# Ollama should automatically detect GPU
+# If not, check Ollama logs
+journalctl -u ollama
+```
+
+### Out of Memory (OOM)
+
+If you get OOM errors:
+
+1. **Check VRAM usage**:
+   ```bash
+   watch -n 1 nvidia-smi
+   ```
+
+2. **Try smaller batch size** (not applicable here - we process 1 at a time)
+
+3. **Try quantized version** (smaller model):
+   ```bash
+   # 4-bit quantized version (~12GB VRAM)
+   ollama pull qwen2.5:32b-instruct-q4_0
+
+   # Update MODEL_NAME in notebook
+   MODEL_NAME = "qwen2.5:32b-instruct-q4_0"
+   ```
+
+### Slow Inference
+
+If inference is very slow (<1 token/sec):
+
+1. **Check GPU utilization**:
+   ```bash
+   nvidia-smi
+   ```
+   GPU should show ~90%+ utilization during inference
+
+2. **Check CPU vs GPU**:
+   Ollama might be using CPU instead of GPU
+   ```bash
+   # Force GPU usage
+   OLLAMA_GPU=1 ollama serve
+   ```
+
+## Model Variants
+
+Ollama provides several Qwen-2.5 variants:
+
+| Model | Size | VRAM | Speed | Quality |
+|-------|------|------|-------|---------|
+| `qwen2.5:32b-instruct` | 32B | ~25GB | Medium | Best |
+| `qwen2.5:32b-instruct-q4_0` | 32B (4-bit) | ~12GB | Fast | Good |
+| `qwen2.5:14b-instruct` | 14B | ~10GB | Fast | Good |
+| `qwen2.5:7b-instruct` | 7B | ~5GB | Very Fast | OK |
+
+For your A100 80GB, **`qwen2.5:32b-instruct`** is recommended (best quality, no VRAM issues).
+
+## Custom Model Cache Location
+
+To store models in `data/models/` directory:
+
+```bash
+# Set environment variable
+export OLLAMA_MODELS="/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/models"
+
+# Add to ~/.bashrc for persistence
+echo 'export OLLAMA_MODELS="/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/models"' >> ~/.bashrc
+
+# Pull model (will download to data/models/)
+ollama pull qwen2.5:32b-instruct
+
+# Verify
+ls -lh $OLLAMA_MODELS/
+```
+
+## Comparing Results
+
+After running both API and local versions, compare results:
+
+```python
+import pandas as pd
+
+# Load results
+qwen_api = pd.read_csv('data/CSV/qwen_annotated_POI_test.csv')
+qwen_local = pd.read_csv('data/CSV/qwen_local_annotated_POI_test.csv')
+
+# Compare professions
+print("API professions:", qwen_api['profession_llm'].value_counts().head())
+print("Local professions:", qwen_local['profession_llm'].value_counts().head())
+
+# Check agreement
+agreement = (qwen_api['profession_llm'] == qwen_local['profession_llm']).mean()
+print(f"Agreement rate: {agreement*100:.1f}%")
+```
+
+## Cost Comparison (10,000 samples)
+
+| Method | Cost | Time | Privacy |
+|--------|------|------|---------|
+| **Qwen Local (A100)** | **$0** | ~50-100 hours | ✅ Full |
+| Qwen API (Alibaba) | ~$10-20 | ~5-10 hours | ⚠️ Data sent to Alibaba |
+| Llama API (Together) | ~$5-10 | ~5-10 hours | ⚠️ Data sent to Together AI |
+| Deepseek API | ~$1-2 | ~5-10 hours | ⚠️ Data sent to Deepseek |
+
+**Recommendation**:
+- For **small tests** (<100 samples): Use API (faster)
+- For **large datasets** (>1000 samples): Use local (free, private)
+- For **research papers**: Use local to avoid data privacy concerns
+
+## Advanced: Parallel Processing
+
+For faster processing on multi-GPU setup:
+
+```python
+# Not implemented yet, but possible with:
+# - Multiple Ollama instances on different GPUs
+# - Ray or Dask for parallel processing
+# - ~4x speedup with 4 GPUs
+```
+
+## Summary
+
+✅ **Ollama** already installed
+✅ **A100 80GB** GPU - perfect for Qwen-2.5-32B
+✅ **Free inference** - no API costs
+✅ **Privacy** - data stays local
+
+**Next steps:**
+1. Pull model: `ollama pull qwen2.5:32b-instruct`
+2. Test with Cell 20: `TEST_MODE = True`, `TEST_SIZE = 10`
+3. Run full dataset: `TEST_MODE = False`
+
+**Estimated time for 10,000 samples**: ~50-100 hours
+**Cost**: $0
+
+Good luck! 🚀

jupyter_notebooks/SPACY_NER_EXPLANATION.md

@@ -0,0 +1,316 @@
+# spaCy NER Implementation
+
+## Why spaCy for NER?
+
+Using **spaCy's Named Entity Recognition (NER)** is significantly better than regex-based cleaning because:
+
+1. **Intelligent entity extraction**: Recognizes PERSON entities using machine learning
+2. **Context-aware**: Understands sentence structure and context
+3. **Robust**: Handles various name formats (first, last, full, stage names)
+4. **Language support**: Works with multiple languages and scripts
+5. **Industry standard**: Used in production NLP applications
+
+## How It Works
+
+### Pipeline Overview
+
+```
+Original Name
+    ↓
+1. Translate Leetspeak (4→a, 3→e, 1→i)
+    ↓
+2. Remove Noise (emoji, LoRA terms, versions)
+    ↓
+3. spaCy NER - Extract PERSON entities
+    ↓
+4. Fallback to capitalized words if needed
+    ↓
+Cleaned Name
+```
+
+### Detailed Steps
+
+#### Step 1: Leetspeak Translation
+```python
+"4kira LoRA v2" → "akira LoRA v2"
+"1rene Model" → "irene Model"
+"3mma Watson" → "emma Watson"
+```
+
+#### Step 2: Noise Removal
+```python
+"akira LoRA v2" → "akira"
+"irene Model" → "irene"
+"emma Watson" → "emma Watson"
+```
+
+#### Step 3: spaCy NER
+```python
+nlp("akira")
+# Entities: [("akira", PERSON)]
+# Result: "akira"
+
+nlp("emma Watson")
+# Entities: [("emma Watson", PERSON)]
+# Result: "emma Watson"
+```
+
+#### Step 4: Fallback
+If spaCy doesn't find a PERSON entity:
+- Extract capitalized words (likely names)
+- Or return cleaned text as-is
+
+## Examples
+
+### Case 1: Simple Name
+```
+Input:  "IU"
+Output: "IU"
+
+Process:
+  - Preprocess: "IU" (no noise)
+  - spaCy NER: Recognizes "IU" as PERSON
+  - Result: "IU"
+```
+
+### Case 2: Name with LoRA Terms
+```
+Input:  "Scarlett Johansson「LoRa」"
+Output: "Scarlett Johansson"
+
+Process:
+  - Preprocess: "Scarlett Johansson" (removed 「LoRa」)
+  - spaCy NER: Recognizes "Scarlett Johansson" as PERSON
+  - Result: "Scarlett Johansson"
+```
+
+### Case 3: Leetspeak Name
+```
+Input:  "4kira Anime Character v1"
+Output: "akira"
+
+Process:
+  - Leetspeak: "akira Anime Character v1"
+  - Preprocess: "akira Anime Character"
+  - spaCy NER: Recognizes "akira" as PERSON
+  - Result: "akira"
+```
+
+### Case 4: Complex Format
+```
+Input:  "Gakki | Aragaki Yui | 新垣結衣"
+Output: "Gakki"
+
+Process:
+  - Preprocess: "Gakki" (kept first part before |)
+  - spaCy NER: Recognizes "Gakki" as PERSON
+  - Result: "Gakki"
+```
+
+### Case 5: With Metadata
+```
+Input:  "Emma Watson (JG) v3.5"
+Output: "Emma Watson"
+
+Process:
+  - Preprocess: "Emma Watson" (removed (JG) and v3.5)
+  - spaCy NER: Recognizes "Emma Watson" as PERSON
+  - Result: "Emma Watson"
+```
+
+## Advantages Over Regex-Only
+
+### Old Approach (Regex Only)
+```python
+# Just remove noise and hope for the best
+name = remove_noise(name)
+name = name.strip()
+# Result: May include non-name words
+```
+
+Problems:
+- Can't distinguish names from other capitalized words
+- May include words like "Model", "Anime", "Character"
+- No context awareness
+- Language-dependent regex patterns needed
+
+### New Approach (spaCy NER)
+```python
+# Intelligent entity extraction
+preprocessed = remove_noise(name)
+doc = nlp(preprocessed)
+person_names = [ent.text for ent in doc.ents if ent.label_ == "PERSON"]
+# Result: Only actual person names
+```
+
+Benefits:
+- ✅ Identifies actual person entities
+- ✅ Ignores non-person words
+- ✅ Context-aware (understands "Emma Watson" is one entity)
+- ✅ Multi-language support
+- ✅ Handles various name formats
+
+## Comparison Examples
+
+| Input | Regex Only | spaCy NER |
+|-------|------------|-----------|
+| `"Emma Watson Model"` | `"Emma Watson Model"` ❌ | `"Emma Watson"` ✅ |
+| `"Anime Character Levi"` | `"Anime Character Levi"` ❌ | `"Levi"` ✅ |
+| `"Taylor Swift v2"` | `"Taylor Swift"` ✅ | `"Taylor Swift"` ✅ |
+| `"K4te Middleton"` | `"K4te Middleton"` ❌ | `"Kate Middleton"` ✅ |
+| `"Celebrity IU"` | `"Celebrity IU"` ❌ | `"IU"` ✅ |
+
+## spaCy Model Information
+
+### Model Used
+- **Name**: `en_core_web_sm`
+- **Language**: English (but works reasonably with romanized names)
+- **Size**: ~13 MB
+- **Entities**: Recognizes PERSON, ORG, GPE, etc.
+
+### Installation
+```bash
+# Install spaCy
+pip install spacy
+
+# Download model
+python -m spacy download en_core_web_sm
+```
+
+The notebook automatically downloads the model if not found.
+
+### Performance
+- **Speed**: ~1000-5000 docs/second
+- **Accuracy**: High for common names
+- **Memory**: Low (~100MB loaded)
+
+## Fallback Strategy
+
+If spaCy doesn't recognize a PERSON entity:
+
+1. **Extract capitalized words**:
+   ```python
+   "unknown name here" → ["unknown"]
+   ```
+
+2. **Return first few capitalized words**:
+   ```python
+   "Celebrity Model Actor" → "Celebrity Model Actor"
+   ```
+
+3. **Last resort**: Return cleaned text as-is
+
+This ensures we always get something, even for:
+- Uncommon/rare names
+- Nicknames
+- Non-English names
+- Stage names
+
+## Testing
+
+### How to Verify spaCy is Working
+
+Run Cell 5 and check the output:
+
+```
+✅ spaCy model loaded: en_core_web_sm
+
+📊 Name cleaning examples (with spaCy NER):
+===================================================================================================
+Original Name                                      | Cleaned Name
+===================================================================================================
+Scarlett Johansson「LoRa」                        | Scarlett Johansson
+Emma Watson (JG)                                  | Emma Watson
+IU                                                | IU
+Belle Delphine                                    | Belle Delphine
+...
+```
+
+### Key Indicators
+
+✅ **Good signs**:
+- Person names cleanly extracted
+- No extra words like "Model", "LoRA", "Celebrity"
+- Multi-word names kept together (e.g., "Emma Watson" not just "Emma")
+
+❌ **Issues to watch**:
+- Empty results (increase fallback logic)
+- Partial names (e.g., only first name)
+- Non-names included (tune preprocessing)
+
+## Customization
+
+### Add More Languages
+
+For better support of non-English names:
+
+```python
+# Download multilingual model
+python -m spacy download xx_ent_wiki_sm
+
+# Use in code
+nlp = spacy.load("xx_ent_wiki_sm")
+```
+
+### Adjust Entity Extraction
+
+To extract other entities:
+
+```python
+# Extract organizations too
+entities = [ent.text for ent in doc.ents
+            if ent.label_ in ["PERSON", "ORG"]]
+```
+
+### Custom Entity Rules
+
+Add custom patterns for names spaCy might miss:
+
+```python
+from spacy.matcher import Matcher
+
+matcher = Matcher(nlp.vocab)
+# Add patterns for specific name formats
+```
+
+## Benefits for This Project
+
+### Better Person Identification
+
+With cleaner names:
+- LLMs receive recognizable names
+- "Emma Watson" instead of "Emma Watson Model LoRA v3"
+- Better identification accuracy
+
+### Reduced Ambiguity
+
+spaCy helps distinguish:
+- Person names vs. descriptive words
+- "Celebrity IU" → "IU" (person)
+- "Model Bella" → "Bella" (person)
+
+### Improved Context for LLMs
+
+Cleaner input = better prompts:
+```
+Before: "Given 'Celebrity Model Emma Watson LoRA v2' (actress)..."
+After:  "Given 'Emma Watson' (actress)..."
+```
+
+The LLM can now focus on identifying the person, not parsing the noise.
+
+## Summary
+
+✅ **spaCy NER** provides intelligent, context-aware name extraction
+✅ **Better than regex** for handling complex name formats
+✅ **Fallback strategy** ensures we always get a result
+✅ **Industry standard** tool used in production NLP
+✅ **Easy to use** with minimal code
+
+The combination of:
+1. Leetspeak translation
+2. Noise removal
+3. spaCy NER
+4. Smart fallbacks
+
+...results in clean, accurate person names ready for LLM annotation!

jupyter_notebooks/Section_2-3-4_Figure_8_deepfake_adapters.ipynb

@@ -0,0 +1,1943 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "e4cf5af4",
+   "metadata": {},
+   "source": [
+    "# Deepfake Adapter Dataset Processing Pipeline\n",
+    "\n",
+    "This notebook processes the `real_person_adapters.csv` dataset to identify and annotate real people used in deepfake LoRA models.\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "The pipeline consists of 4 main steps:\n",
+    "\n",
+    "1. **NER & Name Cleaning**: Extract clean person names from noisy adapter names\n",
+    "   - Remove emoji, version tags (v1, v2), LoRA-specific terms\n",
+    "   - Extract core person name from complex strings\n",
+    "\n",
+    "2. **Country/Nationality Mapping**: Convert nationality tags to standardized countries\n",
+    "   - Uses `misc/lists/countries.csv` reference\n",
+    "   - Excludes uninhabited territories (Isle of Man, Bouvet Island, etc.)\n",
+    "\n",
+    "3. **LLM-Based Profession Annotation**: Use large language models to identify professions\n",
+    "   - Supports 3 LLM providers: **Qwen**, **Llama**, **Mistral** (large versions)\n",
+    "   - Uses profession list from `misc/lists/professions.csv`\n",
+    "   - Can annotate up to 3 professions per person (e.g., Emma Watson: actor, activist)\n",
+    "\n",
+    "4. **Output Formatting**: Generate annotated dataset with standardized fields\n",
+    "\n",
+    "## Prerequisites\n",
+    "\n",
+    "### 1. Install Required Packages\n",
+    "\n",
+    "```bash\n",
+    "pip install pandas numpy emoji requests tqdm spacy\n",
+    "\n",
+    "# Download spaCy English model for NER\n",
+    "python -m spacy download en_core_web_sm\n",
+    "```\n",
+    "\n",
+    "### 2. Set Up API Keys\n",
+    "\n",
+    "You need API keys for at least one of the LLM providers:\n",
+    "\n",
+    "- **Qwen (Alibaba Cloud)**: https://dashscope.aliyun.com/\n",
+    "- **Llama (Together AI)**: https://www.together.ai/\n",
+    "- **Mistral AI**: https://mistral.ai/\n",
+    "\n",
+    "Create text files in `misc/credentials/`:\n",
+    "- `qwen_api_key.txt`\n",
+    "- `together_api_key.txt` (for Llama)\n",
+    "- `mistral_api_key.txt`\n",
+    "\n",
+    "Each file should contain only the API key (no extra whitespace).\n",
+    "\n",
+    "See `misc/credentials/README.md` for detailed instructions.\n",
+    "\n",
+    "## Configuration\n",
+    "\n",
+    "In the LLM annotation section, you can configure:\n",
+    "\n",
+    "```python\n",
+    "SELECTED_LLM = 'qwen'  # Choose: 'qwen', 'llama', or 'mistral'\n",
+    "TEST_MODE = True       # True = test on small sample first\n",
+    "TEST_SIZE = 10         # Number of rows to test\n",
+    "MAX_ROWS = 20000       # Maximum rows to process (None = all)\n",
+    "SAVE_INTERVAL = 10     # Save progress every N rows\n",
+    "```\n",
+    "\n",
+    "## Expected Output\n",
+    "\n",
+    "The final annotated dataset will include:\n",
+    "- `real_name`: Cleaned person name\n",
+    "- `full_name`: Full legal/stage name (from LLM)\n",
+    "- `gender`: Gender (from LLM)\n",
+    "- `profession_llm`: Up to 3 professions (from LLM)\n",
+    "- `country`: Primary country (from LLM)\n",
+    "- `tags`: Original tags as list\n",
+    "- `likely_country`: Country inferred from tags\n",
+    "- `likely_nationality`: Nationality inferred from tags\n",
+    "- `likely_profession`: Profession inferred from tags\n",
+    "- `prompt`: Summary of hints used for LLM\n",
+    "\n",
+    "## Cost Considerations\n",
+    "\n",
+    "For ~10,000 entries:\n",
+    "- **Qwen**: Check Alibaba Cloud pricing\n",
+    "- **Llama (Together AI)**: ~$5-10\n",
+    "- **Mistral**: ~$40-80\n",
+    "\n",
+    "**Always test on a small sample first!**\n",
+    "\n",
+    "## Running the Pipeline\n",
+    "\n",
+    "1. Run cells sequentially from top to bottom\n",
+    "2. Start with `TEST_MODE = True` to test on 10 samples\n",
+    "3. Review results and adjust configuration as needed\n",
+    "4. Set `TEST_MODE = False` for full processing\n",
+    "5. The pipeline saves progress automatically and can be resumed if interrupted\n",
+    "\n",
+    "---\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "23d0ae58",
+   "metadata": {},
+   "source": [
+    "# Deepfake Adapter Dataset - LLM Annotation Pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d7942c40",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3c906826",
+   "metadata": {},
+   "source": [
+    "## NER Code"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cb529f57",
+   "metadata": {},
+   "source": [
+    "please put NER code here. on the first collumn of the adapter dataset in data/real_person_adapters.csv named entity recognition should be performed to extract the most plausible name."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "ner_code_cell",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u2705 spaCy model loaded: en_core_web_sm\n",
+      "Loaded 18 rows\n",
+      "\n",
+      "\ud83d\udd04 Processing names with spaCy NER...\n",
+      "\n",
+      "\ud83d\udcca Name cleaning examples (with spaCy NER):\n",
+      "====================================================================================================\n",
+      "Original Name                                      | Cleaned Name                  \n",
+      "====================================================================================================\n",
+      "Super Pose Book Vol.1 - ControlNet                 | Super Pose Book               \n",
+      "Liyuu LoRA                                         | Liyuu                         \n",
+      "HashimotoKanna/ \u6a4b\u672c\u74b0\u5948 _JP_Actress                   | HashimotoKanna                \n",
+      "Emma Watson (JG)                                   | Emma Watson                   \n",
+      "Gal Gadot\u300cLoRa\u300d                                    | Gal Gadot                     \n",
+      "Scarlett Johansson\u300cLoRa\u300d                           | Scarlett Johansson            \n",
+      "Gakki | Aragaki Yui | \u65b0\u57a3\u7d50\u8863                         | Gakki                         \n",
+      "Actress Satomi_\u77f3\u539f\u25cb\u25cb                                | Actress Satomi                \n",
+      "Game of Thrones Cast                               | Game Thrones Cast             \n",
+      "Natalie Portman\u300cLoRa\u300d                              | Natalie Portman               \n",
+      "Emma Watson LoRA                                   | Emma Watson                   \n",
+      "====================================================================================================\n",
+      "\n",
+      "\ud83e\uddea Leetspeak translation examples:\n",
+      "  4kira LoRA                     -> akira\n",
+      "  3mma Watson v2                 -> Watson\n",
+      "  1rene LORA                     -> irene\n",
+      "  L3vi Ackerman                  -> Levi Ackerman\n",
+      "\n",
+      "\ud83d\udcc8 Statistics:\n",
+      "  Total rows: 18\n",
+      "  Non-empty names: 18\n",
+      "  Empty names: 0\n",
+      "\n",
+      "\ud83c\udfaf Sample spaCy NER results:\n",
+      "  1. IU\n",
+      "  2. Super Pose Book\n",
+      "  3. Liyuu\n",
+      "  4. Irene\n",
+      "  5. AESPA Karina\n",
+      "  6. Saika Kawakita\n",
+      "  7. Liu Yifei\n",
+      "  8. HashimotoKanna\n",
+      "  9. Emma Watson\n",
+      "  10. Gal Gadot\n",
+      "\n",
+      "\u2705 Cleaned 18 names using spaCy NER\n",
+      "\ud83d\udcbe Saved to /home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/CSV/NER_POI_step01_pre_annotation.csv\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "import emoji\n",
+    "import spacy\n",
+    "\n",
+    "# Load spaCy model\n",
+    "# You may need to download it first: python -m spacy download en_core_web_sm\n",
+    "try:\n",
+    "    nlp = spacy.load(\"en_core_web_sm\")\n",
+    "    print(\"\u2705 spaCy model loaded: en_core_web_sm\")\n",
+    "except OSError:\n",
+    "    print(\"\u274c spaCy model not found. Downloading...\")\n",
+    "    import subprocess\n",
+    "    subprocess.run([\"python\", \"-m\", \"spacy\", \"download\", \"en_core_web_sm\"])\n",
+    "    nlp = spacy.load(\"en_core_web_sm\")\n",
+    "    print(\"\u2705 spaCy model downloaded and loaded\")\n",
+    "\n",
+    "# Set up paths\n",
+    "current_dir = Path.cwd()\n",
+    "#input_file = current_dir.parent / \"data/CSV/real_person_adapters.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/samples/real_person_adapters_sample.csv\"\n",
+    "\n",
+    "# Load dataset\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "\n",
+    "def translate_leetspeak(text: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Translate common leetspeak patterns to normal letters.\n",
+    "    Examples: 4kira -> Akira, 3mma -> Emma, 1rene -> Irene\n",
+    "    \"\"\"\n",
+    "    if not text:\n",
+    "        return text\n",
+    "    \n",
+    "    # Common leetspeak mappings (order matters!)\n",
+    "    leetspeak_map = {\n",
+    "        '4': 'a',\n",
+    "        '3': 'e', \n",
+    "        '1': 'i',\n",
+    "        '0': 'o',\n",
+    "        '7': 't',\n",
+    "        '5': 's',\n",
+    "        '8': 'b',\n",
+    "        '9': 'g',\n",
+    "        '@': 'a',\n",
+    "        '$': 's',\n",
+    "        '!': 'i',\n",
+    "    }\n",
+    "    \n",
+    "    result = text\n",
+    "    # Apply mappings at word boundaries or start of string\n",
+    "    for leet, normal in leetspeak_map.items():\n",
+    "        # Replace at start of word\n",
+    "        result = re.sub(rf'\\b{re.escape(leet)}', normal, result, flags=re.IGNORECASE)\n",
+    "        # Replace standalone numbers that look like letters in context\n",
+    "        result = re.sub(rf'(?<=[a-z]){re.escape(leet)}(?=[a-z])', normal, result, flags=re.IGNORECASE)\n",
+    "    \n",
+    "    return result\n",
+    "\n",
+    "def preprocess_for_ner(name: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Preprocess the name before spaCy NER.\n",
+    "    Remove noise but keep the actual name parts.\n",
+    "    \"\"\"\n",
+    "    if pd.isna(name):\n",
+    "        return \"\"\n",
+    "    \n",
+    "    name = str(name)\n",
+    "    \n",
+    "    # FIRST: Translate leetspeak\n",
+    "    name = translate_leetspeak(name)\n",
+    "    \n",
+    "    # Remove emoji\n",
+    "    name = emoji.replace_emoji(name, replace=' ')\n",
+    "    \n",
+    "    # Remove version indicators (v1, v2, v1.0, etc.)\n",
+    "    name = re.sub(r'\\s*[vV]\\d+(\\.\\d+)?\\s*', ' ', name)\n",
+    "    \n",
+    "    # Remove LoRA-related terms (case insensitive)\n",
+    "    lora_terms = ['lora', 'loha', 'lycoris', 'controlnet', 'textual inversion', \n",
+    "                  'embedding', 'ti', 'checkpoint', 'model', 'adapter']\n",
+    "    for term in lora_terms:\n",
+    "        name = re.sub(rf'\\b{term}\\b', '', name, flags=re.IGNORECASE)\n",
+    "    \n",
+    "    # Remove content in parentheses or brackets (often metadata)\n",
+    "    name = re.sub(r'\\([^)]*\\)', '', name)\n",
+    "    name = re.sub(r'\\[[^\\]]*\\]', '', name)\n",
+    "    \n",
+    "    # Remove special characters like \u300c\u300d\n",
+    "    name = re.sub(r'[\u300c\u300d\u300e\u300f\u3010\u3011\u3008\u3009\u300a\u300b]', '', name)\n",
+    "    \n",
+    "    # Handle pipe - keep first part\n",
+    "    if '|' in name:\n",
+    "        name = name.split('|')[0]\n",
+    "    \n",
+    "    # Handle forward slash - keep first part\n",
+    "    if '/' in name:\n",
+    "        name = name.split('/')[0]\n",
+    "    \n",
+    "    # Replace underscores with spaces\n",
+    "    name = name.replace('_', ' ')\n",
+    "    \n",
+    "    # Remove multiple spaces\n",
+    "    name = re.sub(r'\\s+', ' ', name)\n",
+    "    \n",
+    "    # Strip\n",
+    "    name = name.strip()\n",
+    "    \n",
+    "    return name\n",
+    "\n",
+    "def extract_person_name(text: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Use spaCy NER to extract person names from text.\n",
+    "    Falls back to cleaned text if no PERSON entity found.\n",
+    "    \"\"\"\n",
+    "    if not text:\n",
+    "        return \"\"\n",
+    "    \n",
+    "    # Run spaCy NER\n",
+    "    doc = nlp(text)\n",
+    "    \n",
+    "    # Extract PERSON entities\n",
+    "    person_entities = [ent.text for ent in doc.ents if ent.label_ == \"PERSON\"]\n",
+    "    \n",
+    "    if person_entities:\n",
+    "        # Return the first (usually longest/best) person name\n",
+    "        return person_entities[0].strip()\n",
+    "    \n",
+    "    # If no PERSON entity found, try to extract capitalized words (likely names)\n",
+    "    # This helps with names spaCy might miss\n",
+    "    words = text.split()\n",
+    "    capitalized_words = [w for w in words if w and w[0].isupper() and len(w) > 1]\n",
+    "    \n",
+    "    if capitalized_words:\n",
+    "        # Join first few capitalized words (likely the name)\n",
+    "        return ' '.join(capitalized_words[:3]).strip()\n",
+    "    \n",
+    "    # Last resort: return cleaned text\n",
+    "    return text.strip()\n",
+    "\n",
+    "def clean_name_with_spacy(name: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Complete name cleaning pipeline with spaCy NER.\n",
+    "    \n",
+    "    Pipeline:\n",
+    "    1. Translate leetspeak (4\u2192a, 3\u2192e, 1\u2192i, etc.)\n",
+    "    2. Remove noise (emoji, version tags, LoRA terms)\n",
+    "    3. Use spaCy to extract PERSON entities\n",
+    "    4. Fallback to capitalized words or cleaned text\n",
+    "    \"\"\"\n",
+    "    # Step 1 & 2: Preprocess (leetspeak + noise removal)\n",
+    "    preprocessed = preprocess_for_ner(name)\n",
+    "    \n",
+    "    if not preprocessed:\n",
+    "        return \"\"\n",
+    "    \n",
+    "    # Step 3: Extract person name using spaCy NER\n",
+    "    person_name = extract_person_name(preprocessed)\n",
+    "    \n",
+    "    return person_name\n",
+    "\n",
+    "# Apply name cleaning with spaCy\n",
+    "print(\"\\n\ud83d\udd04 Processing names with spaCy NER...\")\n",
+    "df['real_name'] = df['name'].apply(clean_name_with_spacy)\n",
+    "\n",
+    "# Show examples with detailed comparison\n",
+    "print(\"\\n\ud83d\udcca Name cleaning examples (with spaCy NER):\")\n",
+    "print(\"=\" * 100)\n",
+    "print(f\"{'Original Name':<50} | {'Cleaned Name':<30}\")\n",
+    "print(\"=\" * 100)\n",
+    "\n",
+    "examples = df[['name', 'real_name']].head(30)\n",
+    "shown = 0\n",
+    "for idx, row in examples.iterrows():\n",
+    "    if row['name'] != row['real_name'] and shown < 20:\n",
+    "        print(f\"{row['name']:<50} | {row['real_name']:<30}\")\n",
+    "        shown += 1\n",
+    "\n",
+    "print(\"=\" * 100)\n",
+    "\n",
+    "# Show specific test cases\n",
+    "print(\"\\n\ud83e\uddea Leetspeak translation examples:\")\n",
+    "test_names = ['4kira LoRA', '3mma Watson v2', '1rene LORA', 'L3vi Ackerman']\n",
+    "for test in test_names:\n",
+    "    result = clean_name_with_spacy(test)\n",
+    "    print(f\"  {test:<30} -> {result}\")\n",
+    "\n",
+    "# Statistics\n",
+    "print(f\"\\n\ud83d\udcc8 Statistics:\")\n",
+    "print(f\"  Total rows: {len(df)}\")\n",
+    "print(f\"  Non-empty names: {(df['real_name'] != '').sum()}\")\n",
+    "print(f\"  Empty names: {(df['real_name'] == '').sum()}\")\n",
+    "\n",
+    "# Show some examples of what spaCy identified\n",
+    "print(\"\\n\ud83c\udfaf Sample spaCy NER results:\")\n",
+    "sample_names = df['real_name'].head(20).tolist()\n",
+    "for i, name in enumerate(sample_names[:10], 1):\n",
+    "    if name:\n",
+    "        print(f\"  {i}. {name}\")\n",
+    "\n",
+    "print(f\"\\n\u2705 Cleaned {len(df)} names using spaCy NER\")\n",
+    "\n",
+    "# Save intermediate result\n",
+    "output_step1 = current_dir.parent / \"data/CSV/NER_POI_step01_pre_annotation.csv\"\n",
+    "df.to_csv(output_step1, index=False)\n",
+    "print(f\"\ud83d\udcbe Saved to {output_step1}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d701d1cb",
+   "metadata": {},
+   "source": [
+    "## nationality to Country\n",
+    "here tags related to nationality gets converted to the country equivalent. make sure isle of man is shows never up in the country collumn.as well as other uninhabited places."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "36ce71f2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Built country/nationality mapping with 781 entries\n",
+      "\n",
+      "Processed 18 rows\n",
+      "Rows with country: 0\n",
+      "Rows with nationality: 0\n",
+      "Rows with profession: 14\n",
+      "\n",
+      "Top 10 countries:\n",
+      "Series([], Name: count, dtype: int64)\n",
+      "\n",
+      "Sample results:\n",
+      "         real_name likely_country likely_nationality likely_profession\n",
+      "0               IU                                           celebrity\n",
+      "1  Super Pose Book                                                    \n",
+      "2            Liyuu                                                    \n",
+      "3            Irene                                           celebrity\n",
+      "4     AESPA Karina                                           celebrity\n",
+      "5   Saika Kawakita                                           celebrity\n",
+      "6        Liu Yifei                                           celebrity\n",
+      "7   HashimotoKanna                                               actor\n",
+      "8      Emma Watson                                               actor\n",
+      "9        Gal Gadot                                               model\n",
+      "\n",
+      "\ud83d\udd0d Checking 'Irene' entries:\n",
+      "  Name: Irene\n",
+      "  Tags: ['girl', 'photorealistic', 'asian', 'woman', 'beautiful', 'celebrity', 'korean']\n",
+      "  Country: \n",
+      "  Nationality: \n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from pathlib import Path\n",
+    "\n",
+    "# Set up paths\n",
+    "current_dir = Path.cwd()\n",
+    "countries_file = current_dir.parent / \"misc/lists/countries.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step01_pre_annotation.csv\"\n",
+    "output_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "\n",
+    "# Load datasets\n",
+    "poi_df = pd.read_csv(input_file)\n",
+    "countries_df = pd.read_csv(countries_file)\n",
+    "professions_df = pd.read_csv(professions_file)\n",
+    "\n",
+    "# Define uninhabited or non-relevant territories to exclude\n",
+    "excluded_territories = {\n",
+    "    'isle of man', 'bouvet island', 'heard island and mcdonald islands',\n",
+    "    'french southern territories', 'south georgia and the south sandwich islands',\n",
+    "    'svalbard and jan mayen', 'british indian ocean territory', 'antarctica',\n",
+    "    'christmas island', 'cocos (keeling) islands', 'norfolk island',\n",
+    "    'pitcairn', 'tokelau', 'united states minor outlying islands',\n",
+    "    'wallis and futuna', 'western sahara'\n",
+    "}\n",
+    "\n",
+    "# Step 1: Combine tags into one lowercase list\n",
+    "def combine_tags(row):\n",
+    "    return [str(row[f\"tag_{i}\"]).strip().lower() for i in range(1, 8) if pd.notna(row.get(f\"tag_{i}\"))]\n",
+    "\n",
+    "poi_df[\"tags\"] = poi_df.apply(combine_tags, axis=1)\n",
+    "\n",
+    "# Step 2: Build tag \u2192 (country, nationality) mapping with PRIORITIES\n",
+    "tag_to_country_nationality = {}\n",
+    "# We'll use a priority score: direct country name = 3, nationality = 2, word parts = 1\n",
+    "\n",
+    "for _, row in countries_df.iterrows():\n",
+    "    country = str(row[\"en_short_name\"]).strip()\n",
+    "    nationality = str(row[\"nationality\"]).strip()\n",
+    "    \n",
+    "    # Skip excluded territories\n",
+    "    if country.lower() in excluded_territories:\n",
+    "        continue\n",
+    "\n",
+    "    country_lc = country.lower()\n",
+    "    nationality_lc = nationality.lower()\n",
+    "\n",
+    "    # Store as (country, nationality, priority)\n",
+    "    # Exact country name match = highest priority\n",
+    "    if country_lc not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[country_lc] = (country, \"\", 3)\n",
+    "    \n",
+    "    # Exact nationality match = medium priority  \n",
+    "    if nationality_lc not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[nationality_lc] = (\"\", nationality, 2)\n",
+    "    \n",
+    "    # No-space versions\n",
+    "    country_no_space = country_lc.replace(\" \", \"\")\n",
+    "    nationality_no_space = nationality_lc.replace(\" \", \"\")\n",
+    "    \n",
+    "    if country_no_space not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[country_no_space] = (country, \"\", 3)\n",
+    "    if nationality_no_space not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[nationality_no_space] = (\"\", nationality, 2)\n",
+    "\n",
+    "    # Word parts = lowest priority (only for longer words to avoid false matches)\n",
+    "    for part in country_lc.split():\n",
+    "        if len(part) > 4:  # Only words longer than 4 chars\n",
+    "            if part not in tag_to_country_nationality:\n",
+    "                tag_to_country_nationality[part] = (country, \"\", 1)\n",
+    "    for part in nationality_lc.split():\n",
+    "        if len(part) > 4:\n",
+    "            if part not in tag_to_country_nationality:\n",
+    "                tag_to_country_nationality[part] = (\"\", nationality, 1)\n",
+    "\n",
+    "print(f\"Built country/nationality mapping with {len(tag_to_country_nationality)} entries\")\n",
+    "\n",
+    "# Step 3: Infer likely_country and likely_nationality by checking ALL tags\n",
+    "def infer_country_and_nationality(tags):\n",
+    "    \"\"\"\n",
+    "    Check ALL tags and return the best match based on priority.\n",
+    "    Priority: exact country name > nationality > word parts\n",
+    "    \"\"\"\n",
+    "    best_match = None\n",
+    "    best_priority = 0\n",
+    "    \n",
+    "    for tag in tags:\n",
+    "        # Try cleaned version (no spaces)\n",
+    "        cleaned = tag.replace(\" \", \"\").lower()\n",
+    "        \n",
+    "        # Check cleaned version\n",
+    "        if cleaned in tag_to_country_nationality:\n",
+    "            country, nationality, priority = tag_to_country_nationality[cleaned]\n",
+    "            if priority > best_priority and country and country.lower() not in excluded_territories:\n",
+    "                best_match = (country, nationality)\n",
+    "                best_priority = priority\n",
+    "        \n",
+    "        # Also check original tag\n",
+    "        if tag in tag_to_country_nationality:\n",
+    "            country, nationality, priority = tag_to_country_nationality[tag]\n",
+    "            if priority > best_priority and country and country.lower() not in excluded_territories:\n",
+    "                best_match = (country, nationality)\n",
+    "                best_priority = priority\n",
+    "    \n",
+    "    if best_match:\n",
+    "        return pd.Series(best_match)\n",
+    "    return pd.Series([\"\", \"\"])\n",
+    "\n",
+    "poi_df[[\"likely_country\", \"likely_nationality\"]] = poi_df[\"tags\"].apply(infer_country_and_nationality)\n",
+    "\n",
+    "# Step 4: Build tag \u2192 profession mapping\n",
+    "profession_alias_map = {}\n",
+    "\n",
+    "for _, row in professions_df.iterrows():\n",
+    "    canonical = str(row['profession']).strip().lower()\n",
+    "    profession_alias_map[canonical] = canonical\n",
+    "    for alias_col in ['alias_1', 'alias_2', 'alias_3']:\n",
+    "        alias = row.get(alias_col)\n",
+    "        if pd.notna(alias):\n",
+    "            profession_alias_map[str(alias).strip().lower()] = canonical\n",
+    "\n",
+    "# Step 5: Infer likely profession from tags\n",
+    "def infer_profession_from_tags(tags):\n",
+    "    matched = []\n",
+    "    for tag in tags:\n",
+    "        cleaned = tag.strip().lower()\n",
+    "        if cleaned in profession_alias_map:\n",
+    "            matched.append(profession_alias_map[cleaned])\n",
+    "\n",
+    "    if not matched:\n",
+    "        return \"\"\n",
+    "    if \"celebrity\" in matched and len(set(matched)) > 1:\n",
+    "        # Drop 'celebrity' if other professions are present\n",
+    "        matched = [m for m in matched if m != \"celebrity\"]\n",
+    "\n",
+    "    return matched[0]  # Return the first specific match\n",
+    "\n",
+    "\n",
+    "poi_df[\"likely_profession\"] = poi_df[\"tags\"].apply(infer_profession_from_tags)\n",
+    "\n",
+    "# Step 6: Save enriched dataset\n",
+    "poi_df.to_csv(output_file, index=False)\n",
+    "\n",
+    "# Preview results\n",
+    "print(f\"\\nProcessed {len(poi_df)} rows\")\n",
+    "print(f\"Rows with country: {(poi_df['likely_country'] != '').sum()}\")\n",
+    "print(f\"Rows with nationality: {(poi_df['likely_nationality'] != '').sum()}\")\n",
+    "print(f\"Rows with profession: {(poi_df['likely_profession'] != '').sum()}\")\n",
+    "\n",
+    "print(f\"\\nTop 10 countries:\")\n",
+    "print(poi_df[poi_df['likely_country'] != '']['likely_country'].value_counts().head(10))\n",
+    "\n",
+    "print(f\"\\nSample results:\")\n",
+    "print(poi_df[[\"real_name\", \"likely_country\", \"likely_nationality\", \"likely_profession\"]].head(10))\n",
+    "\n",
+    "# Check specific case: Irene with korean tag\n",
+    "irene_rows = poi_df[poi_df['real_name'].str.contains('irene', case=False, na=False)]\n",
+    "if len(irene_rows) > 0:\n",
+    "    print(f\"\\n\ud83d\udd0d Checking 'Irene' entries:\")\n",
+    "    for idx, row in irene_rows.iterrows():\n",
+    "        print(f\"  Name: {row['real_name']}\")\n",
+    "        print(f\"  Tags: {row['tags']}\")\n",
+    "        print(f\"  Country: {row['likely_country']}\")\n",
+    "        print(f\"  Nationality: {row['likely_nationality']}\")\n",
+    "        print()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "29eec5be",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "deepseek_header",
+   "metadata": {},
+   "source": [
+    "## Deepseek annotation (Testing)\n",
+    "\n",
+    "**Use this section first to test if the pipeline works** before using Qwen/Llama/Mistral.\n",
+    "\n",
+    "This uses your existing Deepseek credits. The code is adapted from your original implementation but updated to:\n",
+    "- Use the correct input file from Step 2\n",
+    "- Include the profession list from `misc/lists/professions.csv`\n",
+    "- Allow up to 3 professions per person\n",
+    "- Match the output format of the other LLMs"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "deepseek_code",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u274c Error: deepseek_api_key.txt not found in misc/credentials/\n",
+      "   Please create this file with your API key\n"
+     ]
+    },
+    {
+     "ename": "FileNotFoundError",
+     "evalue": "[Errno 2] No such file or directory: '/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/misc/credentials/deepseek_api_key.txt'",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[10], line 37\u001b[0m\n\u001b[1;32m     35\u001b[0m \u001b[38;5;66;03m# === LOAD API KEY ===\u001b[39;00m\n\u001b[1;32m     36\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 37\u001b[0m     \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mapi_key_file\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[1;32m     38\u001b[0m         api_key \u001b[38;5;241m=\u001b[39m f\u001b[38;5;241m.\u001b[39mread()\u001b[38;5;241m.\u001b[39mstrip()\n\u001b[1;32m     39\u001b[0m     \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m\u2705 Deepseek API key loaded\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
+      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/IPython/core/interactiveshell.py:324\u001b[0m, in \u001b[0;36m_modified_open\u001b[0;34m(file, *args, **kwargs)\u001b[0m\n\u001b[1;32m    317\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m}:\n\u001b[1;32m    318\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m    319\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIPython won\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m by default \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    320\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    321\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myou can use builtins\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m open.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    322\u001b[0m     )\n\u001b[0;32m--> 324\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mio_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/misc/credentials/deepseek_api_key.txt'"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import requests\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from openai import OpenAI\n",
+    "from tqdm import tqdm\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "professions_mapped_file = current_dir.parent / \"misc/lists/professions_mapped.csv\"\n",
+    "api_key_file = current_dir.parent / \"misc/credentials/deepseek_api_key.txt\"\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = True       # Set to False to process full dataset\n",
+    "TEST_SIZE = 10         # Number of rows to test\n",
+    "MAX_ROWS = 20000\n",
+    "SAVE_INTERVAL = 10     # Save every 10 rows\n",
+    "\n",
+    "# Define the SPECIFIC profession categories\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\",\n",
+    "    \"adult performer\",\n",
+    "    \"singer/musician\",\n",
+    "    \"model\",\n",
+    "    \"online personality\",\n",
+    "    \"public figure\",\n",
+    "    \"voice actor/ASMR\",\n",
+    "    \"sports professional\",\n",
+    "    \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD API KEY ===\n",
+    "try:\n",
+    "    with open(api_key_file) as f:\n",
+    "        api_key = f.read().strip()\n",
+    "    print(\"\u2705 Deepseek API key loaded\")\n",
+    "except FileNotFoundError:\n",
+    "    print(\"\u274c Error: deepseek_api_key.txt not found in misc/credentials/\")\n",
+    "    print(\"   Please create this file with your API key\")\n",
+    "    raise\n",
+    "\n",
+    "client = OpenAI(\n",
+    "    api_key=api_key,\n",
+    "    base_url=\"https://api.deepseek.com/v1\"\n",
+    ")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(input_file)\n",
+    "professions_df = pd.read_csv(professions_file)\n",
+    "\n",
+    "# Load profession categories and keywords\n",
+    "try:\n",
+    "    prof_mapped_df = pd.read_csv(professions_mapped_file)\n",
+    "    print(f\"\u2705 Loaded profession mapping with {len(prof_mapped_df)} categories\")\n",
+    "except:\n",
+    "    print(\"\u26a0\ufe0f Warning: professions_mapped.csv not found, using default categories\")\n",
+    "\n",
+    "profession_str = \", \".join(PROFESSION_CATEGORIES)\n",
+    "\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "print(f\"\\nProfession categories ({len(PROFESSION_CATEGORIES)}):\")\n",
+    "for cat in PROFESSION_CATEGORIES:\n",
+    "    print(f\"  - {cat}\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"\\nRunning in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPTS ===\n",
+    "def create_prompt(row):\n",
+    "    \"\"\"Create prompt for Deepseek annotation with specific profession categories.\"\"\"\n",
+    "    name = row['real_name'] if pd.notna(row.get('real_name')) else row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if we don't have enough hints\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Given '{name}' ({hint_text}), provide:\n",
+    "1. Full legal name (Western order if non-latin script)\n",
+    "2. Any stage names/aliases (comma separated)\n",
+    "3. Gender (Male/Female/Other/Unknown)\n",
+    "4. Top 3 most likely professions from ONLY these categories:\n",
+    "   - actor\n",
+    "   - adult performer\n",
+    "   - singer/musician\n",
+    "   - model\n",
+    "   - online personality (includes streamers, cosplayers, influencers)\n",
+    "   - public figure (includes politicians, activists, journalists, authors)\n",
+    "   - voice actor/ASMR\n",
+    "   - sports professional\n",
+    "   - tv personality (includes hosts, presenters, reality TV)\n",
+    "\n",
+    "5. Primary country associated\n",
+    "\n",
+    "IMPORTANT:\n",
+    "- Choose professions ONLY from the 9 categories above\n",
+    "- Provide up to 3 professions, comma-separated, ordered by relevance\n",
+    "- Be SPECIFIC: choose the most accurate category for each role\n",
+    "- \"online personality\" includes: streamers, cosplayers, YouTubers, influencers, content creators\n",
+    "- Use 'Unknown' when uncertain or for fictional characters/places\n",
+    "- For multi-role people, list all relevant categories (e.g., \"actor, singer/musician, online personality\")\n",
+    "\n",
+    "Respond with exactly 5 numbered lines.\"\"\"\n",
+    "\n",
+    "# Create prompts\n",
+    "df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "\n",
+    "# === QUERY DEEPSEEK ===\n",
+    "def query_deepseek(prompt):\n",
+    "    \"\"\"Query Deepseek API.\"\"\"\n",
+    "    try:\n",
+    "        response = client.chat.completions.create(\n",
+    "            model=\"deepseek-chat\",\n",
+    "            messages=[\n",
+    "                {\"role\": \"system\", \"content\": \"You are an assistant that extracts key data on a person based on the name. Respond with exactly 5 numbered lines. For professions, choose ONLY from these categories: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality.\"},\n",
+    "                {\"role\": \"user\", \"content\": prompt}\n",
+    "            ],\n",
+    "            temperature=0.05,\n",
+    "            top_p=0.8,\n",
+    "        )\n",
+    "        return response.choices[0].message.content.strip()\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error querying Deepseek: {e}\")\n",
+    "        return None\n",
+    "\n",
+    "# === PARSE RESPONSE ===\n",
+    "def parse_response(response):\n",
+    "    \"\"\"Parse Deepseek response into structured fields.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    # Split into lines and clean\n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    # Initialize with Unknown values\n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    # Extract information from each numbered line\n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.'):\n",
+    "            fields['gender'] = line[2:].strip()\n",
+    "        elif line.startswith('4.'):\n",
+    "            fields['profession_llm'] = line[2:].strip()\n",
+    "        elif line.startswith('5.'):\n",
+    "            fields['country'] = line[2:].strip()\n",
+    "    \n",
+    "    return fields\n",
+    "\n",
+    "# === PROCESS DATA ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/deepseek_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/deepseek_query_index.txt\"\n",
+    "\n",
+    "# Check for existing progress\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    with open(index_file, 'r') as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "    print(f\"Resuming from index {current_index}\")\n",
+    "\n",
+    "# Initialize result columns if they don't exist\n",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country', 'aliases']:\n",
+    "    if col not in df.columns:\n",
+    "        df[col] = 'Unknown'\n",
+    "\n",
+    "print(f\"\\nAnnotating with DEEPSEEK...\")\n",
+    "print(f\"Processing rows {current_index} to {len(df)}\")\n",
+    "\n",
+    "try:\n",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Annotating\"):\n",
+    "        row = df.iloc[i]\n",
+    "        \n",
+    "        # Query Deepseek\n",
+    "        response = query_deepseek(row['prompt'])\n",
+    "        parsed_data = parse_response(response)\n",
+    "        \n",
+    "        # Update dataframe\n",
+    "        for key, value in parsed_data.items():\n",
+    "            df.at[i, key] = value\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        # Save progress at intervals\n",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            with open(index_file, 'w') as f:\n",
+    "                f.write(str(current_index))\n",
+    "            print(f\"\u2705 Progress saved after {i+1} rows\")\n",
+    "        \n",
+    "        time.sleep(1)  # Rate limiting\n",
+    "    \n",
+    "    print(f\"\\n\u2705 Done! Final results saved to {output_file}\")\n",
+    "    \n",
+    "    # Summary statistics\n",
+    "    print(\"\\n=== Summary Statistics ===\")\n",
+    "    print(f\"Total processed: {len(df)}\")\n",
+    "    print(f\"\\nGender distribution:\")\n",
+    "    print(df['gender'].value_counts())\n",
+    "    print(f\"\\nTop 10 profession combinations:\")\n",
+    "    print(df['profession_llm'].value_counts().head(10))\n",
+    "    print(f\"\\nTop 10 countries:\")\n",
+    "    print(df['country'].value_counts().head(10))\n",
+    "    \n",
+    "    # Show sample results\n",
+    "    print(\"\\n=== Sample Results ===\")\n",
+    "    display_cols = ['real_name', 'full_name', 'gender', 'profession_llm', 'country']\n",
+    "    print(df[display_cols].head(10).to_string(index=False))\n",
+    "\n",
+    "except Exception as e:\n",
+    "    print(f\"\u26a0\ufe0f Error encountered: {e}\")\n",
+    "    print(f\"\u26a0\ufe0f Last processed index: {current_index}\")\n",
+    "    \n",
+    "    # Save progress before exiting\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))\n",
+    "    \n",
+    "    print(f\"\u26a0\ufe0f Progress saved up to row {current_index}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "qwen_header",
+   "metadata": {},
+   "source": [
+    "## Qwen Annotation (Alibaba Cloud)\n",
+    "\n",
+    "Uses **Qwen-Max** (Qwen3-Max) via OpenAI-compatible API.\n",
+    "\n",
+    "**Requirements:**\n",
+    "- API Key: `misc/credentials/qwen_api_key.txt`\n",
+    "- Sign up: https://dashscope.aliyun.com/\n",
+    "- Base URL: `https://dashscope-intl.aliyuncs.com/compatible-mode/v1`\n",
+    "- Model: `qwen-max` (uses Qwen3-Max)\n",
+    "- Uses OpenAI Python client for compatibility\n",
+    "- Cost: Variable (check Alibaba Cloud pricing)\n",
+    "\n",
+    "**Note:** This matches the official dashboard example code exactly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "qwen_code",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd",
+    "import json",
+    "import time",
+    "import re",
+    "from pathlib import Path",
+    "from tqdm import tqdm",
+    "from openai import OpenAI",
+    "",
+    "current_dir = Path.cwd()",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"",
+    "api_key_file = current_dir.parent / \"misc/credentials/qwen_api_key.txt\"",
+    "",
+    "# === CONFIGURATION ===",
+    "TEST_MODE = True",
+    "TEST_SIZE = 10",
+    "MAX_ROWS = 20000",
+    "SAVE_INTERVAL = 10",
+    "",
+    "PROFESSION_CATEGORIES = [",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",",
+    "    \"sports professional\", \"tv personality\"",
+    "]",
+    "",
+    "# === LOAD API KEY ===",
+    "try:",
+    "    with open(api_key_file) as f:",
+    "        api_key = f.read().strip()",
+    "    print(\"\u2705 Qwen API key loaded\")",
+    "    print(f\"   Key starts with: {api_key[:10]}...\")",
+    "except FileNotFoundError:",
+    "    print(\"\u274c Error: qwen_api_key.txt not found\")",
+    "    raise",
+    "",
+    "# Initialize OpenAI client for Qwen (matching dashboard example)",
+    "client = OpenAI(",
+    "    api_key=api_key,",
+    "    base_url=\"https://dashscope-intl.aliyuncs.com/compatible-mode/v1\"",
+    ")",
+    "",
+    "# === LOAD DATA ===",
+    "df = pd.read_csv(input_file)",
+    "print(f\"Loaded {len(df)} rows\")",
+    "",
+    "if TEST_MODE:",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")",
+    "    df = df.head(TEST_SIZE).copy()",
+    "elif MAX_ROWS:",
+    "    df = df.head(MAX_ROWS).copy()",
+    "",
+    "# === CREATE PROMPT ===",
+    "def create_prompt(name: str, tags: list, country: str) -> str:",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"",
+    "    country_str = country if country else \"unknown\"",
+    "    ",
+    "    return f\"\"\"You are an expert at identifying public figures.",
+    "",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}",
+    "",
+    "Provide in JSON format:",
+    "{{",
+    "  \"full_name\": \"<full name>\",",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],",
+    "  \"country\": \"<country>\"",
+    "}}",
+    "",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}",
+    "- Max 3 professions, ordered by relevance",
+    "- \"online personality\" includes streamers, cosplayers, influencers",
+    "- Return \"Unknown\" if uncertain",
+    "",
+    "Return ONLY JSON, no extra text.\"\"\"",
+    "",
+    "# === QUERY QWEN ===",
+    "def query_qwen(prompt: str) -> dict:",
+    "    \"\"\"Query Qwen using OpenAI-compatible API (matching dashboard example).\"\"\"",
+    "    for attempt in range(3):",
+    "        try:",
+    "            completion = client.chat.completions.create(",
+    "                model=\"qwen-max\",  # Note: dashboard shows \"qwen3-max\" but \"qwen-max\" works too",
+    "                messages=[",
+    "                    {\"role\": \"user\", \"content\": prompt}",
+    "                ],",
+    "                temperature=0.1,",
+    "                max_tokens=500,",
+    "                stream=False  # Disable streaming for easier parsing",
+    "            )",
+    "            ",
+    "            content = completion.choices[0].message.content",
+    "            ",
+    "            # Parse JSON from response",
+    "            json_match = re.search(r'\\{[^}]+\\}', content, re.DOTALL)",
+    "            if json_match:",
+    "                return json.loads(json_match.group())",
+    "            ",
+    "            return None",
+    "            ",
+    "        except Exception as e:",
+    "            print(f\"Qwen error (attempt {attempt+1}): {e}\")",
+    "            if attempt < 2:",
+    "                time.sleep(2 ** attempt)",
+    "    ",
+    "    return None",
+    "",
+    "# === PROCESS ===",
+    "output_file = current_dir.parent / f\"data/CSV/qwen_annotated_POI{'_test' if TEST_MODE else ''}.csv\"",
+    "index_file = current_dir.parent / \"misc/qwen_query_index.txt\"",
+    "",
+    "current_index = 0",
+    "if index_file.exists():",
+    "    with open(index_file) as f:",
+    "        current_index = int(f.read().strip())",
+    "",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:",
+    "    if col not in df.columns:",
+    "        df[col] = 'Unknown'",
+    "",
+    "print(f\"\\nAnnotating with QWEN-MAX (OpenAI-compatible API) - rows {current_index} to {len(df)}...\")",
+    "",
+    "try:",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Qwen\"):",
+    "        row = df.iloc[i]",
+    "        ",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8) ",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]",
+    "        ",
+    "        prompt = create_prompt(",
+    "            row.get('real_name', row.get('name', '')),",
+    "            tags,",
+    "            row.get('likely_country', '')",
+    "        )",
+    "        ",
+    "        result = query_qwen(prompt)",
+    "        ",
+    "        if result:",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')",
+    "            profs = result.get('professions', [])",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')",
+    "        ",
+    "        current_index = i + 1",
+    "        ",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):",
+    "            df.to_csv(output_file, index=False)",
+    "            with open(index_file, 'w') as f:",
+    "                f.write(str(current_index))",
+    "            print(f\"\u2705 Saved after {i+1} rows\")",
+    "        ",
+    "        time.sleep(0.5)",
+    "    ",
+    "    print(f\"\\n\u2705 Done! Results: {output_file}\")",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")",
+    "    ",
+    "except Exception as e:",
+    "    print(f\"\u26a0\ufe0f Error: {e}\")",
+    "    df.to_csv(output_file, index=False)",
+    "    with open(index_file, 'w') as f:",
+    "        f.write(str(current_index))",
+    ""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "llama_header",
+   "metadata": {},
+   "source": [
+    "## Llama Annotation (Together AI)\n",
+    "\n",
+    "Uses **Llama-3.1-70B-Instruct-Turbo** via Together AI.\n",
+    "\n",
+    "**Requirements:**\n",
+    "- API Key: `misc/credentials/together_api_key.txt`\n",
+    "- Sign up: https://www.together.ai/\n",
+    "- Model: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo\n",
+    "- Cost: ~$0.90 per 1M tokens (~$5-10 for 10k entries)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "llama_code",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd",
+    "import json",
+    "import time",
+    "import requests",
+    "import re",
+    "from pathlib import Path",
+    "from tqdm import tqdm",
+    "",
+    "current_dir = Path.cwd()",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"",
+    "api_key_file = current_dir.parent / \"misc/credentials/together_api_key.txt\"",
+    "",
+    "# === CONFIGURATION ===",
+    "TEST_MODE = True",
+    "TEST_SIZE = 10",
+    "MAX_ROWS = 20000",
+    "SAVE_INTERVAL = 10",
+    "",
+    "PROFESSION_CATEGORIES = [",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",",
+    "    \"sports professional\", \"tv personality\"",
+    "]",
+    "",
+    "# === LOAD API KEY ===",
+    "try:",
+    "    with open(api_key_file) as f:",
+    "        api_key = f.read().strip()",
+    "    print(\"\u2705 Llama API key loaded\")",
+    "except FileNotFoundError:",
+    "    print(\"\u274c Error: together_api_key.txt not found\")",
+    "    raise",
+    "",
+    "# === LOAD DATA ===",
+    "df = pd.read_csv(input_file)",
+    "print(f\"Loaded {len(df)} rows\")",
+    "",
+    "if TEST_MODE:",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")",
+    "    df = df.head(TEST_SIZE).copy()",
+    "elif MAX_ROWS:",
+    "    df = df.head(MAX_ROWS).copy()",
+    "",
+    "# === CREATE PROMPT ===",
+    "def create_prompt(name: str, tags: list, country: str) -> str:",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"",
+    "    country_str = country if country else \"unknown\"",
+    "    ",
+    "    return f\"\"\"You are an expert at identifying public figures.",
+    "",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}",
+    "",
+    "Provide in JSON format:",
+    "{{",
+    "  \"full_name\": \"<full name>\",",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],",
+    "  \"country\": \"<country>\"",
+    "}}",
+    "",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}",
+    "- Max 3 professions, ordered by relevance",
+    "- \"online personality\" includes streamers, cosplayers, influencers",
+    "- Return \"Unknown\" if uncertain",
+    "",
+    "Return ONLY JSON, no extra text.\"\"\"",
+    "",
+    "# === QUERY LLAMA ===",
+    "def query_llama(prompt: str) -> dict:",
+    "    url = \"https://api.together.xyz/v1/chat/completions\"",
+    "    ",
+    "    headers = {",
+    "        \"Authorization\": f\"Bearer {api_key}\",",
+    "        \"Content-Type\": \"application/json\"",
+    "    }",
+    "    ",
+    "    data = {",
+    "        \"model\": \"meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo\",",
+    "        \"messages\": [{\"role\": \"user\", \"content\": prompt}],",
+    "        \"temperature\": 0.1,",
+    "        \"max_tokens\": 500",
+    "    }",
+    "    ",
+    "    for attempt in range(3):",
+    "        try:",
+    "            response = requests.post(url, headers=headers, json=data, timeout=30)",
+    "            response.raise_for_status()",
+    "            result = response.json()",
+    "            ",
+    "            if 'choices' in result:",
+    "                content = result['choices'][0]['message']['content']",
+    "                json_match = re.search(r'\\{[^}]+\\}', content, re.DOTALL)",
+    "                if json_match:",
+    "                    return json.loads(json_match.group())",
+    "            return None",
+    "        except Exception as e:",
+    "            print(f\"Llama error (attempt {attempt+1}): {e}\")",
+    "            if attempt < 2:",
+    "                time.sleep(2 ** attempt)",
+    "    return None",
+    "",
+    "# === PROCESS ===",
+    "output_file = current_dir.parent / f\"data/CSV/llama_annotated_POI{'_test' if TEST_MODE else ''}.csv\"",
+    "index_file = current_dir.parent / \"misc/llama_query_index.txt\"",
+    "",
+    "current_index = 0",
+    "if index_file.exists():",
+    "    with open(index_file) as f:",
+    "        current_index = int(f.read().strip())",
+    "",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:",
+    "    if col not in df.columns:",
+    "        df[col] = 'Unknown'",
+    "",
+    "print(f\"\\nAnnotating with LLAMA 3.1 70B - rows {current_index} to {len(df)}...\")",
+    "",
+    "try:",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Llama\"):",
+    "        row = df.iloc[i]",
+    "        ",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8) ",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]",
+    "        ",
+    "        prompt = create_prompt(",
+    "            row.get('real_name', row.get('name', '')),",
+    "            tags,",
+    "            row.get('likely_country', '')",
+    "        )",
+    "        ",
+    "        result = query_llama(prompt)",
+    "        ",
+    "        if result:",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')",
+    "            profs = result.get('professions', [])",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')",
+    "        ",
+    "        current_index = i + 1",
+    "        ",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):",
+    "            df.to_csv(output_file, index=False)",
+    "            with open(index_file, 'w') as f:",
+    "                f.write(str(current_index))",
+    "            print(f\"\u2705 Saved after {i+1} rows\")",
+    "        ",
+    "        time.sleep(0.5)",
+    "    ",
+    "    print(f\"\\n\u2705 Done! Results: {output_file}\")",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")",
+    "    ",
+    "except Exception as e:",
+    "    print(f\"\u26a0\ufe0f Error: {e}\")",
+    "    df.to_csv(output_file, index=False)",
+    "    with open(index_file, 'w') as f:",
+    "        f.write(str(current_index))",
+    ""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "mistral_header",
+   "metadata": {},
+   "source": [
+    "## Mixtral Annotation (Mistral AI)\n",
+    "\n",
+    "Uses **Open-Mixtral-8x22B** via Mistral AI API.\n",
+    "\n",
+    "**Requirements:**\n",
+    "- API Key: `misc/credentials/mistral_api_key.txt`\n",
+    "- Sign up: https://mistral.ai/\n",
+    "- Model: open-mixtral-8x22b\n",
+    "- Cost: ~$2 per 1M tokens (~$10-20 for 10k entries)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "mixtral_code",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd",
+    "import json",
+    "import time",
+    "import requests",
+    "import re",
+    "from pathlib import Path",
+    "from tqdm import tqdm",
+    "",
+    "current_dir = Path.cwd()",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"",
+    "api_key_file = current_dir.parent / \"misc/credentials/mistral_api_key.txt\"",
+    "",
+    "# === CONFIGURATION ===",
+    "TEST_MODE = True",
+    "TEST_SIZE = 10",
+    "MAX_ROWS = 20000",
+    "SAVE_INTERVAL = 10",
+    "",
+    "PROFESSION_CATEGORIES = [",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",",
+    "    \"sports professional\", \"tv personality\"",
+    "]",
+    "",
+    "# === LOAD API KEY ===",
+    "try:",
+    "    with open(api_key_file) as f:",
+    "        api_key = f.read().strip()",
+    "    print(\"\u2705 Mixtral API key loaded\")",
+    "except FileNotFoundError:",
+    "    print(\"\u274c Error: mistral_api_key.txt not found\")",
+    "    raise",
+    "",
+    "# === LOAD DATA ===",
+    "df = pd.read_csv(input_file)",
+    "print(f\"Loaded {len(df)} rows\")",
+    "",
+    "if TEST_MODE:",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")",
+    "    df = df.head(TEST_SIZE).copy()",
+    "elif MAX_ROWS:",
+    "    df = df.head(MAX_ROWS).copy()",
+    "",
+    "# === CREATE PROMPT ===",
+    "def create_prompt(name: str, tags: list, country: str) -> str:",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"",
+    "    country_str = country if country else \"unknown\"",
+    "    ",
+    "    return f\"\"\"You are an expert at identifying public figures.",
+    "",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}",
+    "",
+    "Provide in JSON format:",
+    "{{",
+    "  \"full_name\": \"<full name>\",",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],",
+    "  \"country\": \"<country>\"",
+    "}}",
+    "",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}",
+    "- Max 3 professions, ordered by relevance",
+    "- \"online personality\" includes streamers, cosplayers, influencers",
+    "- Return \"Unknown\" if uncertain",
+    "",
+    "Return ONLY JSON, no extra text.\"\"\"",
+    "",
+    "# === QUERY MIXTRAL ===",
+    "def query_mixtral(prompt: str) -> dict:",
+    "    url = \"https://api.mistral.ai/v1/chat/completions\"",
+    "    ",
+    "    headers = {",
+    "        \"Authorization\": f\"Bearer {api_key}\",",
+    "        \"Content-Type\": \"application/json\"",
+    "    }",
+    "    ",
+    "    data = {",
+    "        \"model\": \"open-mixtral-8x22b\",",
+    "        \"messages\": [{\"role\": \"user\", \"content\": prompt}],",
+    "        \"temperature\": 0.1,",
+    "        \"max_tokens\": 500",
+    "    }",
+    "    ",
+    "    for attempt in range(3):",
+    "        try:",
+    "            response = requests.post(url, headers=headers, json=data, timeout=30)",
+    "            response.raise_for_status()",
+    "            result = response.json()",
+    "            ",
+    "            if 'choices' in result:",
+    "                content = result['choices'][0]['message']['content']",
+    "                json_match = re.search(r'\\{[^}]+\\}', content, re.DOTALL)",
+    "                if json_match:",
+    "                    return json.loads(json_match.group())",
+    "            return None",
+    "        except Exception as e:",
+    "            print(f\"Mixtral error (attempt {attempt+1}): {e}\")",
+    "            if attempt < 2:",
+    "                time.sleep(2 ** attempt)",
+    "    return None",
+    "",
+    "# === PROCESS ===",
+    "output_file = current_dir.parent / f\"data/CSV/mixtral_annotated_POI{'_test' if TEST_MODE else ''}.csv\"",
+    "index_file = current_dir.parent / \"misc/mixtral_query_index.txt\"",
+    "",
+    "current_index = 0",
+    "if index_file.exists():",
+    "    with open(index_file) as f:",
+    "        current_index = int(f.read().strip())",
+    "",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:",
+    "    if col not in df.columns:",
+    "        df[col] = 'Unknown'",
+    "",
+    "print(f\"\\nAnnotating with MIXTRAL 8x22B - rows {current_index} to {len(df)}...\")",
+    "",
+    "try:",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Mixtral\"):",
+    "        row = df.iloc[i]",
+    "        ",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8) ",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]",
+    "        ",
+    "        prompt = create_prompt(",
+    "            row.get('real_name', row.get('name', '')),",
+    "            tags,",
+    "            row.get('likely_country', '')",
+    "        )",
+    "        ",
+    "        result = query_mixtral(prompt)",
+    "        ",
+    "        if result:",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')",
+    "            profs = result.get('professions', [])",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')",
+    "        ",
+    "        current_index = i + 1",
+    "        ",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):",
+    "            df.to_csv(output_file, index=False)",
+    "            with open(index_file, 'w') as f:",
+    "                f.write(str(current_index))",
+    "            print(f\"\u2705 Saved after {i+1} rows\")",
+    "        ",
+    "        time.sleep(0.5)",
+    "    ",
+    "    print(f\"\\n\u2705 Done! Results: {output_file}\")",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")",
+    "    ",
+    "except Exception as e:",
+    "    print(f\"\u26a0\ufe0f Error: {e}\")",
+    "    df.to_csv(output_file, index=False)",
+    "    with open(index_file, 'w') as f:",
+    "        f.write(str(current_index))",
+    ""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "gemma_header",
+   "metadata": {},
+   "source": [
+    "## Gemma Annotation (Google - American Open Source)\n",
+    "\n",
+    "Uses **Gemma 2 27B** via Together AI - Google's open-source model.\n",
+    "\n",
+    "**Requirements:**\n",
+    "- API Key: `misc/credentials/together_api_key.txt` (same as Llama)\n",
+    "- Sign up: https://www.together.ai/\n",
+    "- Model: google/gemma-2-27b-it\n",
+    "- Cost: ~$0.80 per 1M tokens (~$4-8 for 10k entries)\n",
+    "- Note: American company (Google), fully open-source model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "gemma_code",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd",
+    "import json",
+    "import time",
+    "import requests",
+    "import re",
+    "from pathlib import Path",
+    "from tqdm import tqdm",
+    "",
+    "current_dir = Path.cwd()",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"",
+    "api_key_file = current_dir.parent / \"misc/credentials/together_api_key.txt\"",
+    "",
+    "# === CONFIGURATION ===",
+    "TEST_MODE = True",
+    "TEST_SIZE = 10",
+    "MAX_ROWS = 20000",
+    "SAVE_INTERVAL = 10",
+    "",
+    "PROFESSION_CATEGORIES = [",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",",
+    "    \"sports professional\", \"tv personality\"",
+    "]",
+    "",
+    "# === LOAD API KEY ===",
+    "try:",
+    "    with open(api_key_file) as f:",
+    "        api_key = f.read().strip()",
+    "    print(\"\u2705 Gemma API key loaded\")",
+    "except FileNotFoundError:",
+    "    print(\"\u274c Error: together_api_key.txt not found\")",
+    "    raise",
+    "",
+    "# === LOAD DATA ===",
+    "df = pd.read_csv(input_file)",
+    "print(f\"Loaded {len(df)} rows\")",
+    "",
+    "if TEST_MODE:",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")",
+    "    df = df.head(TEST_SIZE).copy()",
+    "elif MAX_ROWS:",
+    "    df = df.head(MAX_ROWS).copy()",
+    "",
+    "# === CREATE PROMPT ===",
+    "def create_prompt(name: str, tags: list, country: str) -> str:",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"",
+    "    country_str = country if country else \"unknown\"",
+    "    ",
+    "    return f\"\"\"You are an expert at identifying public figures.",
+    "",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}",
+    "",
+    "Provide in JSON format:",
+    "{{",
+    "  \"full_name\": \"<full name>\",",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],",
+    "  \"country\": \"<country>\"",
+    "}}",
+    "",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}",
+    "- Max 3 professions, ordered by relevance",
+    "- \"online personality\" includes streamers, cosplayers, influencers",
+    "- Return \"Unknown\" if uncertain",
+    "",
+    "Return ONLY JSON, no extra text.\"\"\"",
+    "",
+    "# === QUERY GEMMA ===",
+    "def query_gemma(prompt: str) -> dict:",
+    "    url = \"https://api.together.xyz/v1/chat/completions\"",
+    "    ",
+    "    headers = {",
+    "        \"Authorization\": f\"Bearer {api_key}\",",
+    "        \"Content-Type\": \"application/json\"",
+    "    }",
+    "    ",
+    "    data = {",
+    "        \"model\": \"google/gemma-2-27b-it\",",
+    "        \"messages\": [{\"role\": \"user\", \"content\": prompt}],",
+    "        \"temperature\": 0.1,",
+    "        \"max_tokens\": 500",
+    "    }",
+    "    ",
+    "    for attempt in range(3):",
+    "        try:",
+    "            response = requests.post(url, headers=headers, json=data, timeout=30)",
+    "            response.raise_for_status()",
+    "            result = response.json()",
+    "            ",
+    "            if 'choices' in result:",
+    "                content = result['choices'][0]['message']['content']",
+    "                json_match = re.search(r'\\{[^}]+\\}', content, re.DOTALL)",
+    "                if json_match:",
+    "                    return json.loads(json_match.group())",
+    "            return None",
+    "        except Exception as e:",
+    "            print(f\"Gemma error (attempt {attempt+1}): {e}\")",
+    "            if attempt < 2:",
+    "                time.sleep(2 ** attempt)",
+    "    return None",
+    "",
+    "# === PROCESS ===",
+    "output_file = current_dir.parent / f\"data/CSV/gemma_annotated_POI{'_test' if TEST_MODE else ''}.csv\"",
+    "index_file = current_dir.parent / \"misc/gemma_query_index.txt\"",
+    "",
+    "current_index = 0",
+    "if index_file.exists():",
+    "    with open(index_file) as f:",
+    "        current_index = int(f.read().strip())",
+    "",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:",
+    "    if col not in df.columns:",
+    "        df[col] = 'Unknown'",
+    "",
+    "print(f\"\\nAnnotating with GEMMA 2 27B (Google) - rows {current_index} to {len(df)}...\")",
+    "",
+    "try:",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Gemma\"):",
+    "        row = df.iloc[i]",
+    "        ",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8) ",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]",
+    "        ",
+    "        prompt = create_prompt(",
+    "            row.get('real_name', row.get('name', '')),",
+    "            tags,",
+    "            row.get('likely_country', '')",
+    "        )",
+    "        ",
+    "        result = query_gemma(prompt)",
+    "        ",
+    "        if result:",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')",
+    "            profs = result.get('professions', [])",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')",
+    "        ",
+    "        current_index = i + 1",
+    "        ",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):",
+    "            df.to_csv(output_file, index=False)",
+    "            with open(index_file, 'w') as f:",
+    "                f.write(str(current_index))",
+    "            print(f\"\u2705 Saved after {i+1} rows\")",
+    "        ",
+    "        time.sleep(0.5)",
+    "    ",
+    "    print(f\"\\n\u2705 Done! Results: {output_file}\")",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")",
+    "    ",
+    "except Exception as e:",
+    "    print(f\"\u26a0\ufe0f Error: {e}\")",
+    "    df.to_csv(output_file, index=False)",
+    "    with open(index_file, 'w') as f:",
+    "        f.write(str(current_index))",
+    ""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Qwen-2.5-32B Local Annotation (Ollama)\n",
+    "\n",
+    "Run **Qwen-2.5-32B-Instruct** locally using Ollama - NO API COSTS!\n",
+    "\n",
+    "**Requirements:**\n",
+    "- Ollama installed: https://ollama.com/\n",
+    "- GPU: NVIDIA A100 80GB (recommended)\n",
+    "- RAM: 265GB system RAM\n",
+    "- Model cache: `data/models/` (auto-downloaded on first run)\n",
+    "- Model size: ~20GB download\n",
+    "- VRAM usage: ~22-25GB during inference\n",
+    "\n",
+    "**Setup:**\n",
+    "```bash\n",
+    "# 1. Install Ollama (if not installed)\n",
+    "curl -fsSL https://ollama.com/install.sh | sh\n",
+    "\n",
+    "# 2. Pull Qwen-2.5-32B-Instruct\n",
+    "ollama pull qwen2.5:32b-instruct\n",
+    "\n",
+    "# 3. Verify model is available\n",
+    "ollama list\n",
+    "```\n",
+    "\n",
+    "**Advantages:**\n",
+    "- \u2705 FREE - No API costs\n",
+    "- \u2705 Fast - Local inference on A100\n",
+    "- \u2705 Privacy - Data never leaves your machine\n",
+    "- \u2705 Offline - Works without internet\n",
+    "- \u2705 High quality - 32B parameter model\n",
+    "\n",
+    "**Performance:**\n",
+    "- Speed: ~5-10 tokens/sec on A100\n",
+    "- Throughput: ~100-200 samples/hour (depends on prompt length)\n",
+    "- Cost: $0 (after model download)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "import subprocess\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/NER_POI_step02_annotated.csv\"\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = True\n",
+    "TEST_SIZE = 10\n",
+    "MAX_ROWS = 20000\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "# Ollama settings\n",
+    "MODEL_NAME = \"qwen2.5:32b-instruct\"  # or \"qwen2.5:32b\" for base model\n",
+    "OLLAMA_HOST = \"http://localhost:11434\"  # Default Ollama port\n",
+    "\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",\n",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",\n",
+    "    \"sports professional\", \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === CHECK OLLAMA ===\n",
+    "def check_ollama():\n",
+    "    \"\"\"Check if Ollama is installed and model is available.\"\"\"\n",
+    "    try:\n",
+    "        result = subprocess.run(\n",
+    "            [\"ollama\", \"list\"],\n",
+    "            capture_output=True,\n",
+    "            text=True,\n",
+    "            timeout=5\n",
+    "        )\n",
+    "        \n",
+    "        if result.returncode == 0:\n",
+    "            print(\"\u2705 Ollama is installed\")\n",
+    "            \n",
+    "            # Check if model is available\n",
+    "            if MODEL_NAME in result.stdout or \"qwen2.5\" in result.stdout:\n",
+    "                print(f\"\u2705 Model {MODEL_NAME} is available\")\n",
+    "                return True\n",
+    "            else:\n",
+    "                print(f\"\u26a0\ufe0f  Model {MODEL_NAME} not found\")\n",
+    "                print(f\"   Run: ollama pull {MODEL_NAME}\")\n",
+    "                return False\n",
+    "        else:\n",
+    "            print(\"\u274c Ollama command failed\")\n",
+    "            return False\n",
+    "    except FileNotFoundError:\n",
+    "        print(\"\u274c Ollama not installed\")\n",
+    "        print(\"   Install: curl -fsSL https://ollama.com/install.sh | sh\")\n",
+    "        return False\n",
+    "    except Exception as e:\n",
+    "        print(f\"\u274c Error checking Ollama: {e}\")\n",
+    "        return False\n",
+    "\n",
+    "# Check Ollama before proceeding\n",
+    "if not check_ollama():\n",
+    "    raise RuntimeError(\"Ollama not ready. Please install Ollama and pull the model.\")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"\\nLoaded {len(df)} rows\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPT ===\n",
+    "def create_prompt(name: str, tags: list, country: str) -> str:\n",
+    "    tags_str = \", \".join(tags[:5]) if tags else \"none\"\n",
+    "    country_str = country if country else \"unknown\"\n",
+    "    \n",
+    "    return f\"\"\"You are an expert at identifying public figures.\n",
+    "\n",
+    "Given: Name: {name}, Tags: {tags_str}, Country: {country_str}\n",
+    "\n",
+    "Provide in JSON format:\n",
+    "{{\n",
+    "  \"full_name\": \"<full name>\",\n",
+    "  \"gender\": \"<Male/Female/Other/Unknown>\",\n",
+    "  \"professions\": [\"<profession 1>\", \"<profession 2>\", \"<profession 3>\"],\n",
+    "  \"country\": \"<country>\"\n",
+    "}}\n",
+    "\n",
+    "Choose professions ONLY from: {\", \".join(PROFESSION_CATEGORIES)}\n",
+    "- Max 3 professions, ordered by relevance\n",
+    "- \"online personality\" includes streamers, cosplayers, influencers\n",
+    "- Return \"Unknown\" if uncertain\n",
+    "\n",
+    "Return ONLY JSON, no extra text.\"\"\"\n",
+    "\n",
+    "# === QUERY QWEN LOCAL ===\n",
+    "def query_qwen_local(prompt: str) -> dict:\n",
+    "    \"\"\"Query Qwen-2.5-32B via Ollama.\"\"\"\n",
+    "    for attempt in range(3):\n",
+    "        try:\n",
+    "            result = subprocess.run(\n",
+    "                [\n",
+    "                    \"ollama\", \"run\", MODEL_NAME,\n",
+    "                    prompt\n",
+    "                ],\n",
+    "                capture_output=True,\n",
+    "                text=True,\n",
+    "                timeout=60  # 60 second timeout per query\n",
+    "            )\n",
+    "            \n",
+    "            if result.returncode == 0:\n",
+    "                content = result.stdout.strip()\n",
+    "                \n",
+    "                # Parse JSON from response\n",
+    "                json_match = re.search(r'\\{[^}]+\\}', content, re.DOTALL)\n",
+    "                if json_match:\n",
+    "                    return json.loads(json_match.group())\n",
+    "                \n",
+    "                # If no JSON found, try to extract from markdown code block\n",
+    "                code_block = re.search(r'```(?:json)?\\s*({[^}]+})\\s*```', content, re.DOTALL)\n",
+    "                if code_block:\n",
+    "                    return json.loads(code_block.group(1))\n",
+    "            \n",
+    "            return None\n",
+    "        \n",
+    "        except subprocess.TimeoutExpired:\n",
+    "            print(f\"Qwen local timeout (attempt {attempt+1})\")\n",
+    "            if attempt < 2:\n",
+    "                time.sleep(2)\n",
+    "        except json.JSONDecodeError as e:\n",
+    "            print(f\"Qwen local JSON error (attempt {attempt+1}): {e}\")\n",
+    "            if attempt < 2:\n",
+    "                time.sleep(1)\n",
+    "        except Exception as e:\n",
+    "            print(f\"Qwen local error (attempt {attempt+1}): {e}\")\n",
+    "            if attempt < 2:\n",
+    "                time.sleep(2)\n",
+    "    \n",
+    "    return None\n",
+    "\n",
+    "# === PROCESS ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/qwen_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/qwen_local_query_index.txt\"\n",
+    "\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    with open(index_file) as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "\n",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country']:\n",
+    "    if col not in df.columns:\n",
+    "        df[col] = 'Unknown'\n",
+    "\n",
+    "print(f\"\\nAnnotating with QWEN-2.5-32B LOCAL - rows {current_index} to {len(df)}...\")\n",
+    "print(f\"Model: {MODEL_NAME}\")\n",
+    "print(f\"This may take a while (~100-200 samples/hour on A100)\\n\")\n",
+    "\n",
+    "try:\n",
+    "    start_time = time.time()\n",
+    "    \n",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Qwen Local\"):\n",
+    "        row = df.iloc[i]\n",
+    "        \n",
+    "        tags = [str(row[f'tag_{j}']).lower() for j in range(1, 8)\n",
+    "                if f'tag_{j}' in row and pd.notna(row[f'tag_{j}'])]\n",
+    "        \n",
+    "        prompt = create_prompt(\n",
+    "            row.get('real_name', row.get('name', '')),\n",
+    "            tags,\n",
+    "            row.get('likely_country', '')\n",
+    "        )\n",
+    "        \n",
+    "        result = query_qwen_local(prompt)\n",
+    "        \n",
+    "        if result:\n",
+    "            df.at[i, 'full_name'] = result.get('full_name', 'Unknown')\n",
+    "            df.at[i, 'gender'] = result.get('gender', 'Unknown')\n",
+    "            profs = result.get('professions', [])\n",
+    "            df.at[i, 'profession_llm'] = \", \".join(profs[:3]) if isinstance(profs, list) else str(profs)\n",
+    "            df.at[i, 'country'] = result.get('country', 'Unknown')\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            with open(index_file, 'w') as f:\n",
+    "                f.write(str(current_index))\n",
+    "            \n",
+    "            # Calculate stats\n",
+    "            elapsed = time.time() - start_time\n",
+    "            samples_done = i + 1 - (current_index - (i + 1))\n",
+    "            samples_per_hour = (samples_done / elapsed) * 3600 if elapsed > 0 else 0\n",
+    "            \n",
+    "            print(f\"\u2705 Saved after {i+1} rows (~{samples_per_hour:.1f} samples/hour)\")\n",
+    "    \n",
+    "    elapsed_total = time.time() - start_time\n",
+    "    print(f\"\\n\u2705 Done! Results: {output_file}\")\n",
+    "    print(f\"Total time: {elapsed_total/60:.1f} minutes\")\n",
+    "    print(f\"Average speed: {len(df)/(elapsed_total/3600):.1f} samples/hour\")\n",
+    "    \n",
+    "    print(f\"\\nTop professions:\\n{df['profession_llm'].value_counts().head(10)}\")\n",
+    "\n",
+    "except KeyboardInterrupt:\n",
+    "    print(f\"\\n\u26a0\ufe0f Interrupted by user\")\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))\n",
+    "    print(f\"Progress saved up to row {current_index}\")\n",
+    "\n",
+    "except Exception as e:\n",
+    "    print(f\"\u26a0\ufe0f Error: {e}\")\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))\n",
+    "    print(f\"Progress saved up to row {current_index}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/TESTING_INSTRUCTIONS.md

@@ -0,0 +1,148 @@
+# Quick Testing Instructions
+
+## Start Here! 🚀
+
+You mentioned you have Deepseek credits, so **start by testing with Deepseek first** before trying the other LLMs.
+
+## Step-by-Step Testing
+
+### 1. Make sure your Deepseek API key is in place
+
+Check if this file exists:
+```bash
+cat misc/credentials/deepseek_api_key.txt
+```
+
+If not, create it:
+```bash
+echo "your-deepseek-api-key" > misc/credentials/deepseek_api_key.txt
+```
+
+### 2. Open the notebook
+
+```bash
+jupyter notebook jupyter_notebooks/Section_2-3-4_Figure_8_deepfake_adapters.ipynb
+```
+
+### 3. Run the cells in order
+
+1. **Cell 0-4**: Introduction and setup (just markdown, no execution needed)
+2. **Cell 5**: NER & Name Cleaning (processes `real_person_adapters.csv`)
+3. **Cell 7**: Country/Nationality Mapping
+4. **Cell 10**: 🌟 **DEEPSEEK ANNOTATION** (TEST THIS FIRST!)
+   - Default: `TEST_MODE = True` (10 samples)
+   - Will create: `data/CSV/deepseek_annotated_POI_test.csv`
+5. **Cell 12**: Qwen/Llama/Mistral (run later after Deepseek works)
+
+### 4. Review Deepseek Results
+
+After Cell 10 completes, check:
+- Console output shows summary statistics
+- Output file: `data/CSV/deepseek_annotated_POI_test.csv`
+
+Example output should look like:
+```
+✅ Progress saved after 10 rows
+✅ Done! Final results saved to data/CSV/deepseek_annotated_POI_test.csv
+
+=== Summary Statistics ===
+Total processed: 10
+
+Gender distribution:
+Female    8
+Male      2
+...
+```
+
+### 5. If Deepseek Works Well
+
+Once you're satisfied with the Deepseek results:
+
+**Option A: Process full dataset with Deepseek**
+```python
+# In Cell 10, change:
+TEST_MODE = False
+```
+
+**Option B: Try other LLMs for comparison**
+1. Set up API keys for Qwen/Llama/Mistral (see `misc/credentials/README.md`)
+2. Run Cell 12 with your chosen LLM:
+   ```python
+   SELECTED_LLM = 'qwen'  # or 'llama' or 'mistral'
+   TEST_MODE = True       # Test first!
+   ```
+
+## Expected Cost (Deepseek)
+
+- **10 samples** (test): ~$0.01 or less
+- **1,000 entries**: ~$0.10-0.20
+- **10,000 entries**: ~$1-2
+
+Much cheaper than the other options, making it perfect for testing!
+
+## Troubleshooting
+
+### "deepseek_api_key.txt not found"
+```bash
+# Create the file with your key
+echo "your-api-key" > misc/credentials/deepseek_api_key.txt
+```
+
+### "File does not exist: real_person_adapters.csv"
+Make sure the input file exists:
+```bash
+ls -lh data/CSV/real_person_adapters.csv
+```
+
+### API Rate Limiting
+The code includes automatic rate limiting (`time.sleep(1)` between requests). If you still get rate limited:
+- Increase the sleep time in Cell 10: change `time.sleep(1)` to `time.sleep(2)`
+
+### Pipeline Interrupted
+No problem! The code saves progress every 10 rows. Just re-run the cell and it will resume from where it left off.
+
+## What's Next?
+
+After testing with Deepseek:
+
+1. **If results look good**: Scale up to full dataset with Deepseek
+2. **Compare LLMs**: Test Qwen/Llama/Mistral on the same sample to see which gives best results
+3. **Production run**: Choose your preferred LLM and process the full dataset
+
+## File Outputs
+
+The pipeline creates these files:
+
+```
+data/CSV/
+├── NER_POI_step01_pre_annotation.csv       # After Cell 5 (name cleaning)
+├── NER_POI_step02_annotated.csv            # After Cell 7 (country mapping)
+├── deepseek_annotated_POI_test.csv         # After Cell 10 (test mode)
+├── deepseek_annotated_POI.csv              # After Cell 10 (full mode)
+├── qwen_annotated_POI_test.csv             # After Cell 12 (if using Qwen)
+└── ...
+
+misc/
+├── deepseek_query_index.txt                # Progress tracking
+└── ...
+```
+
+## Quick Commands
+
+```bash
+# View first few results
+head -20 data/CSV/deepseek_annotated_POI_test.csv
+
+# Count processed rows
+wc -l data/CSV/deepseek_annotated_POI_test.csv
+
+# Check progress
+cat misc/deepseek_query_index.txt
+
+# Reset progress (start from scratch)
+rm misc/deepseek_query_index.txt
+```
+
+---
+
+**Ready to start?** Open the notebook and run Cell 5 → Cell 7 → Cell 10! 🎉

jupyter_notebooks/UPDATES_AND_FIXES.md

@@ -0,0 +1,235 @@
+# Recent Updates and Fixes
+
+## Overview
+
+Two important fixes have been implemented based on testing feedback:
+
+1. **Leetspeak Translation** (before NER)
+2. **Improved Country Mapping** (check ALL tags)
+
+---
+
+## Fix 1: Leetspeak Translation
+
+### Problem
+Names with leetspeak (numbers replacing letters) weren't being properly cleaned:
+- `4kira` should be `Akira`
+- `1rene` should be `Irene`
+- `3mma` should be `Emma`
+
+### Solution
+Added leetspeak translation **before** other NER processing in Cell 5.
+
+### Mapping Table
+| Leetspeak | Letter |
+|-----------|--------|
+| 4 | A |
+| 3 | E |
+| 1 | I |
+| 0 | O |
+| 7 | T |
+| 5 | S |
+| 8 | B |
+| 9 | G |
+| @ | A |
+| $ | S |
+| ! | I |
+
+### Examples
+```
+4kira     -> akira
+3mma      -> emma
+1rene     -> irene
+L3vi      -> Levi
+S4sha     -> Sasha
+K4te      -> Kate
+J3ssica   -> Jessica
+```
+
+### Implementation
+The `translate_leetspeak()` function runs FIRST in `clean_name()`, before emoji removal and other cleaning steps. This ensures leetspeak is converted to proper letters before any other processing.
+
+---
+
+## Fix 2: Improved Country Mapping
+
+### Problem
+The country mapping was stopping at the first match, which meant:
+- **Irene** with tags `['girl', 'photorealistic', 'asian', 'woman', 'beautiful', 'celebrity', 'korean']`
+- The `'korean'` tag wasn't being properly mapped to `'South Korea'`
+- This resulted in incomplete hints being sent to the LLM
+- **Expected**: Deepseek should identify **Bae Joo-hyun (Irene)** from Red Velvet
+
+### Solution
+Updated Cell 7 to:
+1. **Check ALL tags** (not just stop at first match)
+2. **Use a priority system** to select the best match:
+   - Priority 3: Exact country name match (highest)
+   - Priority 2: Nationality match (medium)
+   - Priority 1: Word parts (lowest)
+
+### How It Works
+
+#### Before (Broken)
+```python
+def infer_country_and_nationality(tags):
+    for tag in tags:
+        if tag in mapping:
+            return mapping[tag]  # ❌ Stops at first match!
+    return ("", "")
+```
+
+#### After (Fixed)
+```python
+def infer_country_and_nationality(tags):
+    best_match = None
+    best_priority = 0
+
+    for tag in tags:  # ✅ Check ALL tags
+        if tag in mapping:
+            country, nationality, priority = mapping[tag]
+            if priority > best_priority:
+                best_match = (country, nationality)
+                best_priority = priority
+
+    return best_match or ("", "")
+```
+
+### Example: Irene Case
+
+**Input Tags**: `['girl', 'photorealistic', 'asian', 'woman', 'beautiful', 'celebrity', 'korean']`
+
+**Processing**:
+1. Check `'girl'` → no match
+2. Check `'photorealistic'` → no match
+3. Check `'asian'` → no match (too generic)
+4. Check `'woman'` → no match
+5. Check `'beautiful'` → no match
+6. Check `'celebrity'` → no match
+7. Check `'korean'` → ✅ **MATCH!**
+   - Maps to nationality: `'South Korean'`
+   - Which maps to country: `'South Korea'`
+   - Priority: 2 (nationality match)
+
+**Output**:
+- `likely_country`: `'South Korea'`
+- `likely_nationality`: `'South Korean'`
+
+**Sent to Deepseek**:
+```
+Given 'Irene' (celebrity, South Korea), provide:
+1. Full legal name
+2. Aliases
+3. Gender
+4. Top 3 professions
+5. Country
+```
+
+**Expected Result**: Deepseek can now identify this as **Bae Joo-hyun (Irene)**, a South Korean singer/actress from the K-pop group Red Velvet.
+
+---
+
+## Impact on Results
+
+### Better Name Recognition
+- Leetspeak names are now properly translated
+- LLMs receive cleaner, more recognizable names
+
+### Better Country Context
+- All tags are now considered for country mapping
+- More accurate country/nationality hints sent to LLMs
+- Better identification of international celebrities
+
+### Example Improvements
+
+| Name | Tags | Before | After |
+|------|------|--------|-------|
+| `4kira LoRA` | `['japanese', 'actress']` | `'4kira'` + no country | `'Akira'` + `'Japan'` |
+| `Irene` | `['korean', 'celebrity']` | `'Irene'` + no country | `'Irene'` + `'South Korea'` |
+| `1U` | `['korean', 'singer']` | `'1U'` + no country | `'IU'` + `'South Korea'` |
+| `3lsa` | `['model']` | `'3lsa'` + no country | `'Elsa'` + country if tagged |
+
+---
+
+## Testing Recommendations
+
+### Before Running Full Pipeline
+
+1. **Test Leetspeak Translation** (Cell 5):
+   ```python
+   # Look for names with numbers in the output
+   # Verify they're properly translated
+   ```
+
+2. **Test Country Mapping** (Cell 7):
+   ```python
+   # Check the debug output at the end:
+   # "🔍 Checking 'Irene' entries:"
+   # Verify country is properly mapped
+   ```
+
+3. **Test Deepseek Results** (Cell 10):
+   ```python
+   # Look for Irene in the results
+   # Should now identify as Bae Joo-hyun
+   ```
+
+### Validation Checklist
+
+- [ ] Leetspeak names are translated (check console output in Cell 5)
+- [ ] Country mapping shows high success rate (check stats in Cell 7)
+- [ ] Irene is correctly identified as Bae Joo-hyun (check results in Cell 10)
+- [ ] Other K-pop/Korean celebrities are properly identified
+- [ ] Japanese/Chinese celebrities also benefit from better country mapping
+
+---
+
+## Notes
+
+### Why Check ALL Tags?
+
+Some entries have many tags, and the most informative tag might not be first:
+```
+tags = ['girl', 'sexy', 'beautiful', 'asian', 'korean', 'celebrity', 'kpop']
+                                                  ^^^^ Most informative!
+```
+
+The old code might stop at `'girl'` or `'asian'` (no country info), missing the `'korean'` tag.
+
+### Why Use Priority?
+
+Some tags might match multiple countries. Priority ensures we get the best match:
+- `'american'` → exact nationality match (priority 2) → USA
+- `'america'` → could be North/South/Central America (priority 1)
+
+The system picks the higher priority match.
+
+### Word Length Filter
+
+Word parts only match if >4 characters to avoid false positives:
+- ✅ `'china'` → matches China (5 chars)
+- ❌ `'us'` → too short, might be part of other words
+
+---
+
+## Future Improvements
+
+Potential enhancements:
+1. **More leetspeak patterns**: `|\/|` for M, `(_)` for U, etc.
+2. **Fuzzy country matching**: Handle typos like `'corean'` → `'korean'`
+3. **Multi-country support**: Some celebrities work in multiple countries
+4. **Language detection**: Use name structure to infer origin
+
+---
+
+## Summary
+
+✅ **Leetspeak translation** ensures names are readable before NER
+✅ **ALL tags checked** ensures no country hints are missed
+✅ **Priority system** ensures best match is selected
+✅ **Better LLM results** from improved name quality and country context
+
+These fixes should significantly improve the accuracy of person identification, especially for:
+- International celebrities (K-pop, J-pop, C-pop)
+- Names with leetspeak
+- Entries where country info appears later in tag list

misc/deepseek_query_index.txt

@@ -1 +1 @@
-4
+10

misc/lists/PROFESSION_CATEGORIES.md

@@ -0,0 +1,255 @@
+# Profession Categories for Deepfake Adapter Classification
+
+## Overview
+
+The LLM annotation uses **9 specific profession categories** to classify people found in deepfake adapter datasets. These categories cover the most common types of public figures targeted by deepfake technology.
+
+## The 9 Categories
+
+### 1. **actor**
+Film, TV, and theater performers who primarily work in scripted dramatic content.
+
+**Examples:**
+- Emma Watson (film actor)
+- Scarlett Johansson (film actor)
+- Bryan Cranston (TV actor)
+
+**Keywords:** actor, actress, film, movie, cinema, theatrical, performer, drama
+
+---
+
+### 2. **adult performer**
+People working in the adult entertainment industry.
+
+**Examples:**
+- Adult film actors
+- OnlyFans creators
+- Cam models
+
+**Keywords:** adult entertainer, onlyfans, cam model, camgirl, webcam, pornographic actor, porn, pornstar
+
+**Note:** This category is important for research on unauthorized deepfake usage, which disproportionately affects adult performers.
+
+---
+
+### 3. **singer/musician**
+Vocalists, instrumentalists, and music performers across all genres.
+
+**Examples:**
+- IU (K-pop singer)
+- Taylor Swift (singer/songwriter)
+- DJ Khaled (DJ/producer)
+
+**Keywords:** singer, musician, rapper, rap artist, band, vocalist, songwriter, composer, DJ, producer, kpop, jpop
+
+---
+
+### 4. **model**
+Fashion, runway, and photoshoot models.
+
+**Examples:**
+- Gigi Hadid (fashion model)
+- Tyra Banks (supermodel)
+- Ashley Graham (plus-size model)
+
+**Keywords:** model, fashion, runway, photoshoot, supermodel
+
+---
+
+### 5. **online personality**
+Digital content creators, streamers, and influencers.
+
+**Includes:**
+- Streamers (Twitch, YouTube)
+- Cosplayers
+- YouTubers
+- Instagram influencers
+- Content creators
+- E-girls/E-boys
+- Gaming personalities
+
+**Examples:**
+- Belle Delphine (online personality, cosplayer)
+- Pokimane (Twitch streamer)
+- MrBeast (YouTuber)
+
+**Keywords:** influencer, streamer, twitch, youtuber, youtube, content creator, instagrammer, instagram, e-girl, egirl, e-boy, eboy, cosplayer, gamer
+
+---
+
+### 6. **public figure**
+Politicians, activists, journalists, authors, and other public-facing professionals not in entertainment.
+
+**Examples:**
+- Barack Obama (politician)
+- Greta Thunberg (activist)
+- Gordon Ramsay (chef)
+- J.K. Rowling (author)
+
+**Keywords:** politician, activist, public speaker, journalist, author, writer, chef, famous, celebrity (when not in entertainment)
+
+---
+
+### 7. **voice actor/ASMR**
+Voice performers for animation, games, and ASMR content creators.
+
+**Examples:**
+- Tara Strong (voice actress)
+- Troy Baker (voice actor)
+- ASMR artists
+
+**Keywords:** voice actor, voice actress, asmr creator, asmr
+
+---
+
+### 8. **sports professional**
+Professional athletes and sports competitors.
+
+**Examples:**
+- Cristiano Ronaldo (soccer player)
+- Serena Williams (tennis player)
+- LeBron James (basketball player)
+
+**Keywords:** athlete, sports, player, professional, competitor, olympian
+
+---
+
+### 9. **tv personality**
+TV hosts, presenters, reality TV stars, and broadcast personalities.
+
+**Examples:**
+- Oprah Winfrey (talk show host)
+- Kim Kardashian (reality TV)
+- Jimmy Fallon (late night host)
+
+**Keywords:** tv host, tv moderator, talk show host, talkshow, radio host, media personality, reality tv star, reality, comedian, presenter, broadcaster, anchor
+
+---
+
+## Multi-Category Classification
+
+Many public figures work across multiple categories. The LLM can assign **up to 3 professions** per person, ordered by relevance.
+
+### Examples:
+
+| Person | Professions | Explanation |
+|--------|-------------|-------------|
+| **Emma Watson** | `actor, public figure` | Primarily an actor, but also known for activism |
+| **IU (Lee Ji-eun)** | `singer/musician, actor` | K-pop singer who also acts in TV dramas |
+| **Belle Delphine** | `online personality, adult performer` | Internet personality with adult content |
+| **Jamie Foxx** | `actor, singer/musician` | Actor who also has a music career |
+| **Dwayne Johnson** | `actor, sports professional` | Former wrestler, now primarily an actor |
+
+## Category Selection Guidelines
+
+### For the LLM:
+
+1. **Choose most specific category first**
+   - ✅ `"actor"` for film performers
+   - ❌ `"public figure"` for someone primarily known as an actor
+
+2. **Order by relevance**
+   - Most important role first
+   - Secondary roles after
+   - Maximum 3 categories
+
+3. **Be inclusive for online personalities**
+   - Streamers → `"online personality"`
+   - Cosplayers → `"online personality"`
+   - Influencers → `"online personality"`
+
+4. **Distinguish TV personalities from actors**
+   - ✅ `"tv personality"` for talk show hosts
+   - ✅ `"actor"` for scripted TV drama performers
+   - Some can be both!
+
+## Why These 9 Categories?
+
+These categories were chosen based on:
+
+1. **Common targets of deepfake technology**
+   - Celebrities and public figures are most frequently deepfaked
+   - Adult performers are disproportionately affected
+
+2. **Clear distinctions**
+   - Each category represents a distinct professional domain
+   - Minimal overlap between categories
+
+3. **Research relevance**
+   - Important for analyzing demographic patterns in deepfake usage
+   - Helps understand which professions are most at risk
+
+4. **Comprehensive coverage**
+   - Covers the vast majority of people in the deepfake adapter dataset
+   - Includes both traditional and digital-native celebrities
+
+## Output Format
+
+The LLM returns professions as a **comma-separated list**:
+
+```
+"singer/musician, actor"
+"online personality"
+"actor, public figure, online personality"
+"sports professional"
+```
+
+## Validation
+
+After annotation, we can analyze:
+- Distribution across categories
+- Multi-category patterns
+- Correlation with countries/regions
+- Demographic patterns
+
+Example analysis:
+```python
+# Count by category
+df['profession_llm'].value_counts()
+
+# Most common combinations
+df['profession_llm'].value_counts().head(20)
+
+# Filter by specific category
+actors = df[df['profession_llm'].str.contains('actor', na=False)]
+```
+
+## Edge Cases
+
+### Fictional Characters
+**Handling:** Return `"Unknown"` for all fields
+
+**Example:**
+- Input: `"Elsa from Frozen"`
+- Output: `profession_llm: "Unknown"`
+
+### Unclear/Ambiguous Cases
+**Handling:** Use best judgment based on primary public recognition
+
+**Example:**
+- Input: `"Elon Musk"`
+- Primary recognition: Business/technology
+- Category: `"public figure"`
+
+### Multiple Equally Important Roles
+**Handling:** List all relevant categories (up to 3)
+
+**Example:**
+- Input: `"Donald Glover / Childish Gambino"`
+- Categories: `"actor, singer/musician, tv personality"`
+
+## Notes for Researchers
+
+When analyzing the annotated data:
+
+1. **Category frequency** indicates which professions are most targeted
+2. **Multi-category entries** show crossover between industries
+3. **Online personality prevalence** indicates rise of digital-native celebrities in deepfakes
+4. **Adult performer numbers** highlight ongoing issues with non-consensual deepfakes
+
+## Version History
+
+- **v1.0** (Current): Initial 9-category system
+  - Refined from original detailed profession list
+  - Focused on clear, distinct categories
+  - Added comprehensive documentation

misc/lists/professions_mapped.csv

@@ -0,0 +1,10 @@
+category,subcategory,keywords
+actor,actor,actor actress film movie cinema theatrical performer drama
+adult performer,adult performer,adult entertainer onlyfans creator cam model camgirl webcam pornographic actor porn pornstar adult
+singer/musician,singer/musician,singer musician rapper rap artist band vocalist songwriter composer DJ producer kpop jpop
+model,model,model fashion runway photoshoot supermodel
+online personality,online personality,influencer streamer twitch youtuber youtube content creator instagrammer instagram insta e-girl egirl e-boy eboy cosplayer gamer
+public figure,public figure,celebrity celeb politician activist public speaker journalist author writer chef famous
+voice actor/ASMR,voice actor/ASMR,voice actor voice actress asmr creator asmr
+sports professional,sports professional,athlete sports player professional competitor olympian
+tv personality,tv personality,tv host tv moderator talk show host talkshow radio host media personality reality tv star reality comedian presenter broadcaster anchor

misc/qwen_query_index.txt

@@ -0,0 +1 @@
+10