Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
4d0a4e9
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add Kaggle silver training data, retrain model, reorganize data directory
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
ae7305b
- Add 2,483 Kaggle resumes with Gemini-extracted labels + BIO tagging + noise augmentation
- Retrain model (25 epochs): entity F1 97.77%, clean F1 99.18%
- Reorganize training/data/ into sources/, gold/ subdirectories
- Add implementation guide (docs/implementation_guide.md)
- Remove one-off scripts (build_long_resumes, convert_dataturks, label_resume_pdfs)
- Fix Windows UTF-8 encoding in structured_postprocess and train_ner
- Add --extra-train flag to train_ner.py for loading additional training files
- Export updated ONNX + quantized model
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Best model: noise-augmented training, ONNX export, README update
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
4553d68
Model trained with 3x noise augmentation (separator swaps, char
corruption, case changes) for OCR robustness. 25 epochs on 10941
examples (3647 clean + 7294 augmented).
Benchmark results (best across all rounds):
- Val micro F1: 98.16%
- Val noisy F1: 72.67% (was 60.91% at baseline)
- Long resume micro F1: 91.69% (was 38.5% at baseline)
ONNX validation:
- PT vs ONNX: allclose=True, argmax_equal=True
README updated with current metrics, pre/post-processing docs,
training data sources, and repo layout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract city_country_map to separate file with 317 cities
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
a05717f
Moved from inline resume_config.json (54 entries) to dedicated
city_country_map.json (317 entries) organized by region. Added coverage
for India (45 cities), US (80+), UK (25), Canada, Germany, France,
Australia, East/Southeast Asia, Middle East, and broader Europe.
Post-processor loads from file path in config, falls back to inline map.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Make text preprocessor config-driven via resume_config.json
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
e6b7b6f
All preprocessing rules now in pre_processing config section:
- dash_replacements: em-dash → hyphen mappings
- bullet_chars + bullet_replacement: bullet normalization
- strip_labels: remove "Phone:", "Email:" prefixes
- skill_table_categories: known category keywords for table expansion
- table_prose_max_words: threshold to skip prose lines
- table_continuation_max_chars: max length for continuation lines
ResumeTextPreprocessor class reads config on init. Any language
implementation can apply same rules from the JSON config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add noise augmentation for OCR robustness training
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
1652579
Generates 2 noisy variants per training example by applying:
- Separator swaps (| ↔ + ↔ - ↔ ● at 20% per token)
- Character corruption on O-tagged tokens (5%) and entities (3%)
- Token merging to simulate missing spaces (3%)
- Random case changes on non-entity tokens (5%)
- Random line break insertion (2%)
Training data grows from 3647 → 10941 with augmentation.
BIO tag alignment preserved through all transforms.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add DATE config and widen entity punctuation cleanup
Somasundaram Ayyappan
03116a3
Add section detection for hybrid NER entity extraction
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
613cc9b
Rule-based section header detection identifies SKILLS, EXPERIENCE,
EDUCATION, CERTIFICATIONS, LANGUAGES, and PROJECTS sections.
Fills entities the NER model missed using section context:
- Skills: extracts from bullet/comma/dash/pipe-separated lists
- Certifications: extracts from cert section lines
- Languages: extracts language names from language section
Tested results:
- Muthu resume: 23 → 38 skills (added Docker, Kubernetes, Jenkins, etc.)
- Accounting resume: 0 → 14 skills (was completely missing)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix gold labels: 100% match rate (was 88.8%)
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
b5f1c8a
- Removed 194 hallucinated annotations that didn't exist in extracted text
- Corrected 33 annotations with wrong spacing/quotes
- Fixed exbc16/exbc17 key swap (annotations were under wrong resume)
- Match rate: 2325/2325 (100%)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add text preprocessor for PDF extraction artifacts
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
b34f048
Normalizes CRLF, em-dashes, bullets, phone/email labels, and detects
flattened two-column skill tables using known category keywords.
Converts 'Category Skill1, Skill2' to 'Category: Skill1, Skill2'.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 93 gold-labeled resume resource training examples
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
1fcd140
Hand-annotated by Claude (not model-predicted) from 98 resume PDFs.
2542 total entities, 2196 matched (86.4%). Compared to silver labels:
- Skills: 229 → 3293 (14x improvement)
- Dates: 174 → 581 (3.3x)
- Companies: 330 → 504 (1.5x)
- Languages: 0 → 22 (new)
Replaces silver-labeled resume_resource_labeled.json in training pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Retrain model with 98 real-world PDF resumes and export ONNX
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
64f9fd5
Training: 3673 examples (4000 synthetic + 200 DataTurks + 12 manual +
50 long resumes + 98 resume-resource PDFs). Best entity F1: 97.44%.
Benchmark:
- Val micro F1: 98.13% (stable)
- Long resume micro F1: 85.7% (up from 38.5% baseline)
ONNX validation:
- PT vs ONNX: allclose=True, argmax_equal=True
- PT vs Quantized: 1 token diff on punctuation (acceptable)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Make post-processing fully config-driven from resume_config.json
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
a56f909
Every tunable parameter now lives in resume_config.json so any language
implementation can apply identical post-processing rules:
- entity_rules: per-entity min_length, exceptions, blocked_words,
uppercase_bypass, gazetteer_bypass, aliases, reject_patterns
- date_words, present_words for date parsing
- space_collapse_pairs for token cleanup
- title_company_separators for splitting "Title at Company"
- company_gazetteer_match_max_words for prefix matching
- max_experience_months for sanity checking
- country_name_aliases for location inference
- seniority_by_experience_count for fallback seniority
No hardcoded domain values remain in Python — all behavior is
configurable via the JSON file.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move post-processing rules from Python to resume_config.json
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
cdacc8d
Externalizes all tunable post-processing parameters so any language
consumer (JS/TS, etc.) can apply the same rules from config:
- min_lengths for COMPANY, TITLE, SKILL
- proficiency_words filter list
- skill_aliases and cert_aliases normalization maps
- span_merge_max_gap and span_merge_labels
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove hand-picked short skills allowlist for generic post-processing
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
f37acde
Use uppercase-only gate for short skills (<=3 chars) instead of
maintaining a curated allowlist. Trades off Go/R/C detection for
fully generic rules with no benchmark-specific tuning.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Improve post-processing and add 98 real-world resume training data
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
a11ed90
Post-processing fixes:
- Drop short COMPANY fragments (<=3 chars) not in gazetteer
- Drop short SKILL fragments (<=3 chars lowercase) with allowlist for
known short skills (Go, R, C, Bun, Vue, dbt, etc.)
- Reject URLs misclassified as EMAIL (// or www. without @)
- Filter language proficiency words from SKILL (Native, Fluent, etc.)
- Drop TITLE spans <=2 chars
Training data:
- Add 98 silver-labeled resumes from resume-resource.com PDFs
- Wire resume_resource_labeled.json into training pipeline
- Add label_resume_pdfs.py for PDF extraction and auto-labeling
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 50 long dense resume training examples and new synthetic formats
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
efba759
- 50 hand-crafted resumes (38/50 exceed 512 subword tokens, avg 549)
- Diverse: 15+ industries, 10+ countries, dense OCR-style formatting
- Uses + and | separators, bullet points, PROJECTS/LANGUAGES sections
- 3 new synthetic formats (F, G, H) with dense layouts
- Remove 512-token truncation from generated data
- Wire long resumes into training data pipeline
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add section-aware chunked inference for resumes exceeding 512 tokens
Somasundaram Ayyappan+1: Claude Opus 4.6 (1M context)
f10912e
Splits text at paragraph boundaries (double newlines) and packs sections
into chunks that fit within the model's context window. Character offsets
are mapped back to the original text. Falls back to single-pass for
short inputs. Benchmark uses chunked inference — no regression on val set.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Track model weights with Git LFS
Somasundaram Ayyappan
0d6cd8e
Improve structured benchmark analysis and robustness
Somasundaram Ayyappan
4129d85
Clean up training pipeline and add export benchmarks
