filippogreppi's picture
README: drop the project name, exact typed F1 for the first phase
6a9f941 verified
|
Raw History Blame Contribute Delete
4.05 kB
---
base_model: HuggingFaceTB/SmolLM3-3B
library_name: peft
tags: [named-entity-recognition, chain-of-thought, lora]
language: [en]
pipeline_tag: text-generation
datasets: [disi-unibo-nlp-students/pilener-teacher-cot-agreement-v1]
---
# SmolLM3-3B · Structured Reasoning for NER
Part of the code, data and models of the bachelor thesis *Structuring Universal Chain-of-thought for Named Entity Recognition* (Department of Computer Science and Engineering, University of Bologna, 2026). All the repositories cited by name belong to the same organization.
**Second phase** of the project: a teacher reasons twice on the same queries, freely and in a structured format, and
students that differ only in the format of their reasoning are trained on the traces. This is the **structured** student; the free one is `smollm3-3b-free-cot-lora`.
Given a text and a list of entity types (some present, some absent), the model reasons inside its `<think>` block
with three one-line fields: `TYPES` (which requested types occur), `MENTIONS` (mentions to extract and candidates rejected) and `EDGE` (ambiguous boundaries or type assignments), then answers with a JSON object that maps every requested type to the mentions copied verbatim from
the text (an empty list for an absent type).
## Training
- **Base model**: `HuggingFaceTB/SmolLM3-3B`.
- **Data**: the `structured_cot` traces of `disi-unibo-nlp-students/pilener-teacher-cot-agreement-v1` (train, 3,822 queries), each with its own prompt (`structured_prompt`); target: the teacher's answer (`answer`). The free and structured
traces of a query lead to the same answer, so all the students see the same queries and the same targets.
- **Method**: LoRA r 16, alpha 32, dropout 0.05 on the q, k, v, o, gate, up and down projections; lr 1e-4, cosine,
warmup 3%; 2 epochs; batch 1 x 8 gradient accumulation; bf16; maximum sequence length 5,120;
loss on the assistant turn only; 5% validation, best checkpoint by validation loss; seed 42.
## Evaluation
Metric: micro F1 with exact, case-insensitive mention matching per type, over the requested types; reasoning length
in tokens (generated before the end of the thinking block), mean per query. Queries: every gold type plus 1-3 distractors of the same domain, fixed seed
(`free-vs-structured-cot-eval`). vLLM, T 0.6, top-p 0.95, top-k 20, seed 0, up to 12,288 new tokens.
Base = the same model without fine-tuning, with the free prompt.
**CrossNER** (full test sets)
| Set | Base F1 | Free F1 | Structured F1 | Base tokens | Free tokens | Structured tokens |
|---|---|---|---|---|---|---|
| AI | 0.565 | 0.674 | 0.660 | 699 | 433 | 351 |
| Literature | 0.597 | 0.707 | 0.694 | 919 | 495 | 377 |
| Music | 0.677 | 0.756 | 0.735 | 849 | 517 | 363 |
| Politics | 0.664 | 0.713 | 0.690 | 974 | 568 | 429 |
| Science | 0.661 | 0.734 | 0.733 | 826 | 478 | 352 |
| **Mean** | **0.633** | **0.717** | **0.703** | **853** | **498** | **374** |
**Pile-NER test split** (in-domain, 2,000 queries)
| Set | Base F1 | Free F1 | Structured F1 | Base tokens | Free tokens | Structured tokens |
|---|---|---|---|---|---|---|
| Pile-NER | 0.406 | 0.539 | 0.536 | 1,551 | 1,097 | 724 |
**Biomedical benchmarks** (500 sentences each, 465 for JNLPBA-Rare)
| Set | Base F1 | Free F1 | Structured F1 | Base tokens | Free tokens | Structured tokens |
|---|---|---|---|---|---|---|
| NCBI | 0.507 | 0.636 | 0.602 | 431 | 221 | 239 |
| BC5CDR | 0.651 | 0.775 | 0.773 | 485 | 287 | 227 |
| BC2GM | 0.381 | 0.481 | 0.482 | 534 | 238 | 232 |
| BC4CHEMD | 0.448 | 0.576 | 0.572 | 494 | 284 | 307 |
| AnatEM | 0.346 | 0.362 | 0.376 | 484 | 251 | 248 |
| JNLPBA | 0.506 | 0.539 | 0.516 | 514 | 362 | 286 |
| JNLPBA-Rare | 0.448 | 0.429 | 0.415 | 449 | 301 | 226 |
| MedMentions-Rare | 0.466 | 0.564 | 0.544 | 587 | 452 | 288 |
| BioRED | 0.531 | 0.670 | 0.640 | 656 | 469 | 332 |
| **Mean** | **0.476** | **0.559** | **0.547** | **515** | **318** | **265** |
All predictions and metrics: `eval-results` (`free-vs-structured-cot/` and `free-vs-structured-cot-bio/`).