filippogreppi's picture
README: drop the project name, exact typed F1 for the first phase
1dfacde verified
|
Raw History Blame Contribute Delete
4.1 kB
metadata
base_model: Qwen/Qwen3-4B-Thinking-2507
library_name: peft
tags:
  - named-entity-recognition
  - chain-of-thought
  - lora
language:
  - en
pipeline_tag: text-generation
datasets:
  - disi-unibo-nlp-students/pilener-teacher-cot-agreement-v1

Qwen3-4B-Thinking-2507 路 Structured Reasoning for NER

Part of the code, data and models of the bachelor thesis Structuring Universal Chain-of-thought for Named Entity Recognition (Department of Computer Science and Engineering, University of Bologna, 2026). All the repositories cited by name belong to the same organization.

Second phase of the project: a teacher reasons twice on the same queries, freely and in a structured format, and students that differ only in the format of their reasoning are trained on the traces. This is the structured student; the free one is qwen3-4b-thinking-free-cot-lora.

Given a text and a list of entity types (some present, some absent), the model reasons inside its <think> block with three one-line fields: TYPES (which requested types occur), MENTIONS (mentions to extract and candidates rejected) and EDGE (ambiguous boundaries or type assignments), then answers with a JSON object that maps every requested type to the mentions copied verbatim from the text (an empty list for an absent type).

Training

  • Base model: Qwen/Qwen3-4B-Thinking-2507.
  • Data: the structured_cot traces of disi-unibo-nlp-students/pilener-teacher-cot-agreement-v1 (train, 3,822 queries), each with its own prompt (structured_prompt); target: the teacher's answer (answer). The free and structured traces of a query lead to the same answer, so all the students see the same queries and the same targets.
  • Method: LoRA r 16, alpha 32, dropout 0.05 on the q, k, v, o, gate, up and down projections; lr 1e-4, cosine, warmup 3%; 2 epochs; batch 1 x 8 gradient accumulation; bf16; maximum sequence length 5,120; loss on the assistant turn only; 5% validation, best checkpoint by validation loss; seed 42.

Evaluation

Metric: micro F1 with exact, case-insensitive mention matching per type, over the requested types; reasoning length in tokens (generated before the end of the thinking block), mean per query. Queries: every gold type plus 1-3 distractors of the same domain, fixed seed (free-vs-structured-cot-eval). vLLM, T 0.6, top-p 0.95, top-k 20, seed 0, up to 12,288 new tokens. Base = the same model without fine-tuning, with the free prompt.

CrossNER (full test sets)

Set Base F1 Free F1 Structured F1 Base tokens Free tokens Structured tokens
AI 0.635 0.686 0.652 3,903 469 325
Literature 0.665 0.733 0.700 4,866 563 415
Music 0.755 0.789 0.764 5,010 583 392
Politics 0.703 0.714 0.694 5,477 650 436
Science 0.720 0.765 0.749 4,644 536 363
Mean 0.696 0.737 0.712 4,780 560 386

Pile-NER test split (in-domain, 2,000 queries)

Set Base F1 Free F1 Structured F1 Base tokens Free tokens Structured tokens
Pile-NER 0.537 0.573 0.562 7,313 1,029 564

Biomedical benchmarks (500 sentences each, 465 for JNLPBA-Rare)

Set Base F1 Free F1 Structured F1 Base tokens Free tokens Structured tokens
NCBI 0.594 0.699 0.620 1,889 233 229
BC5CDR 0.733 0.765 0.738 2,571 311 246
BC2GM 0.433 0.521 0.462 1,696 257 226
BC4CHEMD 0.594 0.582 0.607 2,565 291 307
AnatEM 0.263 0.382 0.310 1,318 292 267
JNLPBA 0.513 0.569 0.563 3,013 397 280
JNLPBA-Rare 0.219 0.463 0.509 2,475 336 238
MedMentions-Rare 0.546 0.567 0.585 2,726 426 291
BioRED 0.707 0.706 0.674 3,420 522 328
Mean 0.511 0.584 0.563 2,408 341 268

All predictions and metrics: eval-results (free-vs-structured-cot/ and free-vs-structured-cot-bio/).