SmolLM3-3B · Free Reasoning for NER

Part of the code, data and models of the bachelor thesis Structuring Universal Chain-of-thought for Named Entity Recognition (Department of Computer Science and Engineering, University of Bologna, 2026). All the repositories cited by name belong to the same organization.

Second phase of the project: a teacher reasons twice on the same queries, freely and in a structured format, and students that differ only in the format of their reasoning are trained on the traces. This is the free student; the structured one is smollm3-3b-structured-cot-lora.

Given a text and a list of entity types (some present, some absent), the model reasons inside its <think> block with its own unconstrained thinking, then answers with a JSON object that maps every requested type to the mentions copied verbatim from the text (an empty list for an absent type).

Training

  • Base model: HuggingFaceTB/SmolLM3-3B.
  • Data: the free_cot traces of disi-unibo-nlp-students/pilener-teacher-cot-agreement-v1 (train, 3,822 queries), each with its own prompt (free_prompt); target: the teacher's answer (answer). The free and structured traces of a query lead to the same answer, so all the students see the same queries and the same targets.
  • Method: LoRA r 16, alpha 32, dropout 0.05 on the q, k, v, o, gate, up and down projections; lr 1e-4, cosine, warmup 3%; 2 epochs; batch 1 x 8 gradient accumulation; bf16; maximum sequence length 5,120; loss on the assistant turn only; 5% validation, best checkpoint by validation loss; seed 42.

Evaluation

Metric: micro F1 with exact, case-insensitive mention matching per type, over the requested types; reasoning length in tokens (generated before the end of the thinking block), mean per query. Queries: every gold type plus 1-3 distractors of the same domain, fixed seed (free-vs-structured-cot-eval). vLLM, T 0.6, top-p 0.95, top-k 20, seed 0, up to 12,288 new tokens. Base = the same model without fine-tuning, with the free prompt.

CrossNER (full test sets)

Set Base F1 Free F1 Structured F1 Base tokens Free tokens Structured tokens
AI 0.565 0.674 0.660 699 433 351
Literature 0.597 0.707 0.694 919 495 377
Music 0.677 0.756 0.735 849 517 363
Politics 0.664 0.713 0.690 974 568 429
Science 0.661 0.734 0.733 826 478 352
Mean 0.633 0.717 0.703 853 498 374

Pile-NER test split (in-domain, 2,000 queries)

Set Base F1 Free F1 Structured F1 Base tokens Free tokens Structured tokens
Pile-NER 0.406 0.539 0.536 1,551 1,097 724

Biomedical benchmarks (500 sentences each, 465 for JNLPBA-Rare)

Set Base F1 Free F1 Structured F1 Base tokens Free tokens Structured tokens
NCBI 0.507 0.636 0.602 431 221 239
BC5CDR 0.651 0.775 0.773 485 287 227
BC2GM 0.381 0.481 0.482 534 238 232
BC4CHEMD 0.448 0.576 0.572 494 284 307
AnatEM 0.346 0.362 0.376 484 251 248
JNLPBA 0.506 0.539 0.516 514 362 286
JNLPBA-Rare 0.448 0.429 0.415 449 301 226
MedMentions-Rare 0.466 0.564 0.544 587 452 288
BioRED 0.531 0.670 0.640 656 469 332
Mean 0.476 0.559 0.547 515 318 265

All predictions and metrics: eval-results (free-vs-structured-cot/ and free-vs-structured-cot-bio/).

Downloads last month
10
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for structured-cot-ner/smollm3-3b-free-cot-lora

Adapter
(83)
this model