template-tagger v0.0.1

Status: proof of concept. Do not put this in production.

This is a first public version, published so the approach and the numbers are visible and so anyone who wants to push it further has a starting point. It works, it is honest about what it does, and it is not accurate enough to run unattended over real contracts.

A LoRA adapter for Qwen3.5-4B that takes a document plus a list of template tags and says which tag goes where. It never re-emits the document.

It exists because filling in a template by hand is tedious and error prone: a purchase agreement has thirty fields, each behind its own label, and the person importing the document has to find every one of them again.

What it does

Input: a tag list, each field tag with the label it follows, and a document.

TAGS
- voornaam (label: Voornamen:)
- achternaam (label: Achternaam:)
- wil_pp_kopen_ja (blok)

DOCUMENT
Naam: Jan de Vries
Achternaam: De Vries

Output: one line per tag, the tag name and the anchor it follows.

voornaam	Naam:
achternaam	Achternaam:

The caller finds each anchor in the document and inserts the rendered tag after it. The document text is therefore the input, byte for byte, and cannot drift: the model never writes prose. An anchor that is not found is reported as unplaced rather than guessed at.

Rules the model was trained on:

  • only tag names from the list, without delimiters; the caller renders them, so the delimiter style is a setting (${}, $[], $<>, {{}}, <<>>, [], {} and any other pair);
  • tags in the list that do not belong in the document are left out;
  • a block tag appears twice, once as the name and once with a / in front;
  • the anchor is the tail of the line the tag follows, at most 40 characters, copied from the document.

How to use

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-4B", torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "MeerDevelopment/template-tagger-qwen3.5-4b-lora")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")

The merged bf16 weights are in the merged/ folder of this same repo if you would rather not load PEFT:

model = AutoModelForCausalLM.from_pretrained("MeerDevelopment/template-tagger-qwen3.5-4b-lora", subfolder="merged", torch_dtype=torch.bfloat16)

Use enable_thinking=False. Documents longer than about 1200 characters should be split at paragraph boundaries and processed window by window; the tag list travels with every window.

Evaluation

A tag counts as correct when the tag name matches the gold and the insertion point is within two characters. text intact is byte equality of the document after stripping the tags.

run examples precision recall F1 text intact
base Qwen3.5-4B, real contracts 26 0.000 0.000 0.000 100%
this adapter, real contracts 26 0.238 0.238 0.238 92.3%
this adapter, realistic synthetic 167 0.850 0.834 0.842 95.2%
this adapter, synthetic 128 0.647 0.629 0.638 75.0%

The base model scores zero because it cannot produce the answer format at all; every point above is learned.

The gap between 0.842 on realistic synthetic and 0.238 on real contracts is the honest headline of this release. The synthetic documents are generated, the real ones are scanned two-column forms with their own conventions, and the model does not generalise across that gap yet. Where it fails on real contracts, in order:

outcome share of gold tags
correct 28%
plain field named as its _partner variant, or the reverse 34%
tag not emitted at all 16%
right position, wrong tag name 12%
right tag, position more than two characters off 10%

The partner confusion is the largest single failure. Dutch contracts mark the second person's block with a line En and then repeat the same labels; the synthetic corpus mostly wrote the word "partner" into the label instead. The model learned the synthetic convention.

A dropped tag is worse than it looks: one omission shifts every following tag in a repeated form up by one slot, so a single miss can turn six correct tags into six wrong ones.

Training data

469 synthetic documents and 3964 training windows, generated from specs built out of the tag inventory and document skeletons of 18 real Dutch templates. Five generator models contributed: Qwen3.8 Flash, DeepSeek V4.1 Flash, GLM 5.3, Claude Opus 5.5 and Claude Sonnet 5.5.

Every generated document passed a PII gate (email, phone, postcode, IBAN, BSN, chamber-of-commerce number, plus a whitelist of fictional values), an 8-gram overlap check against the real documents, and a structural validator.

The real templates themselves are not in this release and were not trained on; they are only used as an evaluation set. The synthetic corpus is not published with this model.

Limits

  • Not production ready. 24% F1 on real contracts is not enough to run unattended. Expect to correct roughly three of every four tags.
  • Dutch is the main language; English works but the corpus is about 20 percent English.
  • The domain is Dutch new-build property paperwork: purchase agreements, construction contracts, guarantees, handover letters, payment reminders. Other domains will degrade.
  • Partner blocks and repeated forms are the weak point, for the reason above.
  • Block tags are the second weak point: the model finds where a conditional section starts more reliably than where it ends, and it sometimes wraps the block content in stray newlines.
  • It inserts tags, it does not decide which tags a document needs. The tag list is the contract.
  • Trained in bf16. No quantised build was evaluated.

What would improve it

In the order the evidence supports:

  1. More real, hand-verified contracts in training. There is exactly one real document in the training set and it is a blank form; everything else is synthetic.
  2. Synthetic documents that follow the real partner convention (a line En, then the same labels again) instead of writing "partner" into the label.
  3. Per-character boundary labelling rather than windowed generate-and-place, which is the only thing that would touch the localisation errors.

License

Apache 2.0, matching the base model.

Downloads last month
41
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for MeerDevelopment/template-tagger-qwen3.5-4b-lora

Finetuned
Qwen/Qwen3.5-4B
Adapter
(1,222)
this model