developmentseed/gazet-tutorial-qwen3.5-0.8b-lora

gazet-tutorial: Qwen3.5-0.8B LoRA for natural-language geocoding (India)

A LoRA adapter trained in the gazet-tutorial workshop, a teaching-sized reproduction of gazet. One 0.8B model learns two tasks over India's administrative boundaries (Overture Maps divisions: country, states and union territories, districts), distinguished by the system prompt:

1. extract the place names in a question as JSON, and
2. write DuckDB spatial SQL given a candidate table (from fuzzy search) and the question.

"which districts border Nagpur?"
   -> {"places": [{"place": "Nagpur"}]}
   -> fuzzy search returns candidate rows (Nagpur, Nagaur, Nagapattinam, ...)
   -> WITH a AS (SELECT geometry FROM read_parquet('divisions_area') WHERE id = '...')
      SELECT b.id, b.names."primary" AS name, b.subtype, ST_AsGeoJSON(b.geometry) AS geometry
      FROM read_parquet('divisions_area') AS b, a
      WHERE b.subtype = 'county' AND ST_Touches(a.geometry, b.geometry)

This is the workshop's reference run, the one notebook 2 reproduces with its defaults. The same model as a q8_0 GGUF for llama.cpp is at developmentseed/gazet-tutorial-qwen3.5-0.8b-GGUF.

Training

base model	unsloth/Qwen3.5-0.8B, text layers only
data	2,819 SQL + 2,819 extraction examples, synthetic, verified by execution (no LLM involved)
method	SFT with Unsloth + TRL, loss on the assistant turn only, thinking off (<think>\n\n</think>)
LoRA	r 16, alpha 32, dropout 0, attention + MLP projections
schedule	1 epoch, lr 2e-4 linear, effective batch 16, max length 1024
hardware	~20 min on a free Colab T4

The prompts, the dataset and the benchmark are in the GitHub repo under artifacts/ (bundled with pip install git+https://github.com/developmentseed/gazet-tutorial).

Benchmark

Execution accuracy on the workshop benchmark (predicted and reference SQL must return the same feature ids), 40 SQL + 20 extraction questions per split, next to the untouched base model and zero-shot gpt-oss:20b, the model gazet runs in production:

split	gpt-oss:20b SQL	base SQL	this adapter SQL	gpt-oss:20b F1	base F1	this adapter F1
iid	0.53	0.00	1.00	0.92	0.77	1.00
unseen_region	0.55	0.00	1.00	0.91	0.73	1.00
unseen_anchor	0.55	0.00	1.00	0.82	0.86	1.00
noisy	0.25	0.00	0.85	0.55	0.41	0.45
compositional	0.50	0.00	0.18	1.00	0.90	1.00
paraphrase	0.57	0.03	0.98	0.90	0.88	1.00

unseen_region is Kerala, Odisha and Goa, held out entirely from training. F1 is set precision/recall over extracted place names. See the repo README for how the splits are built.

Use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "unsloth/Qwen3.5-0.8B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "developmentseed/gazet-tutorial-qwen3.5-0.8b-lora")

messages = [
    {"role": "system", "content": PLACE_SYSTEM},   # artifacts/dataset/prompts.json in the repo
    {"role": "user", "content": "districts within 50 km of Pune"},
]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False,
                              return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=64, do_sample=False)[0, ids.shape[1]:], skip_special_tokens=True))

The full pipeline (extract, fuzzy search, SQL, execute in DuckDB, map) is in notebook 2 of the tutorial and in the repo's local playground app. Generate with thinking off and greedy decoding, as in training.

Limitations

• India only, eight query shapes (lookup, disambiguation, adjacency, containment, buffer, aggregation). Compositions of two operators are mostly beyond it (see compositional).
• Old names and abbreviations are copied rather than normalised (Orissa stays Orissa, UP stays UP), which is why noisy extraction is low. The training data never shows them.
• A toy for teaching, not a product. The production model is gazet.

Licenses

Adapter weights: Apache-2.0 (same as the base model). Tutorial code: MIT. The training data contains feature ids and names from Overture Maps divisions, derived from OpenStreetMap under the ODbL.