oguzY/text-to-sql-qwen3-0.6B

Text-to-SQL Qwen3-0.6B

LoRA adapter fine-tuned from unsloth/Qwen3-0.6B on gretelai/synthetic_text_to_sql. Given a database schema and a natural-language question, the model is trained to return JSON with sql and explanation fields. Outputs are not guaranteed correct or safe to execute.

This repository publishes checkpoint-3750 (step 3,750, epoch 3.0). It contains adapter weights, tokenizer, chat template, and training state. Load the adapter with the base model; the full base model weights are not included.

Training

The supplied training notebook selects 25,000 training examples and up to 2,500 test examples with seed 42. It uses LoRA rank 16, alpha 16, dropout 0, learning rate 1e-4, and FP16 training. The adapter targets q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. The run was configured for 10 epochs; this release is the epoch-3 checkpoint.

Trained on the default Qwen3 chat template with thinking enabled. Use the same template during inference. The training notebook appends the JSON target directly after the generation prompt; it does not supply explicit reasoning traces.

Usage

Install PyTorch, Transformers with Qwen3 support (transformers>=4.51.0), PEFT compatible with the saved adapter configuration (created with PEFT 0.20.0), and Accelerate.

import json
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "oguzY/text-to-sql-qwen3-0.6B"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Qwen3-0.6B",
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, repo_id)
model.eval()

content = json.dumps({
    "context": "CREATE TABLE employees (id INTEGER, name TEXT, salary INTEGER);",
    "prompt": "Find the names of employees earning more than 50000."
})
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": content}],
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )
response = tokenizer.decode(
    output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True
)
print(response)
# Inspect the response before parsing. If present, the JSON follows </think>.

The intended response format is {"sql": "...", "explanation": "..."}. Validate the response format and SQL before use.

Recorded evaluation

The supplied benchmark notebook records the following results on a custom 70-question SQLite benchmark:

Model	Valid JSON	Executable SQL	Correct execution result
Qwen3-0.6B base	2/70	15/70	7/70
checkpoint-3750 (this release)	70/70	17/70	4/70
checkpoint-5000 (comparison only)	70/70	13/70	3/70

These are saved notebook outputs, not a newly reproduced evaluation. The benchmark question/fixture package is not included in the supplied files. The base model receives an additional JSON-format instruction, while adapters use the training input format, so prompting differs. Generation is capped at 512 new tokens. These results show stronger JSON formatting for the adapter, but do not show improved SQL correctness over the base model on this benchmark.

Files and limitations

• adapter_model.safetensors and adapter_config.json: LoRA adapter.
• Tokenizer files and chat_template.jinja: inference formatting.
• train.ipynb and benchmark.ipynb: supplied notebooks with saved outputs. Their original /content/ paths and external inputs may need adjustment. The benchmark also references checkpoint-5000, which is not part of this release.
• Optimizer, scheduler, RNG, trainer state, and training arguments: original checkpoint files for training continuity.

The model may generate invalid SQL, incorrect joins, unsupported SQL dialects, or incorrect explanations. Review generated queries and use a restricted, read-only database environment when evaluating them.