| ---
|
| license: apache-2.0
|
| language:
|
| - en
|
| tags:
|
| - text-to-sql
|
| - nl2sql
|
| - bird
|
| - sqlite
|
| - darwin
|
| pipeline_tag: text-generation
|
| base_model: FINAL-Bench/Darwin-180B-RSI
|
| ---
|
|
|
| # Darwin-180B-NL2SQL
|
|
|
| <p align="center">
|
| <img src="https://img.shields.io/badge/BIRD_Mini--Dev_SQLite-83.80_EX-gold?style=for-the-badge" alt="BIRD Mini-Dev SQLite 83.80">
|
| <a href="https://huggingface.co/FINAL-Bench/Darwin-180B-RSI"><img src="https://img.shields.io/badge/Model-Darwin--180B--RSI-blue?style=for-the-badge" alt="Darwin-180B-RSI"></a>
|
| <a href="https://huggingface.co/FINAL-Bench"><img src="https://img.shields.io/badge/Org-FINAL--Bench-black?style=for-the-badge" alt="FINAL-Bench"></a>
|
| </p>
|
|
|
| Darwin-180B-NL2SQL is VIDRAFT's text-to-SQL system built on [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI). It turns a natural-language question into a SQLite query, runs it against the database and revises the query from what the execution shows, then returns one final query per question.
|
|
|
| This repository holds the system's predictions on BIRD Mini-Dev (SQLite) and the script used to score them.
|
|
|
| > **Weights:** the model is [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI), unchanged. Download the weights from that repository; no weights were changed or trained for this result.
|
|
|
| ```python
|
| from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
| repo = "FINAL-Bench/Darwin-180B-RSI" # weights for Darwin-180B-NL2SQL
|
| tok = AutoTokenizer.from_pretrained(repo)
|
| model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
|
| ```
|
|
|
| ## Result
|
|
|
| **BIRD Mini-Dev, SQLite, 500 questions: 83.80% execution accuracy (419 / 500).**
|
|
|
| | Difficulty | Correct | Accuracy |
|
| |---|---|---|
|
| | Challenging | 384 / 460 | 83.5% |
|
| | Moderate | 29 / 32 | 90.6% |
|
| | Simple | 6 / 8 | 75.0% |
|
| | **All** | **419 / 500** | **83.80%** |
|
|
|
| - Data: the current canonical release on Hugging Face, [`birdsql/bird_mini_dev`](https://huggingface.co/datasets/birdsql/bird_mini_dev), release "Mini-Dev 1024", `data/mini_dev_sqlite.jsonl` (sha256 `926af2a0d1b47e5d0d15a7aa8a34d4721007f4550a881555b6a89496b54a2606`), with the SQLite databases from `minidev.zip`.
|
| - Metric: strict execution accuracy. A prediction is correct when the set of rows it returns equals the set of rows the reference SQL returns. Each predicted query may run for 60 seconds.
|
| - Measured on 29 October 2026 (KST). The score was computed twice on an idle machine and was identical both times.
|
|
|
| ## Evaluation conditions
|
|
|
| | Condition | Setting |
|
| |---|---|
|
| | Reference (gold) SQL used at any stage of inference | **No** |
|
| | Oracle knowledge beyond the question and evidence given | No |
|
| | Fine-tuning on BIRD | No |
|
| | Final candidates per question | 1 |
|
| | Database access during inference | Read-only execution of the system's own queries |
|
|
|
| The question and evidence in each item are the only task inputs. The reference SQL was used only to score the final predictions.
|
|
|
| ## Files
|
|
|
| | File | Contents |
|
| |---|---|
|
| | `predict_mini_dev_darwin-180b-nl2sql_SQLite.json` | Predictions in the BIRD Mini-Dev submission format (`SQL\t----- bird -----\tdb_id`, keyed by question ID) |
|
| | `predictions.jsonl` | The same predictions, one JSON object per line |
|
| | `score_ex.py` | Strict execution-accuracy scorer used for the numbers above |
|
|
|
| To reproduce the score, place `mini_dev_sqlite.jsonl` next to `score_ex.py` and run:
|
|
|
| ```bash
|
| python score_ex.py predictions.jsonl path/to/MINIDEV/dev_databases
|
| ```
|
|
|
| ## About the system
|
|
|
| The system pairs Darwin-180B-RSI with VIDRAFT's NL2SQL verification engine, which checks each generated query against the live database before an answer is returned. The engine is VIDRAFT's own implementation and is not released.
|
|
|
| A public demo of VIDRAFT's NL2SQL engine (Korean and English, read-only, synthetic data) is available at [final-bench-nl2sql-demo.hf.space](https://final-bench-nl2sql-demo.hf.space/).
|
|
|
| ## Contact
|
|
|
| VIDRAFT · [vidraft.net](https://vidraft.net) · vidraft@mail.vidraft.net
|
|
|
