Darwin-180B-NL2SQL: BIRD Mini-Dev SQLite 83.80 EX predictions and scorer

README.md

@@ -0,0 +1,75 @@
+---
+license: apache-2.0
+language:
+  - en
+tags:
+  - text-to-sql
+  - nl2sql
+  - bird
+  - sqlite
+  - darwin
+pipeline_tag: text-generation
+---
+
+# Darwin-180B-NL2SQL
+
+<p align="center">
+  <img src="https://img.shields.io/badge/BIRD_Mini--Dev_SQLite-83.80_EX-gold?style=for-the-badge" alt="BIRD Mini-Dev SQLite 83.80">
+  <a href="https://huggingface.co/FINAL-Bench/Darwin-180B-RSI"><img src="https://img.shields.io/badge/Model-Darwin--180B--RSI-blue?style=for-the-badge" alt="Darwin-180B-RSI"></a>
+  <a href="https://huggingface.co/FINAL-Bench"><img src="https://img.shields.io/badge/Org-FINAL--Bench-black?style=for-the-badge" alt="FINAL-Bench"></a>
+</p>
+
+Darwin-180B-NL2SQL is VIDRAFT's text-to-SQL system built on [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI). It turns a natural-language question into a SQLite query, runs it against the database and revises the query from what the execution shows, then returns one final query per question.
+
+This repository holds the system's predictions on BIRD Mini-Dev (SQLite) and the script used to score them. The model weights are those of Darwin-180B-RSI; no weights were changed or trained for this result.
+
+## Result
+
+**BIRD Mini-Dev, SQLite, 500 questions: 83.80% execution accuracy (419 / 500).**
+
+| Difficulty | Correct | Accuracy |
+|---|---|---|
+| Challenging | 384 / 460 | 83.5% |
+| Moderate | 29 / 32 | 90.6% |
+| Simple | 6 / 8 | 75.0% |
+| **All** | **419 / 500** | **83.80%** |
+
+- Data: the current canonical release on Hugging Face, [`birdsql/bird_mini_dev`](https://huggingface.co/datasets/birdsql/bird_mini_dev), release "Mini-Dev 1024", `data/mini_dev_sqlite.jsonl` (sha256 `926af2a0d1b47e5d0d15a7aa8a34d4721007f4550a881555b6a89496b54a2606`), with the SQLite databases from `minidev.zip`.
+- Metric: strict execution accuracy. A prediction is correct when the set of rows it returns equals the set of rows the reference SQL returns. Each predicted query may run for 60 seconds.
+- Measured on 29 October 2026 (KST). The score was computed twice on an idle machine and was identical both times.
+
+## Evaluation conditions
+
+| Condition | Setting |
+|---|---|
+| Reference (gold) SQL used at any stage of inference | **No** |
+| Oracle knowledge beyond the question and evidence given | No |
+| Fine-tuning on BIRD | No |
+| Final candidates per question | 1 |
+| Database access during inference | Read-only execution of the system's own queries |
+
+The question and evidence in each item are the only task inputs. The reference SQL was used only to score the final predictions.
+
+## Files
+
+| File | Contents |
+|---|---|
+| `predict_mini_dev_darwin-180b-nl2sql_SQLite.json` | Predictions in the BIRD Mini-Dev submission format (`SQL\t----- bird -----\tdb_id`, keyed by question ID) |
+| `predictions.jsonl` | The same predictions, one JSON object per line |
+| `score_ex.py` | Strict execution-accuracy scorer used for the numbers above |
+
+To reproduce the score, place `mini_dev_sqlite.jsonl` next to `score_ex.py` and run:
+
+```bash
+python score_ex.py predictions.jsonl path/to/MINIDEV/dev_databases
+```
+
+## About the system
+
+The system pairs Darwin-180B-RSI with VIDRAFT's NL2SQL verification engine, which checks each generated query against the live database before an answer is returned. The engine is VIDRAFT's own implementation and is not released.
+
+A public demo of VIDRAFT's NL2SQL engine (Korean and English, read-only, synthetic data) is available at [final-bench-nl2sql-demo.hf.space](https://final-bench-nl2sql-demo.hf.space/).
+
+## Contact
+
+VIDRAFT · [vidraft.net](https://vidraft.net) · vidraft@mail.vidraft.net

score_ex.py

@@ -0,0 +1,56 @@
+# Strict BIRD execution accuracy: set(pred rows) == set(gold rows). usage: python score.py preds.jsonl db_root
+import json, os, sqlite3, sys, collections, time
+from concurrent.futures import ProcessPoolExecutor, TimeoutError as TO
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+preds_path, db_root = sys.argv[1], sys.argv[2]
+gold = {d["question_id"]: d for d in (json.loads(l) for l in open(os.path.join(HERE, "mini_dev_sqlite.jsonl"), encoding="utf-8"))}
+preds = {}
+for l in open(preds_path, encoding="utf-8"):
+    p = json.loads(l); preds[p["question_id"]] = p
+
+
+def run(db, sql, limit=None):
+    con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
+    if limit:  # abort runaway queries inside the worker; a pool timeout alone leaves them running
+        end = time.time() + limit
+        con.set_progress_handler(lambda: time.time() > end, 100000)
+    try:
+        return set(con.execute(sql).fetchall())
+    finally:
+        con.close()
+
+
+def judge(args):
+    db, psql, gsql = args
+    try:
+        g = run(db, gsql)
+    except Exception as e:
+        return "gold_error"
+    if not psql: return "no_sql"
+    try:
+        return "ok" if run(db, psql, 60) == g else "wrong"
+    except sqlite3.OperationalError as e:
+        return "timeout" if "interrupted" in str(e) else "exec_error"
+    except Exception:
+        return "exec_error"
+
+
+if __name__ == "__main__":
+    jobs = []
+    for qid, g in gold.items():
+        db = os.path.join(db_root, g["db_id"], g["db_id"] + ".sqlite")
+        jobs.append((qid, (db, preds.get(qid, {}).get("sql", ""), g["SQL"])))
+    res = {}
+    with ProcessPoolExecutor(8) as ex:
+        futs = {qid: ex.submit(judge, a) for qid, a in jobs}
+        for qid, fu in futs.items():
+            try: res[qid] = fu.result(timeout=600)  # the 60 s per-query limit is enforced inside the worker; this only guards a hung pool
+            except TO: res[qid] = "timeout"
+    c = collections.Counter(res.values())
+    by = collections.defaultdict(lambda: [0, 0])
+    for qid, v in res.items():
+        d = gold[qid]["difficulty"]; by[d][1] += 1; by[d][0] += v == "ok"
+    print(f"EX = {c['ok']}/{len(res)} = {100 * c['ok'] / len(res):.2f}%   ", dict(c))
+    for d, (a, n) in sorted(by.items()): print(f"  {d:12s} {a}/{n} = {100 * a / n:.1f}%")
+    json.dump(res, open(preds_path.replace(".jsonl", "_score.json"), "w"), indent=0)