Add model card

README.md

@@ -1,199 +1,190 @@
 ---
+license: bsd-3-clause
+base_model: Salesforce/codet5p-220m
 library_name: transformers
-tags: []
+pipeline_tag: text2text-generation
+language:
+- en
+datasets:
+- gretelai/synthetic_text_to_sql
+tags:
+- sql
+- sql-to-text
+- code
+- codet5p
+- text2text-generation
+model-index:
+- name: codet5p-sql2text
+  results:
+  - task:
+      type: text2text-generation
+      name: SQL-to-Text
+    dataset:
+      name: gretelai/synthetic_text_to_sql
+      type: gretelai/synthetic_text_to_sql
+      split: test
+    metrics:
+    - type: bleu
+      name: BLEU
+      value: 33.1003
+    - type: rouge1
+      name: ROUGE-1
+      value: 66.8589
+    - type: rouge2
+      name: ROUGE-2
+      value: 45.1124
+    - type: rougel
+      name: ROUGE-L
+      value: 57.0609
 ---
 
-# Model Card for Model ID
+# SQL-to-Text (Salesforce/codet5p-220m)
 
-<!-- Provide a quick summary of what the model is/does. -->
+`Salesforce/codet5p-220m` fine-tuned to explain a SQL query in plain English.
 
+The direction is **SQL -> natural language**: the model takes a query (and,
+optionally, the DDL of the tables it touches) and returns a sentence describing
+what that query does. It does *not* generate SQL from a question.
 
+## Prompt format
 
-## Model Details
+Inputs follow one fixed template; training, evaluation and inference all build
+it with the same function, so they cannot drift apart. The schema block is
+dropped when no DDL is supplied, and when it is supplied only `CREATE TABLE ...` statements are kept.
 
-### Model Description
+```
+Explain the following SQL query.
 
-<!-- Provide a longer summary of what this model is. -->
+Schema:
+<CREATE TABLE statements>
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+SQL:
+<the query>
+```
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+## Usage
 
-### Model Sources [optional]
+```python
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
 
-<!-- Provide the basic links for the model. -->
+model_id = "thealper2/codet5p-sql2text"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+schema = "CREATE TABLE employees (id INT, name TEXT, salary INT, dept_id INT);"
+sql = "SELECT dept_id, AVG(salary) FROM employees GROUP BY dept_id;"
+prompt = f"Explain the following SQL query.\n\nSchema:\n{schema}\n\nSQL:\n{sql}"
 
-## Uses
+inputs = tokenizer(
+    prompt,
+    return_tensors="pt",
+    truncation=True,
+    max_length=256,
+)
+outputs = model.generate(
+    **inputs,
+    num_beams=4,
+    max_new_tokens=128,
+    min_new_tokens=5,
+    early_stopping=True,
+)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+## Training data
 
-### Direct Use
+[`gretelai/synthetic_text_to_sql`](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql), mapping
+`sql` + `sql_context`
+to `sql_explanation`.
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+Preprocessing drops rows that are too short to be a real explanation, removes
+exact duplicates and duplicate *inputs*, and removes any training row whose
+input also appears in the official test split, so the reported test scores are
+not inflated by leakage. The validation split is
+3% of the cleaned train
+split (seed 42).
 
-[More Information Needed]
+Sequence lengths were chosen from the measured token-length distribution:
+source 256 tokens, target
+128 tokens.
 
-### Downstream Use [optional]
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+## Training procedure
 
-[More Information Needed]
+| Hyper-parameter | Value |
+| --- | --- |
+| Epochs | 3.00 |
+| Learning rate | 0.0001 |
+| LR schedule | linear |
+| Warmup ratio | 0.0500 |
+| Weight decay | 0.0100 |
+| Optimiser | adamw_torch |
+| Per-device train batch size | 16 |
+| Gradient accumulation | 4 |
+| Max gradient norm | 1.00 |
+| Model selection | eval_rougeL |
+| Seed | 42 |
+| Effective batch size | 64 |
 
-### Out-of-Scope Use
+Trained on a single NVIDIA GeForce RTX 5060 Ti (15.9 GB) with torch 2.11.0+cu128, bf16 mixed precision.
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
+Wall-clock training time: 84 minutes.
 
 ## Evaluation
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+Scored by `evaluate.py` on the full splits with beam search (num_beams=4).
+
+### Generation quality
+
+| Metric | Validation | Test | _meta |
+| --- | --- | --- | --- |
+| Examples | 2989 | 5850 | - |
+| BLEU | 33.65 | 33.10 | - |
+| ROUGE-1 | 67.15 | 66.86 | - |
+| ROUGE-2 | 45.76 | 45.11 | - |
+| ROUGE-L | 57.58 | 57.06 | - |
+| Mean generated length | 36.37 | 35.69 | - |
+| Loss | 0.5829 | 0.5923 | - |
+
+### SQL-aware faithfulness
+
+Recall metrics ask whether the explanation mentions what the query actually does; the *rate* metrics are error rates, where lower is better -- they measure claims the query does not support.
+
+| Metric | Validation | Test |
+| --- | --- | --- |
+| Examples | 2989 | 5850 |
+| Operation recall | 98.53 | 98.40 |
+| Aggregation recall | 98.72 | 98.57 |
+| Join mention recall | 98.21 | 98.74 |
+| Join table coverage | 98.34 | 98.56 |
+| Condition column coverage | 82.98 | 81.99 |
+| Condition value coverage | 86.18 | 87.24 |
+| Operation over-claim rate | 5.47 | 5.29 |
+| Unsupported number rate | 3.98 | 3.18 |
+| Unsupported quoted-string rate | 3.20 | 3.18 |
+| Unsupported entity rate | 1.38 | 1.72 |
+
+## Limitations
+
+* Trained on synthetic queries and synthetic explanations, so the phrasing
+  reflects that generator's style rather than how a particular team documents
+  its own queries.
+* Explanations are grounded in the query text, not in the data: the model
+  cannot know what a column means beyond its name.
+* Condition coverage is the weakest area -- long `WHERE` clauses lose some
+  columns and literals -- so an explanation may describe a filter less
+  precisely than the query applies it. Do not rely on it as an audit of what a
+  query returns.
+* Inputs are truncated past the configured source length, so very large schemas
+  are only partially visible to the model.
+* English only.
+
+## Reproduction
+
+```bash
+make preprocess
+make train
+make evaluate
+```
+
+Base model: [`Salesforce/codet5p-220m`](https://huggingface.co/Salesforce/codet5p-220m).