| --- |
| base_model: mistralai/Mistral-7B-Instruct-v0.2 |
| library_name: peft |
| pipeline_tag: text-generation |
| license: apache-2.0 |
| language: |
| - en |
| datasets: |
| - Kamyar-zeinalipour/Essay-Syntax-Instruct |
| tags: |
| - education |
| - writing-feedback |
| - syntax |
| - lora |
| - peft |
| --- |
| |
| <div align="center"> |
|
|
| # Mistral7B-Syntax-Instruct |
|
|
| **Mistral-7B-Instruct-v0.2 fine-tuned to give students feedback on essay syntax** |
|
|
| [](https://arxiv.org/abs/2501.07740) [](https://github.com/KamyarZeinalipour/Essay-Syntax-Feedback) [](https://huggingface.co/collections/Kamyar-zeinalipour/automated-syntax-feedback-aieer-2024-6af487599293af6e63c227f9) |
|
|
| </div> |
|
|
| Reads an English student essay and returns structured feedback on its syntax across eight |
| categories β spelling, conjunctions and connectives, modifiers, prepositions, modal verbs, |
| punctuation, articles and pronouns β with examples from the essay and suggestions for each. |
| A LoRA adapter for [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) trained on |
| [Essay-Syntax-Instruct](https://huggingface.co/datasets/Kamyar-zeinalipour/Essay-Syntax-Instruct). |
|
|
| From [*Advancing Student Writing Through Automated Syntax Feedback*](https://arxiv.org/abs/2501.07740) (AIEER 2024). Code: [Essay-Syntax-Feedback](https://github.com/KamyarZeinalipour/Essay-Syntax-Feedback). |
|
|
| ## Results |
|
|
| ROUGE against the reference feedback, base β fine-tuned: |
|
|
| | Model | ROUGE-1 | ROUGE-2 | ROUGE-L | |
| |---|---|---|---| |
| | Llama-2-7b-chat | 0.351 β **0.477** | 0.146 β **0.337** | 0.185 β **0.343** | |
| | Llama-2-13b-chat | 0.328 β **0.476** | 0.148 β **0.340** | 0.214 β **0.347** | |
| | Mistral-7B-Instruct-v0.2 | 0.366 β **0.514** | 0.144 β **0.375** | 0.205 β **0.369** | |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from peft import PeftModel |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2") |
| model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", torch_dtype=torch.bfloat16, device_map="auto") |
| model = PeftModel.from_pretrained(model, "Kamyar-zeinalipour/Mistral7B-Syntax-Instruct").merge_and_unload() |
| ``` |
|
|
| ## Training |
|
|
| LoRA r = 32, Ξ± = 64, dropout 0.1; 3 epochs; AdamW, learning rate 1e-4 with cosine decay; |
| 8 sequences per GPU Γ 4 accumulation steps; up to 2,048 tokens; bf16, DeepSpeed. |
| Script: [`scripts/`](https://github.com/KamyarZeinalipour/Essay-Syntax-Feedback/tree/main/scripts). |
|
|
| ## Models from this paper |
|
|
| | Base model | Adapter | ROUGE-L (base β fine-tuned) | |
| |---|---|---| |
| | [Llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | [Llama2-7B-Syntax-Instruct](https://huggingface.co/Kamyar-zeinalipour/Llama2-7B-Syntax-Instruct) | 0.185 β **0.343** | |
| | [Llama-2-13b-chat](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) | [Llama2-13B-Syntax-Instruct](https://huggingface.co/Kamyar-zeinalipour/Llama2-13B-Syntax-Instruct) | 0.214 β **0.347** | |
| | [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) | **Mistral7B-Syntax-Instruct** (this model) | 0.205 β **0.369** | |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{zeinalipour2024advancing, |
| title = {Advancing Student Writing Through Automated Syntax Feedback}, |
| author = {Kamyar Zeinalipour and Mehak Mehak and Fatemeh Parsamotamed and Marco Maggini and Marco Gori}, |
| booktitle = {AI in Education and Educational Research (AIEER 2024)}, |
| pages = {52--66}, |
| year = {2024}, |
| publisher = {Springer}, |
| doi = {10.1007/978-3-031-93409-4_5}, |
| url = {https://arxiv.org/abs/2501.07740} |
| } |
| ``` |
|
|
| ## License |
|
|
| Released under the base model's license (`apache-2.0`). |
|
|
