---
base_model: mistralai/Mistral-7B-Instruct-v0.2
library_name: peft
pipeline_tag: text-generation
license: apache-2.0
language:
- en
datasets:
- Kamyar-zeinalipour/Essay-Syntax-Instruct
tags:
- education
- writing-feedback
- syntax
- lora
- peft
---

<div align="center">

# Mistral7B-Syntax-Instruct

**Mistral-7B-Instruct-v0.2 fine-tuned to give students feedback on essay syntax**

[![Paper](https://img.shields.io/badge/AIEER_2024-paper-b31b1b)](https://arxiv.org/abs/2501.07740) [![GitHub](https://img.shields.io/badge/GitHub-Essay--Syntax--Feedback-181717?logo=github)](https://github.com/KamyarZeinalipour/Essay-Syntax-Feedback) [![Collection](https://img.shields.io/badge/%F0%9F%A4%97_Collection-Syntax_Feedback-orange)](https://huggingface.co/collections/Kamyar-zeinalipour/automated-syntax-feedback-aieer-2024-6af487599293af6e63c227f9)

</div>

Reads an English student essay and returns structured feedback on its syntax across eight
categories — spelling, conjunctions and connectives, modifiers, prepositions, modal verbs,
punctuation, articles and pronouns — with examples from the essay and suggestions for each.
A LoRA adapter for [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) trained on
[Essay-Syntax-Instruct](https://huggingface.co/datasets/Kamyar-zeinalipour/Essay-Syntax-Instruct).

From [*Advancing Student Writing Through Automated Syntax Feedback*](https://arxiv.org/abs/2501.07740) (AIEER 2024). Code: [Essay-Syntax-Feedback](https://github.com/KamyarZeinalipour/Essay-Syntax-Feedback).

## Results

ROUGE against the reference feedback, base → fine-tuned:

| Model | ROUGE-1 | ROUGE-2 | ROUGE-L |
|---|---|---|---|
| Llama-2-7b-chat | 0.351 → **0.477** | 0.146 → **0.337** | 0.185 → **0.343** |
| Llama-2-13b-chat | 0.328 → **0.476** | 0.148 → **0.340** | 0.214 → **0.347** |
| Mistral-7B-Instruct-v0.2 | 0.366 → **0.514** | 0.144 → **0.375** | 0.205 → **0.369** |

## Usage

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "Kamyar-zeinalipour/Mistral7B-Syntax-Instruct").merge_and_unload()
```

## Training

LoRA r = 32, α = 64, dropout 0.1; 3 epochs; AdamW, learning rate 1e-4 with cosine decay;
8 sequences per GPU × 4 accumulation steps; up to 2,048 tokens; bf16, DeepSpeed.
Script: [`scripts/`](https://github.com/KamyarZeinalipour/Essay-Syntax-Feedback/tree/main/scripts).

## Models from this paper

| Base model | Adapter | ROUGE-L (base → fine-tuned) |
|---|---|---|
| [Llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | [Llama2-7B-Syntax-Instruct](https://huggingface.co/Kamyar-zeinalipour/Llama2-7B-Syntax-Instruct) | 0.185 → **0.343** |
| [Llama-2-13b-chat](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) | [Llama2-13B-Syntax-Instruct](https://huggingface.co/Kamyar-zeinalipour/Llama2-13B-Syntax-Instruct) | 0.214 → **0.347** |
| [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) | **Mistral7B-Syntax-Instruct** (this model) | 0.205 → **0.369** |

## Citation

```bibtex
@inproceedings{zeinalipour2024advancing,
  title     = {Advancing Student Writing Through Automated Syntax Feedback},
  author    = {Kamyar Zeinalipour and Mehak Mehak and Fatemeh Parsamotamed and Marco Maggini and Marco Gori},
  booktitle = {AI in Education and Educational Research (AIEER 2024)},
  pages     = {52--66},
  year      = {2024},
  publisher = {Springer},
  doi       = {10.1007/978-3-031-93409-4_5},
  url       = {https://arxiv.org/abs/2501.07740}
}
```

## License

Released under the base model's license (`apache-2.0`).