| # Essay and IELTS writing coach (Labs build #6, started 2026-10-26) |
|
|
| Paste an essay, get back: six skill scores, the argument parts tagged (lead, position, claims, evidence, counterclaim, |
| rebuttal, conclusion), weak parts flagged, and a band. Build #5 on the agreed list; picked up after outfit swap was paused. |
|
|
| ## What the page does |
| 1. You paste your essay (and optionally the question it answers). |
| 2. Six scores from 1 to 5: cohesion, syntax, vocabulary, phraseology, grammar, conventions. Shown as a radar, with the |
| weakest two called out in plain words. |
| 3. The essay comes back with its argument parts coloured, and each part marked effective / adequate / ineffective. That |
| is where the "weak sentences flagged" feedback comes from: an ineffective claim with no evidence after it, a missing |
| position, no counterclaim, a lead that is one sentence long. |
| 4. A band: an IELTS-style overall band for Task 2 essays. Honest about its limits (see data). |
|
|
| ## Data (all public, all human-scored except where said) |
| - ELLIPSE corpus (Crossley et al.): ~6.5k essays by English language learners, grades 8-12, overall + the six analytic |
| scores, CC BY-NC-SA 4.0. The Kaggle "Feedback Prize - English Language Learning" train set (3.9k) is a subset. |
| - PERSUADE 2.0 corpus: 25k argumentative essays, holistic score 1-6, every argument part annotated with its type and its |
| effectiveness. CC BY-NC-SA 4.0. The Kaggle Feedback Prize 2021 / Effectiveness / AES 2.0 data come from it. |
| - IELTS: "IELTS Writing Scored Essays Dataset" (Kaggle, mazlumi), ~1,200 essays with an overall band and an examiner |
| comment, licence "other", origin not stated. A second set (xntrng15) has criterion scores produced by GPT-5 Mini, so |
| it is not ground truth; at most a sanity check. The band head is therefore small-data and will be labelled as an |
| estimate on the page. |
| - Licence consequence: CC BY-NC-SA means the model weights are shared under the same licence and the page is |
| non-commercial. That is fine for Labs. |
|
|
| ## Recipe (from the Kaggle winners, cut down to one small GPU) |
| - One backbone: deberta-v3-base (large if the budget allows), max length 1024, newline kept as a special token |
| (every winner found DeBERTa's tokenizer drops "\n"; +0.01). |
| - Head A, six scores: regression, MCRMSE, dropout off, layer-wise lr decay, AWP optional. Winners' single |
| deberta-v3-large: ~0.45 MCRMSE; base ~0.46-0.47. Target: ~0.47 on a held-out ELLIPSE fold. |
| - Head B, argument parts: token classification over the essay (BIO over 7 types), decode with "repair broken spans, |
| one lead/position/conclusion per essay". Winners' single model ~0.70 F1 (span overlap); target ~0.65. |
| - Head C, effectiveness: whole essay in once, special start/end token per part, one vector per part -> 3 classes |
| (the "essay group model" every top team converged on). Target log loss ~0.60 single model. |
| - Band: a small model on top of the six scores + part statistics + length, fit on the 1.2k IELTS essays with |
| cross-validation; report the within-half-band accuracy honestly. |
| - Budget: each head is one or two hours on an A10G at ~$1.10/h. Whole build well under $10 of the $30 on labs3. |
|
|
| ## Where things are |
| - Code: this folder (`coach/` model + data, `scripts/` Modal jobs). Data: Modal volume `essay-data` (labs3). |
| - Runs and weights: HF nonsodev/essay-coach (CC BY-NC-SA, public is fine). |
|
|
| ## Results |
| - scores-v1 (2026-10-27): deberta-v3-base, 4 epochs, 24 min on an A10G. MCRMSE 0.446 on the 2,571 ELLIPSE test essays |
| (per trait 0.41-0.48). The Kaggle ELL winners' single deberta-v3-large was ~0.45, so this is at that level. |
| - Serving: scripts/serve.py is a CPU web endpoint on Modal (scales to zero); web/index.html takes ?api=<url>. |
| - parts-v1 (2026-10-27): deberta-v3-base token model, 3 epochs on 24k PERSUADE 2.0 essays, 103 min on an A10G. On 2,000 |
| held-out essays: argument-part F1 0.656 (Feedback Prize 2021 metric; winners' single models ~0.70), effectiveness |
| log loss 0.478 on gold parts (not the competition's test set, so only roughly comparable to their 0.57-0.60). |
| - band-v1: ridge on the six raw scores + part counts + length, 5-fold CV on 1,435 IELTS essays: RMSE 0.79 bands, |
| within half a band 67%, within one band 89%, correlation 0.66 (predicting the mean would be RMSE 1.06). Shown on the |
| page as an estimate with that error rate. |
| - Endpoint: https://sylvayoutube--essay-coach-server-read.us-east.modal.run (CPU, ~0.8 s per essay). |
|
|
