Merge the Space's 8-9 October work (origin/main) into the local changes

The Space had moved on (adaptive learning, detector-guided rewriting,
token-budget handling in groqai, shared integrity checks, proofread
changes). Conflicts in groqai, humanizer and textparse were resolved by
keeping both sides:

- groqai: the Space's token-budget chat loop, with the per-model
fallback on a rate limit added to it.
- humanizer: the Space's PARA_MODEL fallback inside the lazy
load_model(); the automatic deeper rewrite (BOOST) feeds the adaptive
candidate count and the plain-wording source.
- textparse: both sets of guard checks run; strict_check keeps its
rejection reasons and gains the Space's integrity checks.
- integrity_issue: "bracket type changed" no longer fires when an
author-year citation is dropped (it has no "["), so the guard can
re-attach the citation again.

53 unit tests pass; text and Word end-to-end runs complete on the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

README.md

@@ -75,7 +75,7 @@ Optional variables: `PAY_WHATSAPP` (digits with country code), `PAY_EMAIL`, `PRO
 | `APP_NAME` | variable (optional) | Header title (default "CS AI Writing Suite") |
 | `REPORT_BRAND` | variable (optional) | Header text on PDF report pages |
 | `MAX_PASTE_WORDS` | variable (optional) | Paste limit |
-| `DETECTOR_MODEL`, `PARROT_MODEL` | variable (optional) | Swap models |
+| `DETECTOR_MODEL`, `PARA_MODEL` | variable (optional) | Swap models (`PARA_MODEL` = paraphraser; the older name `PARROT_MODEL` still works) |
 | `USERS_REPO` / `USERS_REPO_TOKEN` | secret | Private dataset (e.g. `you/suite-users`) + write token: backs up accounts so they survive restarts |
 | `DAILY_DOC_LIMIT` | variable (optional) | Document runs per signed-in user per day (default 20, 0 = unlimited) |
 | `DAILY_TEXT_LIMIT` | variable (optional) | Pasted-text runs per signed-in user per day (default 100) |
@@ -261,7 +261,7 @@ Scores are indicators from an open-source detector and will differ from Turnitin
 
 ## Licences
 
-Parrot paraphraser: Apache-2.0 · RoBERTa AI detector: Apache-2.0.
+Paraphraser: humarin/chatgpt_paraphraser_on_T5_base, OpenRAIL (default) · Parrot paraphraser (fallback): Apache-2.0 · RoBERTa AI detector: Apache-2.0.
 
 ## Final grammar check (Groq)
 After all rewriting is finished, Groq corrects grammar, spelling and punctuation only (no rewording). A correction is
@@ -308,3 +308,78 @@ paragraph as a whole reads as AI-written again (`POLISH_KEEP_HUMAN=0` switches t
 | `POLISH` | `1` | `0` = grammar-only final check |
 | `POLISH_MAX_EDIT` | `0.6` | Largest share of a sentence's words the polish may change |
 | `POLISH_KEEP_HUMAN` | `1` | `0` = always keep the polished wording, even if the AI score rises |
+
+## Token use and key distribution (groqai.py)
+
+- **Changes-only replies**: the grammar check, quality check and polish return only the sentences they changed;
+  every other id is listed under `"unchanged"`. Ids missing from both lists are retried, so nothing is skipped.
+  This cuts output tokens by roughly 80% on a typical document.
+- **Size-based batches**: a request holds up to `GROQ_BATCH` sentences (default 20) and about `GROQ_BATCH_CHARS`
+  characters of text (default 6000), so the fixed system prompt is shared by more sentences (about 40% fewer requests).
+- **Token-aware key choice**: each Groq reply reports the key's remaining tokens per minute; the next request goes
+  to the key with the most room (requests in flight are booked against it), and waits only when no key has room.
+- **Right-sized `max_tokens`**: sized to the reply, plus a separate allowance for the hidden reasoning of
+  reasoning models (`GROQ_REASONING`: low / medium / high). A reply cut off at the limit is retried once with a
+  larger budget, and complete items are salvaged from a cut-off reply.
+- **Cache**: a sentence already checked (grammar / quality check) is not sent again (`GROQ_CACHE`, default 20000).
+- Admin → Groq status now shows tokens per key and cache hits.
+
+## Rigorous proofread: quality settings
+
+- Repairs of unclear or garbled sentences are judged against the author's original sentence (they used to be
+  rejected for not resembling the garbled text). They count as required fixes and are never undone or reworded
+  by the AI-score steps.
+- A sentence with a grammar error gets the full correction when its AI rise stays within `PROOF_REQ_RISE`;
+  otherwise it gets the minimal grammar fix.
+- The full proofread runs with `PROOF_REASONING` (default `medium`); the grammar audit stays on `low`.
+
+| Variable | Default | Effect |
+|---|---|---|
+| `PROOF_KEEP_RISE` | `0.05` | Largest AI rise allowed for a style-only edit (was 0.03) |
+| `PROOF_ALT_RISE` | `0.10` | A corrected sentence is reworded for the AI score only above this rise |
+| `PROOF_MAX_DOC_RISE` | `2.0` | Document AI points allowed before style edits are undone (was 1.0) |
+| `PROOF_VARIANTS` | `3` | Alternative versions per sentence when rewording (was 4) |
+| `PROOF_REPAIR_SIM` | `0.60` | Minimum similarity of a repair to the current sentence (original known) |
+| `PROOF_REPAIR_SIM_NOREF` | `0.70` | Same, when no original document is given |
+| `PROOF_REASONING` | `medium` | Reasoning effort of the full proofread |
+
+## Paraphrase model
+
+Parrot (`prithivida/parrot_paraphraser_on_T5`) is the paraphraser. `PARA_MODEL` (or the older `PARROT_MODEL`) can
+name another T5 paraphraser with the same `paraphrase:` input, e.g. `humarin/chatgpt_paraphraser_on_T5_base`
+(`PARA_REP_PENALTY`, default 1.0 for Parrot and 1.15 for other models, discourages copying the input back).
+
+## Adversarial Paraphrasing (detector-guided rewriting, adversarial.py)
+
+Technique from Cheng et al. (2025), *Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text*
+(arXiv:2506.07001). While Parrot writes a sentence, the AI detector chooses the words: at every step Parrot proposes
+its most likely next tokens, each one is appended to the text so far and scored by the detector, and the most
+human-reading one is kept. Parrot keeps the sentence fluent (only its own likely tokens are allowed); the detector
+steers the wording. The result is one extra version per sentence next to Parrot's sampled versions, and it goes
+through exactly the same checks (terms, numbers, citations, meaning, fluency, clarity gate, Groq), so a damaged
+version is never chosen.
+
+| Variable | Default | Effect |
+|---|---|---|
+| `HUM_ADV` | `ai` | `ai` = "100% AI-written" box and report-guided rewriting · `always` · `off` |
+| `ADV_K` | `8` | Candidate words scored per step (the paper uses 50; more = stronger, slower) |
+| `ADV_TOP_P` / `ADV_MIN_P` | `0.95` / `0.05` | Only Parrot's likely words are candidates (keeps grammar intact) |
+| `ADV_EVERY` | `1` | Guide every n-th word (`2` halves the detector cost) |
+| `ADV_BATCH` / `ADV_MAX` | `16` / `400` | Sentences decoded together / most sentence pieces per job |
+| `ADV_ON_ZERO` | `0` | ZeroGPU hardware: off (it would need nested GPU calls); `1` = run on CPU |
+
+Cost: about 2-3x the paraphrasing time for the sentences it is used on (dedicated GPU). Admin → Users shows its
+counters next to the Groq status.
+
+## Protection against construct renaming and other rewrite errors (textparse.py)
+
+Checked on every rewrite, in every mode (including report-guided rewriting, where ordinary words are free):
+- **Study constructs are locked.** Terms defined with an abbreviation ("travel satisfaction (TS)") and the Keywords
+  line are read from the whole document; a rewrite that drops one, or brings in an everyday synonym for it
+  ("pleasure"/"contentment" for satisfaction, "closeness" for intimacy, "allegiance" for loyalty), is rejected.
+- **Abbreviations** may not change ("EWB" -> "EWD") or appear from nowhere.
+- **Statistics symbols** (< > ≤ ≥ = ± ×) must survive ("p < .001" never becomes "p .001").
+- **Brackets** must stay balanced ("(TS amongst ...").
+- **Foreign words** ("well-être") and **off-topic vocabulary** (2+ uncommon words that occur nowhere in the
+  document, e.g. "organizational recruitment" in a tourism paper; `HUM_OFFTOPIC`, 0 = off) are rejected.
+A rejected rewrite falls back to another checked version, or to the author's sentence.

adaptive.py

@@ -0,0 +1,344 @@
+"""
+Self-adaptive learning: the humanizer learns from every run and uses what it learned in the next pass and the next
+document. Nothing here can loosen a safety check: every rewrite still has to pass the guard, the clarity gate and
+Groq; learning only decides HOW to generate (which settings, which sources, which words to avoid) so that more of
+what is generated survives the checks and reads as human.
+
+What it learns
+  1. Bad words (cross-document). Every rejected rewrite is examined: the words it brought in that the original
+     did not have are counted against the rejection reason (construct renamed, group swapped, invented name,
+     attribution, off-topic ...). A word that keeps causing rejections ("tribes", "allegiance", "muscle") and is
+     rarely part of an accepted rewrite is banned: Parrot is told never to generate it (bad_words_ids) and the
+     guard rejects it at once.
+  2. Which source wins (cross-document). Parrot samples, detector-guided (adversarial) versions, plain-wording
+     (de-AI) versions and vocabulary-replaced versions compete; how often each source's surviving versions are
+     the one chosen is tracked (Beta posterior). Sources that keep winning get more candidates, sources that
+     never win are skipped.
+  3. How many candidates to sample (cross-document). If few candidates survive the checks, more are sampled.
+  4. Temperature (cross-document, Smart Humanize). Each pass picks a temperature offset by Thompson sampling;
+     the reward is the AI-score drop of that pass minus a penalty for rejected rewrites, so the setting that
+     produces the best accepted output on real documents is used more often.
+  5. Per sentence, within a document (Smart Humanize passes). A sentence whose AI score did not fall gets a
+     bolder setting next pass (higher temperature, detector-guided decoding); a sentence whose rewrites keep
+     being rejected gets a calmer setting, and after ADAPT_MAX_FAILS failures it is left in the author's wording
+     (saves time, avoids damage).
+
+Storage: one JSON file (ADAPT_PATH; default /data/cs_adaptive.json on Spaces with persistent storage, else the temp
+folder). Statistics decay slowly (ADAPT_DECAY) so the system keeps adapting when models or detectors change.
+
+Space variables
+  ADAPT            1 (default) | 0 = off (fixed settings, as before)
+  ADAPT_PATH       file for the learned state
+  ADAPT_BAN_MIN    rejections before a word can be banned (default 4)
+  ADAPT_BAN_RATIO  share of rejections among its uses needed to ban it (default 0.8)
+  ADAPT_MAX_FAILS  passes with only rejected rewrites before a sentence is left as written (default 3)
+  ADAPT_DECAY      per-update decay of the source / temperature statistics (default 0.995)
+"""
+import contextvars
+import json
+import os
+import random
+import re
+import tempfile
+import threading
+import time
+
+ENABLED = os.getenv("ADAPT", "1") != "0"
+BAN_MIN = float(os.getenv("ADAPT_BAN_MIN", "4"))
+BAN_RATIO = float(os.getenv("ADAPT_BAN_RATIO", "0.8"))
+MAX_FAILS = int(os.getenv("ADAPT_MAX_FAILS", "3"))
+DECAY = float(os.getenv("ADAPT_DECAY", "0.995"))
+FLAG = float(os.getenv("ADAPT_FLAG", "0.5"))            # a sentence at/above this AI probability still needs work
+TEMP_ARMS = [-0.2, 0.0, 0.15, 0.3]
+# words that are never right in an academic rewrite of someone else's study (learned list starts from these)
+SEED_BANNED = {"tribes", "tribe", "muscle", "muscles", "être"}
+# rejection reasons that point at the WORDS a rewrite introduced (not at its length or punctuation)
+_WORD_REASONS = ("construct renamed", "group or degree swapped", "name", "attribution", "off-topic", "foreign",
+                 "invented", "non-word", "informal", "lost", "abbreviation", "direction reversed", "causal")
+
+
+def _default_path():
+    p = os.getenv("ADAPT_PATH", "").strip()
+    if p:
+        return p
+    if os.path.isdir("/data") and os.access("/data", os.W_OK):
+        return "/data/cs_adaptive.json"
+    return os.path.join(tempfile.gettempdir(), "cs_adaptive.json")
+
+
+PATH = _default_path()
+_lock = threading.RLock()
+_state = {"words": {}, "sources": {}, "accept": [1.0, 1.0], "temps": {}, "runs": 0, "banned_extra": []}
+_dirty = [False, 0.0]
+
+
+def _load():
+    try:
+        with open(PATH, encoding="utf-8") as f:
+            data = json.load(f)
+        if isinstance(data, dict):
+            _state.update({k: data[k] for k in _state if k in data})
+    except FileNotFoundError:
+        pass
+    except Exception as err:
+        print(f"[adaptive] state not loaded ({err}); starting fresh", flush=True)
+
+
+_load()
+
+
+def save(force=False):
+    """Write the learned state (at most every 30 s unless forced)."""
+    if not ENABLED:
+        return
+    with _lock:
+        if not _dirty[0] or (not force and time.time() - _dirty[1] < 30):
+            return
+        try:
+            tmp = PATH + ".tmp"
+            with open(tmp, "w", encoding="utf-8") as f:
+                json.dump(_state, f)
+            os.replace(tmp, PATH)
+            _dirty[0], _dirty[1] = False, time.time()
+        except Exception as err:
+            print(f"[adaptive] state not saved: {err}", flush=True)
+
+
+def _touch():
+    _dirty[0] = True
+    save()
+
+
+# ------------------------------------------------------------------------------------------- 1. bad words
+def _introduced(original, cand):
+    o = set(re.findall(r"[a-zà-ÿ]{4,}", (original or "").lower()))
+    return {w for w in re.findall(r"[a-zà-ÿ]{4,}", (cand or "").lower()) if w not in o}
+
+
+def record_candidate(original, cand, source, accepted, reason=None):
+    """One generated rewrite and whether it survived the checks."""
+    if not ENABLED or not cand or not original or cand.strip() == original.strip():
+        return
+    with _lock:
+        a = _state["accept"]
+        a[0 if accepted else 1] += 1
+        new = _introduced(original, cand)
+        words = _state["words"]
+        word_reason = (not accepted) and reason and any(r in reason for r in _WORD_REASONS)
+        for w in new:
+            st = words.setdefault(w, [0.0, 0.0])          # [times in a rejected rewrite (word reason), accepted]
+            if accepted:
+                st[1] += 1
+            elif word_reason:
+                st[0] += 1
+        if len(words) > 50000:                            # forget rare words
+            for w in [w for w, (b, g) in words.items() if b + g <= 1][:20000]:
+                words.pop(w, None)
+        s = _state["sources"].setdefault(source, [1.0, 1.0, 0.0])   # [wins+1, losses+1, survivors]
+        s[2] += 1 if accepted else 0
+    _touch()
+
+
+def banned():
+    """Words never to generate: the seed list + words that keep causing rejections."""
+    if not ENABLED:
+        return set()
+    with _lock:
+        out = set(SEED_BANNED) | set(_state.get("banned_extra") or [])
+        for w, (bad, good) in _state["words"].items():
+            if bad >= BAN_MIN and bad / max(1.0, bad + good) >= BAN_RATIO and not _common(w):
+                out.add(w)
+        return out
+
+
+try:
+    from wordfreq import zipf_frequency as _zipf
+except Exception:
+    _zipf = None
+
+
+def _common(w):
+    """Very common words ("shows", "between") are never banned automatically: they only travel along with the
+    word that really caused a rejection."""
+    return _zipf is not None and _zipf(w, "en") >= 5.0
+
+
+def ban_word(word):
+    """Admin: ban a word by hand (e.g. after reviewing a client's output)."""
+    with _lock:
+        lst = set(_state.get("banned_extra") or [])
+        lst.add(word.strip().lower())
+        _state["banned_extra"] = sorted(lst)
+    _touch()
+    save(force=True)
+
+
+def banned_in(original, cand):
+    """A banned word the rewrite introduces, or None."""
+    b = banned()
+    if not b:
+        return None
+    hit = _introduced(original, cand) & b
+    return sorted(hit)[0] if hit else None
+
+
+_bw_cache = {}
+
+
+def bad_words_ids(tokenizer):
+    """Token sequences of the banned words, for model.generate(bad_words_ids=...)."""
+    b = banned()
+    key = (id(tokenizer), frozenset(b))
+    if key in _bw_cache:
+        return _bw_cache[key]
+    ids = []
+    for w in sorted(b):
+        for form in (w, " " + w, w.capitalize(), " " + w.capitalize()):
+            try:
+                t = tokenizer(form, add_special_tokens=False).input_ids
+            except Exception:
+                continue
+            if t and t not in ids and len(t) <= 4:
+                ids.append(t)
+    _bw_cache.clear()
+    _bw_cache[key] = ids or None
+    return _bw_cache[key]
+
+
+# ------------------------------------------------------------------------------------------- 2-3. sources, k
+def record_win(source, won):
+    """Among the versions that survived the checks for one sentence: did this source's version get chosen?"""
+    if not ENABLED:
+        return
+    with _lock:
+        s = _state["sources"].setdefault(source, [1.0, 1.0, 0.0])
+        s[0], s[1] = s[0] * DECAY, s[1] * DECAY            # slow forgetting
+        s[0 if won else 1] += 1
+    _touch()
+
+
+def source_weight(source):
+    """Thompson sample of how likely this source's version is to be the chosen one (0..1)."""
+    if not ENABLED:
+        return 1.0
+    with _lock:
+        a, b, _ = _state["sources"].get(source, [1.0, 1.0, 0.0])
+    return random.betavariate(max(a, 0.1), max(b, 0.1))
+
+
+def use_source(source, min_share=0.08):
+    """Skip a source only when it has clearly and repeatedly lost (keeps exploring otherwise)."""
+    if not ENABLED:
+        return True
+    with _lock:
+        a, b, _ = _state["sources"].get(source, [1.0, 1.0, 0.0])
+    if a + b < 30:
+        return True
+    return a / (a + b) >= min_share or random.random() < 0.1
+
+
+def candidates(k, cap=8):
+    """More samples when few survive the checks."""
+    if not ENABLED:
+        return k
+    with _lock:
+        ok, bad = _state["accept"]
+    rate = ok / max(1.0, ok + bad)
+    if ok + bad < 50:
+        return k
+    if rate < 0.25:
+        return min(cap, k + 2)
+    if rate < 0.4:
+        return min(cap, k + 1)
+    return k
+
+
+# ------------------------------------------------------------------------------------------- 4. temperature
+def pick_temp_offset():
+    """Thompson sampling over TEMP_ARMS (Gaussian posterior of the reward)."""
+    if not ENABLED:
+        return 0.0
+    with _lock:
+        stats = dict(_state["temps"])
+    best, best_v = 0.0, None
+    for arm in TEMP_ARMS:
+        n, mean = stats.get(str(arm), [0.0, 0.0])
+        v = random.gauss(mean, 1.0 / (1.0 + n) ** 0.5) if n else random.gauss(0.0, 1.0)
+        if best_v is None or v > best_v:
+            best, best_v = arm, v
+    return best
+
+
+def record_pass(offset, ai_before, ai_after, n_jobs, n_changed):
+    """Reward of one Smart Humanize pass: AI-% drop, minus a penalty when most rewrites were rejected."""
+    if not ENABLED or n_jobs <= 0:
+        return
+    drop = (ai_before - ai_after) / 10.0                 # 10 points = reward 1
+    kept = n_changed / max(1, n_jobs)
+    reward = drop - 0.5 * (1.0 - kept)
+    with _lock:
+        n, mean = _state["temps"].get(str(offset), [0.0, 0.0])
+        n = n * DECAY
+        _state["temps"][str(offset)] = [n + 1, mean + (reward - mean) / (n + 1)]
+        _state["runs"] += 1
+    _touch()
+
+
+# ------------------------------------------------------------------------------------------- 5. per sentence
+SENT = contextvars.ContextVar("adapt_sentences", default=None)   # job: anchor -> dict(score, fails, temp, bold)
+
+
+def start_job():
+    SENT.set({})
+
+
+def observe(anchor, score):
+    """Pass k scored this sentence; compare with pass k-1 and adjust its settings for pass k+1."""
+    d = SENT.get()
+    if d is None or not ENABLED:
+        return
+    st = d.setdefault(anchor, {"score": None, "fails": 0, "dt": 0.0, "bold": False, "tried": 0, "changed": False})
+    if st["score"] is not None and st.get("changed") and score >= FLAG:
+        if st["score"] - score < 0.05:                   # rewritten but still reads as AI: be bolder next time
+            st["dt"] = min(0.5, st["dt"] + 0.15)
+            st["bold"] = True
+    st["score"] = score
+
+
+def outcome(anchor, changed):
+    """This pass's rewrite of the sentence was written into the document (changed) or every version failed."""
+    d = SENT.get()
+    if d is None or not ENABLED:
+        return
+    st = d.setdefault(anchor, {"score": None, "fails": 0, "dt": 0.0, "bold": False, "tried": 0, "changed": False})
+    st["tried"] += 1
+    st["changed"] = bool(changed)
+    if changed:
+        st["fails"] = 0
+    else:
+        st["fails"] += 1
+        st["dt"] = max(-0.3, st["dt"] - 0.15)            # its rewrites keep failing the checks: calmer
+
+
+def settings(anchor):
+    """(temperature delta, detector-guided?, skip?) for this sentence in the next pass."""
+    d = SENT.get()
+    if d is None or not ENABLED:
+        return 0.0, False, False
+    st = d.get(anchor)
+    if not st:
+        return 0.0, False, False
+    return st["dt"], st["bold"], st["fails"] >= MAX_FAILS
+
+
+def status():
+    if not ENABLED:
+        return "Adaptive learning: off (ADAPT=0)."
+    with _lock:
+        ok, bad = _state["accept"]
+        src = ", ".join(f"{k} {a / (a + b):.0%}" for k, (a, b, _) in sorted(_state["sources"].items()))
+        temps = ", ".join(f"{k}: {m:+.2f} ({n:.0f})" for k, (n, m) in sorted(_state["temps"].items()))
+        nb = len(banned())
+    return (f"Adaptive learning: {_state['runs']} passes learned from; {ok + bad - 2:.0f} rewrites seen, "
+            f"{ok / max(1, ok + bad):.0%} pass the checks; {nb} banned words; source win rates: {src or '-'}; "
+            f"temperature rewards: {temps or '-'}. State file: {PATH}")

adversarial.py

@@ -0,0 +1,164 @@
+"""
+Adversarial Paraphrasing (detector-guided decoding) for the Parrot paraphraser.
+
+Technique from "Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text" (Cheng et al., 2025,
+arXiv:2506.07001): while the paraphraser writes, the AI detector chooses the words. At each decoding step the
+paraphraser proposes its most likely next tokens (top-k inside top-p); every proposal is appended to the text written
+so far and scored by the detector, and the proposal that reads as the MOST HUMAN is kept. The paraphraser keeps the
+sentence fluent and on-meaning (only its own likely tokens are allowed); the detector steers word choice.
+
+Here it adds one extra candidate per sentence next to Parrot's sampled versions. That candidate goes through exactly
+the same checks as every other version (terms / numbers / citations restored and guarded, meaning, fluency, clarity
+gate, Groq check), so a damaged adversarial sentence is simply not chosen.
+
+Cost: one detector batch per guided step (sentences x candidates texts). On a dedicated GPU this is roughly 2-3x the
+time of normal paraphrasing for the sentences it is used on; on CPU it is much slower, so by default it runs only in
+the "100% AI-written" / report-guided modes and on at most ADV_MAX sentence pieces per job.
+
+Space variables
+  HUM_ADV        ai (default: 100% AI mode and report-guided rewriting) | always | off
+  ADV_K          candidate tokens scored per step (default 8; the paper uses 50)
+  ADV_TOP_P      nucleus for the candidates (default 0.95)
+  ADV_MIN_P      a candidate needs at least this share of the top token's probability (default 0.05): keeps the
+                 detector from picking an unlikely, ungrammatical word
+  ADV_EVERY      guide every n-th token (default 1 = every token, as in the paper; 2 halves the detector cost)
+  ADV_BATCH      sentences decoded together (default 16)
+  ADV_MAX        most sentence pieces per job (default 400)
+  ADV_ON_ZERO    1 = also run on ZeroGPU hardware (on the CPU copy of the model; slow). Default 0.
+"""
+import os
+
+MODE = os.getenv("HUM_ADV", "ai").strip().lower()
+K = max(2, int(os.getenv("ADV_K", "8")))
+TOP_P = float(os.getenv("ADV_TOP_P", "0.95"))
+MIN_P = float(os.getenv("ADV_MIN_P", "0.05"))
+EVERY = max(1, int(os.getenv("ADV_EVERY", "1")))
+BATCH = max(1, int(os.getenv("ADV_BATCH", "16")))
+MAX_PIECES = max(0, int(os.getenv("ADV_MAX", "400")))
+ON_ZERO = os.getenv("ADV_ON_ZERO", "0") == "1"
+STATS = {"runs": 0, "pieces": 0, "steps": 0, "scored": 0, "errors": 0, "last_error": ""}
+
+
+def wanted(full_ai):
+    """Is detector-guided decoding used for this job?"""
+    if MODE in ("off", "0", "no", "false"):
+        return False
+    return MODE == "always" or bool(full_ai)
+
+
+def _no_repeat(prev_ids, n=3):
+    """Tokens that would repeat an n-gram already written (same rule as Parrot's no_repeat_ngram_size=3)."""
+    if len(prev_ids) < n - 1:
+        return set()
+    tail = tuple(prev_ids[-(n - 1):])
+    banned = set()
+    for i in range(len(prev_ids) - n + 1):
+        if tuple(prev_ids[i:i + n - 1]) == tail:
+            banned.add(prev_ids[i + n - 1])
+    return banned
+
+
+def generate(texts, model, tokenizer, device, score_fn, temperature=1.0, prefix="paraphrase: "):
+    """Detector-guided paraphrase of each text (one output per text; the input text where decoding fails).
+    score_fn(list[str]) -> list[float] AI probabilities (lower = more human), or None."""
+    import torch
+    outs = list(texts)
+    if not texts:
+        return outs
+    STATS["runs"] += 1
+    temperature = max(0.3, float(temperature))
+    order = sorted(range(len(texts)), key=lambda i: len(texts[i]))      # similar lengths decode together
+    for b0 in range(0, len(order), BATCH):
+        ids_ = order[b0:b0 + BATCH]
+        batch = [texts[i] for i in ids_]
+        try:
+            res = _generate_batch(batch, model, tokenizer, device, score_fn, temperature, prefix, torch)
+        except Exception as err:                    # never break the job: Parrot's normal versions remain
+            STATS["errors"] += 1
+            STATS["last_error"] = str(err)[:200]
+            print(f"[adversarial] batch skipped: {err}", flush=True)
+            continue
+        for i, r in zip(ids_, res):
+            if r and r.strip():
+                outs[i] = r.strip()
+        STATS["pieces"] += len(batch)
+    return outs
+
+
+def _generate_batch(batch, model, tokenizer, device, score_fn, temperature, prefix, torch):
+    enc = tokenizer([prefix + t for t in batch], return_tensors="pt", padding=True, truncation=True, max_length=256)
+    enc = {k: v.to(device) for k, v in enc.items()}
+    n = len(batch)
+    max_new = min(180, int(enc["input_ids"].shape[1] * 1.35) + 12)
+    eos = tokenizer.eos_token_id
+    start = model.config.decoder_start_token_id
+    if start is None:
+        start = tokenizer.pad_token_id
+    written = [[] for _ in range(n)]
+    done = [False] * n
+    with torch.inference_mode():
+        enc_out = model.get_encoder()(input_ids=enc["input_ids"], attention_mask=enc["attention_mask"])
+        last = torch.full((n, 1), start, dtype=torch.long, device=device)
+        past = None
+        for step in range(max_new):
+            out = model(encoder_outputs=enc_out, attention_mask=enc["attention_mask"], decoder_input_ids=last,
+                        past_key_values=past, use_cache=True)
+            past = out.past_key_values
+            probs = torch.softmax(out.logits[:, -1, :].float() / temperature, dim=-1)
+            top_p, top_i = probs.topk(K, dim=-1)
+            top_p, top_i = top_p.cpu().tolist(), top_i.cpu().tolist()
+            choice = [eos] * n
+            texts, owners = [], []
+            for b in range(n):
+                if done[b]:
+                    continue
+                banned = _no_repeat(written[b])
+                cands, cum = [], 0.0
+                for p, t in zip(top_p[b], top_i[b]):
+                    if t in banned:
+                        continue
+                    if cands and (cum >= TOP_P or p < MIN_P * top_p[b][0]):
+                        break
+                    cands.append((t, p))
+                    cum += p
+                if not cands:
+                    cands = [(top_i[b][0], top_p[b][0])]
+                choice[b] = cands[0][0]                    # most likely token by default
+                if len(cands) > 1 and step % EVERY == 0:
+                    for t, _ in cands:
+                        if t == eos:                           # the sentence so far, as it would end here
+                            texts.append(tokenizer.decode(written[b], skip_special_tokens=True))
+                        else:
+                            texts.append(tokenizer.decode(written[b] + [t], skip_special_tokens=True))
+                        owners.append((b, t))
+            if texts:
+                sc = score_fn([t if t.strip() else "." for t in texts])
+                STATS["scored"] += len(texts)
+                if sc:
+                    best = {}
+                    for (b, t), s in zip(owners, sc):
+                        if b not in best or s < best[b][0] - 1e-6:   # ties keep the more likely (earlier) token
+                            best[b] = (s, t)
+                    for b, (_, t) in best.items():
+                        choice[b] = t
+            STATS["steps"] += 1
+            for b in range(n):
+                if done[b]:
+                    choice[b] = tokenizer.pad_token_id
+                    continue
+                if choice[b] == eos:
+                    done[b] = True
+                else:
+                    written[b].append(choice[b])
+            if all(done):
+                break
+            last = torch.tensor([[c] for c in choice], dtype=torch.long, device=device)
+    return [tokenizer.decode(w, skip_special_tokens=True) for w in written]
+
+
+def status():
+    if not wanted(True):
+        return "Adversarial paraphrasing: off (HUM_ADV=off)."
+    s = STATS
+    return (f"Adversarial paraphrasing ({MODE}, k={K}, every {EVERY} token): {s['pieces']} sentence pieces, "
+            f"{s['scored']:,} detector checks" + (f"; last error: {s['last_error']}" if s["last_error"] else "."))

app.py

@@ -1564,11 +1564,12 @@ with gr.Blocks(title=f"{APP_NAME} · CS Software Solutions", theme=theme, css=CS
                                 sd_status = gr.HTML(H.status_idle("Upload your .docx and press **Detect & humanize document**. Nothing else to set."))
                                 sd_file = gr.File(label="Download humanized .docx", interactive=False)
                                 sd_dl = gr.HTML()
-                                sd_pr_btn = gr.Button("🔍 Rigorous proofread (keeps the AI score)", variant="secondary")
-                                gr.Markdown("<small>After humanizing: every sentence is proofread (grammar first, then "
-                                            "clarity, academic wording and flow) and removes doubled terms and AI "
-                                            "leftovers. Grammar fixes are always made; style edits only if the AI score "
-                                            "holds. Free, does not use a document from your plan.</small>")
+                                sd_pr_btn = gr.Button("🔍 Final check & proofread (keeps the AI score)", variant="secondary")
+                                gr.Markdown("<small>After humanizing: one last full check of the whole document in a "
+                                            "single pass: grammar, unclear sentences, academic wording, flow and "
+                                            "consistent terms, compared with your original. Grammar fixes are always "
+                                            "made; style edits only if the AI score holds. Problems only the author can "
+                                            "decide are listed. Free, does not use a document from your plan.</small>")
                                 sd_pr_status = gr.HTML()
                                 sd_pr_file = gr.File(label="Download proofread .docx", interactive=False)
                                 gr.HTML(next_steps_html())
@@ -1741,7 +1742,7 @@ with gr.Blocks(title=f"{APP_NAME} · CS Software Solutions", theme=theme, css=CS
                                 hd_status = gr.HTML(H.status_idle("Upload a document and press **Paraphrase document**."))
                                 hd_file = gr.File(label="Download paraphrased .docx", interactive=False)
                                 hd_dl = gr.HTML()
-                                hd_pr_btn = gr.Button("🔍 Rigorous proofread (keeps the AI score)", variant="secondary")
+                                hd_pr_btn = gr.Button("🔍 Final check & proofread (keeps the AI score)", variant="secondary")
                                 hd_pr_status = gr.HTML()
                                 hd_pr_file = gr.File(label="Download proofread .docx", interactive=False)
                                 hd_pr_dl = gr.HTML()
@@ -2204,6 +2205,8 @@ with gr.Blocks(title=f"{APP_NAME} · CS Software Solutions", theme=theme, css=CS
     ad_p_credits.click(admin_credit, [user_state, ad_p_email, ad_p_credits_n, ad_p_amount], plan_out, **HID)
     admin_tab.select(lambda u: (P.admin_requests(u), P.admin_payments(u), A.admin_reset_requests(u), M.admin_log(u),
                                 f'<div class="cs-gauge-cap" style="text-align:left">{html.escape(H.G.status())}</div>',
+                                f'<div class="cs-gauge-cap" style="text-align:left">{html.escape(H.ADV.status())}</div>',
+                                f'<div class="cs-gauge-cap" style="text-align:left">{html.escape(H.AD.status())}</div>',
                                 f'<div class="cs-gauge-cap" style="text-align:left">{html.escape(PU.status())}</div>',
                                 P.admin_subscriptions(u), P.admin_history(u)),
                      user_state, [ad_req, ad_pay, ad_rr, ad_m_log, ad_groq, ad_push, ad_subs, ad_hist], **HID)

clarity.py

@@ -165,9 +165,11 @@ def _acronym_repeat(t):
 
 
 def _garbled(o, n):
-    why = TP.quality_issue(o, n)
+    why = TP.quality_issue(o, n) or TP.integrity_issue(o, n)
     if why:
         return why
+    if TP._symbols(n) != TP._symbols(o):
+        return "statistics symbol changed"
     v = PR.violations(o, n)                        # publication English rulebook (pubrules.py)
     if v:
         return "rule: " + v[0]

groqai.py

@@ -38,21 +38,99 @@ def _read_keys():
 KEYS = _read_keys()
 KEY = KEYS[0] if KEYS else ""
 ENABLED = bool(KEYS) and os.getenv("GROQ_ENABLED", "1") != "0"
-# per key: paused until (rate limit / daily quota / rejected key), requests sent, last error
-_KEY_STATE = [{"until": 0.0, "calls": 0, "err": ""} for _ in KEYS]
+# per key: paused until (rate limit / daily quota / rejected key), requests sent, last error, and the token budget
+# Groq reports in its response headers (remaining tokens this minute, when that budget refills, the key's limit),
+# plus the tokens reserved by requests still in flight on that key
+_KEY_STATE = [{"until": 0.0, "calls": 0, "err": "", "tok_left": None, "tok_reset": 0.0, "tok_limit": None,
+               "inflight": 0, "tokens": 0} for _ in KEYS]
 _rr = [0]                                         # round-robin position
 URL = "https://api.groq.com/openai/v1"
 PREF = [m.strip() for m in os.getenv(
     "GROQ_MODELS", "openai/gpt-oss-120b,llama-3.3-70b-versatile,qwen/qwen3.8-27b,openai/gpt-oss-20b").split(",")
     if m.strip()]
-BATCH = max(1, int(os.getenv("GROQ_BATCH", "12")))
+# A batch is limited by sentence count (GROQ_BATCH) AND by size (GROQ_BATCH_CHARS characters of sentence text), so
+# the fixed system prompt is shared by as many sentences as fit, without producing over-long replies.
+BATCH = max(1, int(os.getenv("GROQ_BATCH", "20")))
+BATCH_CHARS = max(800, int(os.getenv("GROQ_BATCH_CHARS", "6000")))
 MAX_WAIT = float(os.getenv("GROQ_MAX_WAIT", "60"))
+MAX_OUT = int(os.getenv("GROQ_MAX_OUT", "12000"))          # hard cap on max_tokens per request
+# reasoning effort for reasoning models (gpt-oss, gpt-5, o-series); a step can raise it with REASONING.set(...)
+REASONING_DEFAULT = os.getenv("GROQ_REASONING", "low").lower()
+_REASON_ALLOW = {"low": 700, "medium": 2200, "high": 5000}  # extra max_tokens reserved for the hidden reasoning
+
+
+def _est_tokens(text):
+    """Rough token count of English text (about 3.6 characters per token; errs on the high side)."""
+    return int(len(text or "") / 3.6) + 1
+
+
+def _out_budget(chars, n_items, factor=1.0):
+    """max_tokens for a reply that, in the worst case, returns every sentence (chars) as JSON."""
+    return int(chars / 3.3 * factor) + 14 * n_items + 120
+
+
+def _pack(sizes, max_items, max_chars):
+    """Contiguous batches (lists of indices): at most max_items items and about max_chars characters each."""
+    out, cur, cc = [], [], 0
+    for i, s in enumerate(sizes):
+        if cur and (len(cur) >= max_items or cc + s > max_chars):
+            out.append(cur)
+            cur, cc = [], 0
+        cur.append(i)
+        cc += s
+    if cur:
+        out.append(cur)
+    return out
+
+
+def _dur(s):
+    """Groq reset header ('7.66s', '2m59.5s', '120ms', '1h2m3s') -> seconds."""
+    tot = 0.0
+    for num, unit in re.findall(r"([\d.]+)(ms|h|m|s)", str(s or "")):
+        try:
+            tot += float(num) * {"ms": 0.001, "h": 3600.0, "m": 60.0, "s": 1.0}[unit]
+        except ValueError:
+            pass
+    return tot
+
+
+def _note_limits(k, headers):
+    """Remember the token budget Groq reports for key k (x-ratelimit-* headers)."""
+    try:
+        left = headers.get("x-ratelimit-remaining-tokens")
+        lim = headers.get("x-ratelimit-limit-tokens")
+        reset = headers.get("x-ratelimit-reset-tokens")
+        req_left = headers.get("x-ratelimit-remaining-requests")
+        req_reset = headers.get("x-ratelimit-reset-requests")
+    except Exception:
+        return
+    now = time.time()
+    with _lock:
+        st = _KEY_STATE[k]
+        if left is not None:
+            try:
+                st["tok_left"] = int(float(left))
+                st["tok_reset"] = now + _dur(reset)
+            except ValueError:
+                pass
+        if lim is not None:
+            try:
+                st["tok_limit"] = int(float(lim))
+            except ValueError:
+                pass
+        if req_left is not None:
+            try:
+                if int(float(req_left)) <= 0:          # requests per day used up on this key
+                    st["until"] = max(st["until"], now + max(60.0, _dur(req_reset)))
+                    st["err"] = "daily request limit"
+            except ValueError:
+                pass
 
 _lock = threading.RLock()
 _models = None                 # models available on this key (ordered by preference)
 _down_until = 0.0              # daily quota exhausted -> skip Groq until then
 STATS = {"calls": 0, "checked": 0, "ok": 0, "fixed": 0, "rejected": 0, "errors": 0, "tokens": 0,
-         "last_error": ""}
+         "last_error": "", "cache_hits": 0, "truncated": 0}
 
 
 class Unavailable(Exception):
@@ -63,17 +141,44 @@ def _headers(k=0):
     return {"Authorization": f"Bearer {KEYS[k] if KEYS else ''}", "Content-Type": "application/json"}
 
 
-def _pick_key():
-    """Next usable key (round robin, so the load is spread over all keys); None if all are paused."""
+def _room(st, now):
+    """Tokens a key can still take this minute (inf = unknown or already refilled), minus in-flight reservations."""
+    if st["tok_left"] is None or now >= st["tok_reset"]:
+        return float("inf")
+    return st["tok_left"] - st["inflight"]
+
+
+def _pick_key(need=0, reserve=False):
+    """Usable key with the most token room for a request of about `need` tokens (ties: round robin, so the load is
+    spread over all keys). With reserve=True the tokens are booked on that key until _release(). None if every
+    key is paused or none has room right now."""
     now = time.time()
     with _lock:
         n = len(KEYS)
+        best = None
         for step in range(n):
             k = (_rr[0] + step) % n
-            if _KEY_STATE[k]["until"] <= now:
-                _rr[0] = (k + 1) % n
-                return k
-    return None
+            st = _KEY_STATE[k]
+            if st["until"] > now:
+                continue
+            room = _room(st, now)
+            want = min(need, int(0.9 * st["tok_limit"])) if st["tok_limit"] else need
+            if room < want:
+                continue
+            if best is None or room > best[0]:
+                best = (room, k)
+        if best is None:
+            return None
+        k = best[1]
+        _rr[0] = (k + 1) % n
+        if reserve:
+            _KEY_STATE[k]["inflight"] += need
+        return k
+
+
+def _release(k, need):
+    with _lock:
+        _KEY_STATE[k]["inflight"] = max(0, _KEY_STATE[k]["inflight"] - need)
 
 
 def _pause_key(k, seconds, why):
@@ -96,10 +201,18 @@ def _models_free_in(k):
     return max(1.0, min((_KM_UNTIL.get((k, m), 0.0) for m in _model_list()), default=now) - now)
 
 
-def _next_free_in():
-    """Seconds until the first paused key is usable again."""
+def _next_free_in(need=0):
+    """Seconds until some key can take a request of about `need` tokens (need=0: until the first paused key is
+    usable again)."""
     now = time.time()
-    return max(0.0, min((st["until"] for st in _KEY_STATE), default=0.0) - now)
+    best = None
+    for st in _KEY_STATE:
+        t = max(0.0, st["until"] - now)
+        want = min(need, int(0.9 * st["tok_limit"])) if st["tok_limit"] else need
+        if want and _room(st, now) < want:
+            t = max(t, st["tok_reset"] - now)
+        best = t if best is None else min(best, t)
+    return max(0.0, best or 0.0)
 
 
 # ---------------------------------------------------------------- OpenAI (ChatGPT) as an alternative provider
@@ -117,6 +230,9 @@ OA_STATS = {"calls": 0, "tokens": 0, "last_error": ""}
 
 
 PAR_LIMIT = _cv.ContextVar("llm_parallel", default=0)    # >0: fewer requests at once (proofread on Groq)
+REASONING = _cv.ContextVar("llm_reasoning", default="")  # "" = GROQ_REASONING / OPENAI_REASONING
+OUT_CAP = _cv.ContextVar("llm_out_cap", default=0)        # >0: larger max_tokens cap for this call
+TIMEOUT = _cv.ContextVar("llm_timeout", default=0)        # >0: longer HTTP timeout (s) for this call
 
 
 def _par():
@@ -124,13 +240,16 @@ def _par():
     return max(1, min(PARALLEL, lim) if lim else PARALLEL)
 
 
-def _in_provider(provider, fn, *args):
+def _in_provider(provider, fn, *args, _reasoning=None):
     """Run fn in a worker thread with the caller's provider (context variables do not cross thread pools)."""
     tok = PROVIDER.set(provider)
+    tok_r = REASONING.set(_reasoning) if _reasoning is not None else None
     try:
         return fn(*args)
     finally:
         PROVIDER.reset(tok)
+        if tok_r is not None:
+            REASONING.reset(tok_r)
 
 
 def openai_available():
@@ -170,11 +289,12 @@ def _chat_openai(messages, temperature, max_tokens):
             "max_completion_tokens": max_tokens + (2000 if model.startswith(("gpt-5", "o")) else 0),
             "response_format": {"type": "json_object"}}
     if model.startswith(("gpt-5", "o3", "o4")):
-        body["reasoning_effort"] = os.getenv("OPENAI_REASONING", "low")
+        body["reasoning_effort"] = REASONING.get() or os.getenv("OPENAI_REASONING", "low")
     waited = 0.0
     for attempt in range(10):
         try:
-            r = httpx.post(OPENAI_URL + "/chat/completions", json=body, headers=_oa_headers(), timeout=180)
+            r = httpx.post(OPENAI_URL + "/chat/completions", json=body, headers=_oa_headers(),
+                           timeout=TIMEOUT.get() or 180)
         except Exception as err:
             OA_STATS["last_error"] = f"network: {err}"
             if attempt >= 2:
@@ -282,22 +402,31 @@ def status():
     down = f" · quota pause until {time.strftime('%H:%M UTC', time.gmtime(_down_until))}" \
         if time.time() < _down_until else ""
     now = time.time()
-    keys = " · ".join(f"key {i + 1}: {st['calls']} calls" + (f", paused {int(st['until'] - now)}s ({st['err']})"
-                                                             if st["until"] > now else "")
+    keys = " · ".join(f"key {i + 1}: {st['calls']} calls, ~{st['tokens']:,} tokens"
+                      + (f", {st['tok_left']:,} tokens/min left" if st["tok_left"] is not None
+                         and st["tok_reset"] > now else "")
+                      + (f", paused {int(st['until'] - now)}s ({st['err']})" if st["until"] > now else "")
                       for i, st in enumerate(_KEY_STATE))
     oa = (f" OpenAI (proofread): ON, {(_oa['models'] or OPENAI_PREF)[0]}, {OA_STATS['calls']} calls, "
           f"~{OA_STATS['tokens']:,} tokens" + (f", last error: {OA_STATS['last_error']}" if OA_STATS["last_error"] else "")
           + ".") if OPENAI_KEY else ""
     return oa + (f" Groq quality check is ON ({m}), {len(KEYS)} key(s) [{keys}]{down}. Since restart: {s['checked']} Parrot sentences checked — "
             f"{s['ok']} fine as written, {s['fixed']} repaired, {s['rejected']} rejected (original kept); "
-            f"~{s['tokens']:,} tokens."
+            f"~{s['tokens']:,} tokens; {s.get('cache_hits', 0)} sentence(s) answered from cache."
             + (f" Last error: {s['last_error']}" if s["last_error"] else ""))
 
 
+def _is_reasoning(model):
+    return "gpt-oss" in model or model.startswith(("gpt-5", "o3", "o4"))
+
+
 def _chat(messages, temperature, max_tokens):
-    """One chat completion (JSON mode). With several keys, a rate-limited key is paused and the request moves to
-    the next key at once; it waits only when every key is paused (short limits), and raises Unavailable when all
-    keys are out for longer than GROQ_MAX_WAIT."""
+    """One chat completion (JSON mode). `max_tokens` is the budget for the visible answer; the hidden reasoning of
+    reasoning models gets its own allowance on top. The key with the most token room is used (Groq reports each
+    key's remaining tokens per minute in its response headers; requests in flight are booked against it), so the
+    load is spread by tokens, not just by request count. A rate-limited key is paused and the request moves to the
+    next key at once; it waits only when no key has room (short limits), and raises Unavailable when all keys are
+    out for longer than GROQ_MAX_WAIT. A reply cut off at max_tokens is retried once with a larger budget."""
     global _down_until
     import httpx
     if PROVIDER.get() == "openai":
@@ -305,36 +434,56 @@ def _chat(messages, temperature, max_tokens):
     if not available():
         raise Unavailable("Groq quota paused")
     waited = 0.0
-    body = {"messages": messages, "temperature": temperature, "max_tokens": max_tokens,
-            "response_format": {"type": "json_object"}}
+    models = _model_list()
+    model = models[0]
+    effort = REASONING.get() or REASONING_DEFAULT
+
+    cap = OUT_CAP.get() or MAX_OUT
+
+    def budget(m, base):
+        return min(cap, base + (_REASON_ALLOW.get(effort, 700) if _is_reasoning(m) else 0))
+    body = {"model": model, "messages": messages, "temperature": temperature,
+            "max_tokens": budget(model, max_tokens), "response_format": {"type": "json_object"}}
+    if "gpt-oss" in model:
+        body["reasoning_effort"] = effort
+    prompt_tok = sum(_est_tokens(m.get("content", "")) for m in messages)
+    grown = 0
     plain = set()                                      # models that do not take reasoning_effort
-    for attempt in range(8 + 2 * len(KEYS) + 2 * len(PREF)):
-        k = _pick_key()
-        if k is None:                                  # every key is paused: wait for the first one, if short
-            wait = _next_free_in()
+    for attempt in range(8 + 2 * len(KEYS)):
+        need = prompt_tok + body["max_tokens"] // 2       # expected use: prompt + about half the reply budget
+        k = _pick_key(need, reserve=True)
+        if k is None:                                  # no key has room: wait for the first one, if short
+            wait = _next_free_in(need)
             if wait > MAX_WAIT or waited + wait > MAX_WAIT:
                 STATS["last_error"] = "rate limit on all Groq keys"
                 raise Unavailable("Groq rate limit (all keys)")
             time.sleep(wait + 0.2)
-            waited += wait
+            waited += wait + 0.2
             continue
-        model = _model_for(k)
-        if model is None:                              # every model of this key is at its limit
+        use = _model_for(k)
+        if use is None:                                # every model of this key is at its limit
+            _release(k, need)
             _pause_key(k, _models_free_in(k), "rate limit (all models)")
             continue
-        body["model"] = model
-        body.pop("reasoning_effort", None)
-        if "gpt-oss" in model and model not in plain:
-            body["reasoning_effort"] = "low"
+        if use != model:                               # the preferred model is at its limit (or free again)
+            model = body["model"] = use
+            body["max_tokens"] = budget(model, max_tokens)
+            body.pop("reasoning_effort", None)
+            if "gpt-oss" in model and model not in plain:
+                body["reasoning_effort"] = effort
         try:
-            r = httpx.post(URL + "/chat/completions", json=body, headers=_headers(k), timeout=120)
-        except Exception as err:
-            STATS["errors"] += 1
-            STATS["last_error"] = f"network: {err}"
-            if attempt >= 2:
-                raise Unavailable(str(err))
-            time.sleep(2 * (attempt + 1))
-            continue
+            try:
+                r = httpx.post(URL + "/chat/completions", json=body, headers=_headers(k), timeout=TIMEOUT.get() or 120)
+            except Exception as err:
+                STATS["errors"] += 1
+                STATS["last_error"] = f"network: {err}"
+                if attempt >= 2:
+                    raise Unavailable(str(err))
+                time.sleep(2 * (attempt + 1))
+                continue
+            _note_limits(k, r.headers)
+        finally:
+            _release(k, need)
         with _lock:
             _KEY_STATE[k]["calls"] += 1
         txt = r.text[:400]
@@ -350,7 +499,15 @@ def _chat(messages, temperature, max_tokens):
             with _lock:
                 _KM_UNTIL[(k, model)] = time.time() + (min(max(ra, 60), 6 * 3600) if daily else ra)
             STATS["last_error"] = f"{'daily quota' if daily else 'rate limit'} ({model}, key {k + 1}): {txt[:120]}"
-            continue
+            continue                                   # next model / key at once
+        if (r.status_code in (400, 413) and "max_tokens" in txt and ("must be" in txt.lower() or
+                                                                      "too large" in txt.lower())):
+            if body["max_tokens"] > 1024:              # budget above what this model accepts: shrink and retry
+                body["max_tokens"] = max(1024, body["max_tokens"] // 2)
+                continue
+        if r.status_code == 413:                       # the request itself is too large for this key / model
+            STATS["last_error"] = f"request too large: {txt[:160]}"
+            raise Unavailable(STATS["last_error"])
         if r.status_code == 401:
             _pause_key(k, 3600, "key rejected (401)")
             STATS["last_error"] = f"Groq key {k + 1} rejected (401) — check that secret."
@@ -362,12 +519,19 @@ def _chat(messages, temperature, max_tokens):
                 w in txt.lower() for w in ("decommission", "not found", "does not exist", "not available",
                                             "no access", "deprecated")):
             _drop_model(model)
-            if _model_list()[0] == model:               # the last model left
+            nxt = _model_list()[0]
+            if nxt == model:
                 STATS["last_error"] = f"model unavailable: {txt[:160]}"
                 raise Unavailable(STATS["last_error"])
+            model = body["model"] = nxt
+            body["max_tokens"] = budget(model, max_tokens)
+            body.pop("reasoning_effort", None)
+            if "gpt-oss" in model:
+                body["reasoning_effort"] = effort
             continue
         if r.status_code == 400 and "reasoning_effort" in txt:
             plain.add(model)
+            body.pop("reasoning_effort", None)
             continue
         if r.status_code == 400 and ("response_format" in txt or "json" in txt.lower()) and "response_format" in body:
             body.pop("response_format", None)            # model without JSON mode, or JSON validation failed
@@ -380,9 +544,19 @@ def _chat(messages, temperature, max_tokens):
             STATS["last_error"] = f"HTTP {r.status_code}: {txt[:160]}"
             raise Unavailable(STATS["last_error"])
         data = r.json()
-        STATS["calls"] += 1
-        STATS["tokens"] += int((data.get("usage") or {}).get("total_tokens") or 0)
-        return data["choices"][0]["message"].get("content") or ""
+        used = int((data.get("usage") or {}).get("total_tokens") or 0)
+        with _lock:
+            STATS["calls"] += 1
+            STATS["tokens"] += used
+            _KEY_STATE[k]["tokens"] += used
+        choice = (data.get("choices") or [{}])[0]
+        content = (choice.get("message") or {}).get("content") or ""
+        if choice.get("finish_reason") == "length" and grown < 1 and body["max_tokens"] < cap:
+            grown += 1                                 # reply cut off: one retry with a larger budget
+            body["max_tokens"] = min(cap, int(body["max_tokens"] * 1.8) + 256)
+            STATS["truncated"] = STATS.get("truncated", 0) + 1
+            continue
+        return content
     raise Unavailable("Groq: too many retries")
 
 
@@ -390,6 +564,12 @@ _AI_WORDS = ("delve, crucial, pivotal, comprehensive, furthermore, moreover, not
              "robust, seamless, intricate, landscape, realm, underscore, showcase, tapestry, multifaceted, "
              "paramount, meticulous, holistic, foster, testament, ever-evolving")
 
+# Replies list ONLY the sentences that were changed; every other id goes into "unchanged". Echoing unchanged
+# sentences was most of the output tokens before; ids missing from both lists are treated as "not checked" and the
+# caller retries them, so nothing is silently skipped. (The AI-word list is in PR.PROMPT, rule D2.)
+_DIFF_RULE = ('Every id must appear exactly once: in "items" if you changed (or rejected) that sentence, otherwise '
+              'in "unchanged". Never put an unchanged sentence in "items".')
+
 SYSTEM = f"""You are the quality checker of an academic paraphrasing tool.
 Each item has A (the author's original sentence) and B (the tool's paraphrase of A).
 Keep B's wording and sentence structure. Make only the minimum edits to B so that:
@@ -401,11 +581,12 @@ Keep B's wording and sentence structure. Make only the minimum edits to B so tha
    Reject B if it contains a non-existent word, a wrong-sense word that a reviewer would notice (e.g. "spotted"
    for "repeated", "tracks" for "reports"), first person singular, "you", contractions, opinions ("absurd to me"),
    or a redundant phrase ("an optimization framework for optimization") that cannot be fixed with small edits.
-Do not copy A back; keep B's paraphrasing wherever it is correct. Do not add these words: {_AI_WORDS}. No em dashes.
-status: "ok" if B needed no change, "fixed" if you edited it, "reject" if B cannot be repaired with small edits.
-Reject B if it breaks any rule below that small edits cannot fix.
+Do not copy A back; keep B's paraphrasing wherever it is correct. No em dashes.
+status: "fixed" if you edited B, "reject" if B cannot be repaired with small edits (or breaks a rule below that small
+edits cannot fix). A B that needs no change is "unchanged".
 {PR.PROMPT}
-Answer only with JSON: {{"items": [{{"id": 1, "status": "ok|fixed|reject", "text": "final B"}}, ...]}} — one per id."""
+Answer only with JSON: {{"items": [{{"id": 2, "status": "fixed", "text": "final B"}}, {{"id": 4, "status": "reject"}}],
+"unchanged": [1, 3]}}. {_DIFF_RULE}"""
 
 
 def _surface_terms(text, glossary):
@@ -425,15 +606,38 @@ def _surface_terms(text, glossary):
     return sorted(terms, key=str.lower)
 
 
-def _parse(content, n):
-    """-> list of n (status, text) tuples; (None, None) when an item is missing."""
+def _load_json(content):
+    """The JSON object of a reply. A reply cut off mid-way is salvaged: every complete item is kept."""
+    content = content or ""
+    m = re.search(r"\{.*\}", content, re.S)
+    if m:
+        try:
+            data = json.loads(m.group(0))
+            if isinstance(data, dict):
+                return data
+        except Exception:
+            pass
+    dec, items = json.JSONDecoder(), []
+    for mm in re.finditer(r'\{\s*"id"\s*:', content):
+        try:
+            obj, _ = dec.raw_decode(content, mm.start())
+            if isinstance(obj, dict):
+                items.append(obj)
+        except Exception:
+            continue
+    data = {"items": items, "_salvaged": True}
+    u = re.search(r'"unchanged"\s*:\s*\[([^\]]*)\]', content)
+    if u:
+        data["unchanged"] = re.findall(r"\d+", u.group(1))
+    return data
+
+
+def _parse(content, n, inputs=None):
+    """-> list of n (status, text) tuples; (None, None) when an item is missing.
+    Ids listed under "unchanged" give ("ok", inputs[i]) (the sentence as sent)."""
     out = [(None, None)] * n
-    m = re.search(r"\{.*\}", content or "", re.S)
-    if not m:
-        return out
-    try:
-        data = json.loads(m.group(0))
-    except Exception:
+    data = _load_json(content)
+    if not data:
         return out
     out = list(out)
     for k, it in enumerate(data.get("items") or data.get("results") or []):
@@ -446,7 +650,80 @@ def _parse(content, n):
         st = str(it.get("status") or "").lower().strip()
         t = it.get("text")
         if 1 <= i <= n:
-            out[i - 1] = (st, t.strip() if isinstance(t, str) else None)
+            t = t.strip() if isinstance(t, str) and t.strip() else None
+            if t is None and st not in ("reject", "rejected") and inputs is not None:
+                st, t = "ok", inputs[i - 1]            # listed without text: nothing to change
+            out[i - 1] = (st, t)
+    for u in data.get("unchanged") or data.get("ok") or []:
+        try:
+            i = int(u)
+        except (TypeError, ValueError):
+            continue
+        if 1 <= i <= n and out[i - 1] == (None, None) and inputs is not None:
+            out[i - 1] = ("ok", inputs[i - 1])
+    if inputs is not None and "unchanged" not in data and "ok" not in data and not data.get("_salvaged") \
+            and isinstance(data.get("items"), list):
+        # a complete reply that listed only its changes (no "unchanged" list): the rest needed no change
+        out = [("ok", inputs[i]) if o == (None, None) else o for i, o in enumerate(out)]
+    return out
+
+
+# ------------------------------------------------------------------ per-sentence cache (deterministic steps only)
+# A sentence already checked (same step, same text) is not sent again: the verification rounds of the proofread,
+# re-runs after a quota stop and repeated Smart Humanize passes re-check many identical sentences.
+from collections import OrderedDict as _OD
+_CACHE = _OD()
+_CACHE_MAX = int(os.getenv("GROQ_CACHE", "20000"))
+
+
+def _cache_get(kind, key):
+    if not _CACHE_MAX:
+        return None
+    with _lock:
+        v = _CACHE.get((kind, key))
+        if v is not None:
+            _CACHE.move_to_end((kind, key))
+            STATS["cache_hits"] = STATS.get("cache_hits", 0) + 1
+        return v
+
+
+def _cache_put(kind, key, value):
+    if not _CACHE_MAX or value is None:
+        return
+    with _lock:
+        _CACHE[(kind, key)] = value
+        while len(_CACHE) > _CACHE_MAX:
+            _CACHE.popitem(last=False)
+
+
+def _run_batches(kind, items, sizes, max_items, max_chars, worker, progress, lo, hi, label):
+    """Shared batch runner: packs `items` by count and size, runs `worker(batch_items)` GROQ_PARALLEL at a time
+    (in the caller's provider and reasoning setting) and returns one result per item (None = not checked).
+    Raises Unavailable only if no batch at all could be done."""
+    from concurrent.futures import ThreadPoolExecutor, as_completed
+    out = [None] * len(items)
+    batches = _pack(sizes, max_items, max_chars)
+    if not batches:
+        return out
+    done, ok_batches, first_err = 0, 0, None
+    prov, reas = PROVIDER.get(), REASONING.get()
+    with ThreadPoolExecutor(max_workers=min(_par(), len(batches))) as pool:
+        futs = {pool.submit(_in_provider, prov, worker, [items[i] for i in b], _reasoning=reas): b for b in batches}
+        for f in as_completed(futs):
+            b = futs[f]
+            done += 1
+            try:
+                res = f.result()
+            except Unavailable as err:
+                first_err = first_err or err
+                continue
+            ok_batches += 1
+            for i, r in zip(b, res):
+                out[i] = r
+            if progress is not None:
+                progress(lo + (hi - lo) * done / len(batches), desc=f"{label}: {done}/{len(batches)} batches")
+    if not ok_batches and first_err is not None:
+        raise first_err
     return out
 
 
@@ -454,9 +731,9 @@ def _check_batch(pairs, glossary):
     terms = _surface_terms(" ".join(a for a, _ in pairs), glossary)
     user = ("PROTECTED TERMS: " + ("; ".join(terms) if terms else "(none)") + "\n\n"
             + "\n\n".join(f"{i}.\nA: {a}\nB: {b}" for i, (a, b) in enumerate(pairs, 1)))
-    max_tokens = min(8000, 400 + int(sum(len(b) for _, b in pairs) * 0.6))
+    max_tokens = _out_budget(sum(len(b) for _, b in pairs), len(pairs))
     content = _chat([{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}], 0.1, max_tokens)
-    return _parse(content, len(pairs))
+    return _parse(content, len(pairs), [b for _, b in pairs])
 
 
 PARALLEL = max(1, int(os.getenv("GROQ_PARALLEL", str(min(12, 4 * max(1, len(KEYS)))))))
@@ -468,41 +745,43 @@ def review(pairs, glossary, progress=None, lo=0.85, hi=0.9):
          None  -> rejected: keep the original sentence
          False -> not checked (Groq unavailable for this batch): caller uses its own rules
     Batches run GROQ_PARALLEL at a time. Raises Unavailable only if no batch at all could be checked."""
-    from concurrent.futures import ThreadPoolExecutor, as_completed
     n = len(pairs)
     out = [False] * n
-    starts = list(range(0, n, BATCH))
-    if not starts:
+    if not n:
+        return out
+    todo = []
+    for k, p in enumerate(pairs):
+        hit = _cache_get("review", p)
+        if hit is not None:
+            out[k] = hit[1] if hit[0] != "reject" else None
+        else:
+            todo.append(k)
+    if not todo:
         return out
-    done, ok_batches, first_err = 0, 0, None
     if progress is not None:
-        progress(lo, desc=f"Groq quality check of {n} sentences")
-    with ThreadPoolExecutor(max_workers=min(_par(), len(starts))) as pool:
-        futs = {pool.submit(_in_provider, PROVIDER.get(), _check_batch, pairs[st:st + BATCH], glossary): st for st in starts}
-        for f in as_completed(futs):
-            st = futs[f]
-            done += 1
-            try:
-                res = f.result()
-            except Unavailable as err:
-                first_err = first_err or err
-                continue
-            ok_batches += 1
-            for k, ((a, b), (status_, t)) in enumerate(zip(pairs[st:st + BATCH], res)):
-                with _lock:
-                    STATS["checked"] += 1
-                    if status_ == "reject":
-                        STATS["rejected"] += 1
-                    elif t:
-                        STATS["fixed" if status_ == "fixed" and t.strip() != b.strip() else "ok"] += 1
-                if status_ == "reject":
-                    out[st + k] = None
-                elif t:
-                    out[st + k] = t
-            if progress is not None:
-                progress(lo + (hi - lo) * done / len(starts), desc=f"Groq quality check: {done}/{len(starts)} batches")
-    if not ok_batches and first_err is not None:
-        raise first_err
+        progress(lo, desc=f"Groq quality check of {len(todo)} sentences")
+    res = _run_batches("review", [pairs[k] for k in todo], [len(pairs[k][0]) + len(pairs[k][1]) for k in todo],
+                       BATCH, BATCH_CHARS, lambda ps: _check_batch(ps, glossary), progress, lo, hi,
+                       "Groq quality check")
+    for k, r in zip(todo, res):
+        if r is None:
+            continue
+        status_, t = r
+        a, b = pairs[k]
+        if status_ is None and t is None:              # missing from the reply: not checked
+            continue
+        with _lock:
+            STATS["checked"] += 1
+            if status_ in ("reject", "rejected"):
+                STATS["rejected"] += 1
+            elif t:
+                STATS["fixed" if status_ == "fixed" and t.strip() != b.strip() else "ok"] += 1
+        if status_ in ("reject", "rejected"):
+            out[k] = None
+            _cache_put("review", pairs[k], ("reject", None))
+        elif t:
+            out[k] = t
+            _cache_put("review", pairs[k], ("ok", t))
     return out
 
 
@@ -514,51 +793,46 @@ singular/plural, prepositions, missing or doubled words, run-on sentences and br
 Use the smallest change that makes the sentence correct; keep its meaning and its wording wherever it is already
 correct. Do NOT add new facts, names, people or numbers. Do NOT replace a correct word with a synonym.
 Keep every technical term, acronym, formula, symbol, number, unit and citation (e.g. [12], (Kumar et al., 2021))
-exactly as written, including every item in PROTECTED TERMS. Do not add these words: {_AI_WORDS}. No em dashes.
-If a sentence has no error, return it unchanged.
+exactly as written, including every item in PROTECTED TERMS. No em dashes.
+Check every sentence carefully; a sentence with no error at all is "unchanged".
 {PR.PROMPT}
-Answer only with JSON: {{"items": [{{"id": 1, "text": "corrected sentence"}}, ...]}} — one per id."""
+Answer only with JSON: {{"items": [{{"id": 2, "text": "full corrected sentence"}}], "unchanged": [1, 3]}}. {_DIFF_RULE}"""
 
 
 def _grammar_batch(sents, glossary):
     terms = _surface_terms(" ".join(sents), glossary)
     user = ("PROTECTED TERMS: " + ("; ".join(terms) if terms else "(none)") + "\n\n"
             + "\n".join(f"{i}. {t}" for i, t in enumerate(sents, 1)))
-    max_tokens = min(8000, 400 + int(sum(len(t) for t in sents) * 0.6))
+    max_tokens = _out_budget(sum(len(t) for t in sents), len(sents))
     content = _chat([{"role": "system", "content": GRAMMAR_SYSTEM}, {"role": "user", "content": user}], 0.0, max_tokens)
-    return [t for _, t in _parse(content, len(sents))]
+    return [t for _, t in _parse(content, len(sents), sents)]
 
 
 def grammar(sentences, glossary, progress=None, lo=0.9, hi=0.95, size=None):
     """Grammar-only correction. Returns one entry per sentence: the corrected text (may equal the input),
     or None if that batch could not be checked. Raises Unavailable only if nothing could be checked."""
-    from concurrent.futures import ThreadPoolExecutor, as_completed
     n = len(sentences)
     out = [None] * n
-    size = size or BATCH
-    starts = list(range(0, n, size))
-    if not starts:
+    if not n:
+        return out
+    prov = PROVIDER.get()
+    todo = []
+    for k, t in enumerate(sentences):
+        hit = _cache_get("grammar:" + prov, t)
+        if hit is not None:
+            out[k] = hit
+        else:
+            todo.append(k)
+    if not todo:
         return out
-    done, ok_batches, first_err = 0, 0, None
     if progress is not None:
-        progress(lo, desc=f"Final grammar check of {n} sentences (Groq)")
-    with ThreadPoolExecutor(max_workers=min(_par(), len(starts))) as pool:
-        futs = {pool.submit(_in_provider, PROVIDER.get(), _grammar_batch, sentences[st:st + size], glossary): st for st in starts}
-        for f in as_completed(futs):
-            st = futs[f]
-            done += 1
-            try:
-                res = f.result()
-            except Unavailable as err:
-                first_err = first_err or err
-                continue
-            ok_batches += 1
-            for k, t in enumerate(res):
-                out[st + k] = t
-            if progress is not None:
-                progress(lo + (hi - lo) * done / len(starts), desc=f"Final grammar check: {done}/{len(starts)} batches")
-    if not ok_batches and first_err is not None:
-        raise first_err
+        progress(lo, desc=f"Final grammar check of {len(todo)} sentences (Groq)")
+    res = _run_batches("grammar", [sentences[k] for k in todo], [len(sentences[k]) for k in todo],
+                       size or BATCH, BATCH_CHARS, lambda ss: _grammar_batch(ss, glossary), progress, lo, hi,
+                       "Final grammar check")
+    for k, t in zip(todo, res):
+        out[k] = t
+        _cache_put("grammar:" + prov, sentences[k], t)
     return out
 
 
@@ -568,12 +842,16 @@ The text was produced by an automatic paraphrasing tool, which sometimes substit
 joins phrases so that a sentence no longer makes sense. It is given paragraph by paragraph; each numbered item is
 one sentence of the CURRENT text. When available, the author's ORIGINAL paragraph is shown first: use it ONLY to
 understand what each sentence is meant to say.
-For every numbered sentence:
-1. Clarity first. If the sentence is unclear, garbled, ungrammatical in a way that hides its meaning, or says
-   something different from the original, REPAIR it: rewrite it so that it says exactly what the original means,
-   in clear academic English, keeping the current wording wherever that wording is correct. status "repaired".
-2. Otherwise edit it lightly (status "edited") or keep it (status "ok"):
-   - grammar: agreement, articles, verb forms, tense, prepositions, missing or doubled words, run-on sentences;
+Read each sentence the way a journal reviewer would and ask: is it grammatical, is every word the right word in
+this technical context, is it clear on first reading, does it say what the original says, and does it follow from
+the sentence before? For every numbered sentence:
+1. Clarity first. If the sentence is unclear, garbled, ungrammatical in a way that hides its meaning, uses a
+   wrong-sense word, or says something different from the original, REPAIR it: rewrite it so that it says exactly
+   what the original means, in clear academic English, keeping the current wording wherever that wording is
+   correct. status "repaired".
+2. Otherwise edit it (status "edited") or leave it unchanged:
+   - grammar: agreement, articles, verb forms, tense, prepositions, punctuation, missing or doubled words,
+     run-on sentences, dangling modifiers;
    - academic register: precise technical words instead of informal, odd or wrong-sense words (e.g. "spotted"
      -> "repeated", "comprehension" -> "understanding" where that is meant); remove redundancy; no first person
      singular, no "you", no contractions, no opinions;
@@ -585,6 +863,7 @@ For every numbered sentence:
    - remove leftovers of chatbot writing: labels ("Revised sentence:", "Here is"), "Certainly", markdown (**, #),
      and AI-typical filler ("It is important to note that", "plays a crucial role", "delve into", "in today's
      world", "a testament to", "navigating the landscape"); say the same thing plainly.
+   A sentence that is already correct, clear and well connected is "unchanged": do not make cosmetic edits.
 Rules: keep the meaning, claims and hedging of the original exactly. Do NOT add facts, names, numbers, results or
 citations that are not in the sentence or the original. Do NOT merge, split, drop or reorder sentences: return
 exactly one sentence per id. Keep every technical term, acronym, formula, symbol, number, unit and citation (e.g.
@@ -592,7 +871,7 @@ exactly one sentence per id. Keep every technical term, acronym, formula, symbol
 original back word for word unless the current sentence cannot be repaired any other way.
 Examples of sentences that MUST be repaired (do this kind of deep correction, not a light touch-up):
 - "The above are necessary to give readers a thorough comprehension of the current landscape of knowledge
-  surrounding a specific subject." -> "These elements are necessary to provide readers with a comprehensive
+  surrounding a specific subject." -> "These elements are necessary to provide readers with a thorough
   understanding of the current state of knowledge on the subject."
 - "Their hybrid purposed to do a 25-item questionnaire with follow-up interviews together qualitative interviews
   combined." -> "They used a mixed-methods design that combined a 25-item questionnaire with follow-up
@@ -603,11 +882,10 @@ Examples of sentences that MUST be repaired (do this kind of deep correction, no
 - "The survey aimed to identify the literature deficiencies which would be corrected." -> "The review aimed to
   identify the gaps in the literature that this study addresses."
 Every author name, year, finding, statistic (β, p, r, R², %, n), variable name, citation, heading number and
-technical meaning must stay exactly as in the sentence / original.
-Do not add these words: {_AI_WORDS}. No em dashes.
+technical meaning must stay exactly as in the sentence / original. No em dashes.
 {PR.PROMPT}
-Answer only with JSON: {{"items": [{{"id": 1, "status": "ok|edited|repaired", "text": "sentence"}}, ...]}} —
-one per id."""
+Answer only with JSON: {{"items": [{{"id": 2, "status": "edited", "text": "sentence"}}, {{"id": 3, "status":
+"repaired", "text": "sentence"}}], "unchanged": [1, 4]}}. {_DIFF_RULE}"""
 
 
 def _polish_batch(items, glossary):
@@ -626,10 +904,10 @@ def _polish_batch(items, glossary):
             last = g
         lines.append(f"{i}. {t}")
     user = "PROTECTED TERMS: " + ("; ".join(terms) if terms else "(none)") + "\n" + "\n".join(lines)
-    max_tokens = min(8000, 600 + int(sum(len(t) for t in sents) * 0.8))
+    max_tokens = _out_budget(sum(len(t) for t in sents), len(sents), 1.1)
     content = _chat([{"role": "system", "content": POLISH_SYSTEM}, {"role": "user", "content": user}], 0.2,
                     max_tokens)
-    return _parse(content, len(sents))
+    return _parse(content, len(sents), sents)
 
 
 def polish(sentences, groups=None, glossary=(), progress=None, lo=0.9, hi=0.95, size=None, originals=None):
@@ -646,28 +924,34 @@ def polish(sentences, groups=None, glossary=(), progress=None, lo=0.9, hi=0.95,
     groups = list(groups) if groups is not None else list(range(n))
     originals = list(originals) if originals is not None else [None] * n
     size = size or max(BATCH, 16)
-    batches, cur = [], []
+    # whole paragraphs per batch (long ones are split), limited by sentence count and by size; the original
+    # paragraph is sent once per paragraph, so it counts once
+    batches, cur, cc = [], [], 0
     i = 0
-    while i < n:                                       # whole paragraphs per batch; long ones are split
+    while i < n:
         j = i
         while j < n and groups[j] == groups[i]:
             j += 1
         para = list(range(i, j))
-        if cur and len(cur) + len(para) > size:
+        pc = sum(len(sentences[k]) for k in para) + len(originals[i] or "")
+        if cur and (len(cur) + len(para) > size or cc + pc > BATCH_CHARS):
             batches.append(cur)
-            cur = []
+            cur, cc = [], 0
         while len(para) > size:
             batches.append(para[:size])
             para = para[size:]
         cur += para
+        cc += pc
         i = j
     if cur:
         batches.append(cur)
     done, ok_batches, first_err = 0, 0, None
     if progress is not None:
         progress(lo, desc=f"Final polish of {n} sentences: clarity, grammar, academic wording and flow (Groq)")
+    prov, reas = PROVIDER.get(), REASONING.get()
     with ThreadPoolExecutor(max_workers=min(_par(), len(batches))) as pool:
-        futs = {pool.submit(_in_provider, PROVIDER.get(), _polish_batch, [(groups[k], sentences[k], originals[k]) for k in b], glossary): b
+        futs = {pool.submit(_in_provider, prov, _polish_batch, [(groups[k], sentences[k], originals[k]) for k in b],
+                            glossary, _reasoning=reas): b
                 for b in batches}
         for f in as_completed(futs):
             b = futs[f]
@@ -808,17 +1092,11 @@ def _variants_batch(items, glossary, n, min_change=0):
                  "from the sentence: use different (correct, common) vocabulary and change the clause order or "
                  "voice where natural. Protected terms, numbers and citations stay exactly as written, and every "
                  "rule above still holds.")
-    max_tokens = min(8000, 500 + int(sum(len(t) for t in sents) * 0.6 * (n + 1)))
+    max_tokens = _out_budget(sum(len(t) for t in sents) * n, len(sents) * n, 1.1)
     content = _chat([{"role": "system", "content": VARIANTS_SYSTEM.replace("{n}", str(n))},
                      {"role": "user", "content": user}], 0.7, max_tokens)
     out = [None] * len(items)
-    m = re.search(r"\{.*\}", content or "", re.S)
-    if not m:
-        return out
-    try:
-        data = json.loads(m.group(0))
-    except Exception:
-        return out
+    data = _load_json(content)
     for k, it in enumerate(data.get("items") or []):
         if not isinstance(it, dict):
             continue
@@ -835,32 +1113,76 @@ def _variants_batch(items, glossary, n, min_change=0):
 def variants(sentences, glossary=(), originals=None, n=3, progress=None, lo=0.8, hi=0.85, min_change=0):
     """Up to n correct, human-sounding alternatives per sentence (list, or None if its batch failed).
     min_change: at least this share (%) of the words should differ (vocabulary-change setting)."""
-    from concurrent.futures import ThreadPoolExecutor, as_completed
     N = len(sentences)
-    out = [None] * N
     if not N:
-        return out
+        return [None] * N
     originals = list(originals) if originals is not None else [None] * N
-    size = 8
-    starts = list(range(0, N, size))
-    ok_batches, first_err = 0, None
     if progress is not None:
         progress(lo, desc=f"Finding natural-sounding correct versions of {N} sentence(s)")
-    with ThreadPoolExecutor(max_workers=min(_par(), len(starts))) as pool:
-        futs = {pool.submit(_in_provider, PROVIDER.get(), _variants_batch,
-                            list(zip(sentences[st:st + size], originals[st:st + size])), glossary, n,
-                            min_change): st
-                for st in starts}
-        for f in as_completed(futs):
-            st = futs[f]
-            try:
-                res = f.result()
-            except Unavailable as err:
-                first_err = first_err or err
-                continue
-            ok_batches += 1
-            for k, r in enumerate(res):
-                out[st + k] = r
-    if not ok_batches and first_err is not None:
-        raise first_err
-    return out
+    # every version is about as long as the sentence, so a batch holds fewer sentences than a checking batch
+    sizes = [len(s) * (n + 1) + len(o or "") for s, o in zip(sentences, originals)]
+    return _run_batches("variants", list(zip(sentences, originals)), sizes, 8, BATCH_CHARS * 2,
+                        lambda its: _variants_batch(its, glossary, n, min_change), None, lo, hi, "Alternatives")
+
+
+# ---------------------------------------------------------------- one-call final check of a whole document
+DOC_SYSTEM = POLISH_SYSTEM.replace(
+    "It is given paragraph by paragraph; each numbered item is\none sentence of the CURRENT text.",
+    "The WHOLE document is given at once, paragraph by paragraph; each numbered item is one sentence of the CURRENT "
+    "text.").replace(
+    "Answer only with JSON:", "DOCUMENT-LEVEL CHECKS (you see the whole text, so also):\n"
+    "- every construct listed under CONSTRUCTS keeps exactly that name everywhere (never pleasure/contentment/"
+    "happiness/enjoyment for satisfaction, closeness for intimacy, allegiance/devotion for loyalty);\n"
+    "- the same group words as the original (men/women, male/female, genders), the same numbers and the same "
+    "statistics, including operators: never add, remove or change <, >, = (if one is missing, leave it and list it "
+    "in notes);\n"
+    "- every sentence complete (no cut-off endings, no fragments, no two sentences run together), sentences flow "
+    "into each other within each paragraph, commas in lists and after introductory phrases.\n"
+    "In \"notes\", list (short, at most 12) problems you must NOT fix yourself because they need the author: "
+    "inconsistent abbreviations (e.g. EWB vs EWD), a construct named two ways, missing statistics operators, "
+    "results described in the wrong direction, an unexplained place or name.\n"
+    "List in \"items\" ONLY the sentences you changed; leave every correct sentence out.\n"
+    "Answer only with JSON:").replace(
+    '"unchanged": [1, 4]}. ' + _DIFF_RULE, '"notes": ["..."]}').replace(
+    'List in \"items\" ONLY the sentences you changed;',
+    'status: "repaired" = unclear/garbled/meaning fixed, "fixed" = a grammar, spelling, punctuation or wrong-term '
+    'error corrected, "edited" = style or flow only. List in \"items\" ONLY the sentences you changed;')
+
+
+def proofread_document(sentences, groups, originals, glossary, constructs=()):
+    """ONE request for the whole document: clarity repair + grammar + academic wording + flow + document-level
+    consistency. Returns ([(status, text) per sentence], notes). Unchanged sentences come back as ("ok", text).
+    Raises Unavailable when the provider cannot take the request (too large, quota): the caller falls back to the
+    batched proofread."""
+    n = len(sentences)
+    terms = _surface_terms(" ".join(sentences), glossary)
+    lines, last, para = [], object(), 0
+    for i, (g, t, orig) in enumerate(zip(groups, sentences, originals), 1):
+        if g != last:
+            para += 1
+            lines.append(f"\n[Paragraph {para}]")
+            if orig:
+                lines.append(f"ORIGINAL: {orig}")
+                lines.append("CURRENT:")
+            last = g
+        lines.append(f"{i}. {t}")
+    user = ("CONSTRUCTS: " + ("; ".join(constructs) if constructs else "(none)") + "\n"
+            + "PROTECTED TERMS: " + ("; ".join(terms) if terms else "(none)") + "\n" + "\n".join(lines))
+    chars = sum(len(t) for t in sentences)
+    max_tokens = int(chars / 3.3 * 0.6) + 20 * n + 800       # a reply that changes up to ~60% of the text
+    tok_c, tok_t = OUT_CAP.set(int(os.getenv("PROOF_SINGLE_MAX_OUT", "32000"))), \
+        TIMEOUT.set(float(os.getenv("PROOF_SINGLE_TIMEOUT", "600")))
+    try:
+        content = _chat([{"role": "system", "content": DOC_SYSTEM}, {"role": "user", "content": user}], 0.2,
+                        max_tokens)
+    finally:
+        OUT_CAP.reset(tok_c)
+        TIMEOUT.reset(tok_t)
+    data = _load_json(content)
+    if not data or ("items" not in data and "notes" not in data):
+        raise Unavailable("one-call proofread: no usable answer")
+    res = _parse(content, n, sentences)
+    if data.get("_salvaged"):                              # cut-off reply: the missing ids were NOT checked
+        res = [r if r != (None, None) else (None, None) for r in res]
+    notes = [str(x).strip() for x in (data.get("notes") or []) if str(x).strip()][:12]
+    return res, notes

humanizer.py

@@ -1,4 +1,4 @@
-"""Humanizer engine (Parrot T5 paraphraser) + stylometric indicators + DOCX helpers."""
+"""Humanizer engine (Parrot T5 paraphraser, optional detector-guided "adversarial" decoding) + stylometric indicators + DOCX helpers."""
 import base64
 import contextvars
 import hashlib
@@ -31,6 +31,8 @@ from docx import Document
 from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
 
 import runtime as RT
+import adversarial as ADV
+import adaptive as AD
 import textparse as TP
 import groqai as G
 import meaning as MEANING
@@ -74,8 +76,15 @@ if LOGO_PATH.exists():
 # ---------------------------------------------------------------------------
 # Model (kept in CPU RAM while idle; moved to GPU only inside @spaces.GPU)
 # ---------------------------------------------------------------------------
-MODEL_ID = os.getenv("PARROT_MODEL", "prithivida/parrot_paraphraser_on_T5")
+# Paraphrase model: Parrot (prithivida/parrot_paraphraser_on_T5) by default. PARA_MODEL (or the older PARROT_MODEL)
+# can name another T5 paraphraser with the same "paraphrase: " input, e.g. humarin/chatgpt_paraphraser_on_T5_base.
+# If the chosen model cannot be loaded, Parrot is used so the Space still starts.
+FALLBACK_MODEL = "prithivida/parrot_paraphraser_on_T5"
+MODEL_ID = (os.getenv("PARA_MODEL") or os.getenv("PARROT_MODEL") or FALLBACK_MODEL).strip()
 tokenizer = model = None
+# mild penalty on repeating input tokens: pushes the model away from copying the sentence back (1.0 = off);
+# set again in load_model() if the chosen model could not be loaded and Parrot is used instead
+REP_PENALTY = float(os.getenv("PARA_REP_PENALTY", "1.0" if "parrot" in MODEL_ID.lower() else "1.15"))
 # ZeroGPU / dedicated GPU: keep the model on "cuda" permanently (ZeroGPU restores it for each
 # @spaces.GPU call), so several users can paraphrase at the same time — no global lock.
 ON_GPU = ZERO or torch.cuda.is_available()
@@ -86,11 +95,21 @@ _load_lock = threading.Lock()
 
 def load_model():
     """Load the paraphraser once (at startup; SKIP_MODEL_LOAD=1 delays it to the first use, for tests and tools)."""
-    global tokenizer, model, _cpu_model
+    global tokenizer, model, _cpu_model, MODEL_ID, REP_PENALTY
     with _load_lock:
         if model is None:
-            tok = AutoTokenizer.from_pretrained(MODEL_ID)
-            m = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID).eval()
+            try:
+                tok = AutoTokenizer.from_pretrained(MODEL_ID)
+                m = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID).eval()
+            except Exception as _err:
+                if MODEL_ID == FALLBACK_MODEL:
+                    raise
+                print(f"[humanizer] {MODEL_ID} could not be loaded ({_err}); using {FALLBACK_MODEL}", flush=True)
+                MODEL_ID = FALLBACK_MODEL
+                REP_PENALTY = float(os.getenv("PARA_REP_PENALTY", "1.0"))
+                tok = AutoTokenizer.from_pretrained(MODEL_ID)
+                m = AutoModelForSeq2SeqLM.from_pretrained(MODEL_ID).eval()
+            print(f"[humanizer] paraphrase model: {MODEL_ID}", flush=True)
             if ON_GPU:
                 m.to("cuda")
             else:
@@ -415,6 +434,8 @@ def _generate(texts, temperature, device, batch_size, m=None, k=1):
                 num_return_sequences=k,
                 max_new_tokens=min(180, int(longest * 1.35) + 12),
                 no_repeat_ngram_size=3,
+                repetition_penalty=REP_PENALTY,
+                bad_words_ids=AD.bad_words_ids(tokenizer),      # words learned to cause rejected rewrites
             )
             decoded = tokenizer.batch_decode(generated.detach().cpu(), skip_special_tokens=True)
             for n_, src in enumerate(chunk):
@@ -466,6 +487,8 @@ AI_CHANGE_TARGET = float(os.getenv("HUM_AI_CHANGE_TARGET", "45"))
 MAX_TEMP = float(os.getenv("HUM_MAX_TEMP", "1.0"))        # highest sampling variation ever used
 MAX_VOCAB = float(os.getenv("HUM_MAX_VOCAB", "45"))       # highest new-wording target per sentence (%)
 AI_ROUNDS = max(0, int(os.getenv("HUM_AI_ROUNDS", "2")))
+# Smart Humanize (adaptive): sentences whose AI score did not fall last pass get detector-guided decoding too.
+FORCE_ADV = contextvars.ContextVar("force_adv", default=False)
 # Smart Humanize sets this per pass (its passes already rewrite again, so fewer inner rounds are needed).
 EXTRA_ROUNDS = contextvars.ContextVar("extra_rounds", default=None)
 # Vocabulary change (%) slider, per job. Box ticked: the new-wording target (None -> AI_CHANGE_TARGET).
@@ -556,6 +579,35 @@ def _detector_scores(texts):
         return None
 
 
+def _adversarial(chunks, flat, cand, temp, progress, at):
+    """Adversarial Paraphrasing: one detector-guided Parrot version per piece, added to cand[piece] (every piece of
+    a sentence gets one, so the pieces stay aligned). Returns True if it ran."""
+    if ZERO and not ADV.ON_ZERO:
+        return False                                   # ZeroGPU: no nested GPU allocation for the detector
+    todo = sorted(set(chunks))
+    if not todo or len(todo) > ADV.MAX_PIECES:         # over the per-job cap (ADV_MAX): Parrot's versions only
+        if todo:
+            print(f"[humanizer] adversarial paraphrasing skipped: {len(todo)} pieces > ADV_MAX={ADV.MAX_PIECES}",
+                  flush=True)
+        return False
+    if progress is not None:
+        progress(at, desc=f"Detector-guided rewriting of {len(todo)} sentence piece(s)")
+    try:
+        if ON_GPU and not ZERO and torch.cuda.is_available():
+            outs = ADV.generate([flat[x] for x in todo], load_model(), tokenizer, "cuda", _detector_scores, temp)
+        else:
+            m = _get_cpu_model()
+            with RT.cpu_job():
+                outs = ADV.generate([flat[x] for x in todo], m, tokenizer, "cpu", _detector_scores, temp)
+    except Exception as err:
+        print(f"[humanizer] adversarial paraphrasing skipped: {err}", flush=True)
+        return False
+    for x, o in zip(todo, outs):
+        cand[x].append(o if o and o.strip() else flat[x])
+    print(f"[humanizer] adversarial paraphrasing: {len(todo)} piece(s)", flush=True)
+    return True
+
+
 def _paraphrase_local(texts, temperature, progress=None, lo=0.15, hi=0.85, mode="Auto"):
     """Returns (paragraphs, engine_note).
     1. Parrot writes CANDIDATES paraphrases per sentence (long sentences clause by clause), batched
@@ -583,8 +635,8 @@ def _paraphrase_local(texts, temperature, progress=None, lo=0.15, hi=0.85, mode=
             ctx[key] = doc_ctx.get(key) or (sent_map[si - 1][0] if si > a else "",
                                             sent_map[si + 1][0] if si + 1 < b else "")
     full_ai = FULL_AI.get()
-    deep = full_ai or BOOST.get()
-    k = CANDIDATES_AI if deep else CANDIDATES
+    deep = full_ai or BOOST.get()                      # deeper rewrite: asked for, or switched on by Smart Humanize
+    k = AD.candidates(CANDIDATES_AI if deep else CANDIDATES)   # more samples when few survive the checks
     target = change_target() if full_ai else min(90.0, float(VOCAB_TARGET.get() or 0))
     er = EXTRA_ROUNDS.get()
     rounds = 1 + ((AI_ROUNDS if er is None else er) if (full_ai or target) else 0)
@@ -625,6 +677,17 @@ def _paraphrase_local(texts, temperature, progress=None, lo=0.15, hi=0.85, mode=
 
     glossary = TP.build_glossary(texts) | TP.GLOSSARY.get()
 
+    def checked(orig, new, source, guarded):
+        """Guard result of one version + what the adaptive learner records about it."""
+        ok = guarded
+        if ok and AD.banned_in(orig, ok):                # a word learned to cause bad rewrites
+            ok = None
+        if new and new.strip() != orig.strip():
+            why = None if ok else (AD.banned_in(orig, new) and "banned word") or TP.integrity_issue(orig, new) \
+                or TP.quality_issue(orig, new) or ("lost term" if TP.lost_terms(orig, new, glossary) else "other")
+            AD.record_candidate(orig, new, source, bool(ok), why)
+        return ok
+
     def build(si):
         """Valid candidates of sentence si (terms restored, guard passed) + first raw paraphrase."""
         s, c0, n = sent_map[si]
@@ -634,23 +697,27 @@ def _paraphrase_local(texts, temperature, progress=None, lo=0.15, hi=0.85, mode=
             pieces = [cand[c][j] for c in range(c0, c0 + n)]
             new = TP.merge_chunks(flat[c0:c0 + n], pieces) if n > 1 else pieces[0]
             raw0 = raw0 or new
-            ok = TP.guard(s, TP.restore_terms(s, new, glossary), glossary)
+            source = "adversarial" if (adv_used and j == k) else "parrot"
+            ok = checked(s, new, source, TP.guard(s, TP.restore_terms(s, new, glossary), glossary))
             if ok and TP._near_repeats(ok) - TP._near_repeats(s):
                 ok = None                                # restored term doubled up ("ethics of care ethics")
             if ok and ok.strip() != s.strip() and ok not in opts:
                 opts.append(ok)
+                src_of[(si, ok)] = source
         # "100% AI-written" mode only: plain-wording versions (chatbot vocabulary / set phrases removed) of each
         # paraphrase and of the original itself, so fully AI-written sentences always get a safe, more human option
         for v in lex[si]:                                # vocabulary-replaced versions of the ORIGINAL sentence
-            ok = TP.guard(ref[si], v, glossary)
+            ok = checked(ref[si], v, "lexsub", TP.guard(ref[si], v, glossary))
             if ok and ok.strip() != s.strip() and ok not in opts:
                 opts.append(ok)
-        for base in ((list(opts) + [s]) if deep else []):
+                src_of[(si, ok)] = "lexsub"
+        for base in ((list(opts) + [s]) if deep and AD.use_source("de_ai") else []):
             v = TP.de_ai(base)
             if v != base:
-                ok = TP.guard(s, TP.restore_terms(s, v, glossary), glossary)
+                ok = checked(s, v, "de_ai", TP.guard(s, TP.restore_terms(s, v, glossary), glossary))
                 if ok and ok.strip() != s.strip() and ok not in opts:
                     opts.append(ok)
+                    src_of[(si, ok)] = "de_ai"
         return opts, raw0 or s
 
     def change(si, o):
@@ -661,6 +728,8 @@ def _paraphrase_local(texts, temperature, progress=None, lo=0.15, hi=0.85, mode=
     todo = list(range(len(sent_map)))
     options, first_raw = [[] for _ in sent_map], [s for s, _, _ in sent_map]
     extra_rounds = meaning_dropped = 0
+    adv_used = False
+    src_of = {}                                          # (sentence, version) -> source, for the adaptive learner
     qstat = dict(n=0, meaning=0, fluency=0, second=0, sims=[])
     ref = [anchor_of(s) or s for s, _, _ in sent_map]          # the author's original sentence
     # stricter for technical sentences; a high vocabulary-change setting allows a little more distance (the
@@ -691,6 +760,9 @@ def _paraphrase_local(texts, temperature, progress=None, lo=0.15, hi=0.85, mode=
         chunks = [c for si in todo for c in range(sent_map[si][1], sent_map[si][1] + sent_map[si][2])]
         generate(chunks, min(1.5, temp0 + 0.25 * r), a_, b_,
                  "Paraphrasing" if r == 0 else f"Extra rewrite round {r} for")
+        if r == 0 and (ADV.wanted(full_ai) or FORCE_ADV.get()) and AD.use_source("adversarial"):
+            if _adversarial(chunks, flat, cand, temp0, progress, b_):
+                adv_used = True
         extra_rounds += r > 0
         for si in todo:
             options[si], first_raw[si] = build(si)
@@ -729,6 +801,8 @@ def _paraphrase_local(texts, temperature, progress=None, lo=0.15, hi=0.85, mode=
         note = "⚡ Processed on GPU."
     else:
         note = "🖥️ Processed on CPU."
+    if adv_used:
+        note += " 🧭 Detector-guided (adversarial) rewriting added one extra version per sentence."
 
     # ---- 3. most human candidate (AI detector); 100% AI mode: versions meeting the new-wording target first
     score_of = {}
@@ -739,8 +813,8 @@ def _paraphrase_local(texts, temperature, progress=None, lo=0.15, hi=0.85, mode=
         sc = _ctx_scores([(sent_map[si][0], o) for si, o in multi], ctx)
         if sc:
             score_of = {key: v for key, v in zip(multi, sc)}
-            note += (f" 🎯 Best of {k} Parrot versions per sentence chosen by the AI detector." if k > 1 else
-                     " 🎯 The AI detector chose between Parrot's version and the plain-wording version.")
+            note += (f" 🎯 Best of {k} rewritten versions per sentence chosen by the AI detector." if k > 1 else
+                     " 🎯 The AI detector chose between the rewritten version and the plain-wording version.")
 
     def rank_key(si, o):
         det = score_of.get((si, o), 1.0 if score_of else 0.0)
@@ -757,6 +831,11 @@ def _paraphrase_local(texts, temperature, progress=None, lo=0.15, hi=0.85, mode=
             for o in opts:
                 print(f"[debug]    {score_of.get((si, o), float('nan')):.2f} chg {change(si, o):3.0f}% | {o}", flush=True)
     chosen = [r_[0] if r_ else None for r_ in ranked]
+    for si, opts in enumerate(options):                  # adaptive learner: which source's version won
+        if len(opts) > 1 and chosen[si]:
+            for o in opts:
+                AD.record_win(src_of.get((si, o), "parrot"), o == chosen[si])
+    AD.save()
 
     # ---- 4. Groq: academic-wording check of the chosen sentences + rescue of sentences with no valid version.
     #      100% AI mode: Groq checks the GROQ_SHORTLIST most human versions of each sentence, and the most human
@@ -1013,7 +1092,7 @@ def _paraphrase_local(texts, temperature, progress=None, lo=0.15, hi=0.85, mode=
     if full_ai:
         hit = sum(1 for si, (s0, _, _) in enumerate(sent_map)
                   if sentences[si].strip() != s0.strip() and change(si, sentences[si]) >= target)
-        note += (f" 🤖 100% AI mode: {k} Parrot version{'s' if k != 1 else ''} per sentence"
+        note += (f" 🤖 100% AI mode: {k} rewritten version{'s' if k != 1 else ''} per sentence"
                  + (f", {extra_rounds} extra round(s)" if extra_rounds else "")
                  + (f"; {hit} of {len(sent_map)} sentences reach {target:.0f}% new wording." if target else ".")) \
             + f" ✍️ {changed} of {len(sent_map)} sentences rewritten."
@@ -1350,6 +1429,7 @@ def _docx_core(file_path, temperature, min_words, mode, progress):
 
     progress(0.05, desc="Opening Word document")
     doc = Document(str(source_path))
+    TP.set_document([p.text for p in collect_paragraphs(doc)])   # constructs / vocabulary of the whole document
 
     import detector as _D
     targets, originals = [], []
@@ -1409,6 +1489,7 @@ def _text_core(text, temperature, min_words, mode, progress, max_words=None):
     progress(0.05, desc="Reading pasted text")
     parts = TP.parse_paste(text)
     paragraphs = [p for p, _ in parts]
+    TP.set_document(paragraphs)
     idx = [i for i, p in enumerate(paragraphs) if should_process(p, min_words)]
     if not idx:
         raise gr.Error(f"No paragraph has at least {int(min_words)} words of prose. Lower the minimum and try again.")

proofread.py

@@ -1,575 +1,678 @@
-"""
-Rigorous proofread of a finished (humanized) Word document. Correct grammar comes first; a moderate AI-score rise
-is accepted for that, style-only edits must keep the AI score.
-
-  0. Local clean-up (no model): chatbot leftovers, markdown, doubled words / repeated phrases, AI filler words.
-  1. Grammar audit: the AI's grammar-only check marks every sentence that contains an error (plus sentences with
-     doubled terms, leftovers or broken English): these are REQUIRED fixes.
-  2. Full proofread (clarity repair against the original upload, grammar, academic wording, flow). For a required
-     sentence the full correction is used if its AI rise is at most PROOF_REQ_RISE, otherwise the minimal grammar
-     fix; a sentence WITH an error is never left uncorrected. Style-only edits must stay within PROOF_MAX_RISE.
-  3. Verification rounds (PROOF_ROUNDS): every corrected sentence is grammar-checked again until no error is found.
-  4. LanguageTool: last mechanical check.
-  5. Document check: if the AI score rose by more than PROOF_MAX_DOC_RISE points, style-only edits are undone
-     (riskiest first). Grammar fixes are never undone.
-Every change keeps terms / numbers / citations and the meaning. Only plain text runs are edited.
-
-Space variables: PROOF_REQ_RISE (0.35), PROOF_MAX_RISE (0.15), PROOF_MAX_DOC_RISE (5 points), PROOF_ROUNDS (2).
-"""
-import difflib
-import os
-import time
-import re
-
-import detector as D
-import docx_report as DX
-import groqai as G
-import humanizer as H
-import meaning as MEANING
-import fluency as FL
-import smart as S
-import textparse as TP
-
-MAX_RISE = float(os.getenv("PROOF_MAX_RISE", "0.15"))
-FLAG = float(os.getenv("PROOF_FLAG", "0.50"))
-MAX_DOC_RISE = float(os.getenv("PROOF_MAX_DOC_RISE", "1.0"))
-# a sentence with a real defect (doubled word / repeated phrase / chatbot leftover / broken English) may rise more
-DEFECT_RISE = float(os.getenv("PROOF_DEFECT_RISE", "0.25"))
-# grammar comes first: a sentence with an error is always corrected; its full correction is used if the AI rise
-# stays within PROOF_REQ_RISE (otherwise the minimal grammar fix)
-REQ_RISE = float(os.getenv("PROOF_REQ_RISE", "0.35"))
-ROUNDS = int(os.getenv("PROOF_ROUNDS", "1"))
-BATCH = int(os.getenv("PROOF_BATCH", "24"))             # sentences per request (fewer, larger requests = faster)
-# keep the AI score: a corrected sentence may be at most this much more AI-like than before; above it, correct
-# natural-sounding alternatives are tried (PROOF_AI_ROUNDS rounds, PROOF_VARIANTS versions each, plus Parrot)
-KEEP_RISE = float(os.getenv("PROOF_KEEP_RISE", "0.03"))
-AI_ROUNDS = int(os.getenv("PROOF_AI_ROUNDS", "1"))
-VARIANTS = int(os.getenv("PROOF_VARIANTS", "4"))
-PAR_TOL = int(os.getenv("PROOF_PAR_TOL", "0"))       # flagged words a paragraph may gain after balancing
-PARROT = os.getenv("PROOF_PARROT", "0") == "1"
-# which AI does the proofreading: Groq by default (PROOF_PROVIDER=openai for ChatGPT). If the chosen one is out of
-# quota, the other one is used automatically when its key is set.
-PROVIDER = os.getenv("PROOF_PROVIDER", "groq").lower()
-MAX_WAIT = float(os.getenv("PROOF_MAX_WAIT", "180"))     # seconds the proofread may wait for rate limits in total
-PARALLEL = int(os.getenv("PROOF_PARALLEL", "3"))          # requests at once (Groq: low, so limits are not hit)
-_NAME = {"groq": "Groq", "openai": "ChatGPT"}
-
-
-class _LLM:
-    """Calls Groq / ChatGPT for the proofread so that EVERY sentence gets an answer: failed batches are retried
-    (only the missing sentences), rate limits are waited out (up to PROOF_MAX_WAIT in total), and if the provider
-    is out of quota the other provider is used when its key is set."""
-
-    def __init__(self, rep):
-        self.rep = rep
-        self.order = [PROVIDER] + [p for p in ("groq", "openai") if p != PROVIDER]
-        self.order = [p for p in self.order if (G.OPENAI_KEY if p == "openai" else G.KEYS)]
-        self.k = 0
-        self.waited = 0.0
-        self.used = set()
-        self.dead = False
-
-    @property
-    def name(self):
-        return _NAME.get(self.order[self.k], self.order[self.k]) if self.order else "AI"
-
-    def _ready(self):
-        return G.available()
-
-    def _switch(self):
-        if self.k + 1 < len(self.order):
-            self.k += 1
-            print(f"[proofread] switching to {self.name}", flush=True)
-            return True
-        return False
-
-    def _blocked_for(self):
-        """Seconds until the current provider can be used again (large = daily quota / billing / bad key)."""
-        now = time.time()
-        if self.order[self.k] == "openai":
-            return max(0.0, G._oa["until"] - now) if G.OPENAI_KEY else 1e9
-        return max(G._down_until - now, G._next_free_in(), 0.0)
-
-    def _wait(self, why):
-        blocked = self._blocked_for()
-        if blocked > 90:                               # quota for the day / billing: waiting will not help
-            print(f"[proofread] {self.name} blocked for {int(blocked)}s ({why}); "
-                  f"groq: {G.STATS.get('last_error', '')[:120]} | openai: {G.OA_STATS.get('last_error', '')[:120]}",
-                  flush=True)
-            return False
-        t = min(30.0, max(3.0, blocked or 5.0))
-        if self.waited + t > MAX_WAIT:
-            return False
-        self.rep(None, f"Waiting {int(t)}s for the {self.name} rate limit ({why})")
-        time.sleep(t)
-        self.waited += t
-        return True
-
-    def run(self, fn, items, *args, missing=lambda r: r is None, **kw):
-        """fn(list, *args, **kw) -> list of results; returns results for all items (None only if impossible)."""
-        out = [None] * len(items)
-        todo = list(range(len(items)))
-        while todo and self.order:
-            tok_p = G.PROVIDER.set(self.order[self.k])
-            tok_l = G.PAR_LIMIT.set(PARALLEL if self.order[self.k] == "groq" else 0)
-            try:
-                if not self._ready():
-                    raise G.Unavailable("paused")
-                res = fn([items[i] for i in todo], *args, **kw)
-                self.used.add(self.order[self.k])
-                for i, r in zip(todo, res):
-                    if not missing(r):
-                        out[i] = r
-                left = [i for i in todo if missing(out[i])]
-                if len(left) == len(todo):              # nothing came back: treat as unavailable
-                    raise G.Unavailable("no answers")
-                todo = left
-                if todo and not self._wait(f"{len(todo)} sentence(s) to retry"):
-                    break
-            except G.Unavailable as err:
-                daily = "quota" in str(err).lower() or "billing" in str(err).lower() or \
-                    (self.order[self.k] == "groq" and G._next_free_in() > 300)
-                if (daily or self._blocked_for() > 90) and self._switch():
-                    continue
-                if not self._wait(str(err)[:60]) and not self._switch():
-                    self.dead = True
-                    break
-            finally:
-                G.PROVIDER.reset(tok_p)
-                G.PAR_LIMIT.reset(tok_l)
-        return out
-
-
-def _jobs(doc, orig_path):
-    blocks = list(D.docx_blocks(doc))
-    orig = None
-    if orig_path:
-        try:
-            ob = list(D.docx_blocks(DX.open_clean(orig_path)))
-            orig = ob if len(ob) == len(blocks) else None
-        except Exception as err:
-            print(f"[proofread] original not used: {err}", flush=True)
-    jobs = []
-    for bi, b in enumerate(blocks):
-        if b.kind != "para" or b.el is None or not b.raw:
-            continue
-        raws = D.split_sentences(b.raw)
-        o_raw = orig[bi].raw if orig is not None and orig[bi].kind == "para" else None
-        for sp, (cp, cs) in zip(DX.sentence_char_spans(b.raw, raws), H.originals_for(raws, o_raw)):
-            if not sp:
-                continue
-            sub = S.safe_subspan(b.el, *sp)
-            if sub and TP.artifact_only(b.raw[sub[0]:sub[1]]):
-                jobs.append(dict(b=b, span=sub, text=b.raw[sub[0]:sub[1]], par=cp, sent=None, drop=True))
-            elif sub and len(b.raw[sub[0]:sub[1]].split()) >= 4:
-                full = tuple(sub) == tuple(sp)
-                jobs.append(dict(b=b, span=sub, text=b.raw[sub[0]:sub[1]], par=cp, sent=cs if full else None))
-    return jobs
-
-
-accept = TP.strict_accept
-
-
-def _checked(jobs, cands, glossary, minimal=False):
-    """Indices whose candidate passes the strict acceptance, the meaning check and the fluency check. A full
-    proofread is compared with the author's original sentence (when known); a minimal correction with the
-    current sentence."""
-    idx = [i for i, c in enumerate(cands) if c and c.strip() != jobs[i].get("base", jobs[i]["text"]).strip()]
-    ok = {}
-    for i in idx:
-        cur = jobs[i].get("base", jobs[i]["text"])
-        a = accept(cur, cands[i], glossary, jobs[i]["sent"])
-        if a:
-            ok[i] = a
-    keys = list(ok)
-    if keys:
-        refs = [jobs[i].get("base", jobs[i]["text"]) if minimal else (jobs[i]["sent"] or jobs[i]["text"])
-                for i in keys]
-        # light edits must keep the meaning closely; repairs of broken sentences are compared with the original
-        lim = MEANING.min_sim() if minimal else MEANING.soft_sim()
-        mean = MEANING.keeps_meaning(list(zip(refs, [ok[i] for i in keys])), min_sim=lim)
-        if not minimal:                                   # and must stay close to the current sentence too
-            mean = [m and k for m, k in zip(mean, MEANING.keeps_meaning(
-                [(jobs[i].get("base", jobs[i]["text"]), ok[i]) for i in keys], min_sim=MEANING.soft_sim()))]
-        flu = FL.acceptable([(jobs[i]["text"], ok[i]) for i in keys])
-        for i, m, f in zip(keys, mean, flu):
-            if not (m and f):
-                ok.pop(i)
-    return ok
-
-
-def _ai_ok(cur_scores, new_texts, defect=None):
-    """Per candidate: does it keep the AI probability in bounds? Returns (flags, new scores). A defective sentence
-    (doubled terms, chatbot leftovers, broken English) may rise up to DEFECT_RISE, and may enter the flagged range
-    only within that allowance."""
-    sc = H._detector_scores(new_texts) or [0.0] * len(new_texts)
-    defect = defect or [False] * len(new_texts)
-    flags = []
-    for c, n, d in zip(cur_scores, sc, defect):
-        if d:
-            flags.append(n <= c + DEFECT_RISE)
-        else:
-            flags.append(n <= c + MAX_RISE and not (n >= FLAG > c))
-    return flags, sc
-
-
-def proofread_docx(path, orig_path=None, rep=None, hi=0.80, mod=0.50):
-    return _proofread(path, orig_path, rep, hi, mod)
-
-
-def _proofread(path, orig_path=None, rep=None, hi=0.80, mod=0.50):
-    rep0 = rep or (lambda *a, **k: None)
-    _last = [0.05]
-
-    def rep(p, desc=""):
-        if p is not None:
-            _last[0] = p
-        rep0(_last[0], desc=desc)
-    llm = _LLM(rep)
-    if not llm.order:
-        raise G.Unavailable("No AI key is set for the proofread: add GROQ_API_KEY (or OPENAI_API_KEY).")
-    rep(0.05, "Opening the humanized document")
-    doc = DX.open_clean(path)
-    before = D.summarise(S._score_doc(doc), hi, mod)
-    jobs = _jobs(doc, orig_path)
-    if not jobs:
-        raise ValueError("No prose sentences found to proofread.")
-    # 0. local clean-up first (no model, no AI risk): chatbot leftovers, markdown, doubled words / phrases,
-    #    AI-typical filler words and spaced dashes
-    texts = [j["text"] for j in jobs]
-    local = {}
-    drops = {i for i, j in enumerate(jobs) if j.get("drop")}
-    for i, t in enumerate(texts):
-        if i in drops:
-            continue
-        c = TP.basic_grammar(TP.de_ai(t))
-        if c.strip() != t.strip():
-            g = accept(t, c, set())
-            if g:
-                local[i] = g
-    src = list(dict.fromkeys(j["par"] or j["sent"] or j["text"] for j in jobs))
-    glossary = TP.build_glossary(src) | TP.GLOSSARY.get()
-    rep(0.12, f"Scoring {len(texts)} sentences")
-    raw_sc = H._detector_scores(texts) or [0.0] * len(texts)
-    if local:                                          # keep a local clean-up unless it raises the AI score
-        keys = list(local)
-        sc = H._detector_scores([local[i] for i in keys]) or [0.0] * len(keys)
-        for i, s_ in zip(keys, sc):
-            if s_ > raw_sc[i] + MAX_RISE and not TP.has_defect(texts[i]):
-                local.pop(i)
-            else:
-                raw_sc[i] = s_
-    base = [local.get(i, t) for i, t in enumerate(texts)]   # what Groq proofreads
-    for i, j in enumerate(jobs):
-        j["base"] = base[i]
-    cur = raw_sc
-    defect = [TP.has_defect(t) for t in texts]
-    for i in FL.bad(base):
-        defect[i] = True
-
-    # 1. grammar audit: Groq's grammar-only correction tells which sentences contain an error (and gives the
-    #    minimal fix for them). Sentences with an error or a defect are REQUIRED fixes: correctness comes first.
-    rep(0.15, "Grammar audit of every sentence")
-    work = [i for i in range(len(jobs)) if i not in drops]
-    minimal = {}
-    g = llm.run(G.grammar, [base[i] for i in work], glossary, rep, 0.15, 0.35, size=BATCH)
-    unchecked = sum(1 for t in g if t is None)
-    if work and unchecked == len(work):
-        raise G.Unavailable("The proofreading engine has reached its usage limit for now. Please try again in a "
-                            "while; nothing in your document was changed.")
-    cands = [None] * len(jobs)
-    for i, t in zip(work, g):
-        cands[i] = t
-    minimal = _checked(jobs, cands, glossary, minimal=True)
-    required = {i for i in work if defect[i] or i in minimal}
-
-    # 2. full proofread (clarity + grammar + academic wording + flow)
-    ctx = [j["par"] or j["sent"] for j in jobs]
-    ctx = [c if c and c.strip() != t.strip() else None for c, t in zip(ctx, texts)]
-    groups = [id(j["b"]) for j in jobs]
-    idx_all = list(range(len(jobs)))
-    res = llm.run(lambda ids, *a: G.polish([base[i] for i in ids], [groups[i] for i in ids], glossary, rep,
-                                            0.35, 0.65, originals=[ctx[i] for i in ids], size=BATCH),
-                  idx_all, missing=lambda r: r is None or r[1] is None)
-    res = [r if r is not None else (None, None) for r in res]
-    full = _checked(jobs, [t for _, t in res], glossary)
-    for i in drops:
-        full.pop(i, None)
-    final, rise = {i: t for i, t in local.items()}, {}
-    for i in drops:                                    # chatbot leftovers on their own: removed
-        final[i] = ""
-    # every correct version of a sentence (full proofread, minimal grammar fix) is scored; a sentence with an error
-    # takes its LEAST AI-like correct version, a correct sentence only takes a style edit that keeps the AI score
-    opts = {}
-    for i in work:
-        for t in (full.get(i), minimal.get(i)):
-            if t and t not in opts.setdefault(i, []):
-                opts[i].append(t)
-    flat = [(i, t) for i, ts in opts.items() for t in ts]
-    sc = H._detector_scores([t for _, t in flat]) or [0.0] * len(flat)
-    score_of = {(i, t): s_ for (i, t), s_ in zip(flat, sc)}
-    n_minimal = 0
-    for i, ts in opts.items():
-        if i in required:
-            best = min(ts, key=lambda t: (score_of[(i, t)], len(t)))
-            final[i], rise[i] = best, score_of[(i, best)] - cur[i]
-            n_minimal += best == minimal.get(i) and best != full.get(i)
-        elif full.get(i):
-            s_ = score_of[(i, full[i])]
-            if s_ - cur[i] <= KEEP_RISE:              # style-only edit: only if the AI score stays the same
-                final[i], rise[i] = full[i], s_ - cur[i]
-
-    # 3. verification rounds: every corrected or required sentence is checked again until Groq finds no error
-    n_verify = 0
-    for rnd in range(ROUNDS):
-        chk = [i for i in work if i in final or i in required]
-        if not chk:
-            break
-        rep(0.65 + 0.08 * rnd, f"Verification round {rnd + 1}: re-checking {len(chk)} sentence(s)")
-        if llm.dead:
-            break
-        g = llm.run(G.grammar, [final.get(i, base[i]) for i in chk], glossary, rep, 0.65 + 0.08 * rnd,
-                    0.72 + 0.08 * rnd, size=BATCH)
-        changed = 0
-        for i, t in zip(chk, g):
-            now = final.get(i, base[i])
-            if not t or t.strip() == now.strip():
-                continue
-            ok = accept(now, t, glossary, jobs[i]["sent"])
-            if ok and H._small_edit(now, ok) and MEANING.keeps_meaning([(now, ok)])[0]:
-                final[i] = ok
-                required.add(i)                        # it had an error: keep this fix
-                changed += 1
-        n_verify += changed
-        if not changed:
-            break
-
-    # 4. LanguageTool: last mechanical check (agreement, confused words, a/an, punctuation)
-    try:
-        import lt_grammar as LT
-        if LT.MODE != "off":
-            ids = [i for i in work if final.get(i, base[i])]
-            for i, t in zip(ids, LT.correct([final.get(i, base[i]) for i in ids])):
-                now = final.get(i, base[i])
-                if t.strip() != now.strip():
-                    ok = accept(now, t, glossary)
-                    if ok and H._small_edit(now, ok):
-                        final[i] = ok
-                        required.add(i)
-    except Exception as err:
-        print(f"[proofread] LanguageTool skipped: {err}", flush=True)
-
-    # 5. keep the AI score: correct, natural-sounding alternatives (AI) and Parrot rewrites of corrected sentences;
-    #    only versions verified error-free (grammar check returns them unchanged, LanguageTool finds nothing,
-    #    fluent, same meaning) are kept as options.
-    verified = {}                                      # sentence -> set of verified correct versions
-
-    def gen_alts(ids, stage):
-        if not ids or llm.dead:
-            return
-        rep(stage, f"Keeping the AI score: natural rewording of {len(ids)} sentence(s)")
-        alts = llm.run(G.variants, [final.get(i, base[i]) for i in ids], glossary,
-                       [jobs[i]["sent"] or None for i in ids], VARIANTS)
-        pool = []
-        for i, vs in zip(ids, alts):
-            for v in (vs or []):
-                ok = accept(final.get(i, base[i]), v, glossary, jobs[i]["sent"])
-                if ok:
-                    pool.append((i, ok))
-        if PARROT:
-            try:
-                pr, _ = H._paraphrase_local([final.get(i, base[i]) for i in ids], 0.9, None, 0.8, 0.85)
-                for i, t in zip(ids, pr):
-                    ok = accept(final.get(i, base[i]), t, glossary, jobs[i]["sent"])
-                    if ok:
-                        pool.append((i, ok))
-            except Exception as err:
-                print(f"[proofread] Parrot rewording skipped: {err}", flush=True)
-        pool = list(dict.fromkeys(pool))
-        if not pool:
-            return
-        mean = MEANING.keeps_meaning([(final.get(i, base[i]), t) for i, t in pool])
-        flu = FL.acceptable([(final.get(i, base[i]), t) for i, t in pool])
-        pool = [(i, t) for (i, t), m, f in zip(pool, mean, flu) if m and f]
-        if not pool:
-            return
-        chk = llm.run(G.grammar, [t for _, t in pool], glossary, rep, stage, stage + 0.02, size=BATCH)
-        clean = [(i, t) for (i, t), g_ in zip(pool, chk) if g_ is not None and g_.strip() == t.strip()]
-        try:
-            import lt_grammar as LT
-            if LT.MODE != "off" and clean:
-                ltc = LT.correct([t for _, t in clean])
-                clean = [c for c, l_ in zip(clean, ltc) if l_.strip() == c[1].strip()]
-        except Exception:
-            pass
-        for i, t in clean:
-            verified.setdefault(i, set()).add(t)
-
-    n_reworded = 0
-    for rnd in range(AI_ROUNDS):
-        ids = [i for i in work if final.get(i)]
-        if not ids:
-            break
-        now_sc = H._detector_scores([final[i] for i in ids]) or [0.0] * len(ids)
-        hot = [i for i, s_ in zip(ids, now_sc) if s_ > cur[i] + KEEP_RISE]
-        if not hot:
-            break
-        now_of = dict(zip(ids, now_sc))
-        gen_alts(hot, 0.78 + 0.02 * rnd)
-        cand = [(i, t) for i in hot for t in verified.get(i, ())]
-        sc = H._detector_scores([t for _, t in cand]) or [1.0] * len(cand)
-        best = {}
-        for (i, t), s_ in zip(cand, sc):
-            if s_ < now_of[i] - 0.02 and (i not in best or s_ < best[i][1]):
-                best[i] = (t, s_)
-        for i, (t, s_) in best.items():
-            final[i], rise[i] = t, s_ - cur[i]
-            required.add(i)
-            n_reworded += 1
-        if not best:
-            break
-
-    # 5b. paragraph balance: the document score is measured over sentence windows inside each paragraph, so
-    #     single-sentence scores are not enough. Every paragraph whose AI-flagged words rose is re-balanced by
-    #     choosing, sentence by sentence, among the CORRECT versions only (current, minimal fix, verified natural
-    #     versions; for a sentence without an error also its unedited form) the combination with the fewest
-    #     flagged words.
-    by_par = {}
-    for i, j in enumerate(jobs):
-        by_par.setdefault(id(j["b"]), []).append(i)
-
-    def par_text(pid, choice):
-        ids = by_par[pid]
-        raw = jobs[ids[0]]["b"].raw
-        for i in sorted(ids, key=lambda i: -jobs[i]["span"][0]):
-            t = choice.get(i, jobs[i]["text"])
-            a_, e_ = jobs[i]["span"]
-            if t == "":
-                while e_ < len(raw) and raw[e_] == " ":
-                    e_ += 1
-            raw = raw[:a_] + t + raw[e_:]
-        return raw
-
-    def flagged(text):
-        try:
-            st = D.summarise(S._score_paras([text]), hi, mod)
-            return st["high"] + st["moderate"]
-        except Exception:
-            return 0
-
-    def options(i):
-        o = [final.get(i, base[i])]
-        for t in [minimal.get(i)] + sorted(verified.get(i, ())):
-            if t and t not in o:
-                o.append(t)
-        if i not in required and jobs[i]["text"] not in o:
-            o.append(jobs[i]["text"])              # no error: its unedited form is allowed too
-        return o
-
-    n_balanced = 0
-    changed_pars = [pid for pid, ids in by_par.items() if any(i in final for i in ids)]
-    rep(0.86, f"Balancing the AI score of {len(changed_pars)} paragraph(s)")
-    for attempt in range(2):
-        worse = []
-        for pid in changed_pars:
-            was = flagged(par_text(pid, {}))
-            if flagged(par_text(pid, final)) > was + PAR_TOL:
-                worse.append((pid, was))
-        if not worse:
-            break
-        if attempt == 1:                               # more correct versions for the sentences in those paragraphs
-            gen_alts([i for pid, _ in worse for i in by_par[pid] if i in final and final[i]], 0.88)
-        for pid, was in worse:
-            choice = {i: final[i] for i in by_par[pid] if i in final}
-            score = flagged(par_text(pid, choice))
-            for _ in range(len(by_par[pid]) * 2):      # greedy: best single swap per step
-                best = None
-                for i in by_par[pid]:
-                    if i in drops:
-                        continue
-                    for t in options(i):
-                        if t == choice.get(i, jobs[i]["text"]):
-                            continue
-                        trial = dict(choice)
-                        if t == jobs[i]["text"]:
-                            trial.pop(i, None)
-                        else:
-                            trial[i] = t
-                        f_ = flagged(par_text(pid, trial))
-                        if f_ < score and (best is None or f_ < best[0]):
-                            best = (f_, trial)
-                if best is None:
-                    break
-                score, choice = best
-                n_balanced += 1
-                if score <= was + PAR_TOL:
-                    break
-            for i in by_par[pid]:
-                if i in drops:
-                    continue
-                if i in choice:
-                    final[i] = choice[i]
-                else:
-                    final.pop(i, None)
-
-    ids = [i for i in work if final.get(i)]
-    still = sum(1 for i, s_ in zip(ids, H._detector_scores([final[i] for i in ids]) or [0.0] * len(ids))
-                if s_ > cur[i] + KEEP_RISE) if ids else 0
-
-    # 6. write back, then document-level AI check (undo style-only edits if the score still rose)
-    rep(0.85, "Writing corrections and re-checking the AI score")
-
-    def write(chosen):
-        d = DX.open_clean(path)
-        js = _jobs(d, None)
-        if len(js) != len(jobs):
-            return None, 0
-        n = 0
-        for i in sorted(chosen, key=lambda i: (id(js[i]["b"]), -js[i]["span"][0])):
-            a_, e_ = js[i]["span"]
-            if chosen[i] == "":                        # deletion: take the following space with it
-                raw = js[i]["b"].raw
-                while e_ < len(raw) and raw[e_] == " ":
-                    e_ += 1
-            if S.replace_span(js[i]["b"].el, a_, e_, chosen[i]):
-                n += 1
-        return d, n
-
-    out_doc, n = write(final)
-    if out_doc is None:
-        raise ValueError("The document layout changed while proofreading; please run it again.")
-    after = D.summarise(S._score_doc(out_doc), hi, mod)
-    undone = 0
-    while after["pct"] > before["pct"] + MAX_DOC_RISE:
-        optional = [i for i in final if i not in required and i not in drops and i not in local]
-        if not optional:                               # only required (grammar) fixes left: they always stay
-            break
-        worst = sorted(optional, key=lambda i: rise.get(i, 0), reverse=True)[:max(1, len(optional) // 5)]
-        for i in worst:
-            final.pop(i)
-            undone += 1
-        out_doc, n = write(final)
-        after = D.summarise(S._score_doc(out_doc), hi, mod)
-    out = H._save_output(out_doc, S._clean_stem(path), "_proofread")
-    who = " + ".join(_NAME.get(p, p) for p in llm.order if p in llm.used) or llm.name
-    note = (f"🔍 Rigorous proofread by {who}: {n} of {len(jobs)} sentences corrected"
-            + (f" ({len(required)} had a grammar error or defect and were always corrected"
-               + (f", {n_minimal} with the minimal grammar fix" if n_minimal else "")
-               + (f"; {n_verify} more fix(es) found in the verification rounds" if n_verify else "")
-               + (f"; {n_reworded} reworded naturally to keep the AI score" if n_reworded else "")
-               + (f"; {n_balanced} swap(s) between correct versions to balance paragraphs" if n_balanced else "")
-               + ")"
-               if required else "")
-            + (f"; {sum(defect)} had doubled terms, chatbot leftovers or broken English, "
-               f"{sum(1 for i, d in enumerate(defect) if d and i in final)} of them fixed" if any(defect) else "")
-            + f". AI score {before['pct']}% → {after['pct']}%."
-            + (f" {undone} style-only edit(s) were undone because they raised the AI score; grammar fixes "
-               "were all kept." if undone else "")
-            + (f" {still} corrected sentence(s) still read slightly more AI-like: no error-free natural version "
-               "was found, and correct grammar was kept." if still else "")
-            + (f" ⚠️ {unchecked} sentence(s) could not be checked (AI quota exhausted); run the proofread again "
-               "later to finish them." if unchecked else "")
-            + ("" if orig_path else " Tip: run it right after humanizing, so your original is used as the meaning "
-                                    "reference for unclear sentences."))
-    print(f"[proofread] {note}", flush=True)
-    return dict(path=out, before=before, after=after, n=n, total=len(jobs), note=note)
+"""
+Rigorous proofread of a finished (humanized) Word document. Correct grammar comes first; a moderate AI-score rise
+is accepted for that, style-only edits must keep the AI score.
+
+  0. Local clean-up (no model): chatbot leftovers, markdown, doubled words / repeated phrases, AI filler words.
+  1. Grammar audit: the AI's grammar-only check marks every sentence that contains an error (plus sentences with
+     doubled terms, leftovers or broken English): these are REQUIRED fixes.
+  2. Full proofread (clarity repair against the original upload, grammar, academic wording, flow). For a required
+     sentence the full correction is used if its AI rise is at most PROOF_REQ_RISE, otherwise the minimal grammar
+     fix; a sentence WITH an error is never left uncorrected. Style-only edits must stay within PROOF_MAX_RISE.
+  3. Verification rounds (PROOF_ROUNDS): every corrected sentence is grammar-checked again until no error is found.
+  4. LanguageTool: last mechanical check.
+  5. Document check: if the AI score rose by more than PROOF_MAX_DOC_RISE points, style-only edits are undone
+     (riskiest first). Grammar fixes are never undone.
+Every change keeps terms / numbers / citations and the meaning. Only plain text runs are edited.
+
+Unclear / garbled sentences the full proofread REPAIRS are judged against the author's original sentence (not
+against the garbled text, which made good repairs fail the meaning check) and are never undone by the AI-score
+steps: clarity wins.
+
+Space variables: PROOF_REQ_RISE (0.35), PROOF_MAX_RISE (0.15), PROOF_MAX_DOC_RISE (2 points), PROOF_ROUNDS (1),
+PROOF_KEEP_RISE (0.05), PROOF_ALT_RISE (0.10), PROOF_REPAIR_SIM (0.60), PROOF_REASONING (medium).
+"""
+import difflib
+import os
+import time
+import re
+
+import detector as D
+import docx_report as DX
+import groqai as G
+import humanizer as H
+import meaning as MEANING
+import fluency as FL
+import smart as S
+import textparse as TP
+
+MAX_RISE = float(os.getenv("PROOF_MAX_RISE", "0.15"))
+FLAG = float(os.getenv("PROOF_FLAG", "0.50"))
+MAX_DOC_RISE = float(os.getenv("PROOF_MAX_DOC_RISE", "2.0"))
+# a sentence with a real defect (doubled word / repeated phrase / chatbot leftover / broken English) may rise more
+DEFECT_RISE = float(os.getenv("PROOF_DEFECT_RISE", "0.25"))
+# grammar comes first: a sentence with an error is always corrected; its full correction is used if the AI rise
+# stays within PROOF_REQ_RISE (otherwise the minimal grammar fix)
+REQ_RISE = float(os.getenv("PROOF_REQ_RISE", "0.35"))
+ROUNDS = int(os.getenv("PROOF_ROUNDS", "1"))
+BATCH = int(os.getenv("PROOF_BATCH", "24"))             # sentences per request (fewer, larger requests = faster)
+# keep the AI score: a corrected sentence may be at most this much more AI-like than before; above it, correct
+# natural-sounding alternatives are tried (PROOF_AI_ROUNDS rounds, PROOF_VARIANTS versions each, plus Parrot)
+KEEP_RISE = float(os.getenv("PROOF_KEEP_RISE", "0.05"))
+# a corrected sentence is re-worded for the AI score only if it rose more than this (each rewording costs tokens and
+# moves the wording away from the checked correction)
+ALT_RISE = float(os.getenv("PROOF_ALT_RISE", "0.10"))
+AI_ROUNDS = int(os.getenv("PROOF_AI_ROUNDS", "1"))
+VARIANTS = int(os.getenv("PROOF_VARIANTS", "3"))
+# a repair of a garbled sentence must keep the author's meaning (MEANING_SOFT against the original sentence) and
+# stay at least this close to the current sentence; without any original it needs PROOF_REPAIR_SIM_NOREF
+REPAIR_SIM = float(os.getenv("PROOF_REPAIR_SIM", "0.60"))
+REPAIR_SIM_NOREF = float(os.getenv("PROOF_REPAIR_SIM_NOREF", "0.70"))
+# reasoning effort of the full proofread on reasoning models (the grammar audit stays on GROQ_REASONING)
+REASONING = os.getenv("PROOF_REASONING", "medium").lower()
+# single (default): the whole document is checked and corrected in ONE request (grammar, clarity, academic wording,
+# flow and document-wide consistency), and the reply lists problems that need the author. Every correction still
+# goes through the local checks. If the provider cannot take one request of that size, or it fails, the batched
+# proofread (several smaller requests) runs instead. batched = always the batched proofread.
+MODE = os.getenv("PROOF_MODE", "single").lower()
+PAR_TOL = int(os.getenv("PROOF_PAR_TOL", "0"))       # flagged words a paragraph may gain after balancing
+PARROT = os.getenv("PROOF_PARROT", "0") == "1"
+# which AI does the proofreading: Groq by default (PROOF_PROVIDER=openai for ChatGPT). If the chosen one is out of
+# quota, the other one is used automatically when its key is set.
+PROVIDER = os.getenv("PROOF_PROVIDER", "groq").lower()
+MAX_WAIT = float(os.getenv("PROOF_MAX_WAIT", "180"))     # seconds the proofread may wait for rate limits in total
+PARALLEL = int(os.getenv("PROOF_PARALLEL", "3"))          # requests at once (Groq: low, so limits are not hit)
+_NAME = {"groq": "Groq", "openai": "ChatGPT"}
+
+
+class _LLM:
+    """Calls Groq / ChatGPT for the proofread so that EVERY sentence gets an answer: failed batches are retried
+    (only the missing sentences), rate limits are waited out (up to PROOF_MAX_WAIT in total), and if the provider
+    is out of quota the other provider is used when its key is set."""
+
+    def __init__(self, rep):
+        self.rep = rep
+        self.order = [PROVIDER] + [p for p in ("groq", "openai") if p != PROVIDER]
+        self.order = [p for p in self.order if (G.OPENAI_KEY if p == "openai" else G.KEYS)]
+        self.k = 0
+        self.waited = 0.0
+        self.used = set()
+        self.dead = False
+
+    @property
+    def name(self):
+        return _NAME.get(self.order[self.k], self.order[self.k]) if self.order else "AI"
+
+    def _ready(self):
+        return G.available()
+
+    def _switch(self):
+        if self.k + 1 < len(self.order):
+            self.k += 1
+            print(f"[proofread] switching to {self.name}", flush=True)
+            return True
+        return False
+
+    def _blocked_for(self):
+        """Seconds until the current provider can be used again (large = daily quota / billing / bad key)."""
+        now = time.time()
+        if self.order[self.k] == "openai":
+            return max(0.0, G._oa["until"] - now) if G.OPENAI_KEY else 1e9
+        return max(G._down_until - now, G._next_free_in(), 0.0)
+
+    def _wait(self, why):
+        blocked = self._blocked_for()
+        if blocked > 90:                               # quota for the day / billing: waiting will not help
+            print(f"[proofread] {self.name} blocked for {int(blocked)}s ({why}); "
+                  f"groq: {G.STATS.get('last_error', '')[:120]} | openai: {G.OA_STATS.get('last_error', '')[:120]}",
+                  flush=True)
+            return False
+        t = min(30.0, max(3.0, blocked or 5.0))
+        if self.waited + t > MAX_WAIT:
+            return False
+        self.rep(None, f"Waiting {int(t)}s for the {self.name} rate limit ({why})")
+        time.sleep(t)
+        self.waited += t
+        return True
+
+    def run(self, fn, items, *args, missing=lambda r: r is None, **kw):
+        """fn(list, *args, **kw) -> list of results; returns results for all items (None only if impossible)."""
+        out = [None] * len(items)
+        todo = list(range(len(items)))
+        while todo and self.order:
+            tok_p = G.PROVIDER.set(self.order[self.k])
+            tok_l = G.PAR_LIMIT.set(PARALLEL if self.order[self.k] == "groq" else 0)
+            try:
+                if not self._ready():
+                    raise G.Unavailable("paused")
+                res = fn([items[i] for i in todo], *args, **kw)
+                self.used.add(self.order[self.k])
+                for i, r in zip(todo, res):
+                    if not missing(r):
+                        out[i] = r
+                left = [i for i in todo if missing(out[i])]
+                if len(left) == len(todo):              # nothing came back: treat as unavailable
+                    raise G.Unavailable("no answers")
+                todo = left
+                if todo and not self._wait(f"{len(todo)} sentence(s) to retry"):
+                    break
+            except G.Unavailable as err:
+                daily = "quota" in str(err).lower() or "billing" in str(err).lower() or \
+                    (self.order[self.k] == "groq" and G._next_free_in() > 300)
+                if (daily or self._blocked_for() > 90) and self._switch():
+                    continue
+                if not self._wait(str(err)[:60]) and not self._switch():
+                    self.dead = True
+                    break
+            finally:
+                G.PROVIDER.reset(tok_p)
+                G.PAR_LIMIT.reset(tok_l)
+        return out
+
+
+def _jobs(doc, orig_path):
+    blocks = list(D.docx_blocks(doc))
+    orig = None
+    if orig_path:
+        try:
+            ob = list(D.docx_blocks(DX.open_clean(orig_path)))
+            orig = ob if len(ob) == len(blocks) else None
+        except Exception as err:
+            print(f"[proofread] original not used: {err}", flush=True)
+    jobs = []
+    for bi, b in enumerate(blocks):
+        if b.kind != "para" or b.el is None or not b.raw:
+            continue
+        raws = D.split_sentences(b.raw)
+        o_raw = orig[bi].raw if orig is not None and orig[bi].kind == "para" else None
+        for sp, (cp, cs) in zip(DX.sentence_char_spans(b.raw, raws), H.originals_for(raws, o_raw)):
+            if not sp:
+                continue
+            sub = S.safe_subspan(b.el, *sp)
+            if sub and TP.artifact_only(b.raw[sub[0]:sub[1]]):
+                jobs.append(dict(b=b, span=sub, text=b.raw[sub[0]:sub[1]], par=cp, sent=None, drop=True))
+            elif sub and len(b.raw[sub[0]:sub[1]].split()) >= 4:
+                full = tuple(sub) == tuple(sp)
+                jobs.append(dict(b=b, span=sub, text=b.raw[sub[0]:sub[1]], par=cp, sent=cs if full else None))
+    return jobs
+
+
+accept = TP.strict_accept
+
+
+def _checked(jobs, cands, glossary, minimal=False, repaired=()):
+    """Indices whose candidate passes the strict acceptance, the meaning check and the fluency check. A full
+    proofread is compared with the author's original sentence (when known) and with the current sentence; a
+    minimal correction with the current sentence only. A REPAIR of a garbled sentence (`repaired`) is judged
+    against the author's original: it must say what the original says, and only loosely resemble the garbled
+    current wording (that wording is what was wrong)."""
+    def cur_of(i):
+        return jobs[i].get("base", jobs[i]["text"])
+    idx = [i for i, c in enumerate(cands) if c and c.strip() != cur_of(i).strip()]
+    ok = {}
+    for i in idx:
+        a = accept(cur_of(i), cands[i], glossary, jobs[i]["sent"])
+        sent = jobs[i]["sent"]
+        if not a and not minimal and i in repaired and sent:
+            # a repair is measured against the author's sentence (back to the original wording is allowed)
+            a = TP.basic_grammar(cands[i].strip()) if cands[i].strip() == sent.strip() else \
+                accept(sent, cands[i], glossary)
+        if a:
+            ok[i] = a
+    keys = list(ok)
+    if not keys:
+        return ok
+    if minimal:
+        mean = MEANING.keeps_meaning([(cur_of(i), ok[i]) for i in keys], min_sim=MEANING.min_sim())
+    else:
+        soft = MEANING.soft_sim()
+        mean = []
+        pairs_o = [(jobs[i]["sent"] or jobs[i]["text"], ok[i]) for i in keys]
+        pairs_c = [(cur_of(i), ok[i]) for i in keys]
+        m_o = MEANING.keeps_meaning(pairs_o, min_sim=soft)
+        sim_c = MEANING.similarities(pairs_c)
+        par_keys = [i for i in keys if i in repaired and not jobs[i]["sent"] and jobs[i]["par"]]
+        par_sim = {}
+        if par_keys:
+            ps = MEANING.similarities([(jobs[i]["par"], ok[i]) for i in par_keys]
+                                      + [(jobs[i]["par"], cur_of(i)) for i in par_keys])
+            par_sim = {i: (ps[n], ps[len(par_keys) + n]) for n, i in enumerate(par_keys)}
+        for i, mo, sc_ in zip(keys, m_o, sim_c):
+            same = ok[i].strip() == cur_of(i).strip()
+            if i not in repaired:                         # light edit: close to the original AND the current text
+                mean.append(mo and (same or sc_ >= soft))
+            elif jobs[i]["sent"]:                         # repair, original sentence known
+                mean.append(mo and (same or sc_ >= REPAIR_SIM))
+            elif i in par_sim:                            # repair, only the original paragraph known
+                new_s, old_s = par_sim[i]
+                mean.append(new_s >= old_s - 0.02 and sc_ >= REPAIR_SIM)
+            else:                                         # repair without any original (standalone proofread)
+                mean.append(same or sc_ >= REPAIR_SIM_NOREF)
+    flu = FL.acceptable([(jobs[i]["text"], ok[i]) for i in keys])
+    for i, m, f in zip(keys, mean, flu):
+        if not (m and f):
+            ok.pop(i)
+    return ok
+
+
+def _ai_ok(cur_scores, new_texts, defect=None):
+    """Per candidate: does it keep the AI probability in bounds? Returns (flags, new scores). A defective sentence
+    (doubled terms, chatbot leftovers, broken English) may rise up to DEFECT_RISE, and may enter the flagged range
+    only within that allowance."""
+    sc = H._detector_scores(new_texts) or [0.0] * len(new_texts)
+    defect = defect or [False] * len(new_texts)
+    flags = []
+    for c, n, d in zip(cur_scores, sc, defect):
+        if d:
+            flags.append(n <= c + DEFECT_RISE)
+        else:
+            flags.append(n <= c + MAX_RISE and not (n >= FLAG > c))
+    return flags, sc
+
+
+def proofread_docx(path, orig_path=None, rep=None, hi=0.80, mod=0.50):
+    return _proofread(path, orig_path, rep, hi, mod)
+
+
+def _proofread(path, orig_path=None, rep=None, hi=0.80, mod=0.50):
+    rep0 = rep or (lambda *a, **k: None)
+    _last = [0.05]
+
+    def rep(p, desc=""):
+        if p is not None:
+            _last[0] = p
+        rep0(_last[0], desc=desc)
+    llm = _LLM(rep)
+    if not llm.order:
+        raise G.Unavailable("No AI key is set for the proofread: add GROQ_API_KEY (or OPENAI_API_KEY).")
+    rep(0.05, "Opening the humanized document")
+    doc = DX.open_clean(path)
+    before = D.summarise(S._score_doc(doc), hi, mod)
+    jobs = _jobs(doc, orig_path)
+    if not jobs:
+        raise ValueError("No prose sentences found to proofread.")
+    # 0. local clean-up first (no model, no AI risk): chatbot leftovers, markdown, doubled words / phrases,
+    #    AI-typical filler words and spaced dashes
+    texts = [j["text"] for j in jobs]
+    local = {}
+    drops = {i for i, j in enumerate(jobs) if j.get("drop")}
+    for i, t in enumerate(texts):
+        if i in drops:
+            continue
+        c = TP.basic_grammar(TP.de_ai(t))
+        if c.strip() != t.strip():
+            g = accept(t, c, set())
+            if g:
+                local[i] = g
+    src = list(dict.fromkeys(j["par"] or j["sent"] or j["text"] for j in jobs))
+    glossary = TP.build_glossary(src) | TP.GLOSSARY.get()
+    TP.set_document([b.raw for b in D.docx_blocks(doc) if b.kind == "para" and b.raw])
+    rep(0.12, f"Scoring {len(texts)} sentences")
+    raw_sc = H._detector_scores(texts) or [0.0] * len(texts)
+    if local:                                          # keep a local clean-up unless it raises the AI score
+        keys = list(local)
+        sc = H._detector_scores([local[i] for i in keys]) or [0.0] * len(keys)
+        for i, s_ in zip(keys, sc):
+            if s_ > raw_sc[i] + MAX_RISE and not TP.has_defect(texts[i]):
+                local.pop(i)
+            else:
+                raw_sc[i] = s_
+    base = [local.get(i, t) for i, t in enumerate(texts)]   # what Groq proofreads
+    for i, j in enumerate(jobs):
+        j["base"] = base[i]
+    cur = raw_sc
+    defect = [TP.has_defect(t) for t in texts]
+    for i in FL.bad(base):
+        defect[i] = True
+
+    work = [i for i in range(len(jobs)) if i not in drops]
+    ctx = [j["par"] or j["sent"] for j in jobs]
+    ctx = [c if c and c.strip() != t.strip() else None for c, t in zip(ctx, texts)]
+    groups = [id(j["b"]) for j in jobs]
+    idx_all = list(range(len(jobs)))
+    minimal, notes, single, unchecked = {}, [], False, 0
+
+    # ONE request for the whole document (PROOF_MODE=single): grammar, clarity, wording, flow and consistency
+    if MODE == "single":
+        rep(0.2, "Final check: the whole document is read and corrected in one pass")
+        constructs = sorted({re.sub(r"\s+", " ", m.group(0)).strip() for m in re.finditer(
+            r"(?:[A-Za-z][\w\-]*\s+){1,4}\([A-Z][A-Za-z0-9\-]{1,10}\)",
+            " ".join(b.raw for b in D.docx_blocks(doc) if b.kind == "para" and b.raw))})[:40]
+        tok_r = G.REASONING.set(REASONING)
+        for prov in llm.order:                         # one call; the other provider only if the first fails
+            tok_p = G.PROVIDER.set(prov)
+            try:
+                res, notes = G.proofread_document([base[i] for i in idx_all], groups, ctx, glossary, constructs)
+                llm.used.add(prov)
+                single = True
+                break
+            except G.Unavailable as err:
+                print(f"[proofread] one-call check not possible on {prov}: {err}", flush=True)
+            except Exception as err:
+                print(f"[proofread] one-call check failed on {prov}: {err}", flush=True)
+            finally:
+                G.PROVIDER.reset(tok_p)
+        G.REASONING.reset(tok_r)
+        if single:
+            unchecked = sum(1 for r in res if r == (None, None))
+            res = [r if r != (None, None) else ("ok", base[i]) for i, r in enumerate(res)]
+            fixed_ids = {i for i, (st, _) in enumerate(res) if st in ("fixed", "repaired")}
+            required = {i for i in work if defect[i] or i in fixed_ids}
+        else:
+            rep(0.2, "The document is too large for one request here: checking it in parts")
+
+    if not single:
+        # 1. grammar audit: Groq's grammar-only correction tells which sentences contain an error (and gives the
+        #    minimal fix for them). Sentences with an error or a defect are REQUIRED fixes: correctness comes first.
+        rep(0.15, "Grammar audit of every sentence")
+        g = llm.run(G.grammar, [base[i] for i in work], glossary, rep, 0.15, 0.35, size=BATCH)
+        unchecked = sum(1 for t in g if t is None)
+        if work and unchecked == len(work):
+            raise G.Unavailable("The proofreading engine has reached its usage limit for now. Please try again in "
+                                "a while; nothing in your document was changed.")
+        cands = [None] * len(jobs)
+        for i, t in zip(work, g):
+            cands[i] = t
+        minimal = _checked(jobs, cands, glossary, minimal=True)
+        required = {i for i in work if defect[i] or i in minimal}
+
+        # 2. full proofread (clarity + grammar + academic wording + flow)
+        tok_r = G.REASONING.set(REASONING)                 # deeper reading for the full proofread
+        try:
+            res = llm.run(lambda ids, *a: G.polish([base[i] for i in ids], [groups[i] for i in ids], glossary,
+                                                    rep, 0.35, 0.65, originals=[ctx[i] for i in ids], size=BATCH),
+                          idx_all, missing=lambda r: r is None or r[1] is None)
+        finally:
+            G.REASONING.reset(tok_r)
+        res = [r if r is not None else (None, None) for r in res]
+    asked_repair = {i for i, (st, _) in enumerate(res) if st == "repaired"}
+    full = _checked(jobs, [t for _, t in res], glossary, repaired=asked_repair)
+    for i in drops:
+        full.pop(i, None)
+    # an accepted repair of an unclear sentence is a required fix: clarity comes before the AI score
+    repaired = {i for i in full if i in asked_repair}
+    required |= repaired
+    final, rise = {i: t for i, t in local.items()}, {}
+    for i in drops:                                    # chatbot leftovers on their own: removed
+        final[i] = ""
+    # every correct version of a sentence (full proofread, minimal grammar fix) is scored. A repaired sentence takes
+    # its repair; a sentence with an error takes the full correction if its AI rise stays within PROOF_REQ_RISE,
+    # otherwise the minimal grammar fix; a correct sentence only takes a style edit that keeps the AI score
+    opts = {}
+    for i in work:
+        for t in (full.get(i), minimal.get(i)):
+            if t and t not in opts.setdefault(i, []):
+                opts[i].append(t)
+    flat = [(i, t) for i, ts in opts.items() for t in ts]
+    sc = H._detector_scores([t for _, t in flat]) or [0.0] * len(flat)
+    score_of = {(i, t): s_ for (i, t), s_ in zip(flat, sc)}
+    n_minimal = 0
+    for i, ts in opts.items():
+        if i in repaired:
+            final[i], rise[i] = full[i], score_of[(i, full[i])] - cur[i]
+        elif i in required:
+            f_, m_ = full.get(i), minimal.get(i)
+            if f_ and (not m_ or score_of[(i, f_)] - cur[i] <= REQ_RISE):
+                best = f_
+            elif m_:
+                best = m_
+            else:
+                best = min(ts, key=lambda t: (score_of[(i, t)], len(t)))
+            final[i], rise[i] = best, score_of[(i, best)] - cur[i]
+            n_minimal += best == m_ and best != f_
+        elif full.get(i):
+            s_ = score_of[(i, full[i])]
+            if s_ - cur[i] <= KEEP_RISE:              # style-only edit: only if the AI score stays the same
+                final[i], rise[i] = full[i], s_ - cur[i]
+
+    # 3. verification rounds: every corrected or required sentence is checked again until Groq finds no error
+    n_verify = 0
+    for rnd in range(0 if single else ROUNDS):
+        # a sentence whose text IS the grammar check's own output is not sent again (it was just checked)
+        chk = [i for i in work if (i in final or i in required) and final.get(i, base[i]) != minimal.get(i)]
+        if not chk:
+            break
+        rep(0.65 + 0.08 * rnd, f"Verification round {rnd + 1}: re-checking {len(chk)} sentence(s)")
+        if llm.dead:
+            break
+        g = llm.run(G.grammar, [final.get(i, base[i]) for i in chk], glossary, rep, 0.65 + 0.08 * rnd,
+                    0.72 + 0.08 * rnd, size=BATCH)
+        changed = 0
+        for i, t in zip(chk, g):
+            now = final.get(i, base[i])
+            if not t or t.strip() == now.strip():
+                continue
+            ok = accept(now, t, glossary, jobs[i]["sent"])
+            if ok and H._small_edit(now, ok) and MEANING.keeps_meaning([(now, ok)])[0]:
+                final[i] = ok
+                required.add(i)                        # it had an error: keep this fix
+                changed += 1
+        n_verify += changed
+        if not changed:
+            break
+
+    # 4. LanguageTool: last mechanical check (agreement, confused words, a/an, punctuation)
+    try:
+        import lt_grammar as LT
+        if LT.MODE != "off":
+            ids = [i for i in work if final.get(i, base[i])]
+            for i, t in zip(ids, LT.correct([final.get(i, base[i]) for i in ids])):
+                now = final.get(i, base[i])
+                if t.strip() != now.strip():
+                    ok = accept(now, t, glossary)
+                    if ok and H._small_edit(now, ok):
+                        final[i] = ok
+                        required.add(i)
+    except Exception as err:
+        print(f"[proofread] LanguageTool skipped: {err}", flush=True)
+
+    # 5. keep the AI score: correct, natural-sounding alternatives (AI) and Parrot rewrites of corrected sentences;
+    #    only versions verified error-free (grammar check returns them unchanged, LanguageTool finds nothing,
+    #    fluent, same meaning) are kept as options.
+    verified = {}                                      # sentence -> set of verified correct versions
+
+    def gen_alts(ids, stage):
+        if not ids or llm.dead:
+            return
+        rep(stage, f"Keeping the AI score: natural rewording of {len(ids)} sentence(s)")
+        alts = llm.run(G.variants, [final.get(i, base[i]) for i in ids], glossary,
+                       [jobs[i]["sent"] or None for i in ids], VARIANTS)
+        pool = []
+        for i, vs in zip(ids, alts):
+            for v in (vs or []):
+                ok = accept(final.get(i, base[i]), v, glossary, jobs[i]["sent"])
+                if ok:
+                    pool.append((i, ok))
+        if PARROT:
+            try:
+                pr, _ = H._paraphrase_local([final.get(i, base[i]) for i in ids], 0.9, None, 0.8, 0.85)
+                for i, t in zip(ids, pr):
+                    ok = accept(final.get(i, base[i]), t, glossary, jobs[i]["sent"])
+                    if ok:
+                        pool.append((i, ok))
+            except Exception as err:
+                print(f"[proofread] Parrot rewording skipped: {err}", flush=True)
+        pool = list(dict.fromkeys(pool))
+        if not pool:
+            return
+        mean = MEANING.keeps_meaning([(final.get(i, base[i]), t) for i, t in pool])
+        flu = FL.acceptable([(final.get(i, base[i]), t) for i, t in pool])
+        pool = [(i, t) for (i, t), m, f in zip(pool, mean, flu) if m and f]
+        if not pool:
+            return
+        chk = llm.run(G.grammar, [t for _, t in pool], glossary, rep, stage, stage + 0.02, size=BATCH)
+        clean = [(i, t) for (i, t), g_ in zip(pool, chk) if g_ is not None and g_.strip() == t.strip()]
+        try:
+            import lt_grammar as LT
+            if LT.MODE != "off" and clean:
+                ltc = LT.correct([t for _, t in clean])
+                clean = [c for c, l_ in zip(clean, ltc) if l_.strip() == c[1].strip()]
+        except Exception:
+            pass
+        for i, t in clean:
+            verified.setdefault(i, set()).add(t)
+
+    n_reworded = 0
+    for rnd in range(0 if single else AI_ROUNDS):
+        ids = [i for i in work if final.get(i)]
+        if not ids:
+            break
+        now_sc = H._detector_scores([final[i] for i in ids]) or [0.0] * len(ids)
+        hot = [i for i, s_ in zip(ids, now_sc) if s_ > cur[i] + ALT_RISE and i not in repaired]
+        if not hot:
+            break
+        now_of = dict(zip(ids, now_sc))
+        gen_alts(hot, 0.78 + 0.02 * rnd)
+        cand = [(i, t) for i in hot for t in verified.get(i, ())]
+        sc = H._detector_scores([t for _, t in cand]) or [1.0] * len(cand)
+        best = {}
+        for (i, t), s_ in zip(cand, sc):
+            if s_ < now_of[i] - 0.02 and (i not in best or s_ < best[i][1]):
+                best[i] = (t, s_)
+        for i, (t, s_) in best.items():
+            final[i], rise[i] = t, s_ - cur[i]
+            required.add(i)
+            n_reworded += 1
+        if not best:
+            break
+
+    # 5b. paragraph balance: the document score is measured over sentence windows inside each paragraph, so
+    #     single-sentence scores are not enough. Every paragraph whose AI-flagged words rose is re-balanced by
+    #     choosing, sentence by sentence, among the CORRECT versions only (current, minimal fix, verified natural
+    #     versions; for a sentence without an error also its unedited form) the combination with the fewest
+    #     flagged words.
+    by_par = {}
+    for i, j in enumerate(jobs):
+        by_par.setdefault(id(j["b"]), []).append(i)
+
+    def par_text(pid, choice):
+        ids = by_par[pid]
+        raw = jobs[ids[0]]["b"].raw
+        for i in sorted(ids, key=lambda i: -jobs[i]["span"][0]):
+            t = choice.get(i, jobs[i]["text"])
+            a_, e_ = jobs[i]["span"]
+            if t == "":
+                while e_ < len(raw) and raw[e_] == " ":
+                    e_ += 1
+            raw = raw[:a_] + t + raw[e_:]
+        return raw
+
+    def flagged(text):
+        try:
+            st = D.summarise(S._score_paras([text]), hi, mod)
+            return st["high"] + st["moderate"]
+        except Exception:
+            return 0
+
+    def options(i):
+        o = [final.get(i, base[i])]
+        if i in repaired:                          # only the repair and its verified natural versions
+            return o + [t for t in sorted(verified.get(i, ())) if t not in o]
+        for t in [minimal.get(i)] + sorted(verified.get(i, ())):
+            if t and t not in o:
+                o.append(t)
+        if i not in required and jobs[i]["text"] not in o:
+            o.append(jobs[i]["text"])              # no error: its unedited form is allowed too
+        return o
+
+    n_balanced = 0
+    changed_pars = [pid for pid, ids in by_par.items() if any(i in final for i in ids)]
+    rep(0.86, f"Balancing the AI score of {len(changed_pars)} paragraph(s)")
+    for attempt in range(2):
+        worse = []
+        for pid in changed_pars:
+            was = flagged(par_text(pid, {}))
+            if flagged(par_text(pid, final)) > was + PAR_TOL:
+                worse.append((pid, was))
+        if not worse:
+            break
+        if attempt == 1 and not single:                # more correct versions for the sentences in those paragraphs
+            gen_alts([i for pid, _ in worse for i in by_par[pid] if i in final and final[i]], 0.88)
+        for pid, was in worse:
+            choice = {i: final[i] for i in by_par[pid] if i in final}
+            score = flagged(par_text(pid, choice))
+            for _ in range(len(by_par[pid]) * 2):      # greedy: best single swap per step
+                best = None
+                for i in by_par[pid]:
+                    if i in drops:
+                        continue
+                    for t in options(i):
+                        if t == choice.get(i, jobs[i]["text"]):
+                            continue
+                        trial = dict(choice)
+                        if t == jobs[i]["text"]:
+                            trial.pop(i, None)
+                        else:
+                            trial[i] = t
+                        f_ = flagged(par_text(pid, trial))
+                        if f_ < score and (best is None or f_ < best[0]):
+                            best = (f_, trial)
+                if best is None:
+                    break
+                score, choice = best
+                n_balanced += 1
+                if score <= was + PAR_TOL:
+                    break
+            for i in by_par[pid]:
+                if i in drops:
+                    continue
+                if i in choice:
+                    final[i] = choice[i]
+                else:
+                    final.pop(i, None)
+
+    ids = [i for i in work if final.get(i)]
+    still = sum(1 for i, s_ in zip(ids, H._detector_scores([final[i] for i in ids]) or [0.0] * len(ids))
+                if s_ > cur[i] + ALT_RISE and i not in repaired) if ids else 0
+
+    # 6. write back, then document-level AI check (undo style-only edits if the score still rose)
+    rep(0.85, "Writing corrections and re-checking the AI score")
+
+    def write(chosen):
+        d = DX.open_clean(path)
+        js = _jobs(d, None)
+        if len(js) != len(jobs):
+            return None, 0
+        n = 0
+        for i in sorted(chosen, key=lambda i: (id(js[i]["b"]), -js[i]["span"][0])):
+            a_, e_ = js[i]["span"]
+            if chosen[i] == "":                        # deletion: take the following space with it
+                raw = js[i]["b"].raw
+                while e_ < len(raw) and raw[e_] == " ":
+                    e_ += 1
+            if S.replace_span(js[i]["b"].el, a_, e_, chosen[i]):
+                n += 1
+        return d, n
+
+    out_doc, n = write(final)
+    if out_doc is None:
+        raise ValueError("The document layout changed while proofreading; please run it again.")
+    after = D.summarise(S._score_doc(out_doc), hi, mod)
+    undone = 0
+    while after["pct"] > before["pct"] + MAX_DOC_RISE:
+        optional = [i for i in final if i not in required and i not in drops and i not in local]
+        if not optional:                               # only required (grammar) fixes left: they always stay
+            break
+        worst = sorted(optional, key=lambda i: rise.get(i, 0), reverse=True)[:max(1, len(optional) // 5)]
+        for i in worst:
+            final.pop(i)
+            undone += 1
+        out_doc, n = write(final)
+        after = D.summarise(S._score_doc(out_doc), hi, mod)
+    out = H._save_output(out_doc, S._clean_stem(path), "_proofread")
+    who = " + ".join(_NAME.get(p, p) for p in llm.order if p in llm.used) or llm.name
+    note = (("🔍 Final check in one request" if single else "🔍 Rigorous proofread")
+            + f" by {who}: {n} of {len(jobs)} sentences corrected"
+            + (f" ({len(required)} had a grammar error or defect and were always corrected"
+               + (f", {len(repaired)} unclear sentence(s) repaired to the original meaning" if repaired else "")
+               + (f", {n_minimal} with the minimal grammar fix" if n_minimal else "")
+               + (f"; {n_verify} more fix(es) found in the verification rounds" if n_verify else "")
+               + (f"; {n_reworded} reworded naturally to keep the AI score" if n_reworded else "")
+               + (f"; {n_balanced} swap(s) between correct versions to balance paragraphs" if n_balanced else "")
+               + ")"
+               if required else "")
+            + (f"; {sum(defect)} had doubled terms, chatbot leftovers or broken English, "
+               f"{sum(1 for i, d in enumerate(defect) if d and i in final)} of them fixed" if any(defect) else "")
+            + f". AI score {before['pct']}% → {after['pct']}%."
+            + (f" {undone} style-only edit(s) were undone because they raised the AI score; grammar fixes "
+               "were all kept." if undone else "")
+            + (f" {still} corrected sentence(s) still read slightly more AI-like: no error-free natural version "
+               "was found, and correct grammar was kept." if still else "")
+            + (f" ⚠️ {unchecked} sentence(s) could not be checked (AI quota exhausted); run the proofread again "
+               "later to finish them." if unchecked else "")
+            + ("" if orig_path else " Tip: run it right after humanizing, so your original is used as the meaning "
+                                    "reference for unclear sentences."))
+    if notes:                                          # document-level problems that need the author
+        note += " 📝 For the author: " + " · ".join(notes)
+    print(f"[proofread] {note}", flush=True)
+    return dict(path=out, before=before, after=after, n=n, total=len(jobs), note=note, notes=notes,
+                single=single)

pubrules.py

@@ -185,9 +185,19 @@ A. Integrity - never broken:
  A4. Keep the strength of every claim: keep may/might/suggests/approximately; never add proves/clearly/always/all.
  A5. One sentence in, one sentence out: never split, merge, drop or reorder sentences.
  A6. Never replace a technical term with a synonym ("accuracy" is not "precision", "sweeps" is not "reaches",
-     "escalates" is not "increases"); when unsure, keep the author's word.
+     "escalates" is not "increases"); when unsure, keep the author's word. The study's own constructs (every term
+     defined with an abbreviation, e.g. "travel satisfaction (TS)", "intimacy (IN)") keep their exact names: never
+     "pleasure", "contentment", "happiness", "closeness", "allegiance", "devotion" for them.
+ A7. Never add a reason, cause, explanation or attribution the original does not state ("according to some
+     experts", "since they have ...", "a systematic analysis found", "researchers contend").
+ A8. Keep every list item and every name or place ("Digha, Darjeeling, Jhargram and Purulia" keeps all four).
+ A9. Keep statistics exactly: "p = .120" stays "p = .120"; never turn "=" into "<" or ">", never add an operator
+     that is missing in the original, never drop one ("p < .001" never becomes "p .001").
+ A10. Never end a sentence or a term half-way ("... and to enhance their.", "well-,").
 B. Grammar - every sentence complete and correct: subject-verb agreement, articles, verb forms, tense, prepositions,
-   punctuation (commas in lists and after introductory clauses), no missing, doubled or run-together words, no
+   punctuation (keep every comma of the original that is still needed: in lists, after introductory phrases such as
+   "After COVID-19," or "However,", and after a bracketed abbreviation in a list; never remove commas to change the
+   style), no missing, doubled or run-together words, no
    fragments, no run-on sentences, a capital only where English needs one ("Greshake et al. generalised", not
    "et al. Generalised").
 C. Academic register: formal and impersonal; no contractions, no "you", no first person singular, no informal or

smart.py

@@ -24,6 +24,7 @@ import humanizer as H
 import groqai as G
 import report_flags as R
 import restructure as RS
+import adaptive as AD
 import meaning as MEANING
 import profile_settings as PS
 import fluency as FL
@@ -304,6 +305,40 @@ def _rewrite_retry(sentences, temperature, mode, rep, lo, hi):
     return new, note
 
 
+def _adaptive(fn, sentences, temperature, mode, rep, lo, hi):
+    """Self-adaptive pass: every sentence gets the setting its own history in this document calls for
+    (adaptive.settings): bolder (higher temperature + detector-guided decoding) when its AI score did not fall last
+    pass, calmer when its rewrites keep failing the checks, left as written after ADAPT_MAX_FAILS failures.
+    Sentences with the same setting are rewritten together."""
+    anchors = [H.anchor_of(t) or t for t in sentences]
+    sets = [AD.settings(a) for a in anchors]
+    groups = {}
+    for i, (dt, bold, skip) in enumerate(sets):
+        if not skip:
+            groups.setdefault((round(dt, 2), bold), []).append(i)
+    new, note = [None] * len(sentences), ""
+    total = max(1, sum(len(v) for v in groups.values()))
+    pos = 0
+    for (dt, bold), idx in sorted(groups.items()):
+        a_ = lo + (hi - lo) * pos / total
+        b_ = lo + (hi - lo) * (pos + len(idx)) / total
+        tok = H.FORCE_ADV.set(bool(bold))
+        try:
+            out, n_ = fn([sentences[i] for i in idx], min(1.5, max(0.3, temperature + dt)), mode, rep, a_, b_)
+        finally:
+            H.FORCE_ADV.reset(tok)
+        note = note or n_
+        for i, o in zip(idx, out):
+            new[i] = o
+        pos += len(idx)
+    for a, t, n in zip(anchors, sentences, new):
+        AD.outcome(a, bool(n and n.strip() != t.strip()))
+    skipped = sum(1 for x in sets if x[2])
+    if skipped:
+        note += f" 🧠 {skipped} sentence(s) kept as written: their rewrites failed the checks in {AD.MAX_FAILS} passes."
+    return new, note
+
+
 def _full_ai():
     return bool(H.FULL_AI.get())
 
@@ -549,12 +584,15 @@ def smart_docx(path, target, passes, temperature, mode, hi, mod, rep):
     log, first, note = [], None, ""
     orig_text = "\n".join(b.text for b in D.docx_blocks(doc) if b.kind == "para")
     TP.GLOSSARY.set(TP.build_glossary([orig_text]))     # the document's key terms stay untouched
+    TP.set_document([orig_text])                        # its constructs (defined terms, keywords) never change
     best = None                                        # (pct, docx bytes, scored blocks)
     rpt, rows = R.ACTIVE.get(), None                   # uploaded report: rewrite only the sentences it flags
     if rpt is not None:
         passes = 1
     acc, from_orig = None, True                        # last version whose AI score went DOWN (passes build on it)
     stall, low, t0 = 0, None, time.time()           # automatic passes: stop when no pass improves any more
+    AD.start_job()                                     # self-adaptive settings per sentence for this document
+    pend = None                                        # (temperature offset, AI % before, jobs, rewritten) of last pass
     step = 0.8 / max(1, passes)
     for k in range(1, int(passes) + 2):                # last iteration = final check only
         base = 0.05 + step * (k - 1)
@@ -562,6 +600,9 @@ def smart_docx(path, target, passes, temperature, mode, hi, mod, rep):
         blocks = _score_doc(doc)
         stats = D.summarise(blocks, hi, mod)
         first = first or stats
+        if pend is not None:                           # learn how well last pass's setting worked
+            AD.record_pass(pend[0], pend[1], stats["pct"], pend[2], pend[3])
+            pend = None
         cur = "\n".join(b.text for b in blocks if b.kind == "para")
         chg = H.text_change(orig_text, cur) if k > 1 else None
         cand_ = (stats["pct"], chg, (_snapshot(doc), blocks))
@@ -577,6 +618,10 @@ def smart_docx(path, target, passes, temperature, mode, hi, mod, rep):
             chg, from_orig = acc[2], acc[3]
         else:
             acc, from_orig = (stats["pct"], _snapshot(doc), chg, k == 1), k == 1
+        for b_ in blocks:                              # per-sentence learning: did its AI score fall?
+            if b_.kind == "para":
+                for s_, p_ in zip(b_.sents, b_.scores):
+                    AD.observe(H.anchor_of(s_) or s_, p_)
         if k > passes:
             break
         if _done_early(k, stats["pct"], target, chg):
@@ -638,16 +683,19 @@ def smart_docx(path, target, passes, temperature, mode, hi, mod, rep):
                 rows = linfo
             log.append(_log_line(k, stats["pct"]) + f" — no rewritable sentences left ({locked} locked)")
             break
-        temp = min(1.5, float(temperature) + 0.15 * (k - 1))
+        t_off = AD.pick_temp_offset()                  # learned across documents (Thompson sampling)
+        temp = min(1.5, max(0.3, float(temperature) + 0.15 * (k - 1) + t_off))
         # pass 1 pushes each sentence to the new-wording level (measured against the ORIGINAL sentence);
         # later passes only lower the AI score: their input is already rewritten, so a per-sentence level there
         # would compare against the previous pass and keep churning. The whole-text level is checked below.
         vtok = H.VOCAB_TARGET.set(0.0) if not from_orig else None
         try:
             if rpt is not None:
-                new, note = _report_rewrite([j[2] for j in jobs], temp, mode, rep, base + 0.1 * step, base + 0.9 * step)
+                new, note = _adaptive(_report_rewrite, [j[2] for j in jobs], temp, mode, rep, base + 0.1 * step,
+                                      base + 0.9 * step)
             else:
-                new, note = _rewrite_retry([j[2] for j in jobs], temp, mode, rep, base + 0.1 * step, base + 0.9 * step)
+                new, note = _adaptive(_rewrite_retry, [j[2] for j in jobs], temp, mode, rep, base + 0.1 * step,
+                                      base + 0.9 * step)
         finally:
             if vtok is not None:
                 H.VOCAB_TARGET.reset(vtok)
@@ -658,12 +706,14 @@ def smart_docx(path, target, passes, temperature, mode, hi, mod, rep):
                 H.remember_anchor(t, n)
                 done += 1
         log.append(_log_line(k, stats["pct"], len(jobs), done, locked))
+        pend = (t_off, stats["pct"], len(jobs), done)
         if rpt is not None:
             rows = [dict(report=ri[0], colour=ri[1], orig=j[2], new=(n if (n and n != j[2]) else j[2]),
                          status="rephrased" if (n and n != j[2]) else "kept")
                     for j, n, ri in zip(jobs, new, jinfo)] + linfo
         if not done:
             break
+    AD.save(force=True)
     # keep the lowest-scoring version seen (a pass can occasionally make things worse)
     rep(0.9, "Preparing results")
     from docx import Document
@@ -691,6 +741,7 @@ def smart_text(text, target, passes, temperature, mode, hi, mod, rep):
     parts = TP.parse_paste(text)
     seps = [sep for _, sep in parts]
     TP.GLOSSARY.set(TP.build_glossary([p for p, _ in parts]))   # the text's key terms stay untouched
+    TP.set_document([p for p, _ in parts])
     paras = [p for p, _ in parts]
     orig = TP.join_paste(parts)
     H.ANCHOR.set({})                                   # rewritten sentence -> author's original sentence
@@ -700,6 +751,8 @@ def smart_text(text, target, passes, temperature, mode, hi, mod, rep):
         passes = 1
     acc, from_orig = None, True                        # last version whose AI score went DOWN (passes build on it)
     stall, low, t0 = 0, None, time.time()           # automatic passes: stop when no pass improves any more
+    AD.start_job()                                     # self-adaptive settings per sentence for this document
+    pend = None                                        # (temperature offset, AI % before, jobs, rewritten) of last pass
     step = 0.8 / max(1, passes)
     for k in range(1, int(passes) + 2):
         base = 0.05 + step * (k - 1)
@@ -707,6 +760,9 @@ def smart_text(text, target, passes, temperature, mode, hi, mod, rep):
         blocks = _score_paras(paras)
         stats = D.summarise(blocks, hi, mod)
         first = first or stats
+        if pend is not None:                           # learn how well last pass's setting worked
+            AD.record_pass(pend[0], pend[1], stats["pct"], pend[2], pend[3])
+            pend = None
         chg = H.text_change(orig, TP.join_paste(list(zip(paras, seps)))) if k > 1 else None
         cand_ = (stats["pct"], chg, (list(paras), blocks))
         if _better(cand_, best, k):
@@ -719,6 +775,10 @@ def smart_text(text, target, passes, temperature, mode, hi, mod, rep):
             chg, from_orig = acc[2], acc[3]
         else:
             acc, from_orig = (stats["pct"], list(paras), chg, k == 1, blocks), k == 1
+        for b_ in blocks:                              # per-sentence learning: did its AI score fall?
+            if b_.kind == "para":
+                for s_, p_ in zip(b_.sents, b_.scores):
+                    AD.observe(H.anchor_of(s_) or s_, p_)
         if k > passes:
             break
         if _done_early(k, stats["pct"], target, chg):
@@ -763,16 +823,19 @@ def smart_text(text, target, passes, temperature, mode, hi, mod, rep):
         if not jobs:
             log.append(_log_line(k, stats["pct"]) + " — no rewritable sentences left")
             break
-        temp = min(1.5, float(temperature) + 0.15 * (k - 1))
+        t_off = AD.pick_temp_offset()                  # learned across documents (Thompson sampling)
+        temp = min(1.5, max(0.3, float(temperature) + 0.15 * (k - 1) + t_off))
         # pass 1 pushes each sentence to the new-wording level (measured against the ORIGINAL sentence);
         # later passes only lower the AI score: their input is already rewritten, so a per-sentence level there
         # would compare against the previous pass and keep churning. The whole-text level is checked below.
         vtok = H.VOCAB_TARGET.set(0.0) if not from_orig else None
         try:
             if rpt is not None:
-                new, note = _report_rewrite([j[2] for j in jobs], temp, mode, rep, base + 0.1 * step, base + 0.9 * step)
+                new, note = _adaptive(_report_rewrite, [j[2] for j in jobs], temp, mode, rep, base + 0.1 * step,
+                                      base + 0.9 * step)
             else:
-                new, note = _rewrite_retry([j[2] for j in jobs], temp, mode, rep, base + 0.1 * step, base + 0.9 * step)
+                new, note = _adaptive(_rewrite_retry, [j[2] for j in jobs], temp, mode, rep, base + 0.1 * step,
+                                      base + 0.9 * step)
         finally:
             if vtok is not None:
                 H.VOCAB_TARGET.reset(vtok)
@@ -785,12 +848,14 @@ def smart_text(text, target, passes, temperature, mode, hi, mod, rep):
                 done += 1
         paras = [" ".join(sents[i]) if b.kind == "para" else b.text for i, b in enumerate(blocks)]
         log.append(_log_line(k, stats["pct"], len(jobs), done))
+        pend = (t_off, stats["pct"], len(jobs), done)
         if rpt is not None:
             rows = [dict(report=ri[0], colour=ri[1], orig=j[2], new=(n if (n and n != j[2]) else j[2]),
                          status="rephrased" if (n and n != j[2]) else "kept")
                     for j, n, ri in zip(jobs, new, jinfo)]
         if not done:
             break
+    AD.save(force=True)
     rep(0.9, "Preparing results")
     paras, blocks = best[2]
     after = D.summarise(blocks, hi, mod)

tests/test_groqai.py

@@ -61,7 +61,8 @@ class _Resp:
 def test_rate_limited_model_falls_back_to_the_next_model(monkeypatch):
     import httpx
     monkeypatch.setattr(G, "KEYS", ["k1"])
-    monkeypatch.setattr(G, "_KEY_STATE", [{"until": 0.0, "calls": 0, "err": ""}])
+    monkeypatch.setattr(G, "_KEY_STATE", [dict(G._KEY_STATE[0] if G._KEY_STATE else {}, until=0.0, calls=0, err="",
+                                                tok_left=None, tok_reset=0.0, tok_limit=None, inflight=0, tokens=0)])
     monkeypatch.setattr(G, "ENABLED", True)
     monkeypatch.setattr(G, "_models", ["big", "small"])
     monkeypatch.setattr(G, "_KM_UNTIL", {})

textparse.py

@@ -135,6 +135,7 @@ viscosity density velocity torque load cathode anode electrolyte catalyst enzyme
 hypothesis significance sample respondent mediation moderation reliability validity simulation mesh node
 cluster packet protocol routing entropy matrix vector tensor derivative integral equation theorem spectrum
 wavelength phase charge discharge battery circuit converter inverter sensor compression sparse sparsity
+gender genders male males female females men women participants respondents
 reconstruction segmentation detection estimation prediction forecasting pavement asphalt binder aggregate
 """.split())
 
@@ -201,6 +202,8 @@ def build_glossary(texts):
                     continue
                 if n == 2 and max(len(g[0]), len(g[1])) < 5:
                     continue
+                if n == 3 and g[1] in _STOP_N and g[1] not in ("of", "to", "and", "for", "in", "on", "per"):
+                    continue                              # "support the preservation" is wording, not a term
                 k = " ".join(g)
                 grams[k] = grams.get(k, 0) + 1
     if NARROW.get():                                   # report-flagged rewriting: real technical terms only
@@ -214,6 +217,103 @@ def build_glossary(texts):
     return frozenset(t for t in terms if t)
 
 
+# ---------------------------------------------------------------- document key terms (always protected)
+# The study's own constructs must never be swapped for synonyms ("travel satisfaction" -> "travel pleasure",
+# "intimacy" -> "closeness", "destination loyalty" -> "allegiance"). They are read from the WHOLE document once per
+# job and protected in every mode, including report-guided rewriting where the ordinary glossary is narrowed:
+#   * every term defined with an abbreviation, "travel satisfaction (TS)" -> "travel satisfaction" + "satisfaction"
+#   * the Keywords line, "Keywords: Travel motivation, Intimacy, ..." -> each keyword + its head word
+KEY = contextvars.ContextVar("tp_key_terms", default=frozenset())
+KEY_DEF = contextvars.ContextVar("tp_key_defined", default=frozenset())   # terms defined with an abbreviation
+DOC_VOCAB = contextvars.ContextVar("tp_doc_vocab", default=frozenset())
+DOC_PROPER = contextvars.ContextVar("tp_doc_proper", default=frozenset())   # words the document always capitalises
+_GENERIC_HEADS = {"study", "analysi", "analysis", "difference", "differenc", "economy", "engagement", "modeling",
+                  "modelling", "council", "organization", "organisation", "approach", "model", "method", "framework", "system", "process", "factor",
+                  "factors", "effect", "effects", "impact", "role", "level", "levels", "rate", "index", "test",
+                  "scale", "theory", "research", "development", "management", "performance", "behaviour",
+                  "behavior", "value", "values", "data", "group", "groups", "practice", "practices"}
+
+
+def key_terms(texts):
+    """Construct names of a document (normalised), see KEY."""
+    joined = " ".join(t for t in texts if t)
+    terms = set()
+    for m in re.finditer(r"((?:[A-Za-z][\w\-]*\s+){1,5})\(\s*([A-Z][A-Za-z0-9\-]{1,11}?)s?\s*\)", joined):
+        acr, words = m.group(2), m.group(1).split()
+        core = len(re.findall(r"\b" + re.escape(acr) + r"\b", joined)) >= 4   # a construct the study keeps using
+        n = len(re.findall(r"[A-Z]", acr)) or len(acr)
+        cand = words[-n:]
+        while cand and cand[0].lower() in _STOP:
+            cand = cand[1:]
+        if not cand or len(" ".join(cand)) < 4:
+            continue
+        terms.add(_norm(" ".join(cand)).strip())
+        head = _norm(cand[-1]).strip()
+        if core and len(head) >= 5 and head not in _GENERIC_HEADS and head not in _STOP_N:
+            terms.add(head)
+    KEY_DEF.set(frozenset(terms))
+    for line in re.findall(r"(?im)^\s*key\s*-?\s*words?\s*[:\-–]\s*(.+)$", "\n".join(t for t in texts if t)):
+        for kw in re.split(r"[,;·•]", line):
+            kw = _norm(kw).strip()
+            if len(kw) >= 4 and len(kw.split()) <= 5:
+                terms.add(kw)
+                head = kw.split()[-1]
+                if len(head) >= 5 and head not in _GENERIC_HEADS and head not in _STOP_N:
+                    terms.add(head)
+    return frozenset(t for t in terms if t)
+
+
+# everyday synonyms a paraphraser uses for common research constructs; when the construct is one of the document's
+# key terms, a rewrite may not bring in its synonym ("satisfaction" study: "pleasure", "contentment" are rejected)
+_CONSTRUCT_SYN = {
+    "satisfaction": {"pleasure", "contentment", "happiness", "delight", "gratification", "fulfilment", "fulfillment"},
+    "loyalty": {"allegiance", "faithfulness", "devotion", "fidelity"},
+    "intimacy": {"closeness", "familiarity", "affinity"},
+    "motivation": {"incentive", "incentives", "impetus", "inducement"},
+    "well being": {"welfare", "wellness", "health"},
+    "wellbeing": {"welfare", "wellness", "health"},
+    "trust": {"faith", "confidence", "reliance"},
+    "attachment": {"bond", "fondness", "affection"},
+    "intention": {"willingness", "inclination"},
+    "engagement": {"involvement", "participation", "commitment"},
+    "commitment": {"dedication", "devotion", "pledge"},
+    "performance": {"output", "achievement", "efficiency"},
+    "image": {"picture", "impression", "reputation"},
+    "quality": {"standard", "calibre", "caliber", "grade"},
+    "anxiety": {"worry", "nervousness", "unease"},
+    "stress": {"strain", "pressure", "tension"},
+}
+
+
+def renamed_construct(original, new):
+    """A key construct's everyday synonym the rewrite introduces ("allegiance" in a loyalty study), or None."""
+    key = KEY.get()
+    if not key:
+        return None
+    nn, no = _norm(new), _norm(original)
+    for term, syns in _CONSTRUCT_SYN.items():
+        if term in key:
+            for w in syns:
+                if f" {_norm(w).strip()} " in nn and f" {_norm(w).strip()} " not in no:
+                    return f"{term} -> {w}"
+    return None
+
+
+def set_document(texts):
+    """Call once per job with the whole document: key terms and vocabulary for the checks below."""
+    texts = [t for t in texts if t]
+    KEY.set(key_terms(texts))
+    cap, low = {}, {}
+    for t in texts:
+        for m in re.finditer(r"(?<![.!?:]\s)(?<=\s)([A-Za-z][a-z]{2,})\b", t):   # mid-sentence words only
+            w = m.group(1)
+            d = cap if w[:1].isupper() else low
+            d[w.lower()] = d.get(w.lower(), 0) + 1
+    DOC_PROPER.set(frozenset(w.capitalize() for w, c in cap.items() if c >= 2 and not low.get(w)))
+    vocab = {w.lower() for t in texts for w in re.findall(r"[A-Za-z][A-Za-z\-]+", t)}
+    DOC_VOCAB.set(frozenset(vocab) if sum(len(t.split()) for t in texts) >= 300 else frozenset())
+
+
 def lost_terms(original, new, glossary=None):
     """Technical terms of `original` that `new` no longer contains (empty list = safe)."""
     glossary = GLOSSARY.get() if glossary is None else glossary
@@ -227,6 +327,7 @@ def lost_terms(original, new, glossary=None):
     missing += [g for g in glossary if f" {g} " in no and f" {g} " not in nn]
     # statistical relations in any word form: "does not moderate" must not become "does not modify"
     missing += [st + "…" for st in _STAT_STEMS if f" {st}" in no and f" {st}" not in nn]
+    missing += [g for g in KEY.get() if f" {g} " in no and f" {g} " not in nn]       # study constructs
     return sorted(set(missing))
 
 
@@ -250,11 +351,12 @@ def invented(original, new):
     if any(n not in nums_o for n in re.findall(r"\d+(?:\.\d+)?", CITE.sub(" ", new))):
         return "number"
     for m in re.finditer(r"\b[A-Z][a-z]{2,}\b", new):           # capitalised word mid-sentence = name-like
-        if _SENT_START.search(new[:m.start()]):
-            continue
+        if _SENT_START.search(new[:m.start()]) or (re.search(r"[\])]\s*$", new[:m.start()])
+                                                   and re.fullmatch(r"[A-Z][a-z]+(?:ed|es|s)", m.group(0))):
+            continue                                     # sentence start, or "[56] Underscored" (citation + verb)
         w = m.group(0).lower()
-        if w in o_words:
-            continue
+        if w in o_words or re.search(r"\b" + re.escape(w) + r"\b", o_low):
+            continue                                     # ("Men's" in the original counts as "Men")
         # another form of a name the author uses ("India's" for "Indian", "Bayes" for "Bayesian") is not invented
         if len(w) >= 4 and any(o.startswith(w[:max(4, len(w) - 2)]) or w.startswith(o[:max(4, len(o) - 2)])
                                for o in o_caps):
@@ -300,11 +402,43 @@ def _known(word, o_words):
     return False
 
 
+_FOREIGN = re.compile(r"\b[\w\-]*[À-ÖØ-öø-ÿĀ-žƀ-ɏ][\w\-]*")
+OFFTOPIC_N = int(__import__("os").getenv("HUM_OFFTOPIC", "2"))   # 0 = check off
+
+
+def _offtopic(original, new):
+    """Uncommon content words the rewrite brings in that appear nowhere in the document and are not forms of the
+    sentence's own words ("organizational recruitment" in a tourism paper). None, or the words."""
+    vocab = DOC_VOCAB.get()
+    if not vocab or not OFFTOPIC_N or _zipf is None:
+        return None
+    o_low = original.lower()
+    stems = {w[:5] for w in re.findall(r"[a-z]{5,}", o_low)}
+    odd = []
+    for w in set(re.findall(r"[A-Za-z][a-z\-]{5,}", CITE.sub(" ", new))):
+        lw = w.lower()
+        if lw in vocab or lw[:5] in stems or lw in _STOP or w[:1].isupper():
+            continue
+        if _zipf(lw, "en") < 3.3 and not any(v.startswith(lw[:6]) for v in vocab):
+            odd.append(lw)                                # rare AND unrelated to any word the document uses
+    return odd if len(odd) >= OFFTOPIC_N else None
+
+
 def quality_issue(original, new):
     """Why `new` is not acceptable academic English compared with `original` (None = fine):
-    a non-word the original doesn't contain, informal / conversational wording, first person singular,
-    addressing the reader, contractions, or a question where the original made a statement."""
+    a non-word the original doesn't contain, a foreign word, words foreign to the document's topic, informal /
+    conversational wording, first person singular, addressing the reader, contractions, or a question where the
+    original made a statement."""
     o_words = set(re.findall(r"[a-z][a-z'\-]*", original.lower()))
+    syn = renamed_construct(original, new)
+    if syn:
+        return f"construct renamed ({syn})"
+    for m in _FOREIGN.finditer(new):                      # "well-être": accented words the original does not use
+        if m.group(0).lower() not in original.lower():
+            return f"foreign word '{m.group(0)}'"
+    odd = _offtopic(original, new)
+    if odd:
+        return "off-topic words '" + ", ".join(sorted(odd)) + "'"
     for w in re.findall(r"[A-Za-z][A-Za-z\-]{3,}", CITE.sub(" ", new)):
         if w.isupper() or (w[0].isupper() and w.lower() in o_words):
             continue                                      # acronyms / names already in the original
@@ -376,6 +510,152 @@ def repair_citations(original, new):
             return None
         new = new.replace(c, best, 1)
     return new
+def _symbols(t):
+    """Comparison / statistics symbols outside citations: < > ≤ ≥ = ± × and their order."""
+    return re.findall(r"[<>≤≥=±×]", CITE.sub(" ", t or ""))
+
+
+def _brackets_ok(t):
+    return (t or "").count("(") == (t or "").count(")") and (t or "").count("[") == (t or "").count("]")
+
+
+def _new_acronyms(original, new):
+    acr = lambda t: set(re.findall(r"\b[A-Z][A-Z0-9]{1,}[a-z]?\b", CITE.sub(" ", t or "")))
+    return acr(new) - acr(original)
+
+
+# ---------------------------------------------------------------- integrity checks shared by EVERY rewrite path
+# (Parrot guard, Groq quality check, final polish, proofread, clarity gate). Each one targets an error seen in real
+# humanized papers.
+_ATTRIB = re.compile(r"\b(?:according to|(?:some |many |several )?(?:experts?|scholars?|researchers?|analysts?|"
+                     r"studies|scientists)\s+(?:say|says|said|argue|argues|contend|contends|believe|believes|"
+                     r"suggest|suggests|claim|claims|found|find|show|shows|note|notes)|a (?:systematic |recent )?"
+                     r"(?:analysis|review|study|survey) (?:found|showed|shows|finds|revealed|suggests))\b", re.I)
+_SAID = re.compile(r"\b(?:said|says|told|stated|according to)\b", re.I)
+_CAUSAL = re.compile(r"\b(?:because|since (?:they|it|he|she|we|the|this|these|those|their|its)|due to|owing to|"
+                     r"as a result of|thereby|in turn)\b", re.I)
+_DANGLING_END = re.compile(r"\b(?:the|a|an|their|its|our|his|her|of|to|and|or|on|in|with|for|by|from|as|than|"
+                           r"that|which|this|these|between|among)\s*[.!?]$", re.I)
+_DANGLING_HYPHEN = re.compile(r"[A-Za-z]-(?=[\s,.;:)])")
+_EMDASH = re.compile(r"—|\s–\s|\w\s-\s\w")       # also a spaced hyphen used as a dash
+_SENT_ADV = r"(?:However|Moreover|Furthermore|Therefore|Consequently|Nevertheless|Nonetheless|Additionally|" \
+            r"In addition|In general|In contrast|In particular|In summary|In conclusion|For example|For instance|" \
+            r"Thus|Hence|Instead|Similarly|Likewise|Meanwhile|Simultaneously|Finally|" \
+            r"Accordingly|Specifically|Notably|Importantly|Ultimately|Conversely|Subsequently)"
+# ("Further", "Overall", "First" are left out: "Further studies…", "Overall satisfaction…" need no comma)
+
+
+def _proper_nouns(t):
+    """Capitalised names inside a sentence (Darjeeling, Singapore, Kumar), not its first word."""
+    words = re.findall(r"(?<=[\s(\[,;:])([A-Z][a-z]{2,}(?:-[A-Z][a-z]+)?)\b", " " + CITE.sub(" ", t or "")[1:])
+    return {w for w in words if w.lower() not in _STOP and w not in ("Table", "Figure", "Fig", "Section", "Eq",
+                                                                     "Chapter", "Appendix")}
+
+
+def integrity_issue(original, new):
+    """Damage a rewrite may never do (None = fine):
+      * an attribution or a causal explanation the original does not make ("According to some experts … since they
+        have more muscle"; "a systematic analysis found");
+      * a name / place dropped from the sentence ("Digha, Darjeeling, Jhargram" -> "Digha Jhargram");
+      * a key construct used more often than in the original ("… destination loyalty (DL) and intimacy");
+      * a cut-off ending ("… and to enhance their.") or a cut-off term ("on well-, and");
+      * an em dash, or a doubled conjunction ("and, and");
+      * two sentences run together ("… or mass According to …")."""
+    o, n = original or "", new or ""
+    try:
+        import adaptive as _AD                            # words learned to cause bad rewrites
+        bw = _AD.banned_in(o, n)
+        if bw:
+            return f"banned word: {bw}"
+    except Exception:
+        pass
+    if _ATTRIB.search(n) and not _ATTRIB.search(o):
+        return "attribution added"
+    if len(_CAUSAL.findall(n)) > len(_CAUSAL.findall(o)):
+        return "causal claim added"
+    if len(_SAID.findall(n)) > len(_SAID.findall(o)):
+        return "attribution added"                        # "... beyond local capacity the director said"
+    if set(re.findall(r"\b[A-Z]\.[A-Z]\.?", n)) - set(re.findall(r"\b[A-Z]\.[A-Z]\.?", o)):
+        return "abbreviation added"                       # "Darjeeling, and M.P, and purulia"
+    names = DOC_PROPER.get()
+    for w in _proper_nouns(o) | ({o.split()[0].strip(",.;:")} if o.split() else set()):
+        is_name = w in names if names else (_zipf is not None and _zipf(w.lower(), "en") < 2.0)
+        if len(w) > 3 and is_name and re.search(r"\b" + re.escape(w.lower()) + r"\b", n) and w not in n:
+            return f"name lower-cased: {w}"               # "purulia" (the document always writes "Purulia")
+    for a, b in re.findall(r"\b([A-Z][a-z]+)\s+([A-Z][a-z]+)\b", o):
+        if a in names and re.search(r"\b" + a.lower() + r"\s+" + b + r"\b", n):
+            return f"name lower-cased: {a} {b}"            # "west Bengal"
+    stray = r"[a-z,]\s+(?:The|A|An|Of|For|In|And|To|With|On)\s"
+    if re.search(stray, n) and not re.search(stray, o):
+        return "capital inside a sentence"                # "variance in The TS"
+    if re.search(r"(?:\b[A-Z][a-z]{2,}\s+){3,}", n) and not re.search(r"(?:\b[A-Z][a-z]{2,}\s+){3,}", o):
+        caps = re.findall(r"(?:\b[A-Z][a-z]{2,}\s+){3,}", n)[0].split()
+        if sum(w.lower() in _STOP or (_zipf is not None and _zipf(w.lower(), "en") >= 4.5) for w in caps) >= 2:
+            return "title case in a sentence"             # "The Strong Effect of Intimacy For Both genders"
+    if re.match(r"^\[\d[\d,\s–\-]*\]\s+[A-Z][a-z]+(?:ed|es|s)\b", o) and \
+            re.match(r"^\[\d[\d,\s–\-]*\]\s+(?:The|A|An|This|These|Its)\b", n):
+        return "reporting verb dropped"                   # "[56] Highlighted ..." -> "[56] The importance of ..."
+    if re.search(r"\b(whereas|while|although|though|but|however|and|or|because)\s+(whereas|while|although|"
+                 r"though|but|however|because)\b", n, re.I):
+        return "doubled connective"                       # "whereas while", "but however"
+    sw = [x.lower() for x in re.findall(r"[A-Za-z]{5,}", n)]
+    so = [x.lower() for x in re.findall(r"[A-Za-z]{5,}", o)]
+    rep_n = {a[:5] for a, b in zip(sw, sw[1:]) if a != b and a[:5] == b[:5]}
+    rep_o = {a[:5] for a, b in zip(so, so[1:]) if a != b and a[:5] == b[:5]}
+    if rep_n - rep_o:
+        return "stem repeated"                            # "intimate intimacy intimation"
+    dash = lambda t: re.sub(r"[‐‑–—]", "-", t).lower()
+    lost = {w for w in _proper_nouns(o) if dash(w) not in dash(n)               # (case / dash changes are fine)
+            and not re.search(r"[\])]\s*" + re.escape(w) + r"\b", o)         # "[59] Discussed" is a verb
+            and not (_zipf is not None and _zipf(w.lower(), "en") >= 5.0)}    # "for Men" -> "for males"
+    if lost:
+        return "name dropped: " + ", ".join(sorted(lost))
+    nn, no = _norm(n), _norm(o)
+    for k in KEY_DEF.get():
+        if nn.count(f" {k} ") > max(1, no.count(f" {k} ")):
+            return f"construct repeated: {k}"
+    if _DANGLING_END.search(n.strip()) and not _DANGLING_END.search(o.strip()):
+        return "sentence cut off"
+    if len(_DANGLING_HYPHEN.findall(n)) > len(_DANGLING_HYPHEN.findall(o)):
+        return "term cut off"
+    cut = r"\b(?:of|to|for|with|on|in|by|from|the|a|an|and|or|their|its)\s*,"
+    if len(re.findall(cut, n, re.I)) > len(re.findall(cut, o, re.I)):
+        return "phrase cut off"                           # "the port of, and the forest"
+    op = r"\b(?:Sadly|Unfortunately|Fortunately|Luckily|Interestingly|Surprisingly|Obviously|Clearly|Undoubtedly|" \
+         r"Remarkably|Importantly|Notably|Strikingly)\b"
+    if re.search(op, n) and not re.search(op, o):
+        return "opinion word added"                       # "Sadly loyalty does not ..."
+    letters = lambda t: re.findall(r"(?<![\w.'’-])([b-zB-HJ-Z])(?![\w.'’-])", CITE.sub(" ", t))
+    if len(letters(n)) > len(letters(o)):
+        return "stray letter"                             # "... is d a vital predictor"
+    sq = lambda t: t.count("[") - sum(1 for c in CITE.findall(t) if c.startswith("["))   # "[" outside citations
+    if sq(n) > sq(o):                                     # (author-year citations have no "[": not counted)
+        return "bracket type changed"                     # "(explaining ...)" -> "[explaning ...]"
+    if _EMDASH.search(n) and not _EMDASH.search(o):
+        return "em dash"
+    if re.search(r"\b(\w+),?\s+\1\b", n, re.I) and not re.search(r"\b(\w+),?\s+\1\b", o, re.I):
+        return "doubled word"
+    if re.search(r"\b[a-z]{3,}\s+(?:According|However|Moreover|Furthermore|Therefore|This|These|The|It|In)\s+[a-z]", n) \
+            and not re.search(r"\b[a-z]{3,}\s+(?:According|However|Moreover|Furthermore|Therefore|This|These|The|It|In)\s+[a-z]", o):
+        return "sentences run together"
+    return None
+
+
+def restore_commas(original, new):
+    """Put back the commas a paraphraser dropped between words that keep their order: list items ("airports,
+    trains and taxis"), after an opening phrase ("After COVID-19, tourists") and after an abbreviation in brackets
+    ("(TM), eudaimonic"). Also adds the comma after a sentence-opening linking word ("However the" -> "However,
+    the")."""
+    if not new:
+        return new
+    out = new
+    for m in re.finditer(r"(?<![\w\-])([\w)\]%][\w\-)\]%]*),\s+(?=([\w(\[]+))", original or ""):
+        a, b = m.group(1), m.group(2)
+        pat = re.compile(r"(?<![\w-])" + re.escape(a) + r"(\s+)" + re.escape(b) + r"(?![\w-])")
+        if len(pat.findall(out)) == 1 and not re.search(re.escape(a) + r",\s+" + re.escape(b), out):
+            out = pat.sub(lambda k: a + "," + k.group(1) + b, out, count=1)
+    out = re.sub(rf"^({_SENT_ADV})\s+(?=[a-z])", r"\1, ", out)
+    return out
 
 
 def guard(original, new, glossary=None):
@@ -389,12 +669,21 @@ def guard(original, new, glossary=None):
     new = repair_citations(original, new)
     if new is None or invented(original, new) or quality_issue(original, new) or symbol_issue(original, new):
         return None
+    if _symbols(new) != _symbols(original):              # "p < .001" must not become "p .001"
+        return None
+    if _brackets_ok(original) and not _brackets_ok(new):  # "(TS amongst" - a bracket lost
+        return None
+    if _new_acronyms(original, new):                      # "EWB" must not become "EWD"
+        return None
+    new = restore_commas(original, new)                   # "airports trains and taxis" -> "airports, trains and taxis"
+    if integrity_issue(original, new):
+        return None
     ow, nw = len(original.split()), len(new.split())
     if nw < 0.5 * ow or nw > 1.8 * ow + 5:
         return None
     # numbers (results, years, units) must be kept
-    nums_o = set(re.findall(r"\d+(?:\.\d+)?%?", CITE.sub(" ", original)))
-    nums_n = set(re.findall(r"\d+(?:\.\d+)?%?", new))
+    nums_o = set(re.findall(r"(?<![\d])\.?\d+(?:\.\d+)?%?", CITE.sub(" ", original)))
+    nums_n = set(re.findall(r"(?<![\d])\.?\d+(?:\.\d+)?%?", new))
     if not nums_o <= nums_n:
         return None
     if lost_terms(original, new, glossary):                # technical wording must survive unchanged
@@ -747,6 +1036,8 @@ def basic_grammar(s):
     t = re.sub(r"\s+([,.;:!?%)\]])", r"\1", t)            # space before punctuation
     t = re.sub(r"([(\[])\s+", r"\1", t)
     t = re.sub(r",\s*,+", ",", t)
+    t = re.sub(r"\b(and|or|but|the|of|to|in)\s*,\s*\1\b", r"\1", t, flags=re.I)    # "and, and" -> "and"
+    t = re.sub(rf"^({_SENT_ADV})\s+(?=[a-z])", r"\1, ", t)                     # "However the" -> "However, the"
     t = re.sub(r"([.!?])\1+", r"\1", t)
     t = _outside_cites(t, lambda x: re.sub(r"\b(\w{2,})\s+\1\b", r"\1", x, flags=re.I))   # "the the", "data data"
     t = _outside_cites(t, lambda x: re.sub(r"\b([a-z]{3,})\s+(and|or)\s+\1\b", r"\1", x))  # "informed and informed"
@@ -796,11 +1087,21 @@ def strict_check(cur, cand, glossary, ref=None):
     new = match_ending(cur, basic_grammar(normalize_typography(cur, cand.strip())))
     new = re.sub(r'([.!?])\s*["”“]{2,}', r"\1”", new)                 # .”” -> .”
     new = re.sub(r'["“”]{2,}', "”", new)
+    new = restore_commas(cur, new)
     if new.strip() == cur.strip():
         return None, "unchanged"
     ow, nw = len(cur.split()), len(new.split())
     if nw < 0.55 * ow or nw > 1.6 * ow + 4:
         return None, "length"
+    why = integrity_issue(cur, new)
+    if why:
+        return None, why
+    if _symbols(new) != _symbols(cur):
+        return None, "relation sign changed"
+    if _new_acronyms(cur, new):
+        return None, "acronym added"
+    if _brackets_ok(cur) and not _brackets_ok(new):
+        return None, "unbalanced bracket"
     for base in filter(None, (cur, ref)):
         why = invented(base, new)
         if why:
@@ -808,8 +1109,8 @@ def strict_check(cur, cand, glossary, ref=None):
     why = quality_issue(cur, new) or symbol_issue(cur, new)
     if why:
         return None, why
-    nums_c = set(re.findall(r"\d+(?:\.\d+)?%?", CITE.sub(" ", cur)))
-    if not nums_c <= set(re.findall(r"\d+(?:\.\d+)?%?", new)):
+    nums_c = set(re.findall(r"(?<![\d])\.?\d+(?:\.\d+)?%?", CITE.sub(" ", cur)))
+    if not nums_c <= set(re.findall(r"(?<![\d])\.?\d+(?:\.\d+)?%?", new)):
         return None, "number lost"
     cc, nc = _cites(cur), _cites(new)
     if len(nc) > len(cc) or any(not any(difflib.SequenceMatcher(None, c, n).ratio() >= 0.8 for n in nc) for c in cc):
@@ -834,6 +1135,10 @@ def _near_repeats(t):
     out = set()
     for i, x in enumerate(w):
         if len(x) >= 5 and x in w[i + 1:i + 5]:
+            j = w.index(x, i + 1)
+            a, b = (w[i - 1] if i else ""), w[j - 1]
+            if a != b and len(a) >= 3 and len(b) >= 3 and a not in _STOP and b not in _STOP:
+                continue                                  # contrast, not a repeat: "female travellers than male travellers"
             out.add(x)
     return out