robustnesslab-ai / LIMITATIONS.md
Hugdrigo's picture
Upload 33 files
f4c6961 verified
|
Raw History Blame Contribute Delete
1.4 kB

Scientific and technical limitations

  • Synthetic examples are not evidence of generalization to authentic academic abstracts, policy texts or student writing.
  • Typo, synonym and appended-context edits are deterministic heuristics; semantic label invariance was not verified by human annotators.
  • A spell-change model flip can be informative, but may not constitute an actual error in naturally occurring writing.
  • Destructive truncation can change a statement's rhetorical function; no accuracy is reported for it.
  • Dictionary substitution has low row-change coverage on some splits; a deceptively high subset score is not an overall performance improvement.
  • Conditional subset scores compare on matched changed IDs. They cannot be substituted for full-sample accuracy.
  • The fixed examples are small, particularly cross-language subgroups; bootstrap resampling omits training and annotation uncertainty.
  • Naive Bayes posteriors are not proven calibrated. Similarity margins are not probabilities.
  • Research release gates are arbitrary adjustable checklist thresholds, NOT safety, compliance, fairness or deployment certification.
  • No real-time monitoring, real users, neural models, learned adversarial generator or external validation is implemented.
  • Models are transparent pedagogical lexical baselines; they are not SOTA, and no production deployment is recommended.