modelguard-observatory / LIMITATIONS.md
Hugdrigo's picture
Upload 28 files
85920d3 verified
|
Raw History Blame Contribute Delete
1.23 kB

Limitations

  1. Both bundled evaluation groups are synthetic-curated, not time-series observations; the product name does not imply live telemetry.
  2. Data drift here is descriptive shift between two designed evaluation splits. There is no evidence of model drift over calendar time.
  3. Performance numbers on a small, synthetic task do not generalize automatically to new scientific papers, languages, genres, or institutions.
  4. Bootstrap percentiles assume example resampling and do not resolve annotation dependence, distribution shift, or real-world validation gaps.
  5. PSI and JSD depend on chosen bins and categories; PSI thresholds are heuristics.
  6. No protected-group labels are available, so fairness certification is impossible.
  7. Imported evaluation data is not validated for correctness of human labels or provenance. The user is responsible for rights and permitted use.
  8. Browser-based code is inspectable and modifiable by any user. Release gate overrides only change display criteria, not data quality.
  9. No active monitoring, alerting, automated retraining, model hosting, or remote persistent storage is implemented.
  10. This package does not contain journal PDFs, real student papers, or private data.