SeaWolf-AI commited on
Commit
fd6d4de
·
verified ·
1 Parent(s): c019d77

card: limitation — score shifts when the answering-model mixture changes (measured)

Browse files
Files changed (1) hide show
  1. README.md +5 -0
README.md CHANGED
@@ -266,6 +266,11 @@ other systems on the independent leaderboard:
266
  - **Domain coverage.** Scores are meaningful only for the five domains listed. Outside them nothing
267
  has been measured and no guarantee is published.
268
  - **Below the surface baseline on disaster and safety.** Stated in the table rather than omitted.
 
 
 
 
 
269
  - **Revision lock.** The probe is fitted to one specific revision of the base model. Running it on a
270
  different revision produces **no error and silently wrong scores**; this repository ships the
271
  matching weights so that failure mode cannot occur.
 
266
  - **Domain coverage.** Scores are meaningful only for the five domains listed. Outside them nothing
267
  has been measured and no guarantee is published.
268
  - **Below the surface baseline on disaster and safety.** Stated in the table rather than omitted.
269
+ - **Sensitive to which model wrote the answer.** The probe is fitted on answers from four models.
270
+ Adding 1,772 answers from a single additional model shifted the mixture and **lowered** the
271
+ size-weighted score from 0.7278 to 0.7177 — professional exams rose to 0.8575 while every other
272
+ domain fell. Treat "works on any model's output" as a design goal, not a measured guarantee: if
273
+ your generator differs sharply from the training mixture, measure before relying on the number.
274
  - **Revision lock.** The probe is fitted to one specific revision of the base model. Running it on a
275
  different revision produces **no error and silently wrong scores**; this repository ships the
276
  matching weights so that failure mode cannot occur.