Algorex commited on
Commit
70e6e3c
·
verified ·
1 Parent(s): 5af9018

Card: memorisation caveat stated as a possibility, not a conclusion

Browse files
Files changed (1) hide show
  1. README.md +2 -1
README.md CHANGED
@@ -86,7 +86,8 @@ gain was a label-convention bias that did not carry over, not a better reading o
86
  - **Not a pristine test.** The FinBen FPB test split had been looked at in an earlier experiment by the same author
87
  before this study was pre-registered. The study discloses this; treat the test number as a confirmatory re-use.
88
  - **The base model has seen the test sentences.** Its instruction-tuning data lists all Financial PhraseBank sentences,
89
- of which the FinBen test split is a subset. Both numbers above are therefore partly memorisation.
 
90
  - **Scope.** English financial news sentences in Financial PhraseBank's style. Anything else is a distribution shift.
91
 
92
  ## Prompt format
 
86
  - **Not a pristine test.** The FinBen FPB test split had been looked at in an earlier experiment by the same author
87
  before this study was pre-registered. The study discloses this; treat the test number as a confirmatory re-use.
88
  - **The base model has seen the test sentences.** Its instruction-tuning data lists all Financial PhraseBank sentences,
89
+ of which the FinBen test split is a subset. Both numbers above may therefore include memorisation of the test
90
+ sentences rather than pure generalisation; the size of that effect is not measured here.
91
  - **Scope.** English financial news sentences in Financial PhraseBank's style. Anything else is a distribution shift.
92
 
93
  ## Prompt format