Card: point to the paper and configs for training settings; amplification is not part of COUNT
Browse files
README.md
CHANGED
|
@@ -19,8 +19,12 @@ Training-Free Adaptation** ([paper](https://arxiv.org/abs/XXXX.XXXXX), [code](ht
|
|
| 19 |
COUNT adds an n-gram embedding to a standard transformer without adding learned parameters. For each
|
| 20 |
position it looks up the most frequent next tokens of the longest matching n-grams in a table of corpus
|
| 21 |
counts held in CPU RAM, averages those tokens' rows of the model's own output head (weighted by their
|
| 22 |
-
count scores), and adds the result to the token embedding.
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
## Checkpoints
|
| 26 |
|
|
@@ -41,8 +45,7 @@ weights, optimizer state and trainer state.
|
|
| 41 |
|
| 42 |
Paths follow the code's ladder layout, `model-ladders/count-<size>/<SIZE>/step<N>/`, so pointing
|
| 43 |
`OLMO_ROOT_DIR` at a download lets the code find them. Each step directory has a `release_args.json`
|
| 44 |
-
with the ladder settings it was trained with
|
| 45 |
-
(`--count-k 8`) and averages over orders 1–6 (`--count-max-order 6 --count-num-orders 6`).
|
| 46 |
|
| 47 |
The tables are in [count-ngram-tables](https://huggingface.co/datasets/IanMagnusson/count-ngram-tables).
|
| 48 |
A checkpoint can be evaluated with its training table or with any other table there, including the
|
|
|
|
| 19 |
COUNT adds an n-gram embedding to a standard transformer without adding learned parameters. For each
|
| 20 |
position it looks up the most frequent next tokens of the longest matching n-grams in a table of corpus
|
| 21 |
counts held in CPU RAM, averages those tokens' rows of the model's own output head (weighted by their
|
| 22 |
+
count scores), and adds the result to the token embedding.
|
| 23 |
+
|
| 24 |
+
These checkpoints were also trained with other settings that are not part of COUNT itself, such as
|
| 25 |
+
embedding amplification. For the full configuration of each model, see the paper's appendix, the
|
| 26 |
+
`config.json` and `release_args.json` in each step directory, and the ladder configuration in the
|
| 27 |
+
[code](https://github.com/IanMagnusson/COUNT).
|
| 28 |
|
| 29 |
## Checkpoints
|
| 30 |
|
|
|
|
| 45 |
|
| 46 |
Paths follow the code's ladder layout, `model-ladders/count-<size>/<SIZE>/step<N>/`, so pointing
|
| 47 |
`OLMO_ROOT_DIR` at a download lets the code find them. Each step directory has a `release_args.json`
|
| 48 |
+
with the ladder settings it was trained with.
|
|
|
|
| 49 |
|
| 50 |
The tables are in [count-ngram-tables](https://huggingface.co/datasets/IanMagnusson/count-ngram-tables).
|
| 51 |
A checkpoint can be evaluated with its training table or with any other table there, including the
|