English
n-gram
olmo-core
language-modeling
IanMagnusson commited on
Commit
8a081fa
·
verified ·
1 Parent(s): 88323b1

Card: point to the paper and configs for training settings; amplification is not part of COUNT

Browse files
Files changed (1) hide show
  1. README.md +7 -4
README.md CHANGED
@@ -19,8 +19,12 @@ Training-Free Adaptation** ([paper](https://arxiv.org/abs/XXXX.XXXXX), [code](ht
19
  COUNT adds an n-gram embedding to a standard transformer without adding learned parameters. For each
20
  position it looks up the most frequent next tokens of the longest matching n-grams in a table of corpus
21
  counts held in CPU RAM, averages those tokens' rows of the model's own output head (weighted by their
22
- count scores), and adds the result to the token embedding. A LayerNorm and a √d scale follow
23
- (embedding amplification), whose two small parameter vectors are the only addition to the backbone.
 
 
 
 
24
 
25
  ## Checkpoints
26
 
@@ -41,8 +45,7 @@ weights, optimizer state and trainer state.
41
 
42
  Paths follow the code's ladder layout, `model-ladders/count-<size>/<SIZE>/step<N>/`, so pointing
43
  `OLMO_ROOT_DIR` at a download lets the code find them. Each step directory has a `release_args.json`
44
- with the ladder settings it was trained with: COUNT reads 8 of the table's 32 candidates per order
45
- (`--count-k 8`) and averages over orders 1–6 (`--count-max-order 6 --count-num-orders 6`).
46
 
47
  The tables are in [count-ngram-tables](https://huggingface.co/datasets/IanMagnusson/count-ngram-tables).
48
  A checkpoint can be evaluated with its training table or with any other table there, including the
 
19
  COUNT adds an n-gram embedding to a standard transformer without adding learned parameters. For each
20
  position it looks up the most frequent next tokens of the longest matching n-grams in a table of corpus
21
  counts held in CPU RAM, averages those tokens' rows of the model's own output head (weighted by their
22
+ count scores), and adds the result to the token embedding.
23
+
24
+ These checkpoints were also trained with other settings that are not part of COUNT itself, such as
25
+ embedding amplification. For the full configuration of each model, see the paper's appendix, the
26
+ `config.json` and `release_args.json` in each step directory, and the ladder configuration in the
27
+ [code](https://github.com/IanMagnusson/COUNT).
28
 
29
  ## Checkpoints
30
 
 
45
 
46
  Paths follow the code's ladder layout, `model-ladders/count-<size>/<SIZE>/step<N>/`, so pointing
47
  `OLMO_ROOT_DIR` at a download lets the code find them. Each step directory has a `release_args.json`
48
+ with the ladder settings it was trained with.
 
49
 
50
  The tables are in [count-ngram-tables](https://huggingface.co/datasets/IanMagnusson/count-ngram-tables).
51
  A checkpoint can be evaluated with its training table or with any other table there, including the