| ## Jam-Contextsum |
|
|
| Jam-Contextsum is a GPT2-like model finetuned to generate summary on why the method exists. |
|
|
| ## Jam-Contextsum Training Details |
|
|
| - ckpt_pretrain is the file that we use to finetune the model for generating the summary on why the method exists |
| - Our [GitHub repo](https://github.com/apcl-research/jam-contextsum) contains the code for reproduction using the same [data](https://huggingface.co/datasets/apcl/jam_contextsum). |
| |
| |
| ## ckpt_pretrain.pt |
| | Hyperparameter | Description | Value | |
| | ----------- | ----------- |------------| |
| |e | embedding dimensions | 512 | |
| |L | number of layers | 4 | |
| |h | attention heads | 4 | |
| |c | block size / context length | 1,024 | |
| |b | batch size | 4 | |
| |a | accumulation steps | 32 | |
| |d | dropout | 0.20 | |
| |r | learning rate | 3e-5 | |
| |y | iterations | 1e-5 | |
| |iter | number of iterations after pretraing | 137,900 | |
