O0503HMA20
This model is a fine-tuned version of allenai/OLMo-1B on an unknown dataset.
It achieves the following results on the evaluation set:
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 16
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine_with_restarts
- lr_scheduler_warmup_steps: 60
- num_epochs: 3
- mixed_precision_training: Native AMP
Training results
| Training Loss |
Epoch |
Step |
Validation Loss |
| 2.2681 |
0.09 |
10 |
0.1651 |
| 0.1639 |
0.18 |
20 |
0.1582 |
| 0.1515 |
0.27 |
30 |
0.1589 |
| 0.1564 |
0.36 |
40 |
0.1544 |
| 0.1508 |
0.45 |
50 |
0.1480 |
| 0.1538 |
0.54 |
60 |
0.1484 |
| 0.1505 |
0.63 |
70 |
0.1494 |
| 0.1498 |
0.73 |
80 |
0.1548 |
| 0.1473 |
0.82 |
90 |
0.1507 |
| 0.1488 |
0.91 |
100 |
0.1476 |
| 0.1528 |
1.0 |
110 |
0.1518 |
| 0.1462 |
1.09 |
120 |
0.1495 |
| 0.1478 |
1.18 |
130 |
0.1500 |
| 0.1479 |
1.27 |
140 |
0.1514 |
| 0.1489 |
1.36 |
150 |
0.1483 |
| 0.1382 |
1.45 |
160 |
0.1241 |
| 0.1083 |
1.54 |
170 |
0.0841 |
| 0.3503 |
1.63 |
180 |
0.0684 |
| 0.0812 |
1.72 |
190 |
0.0730 |
| 0.0715 |
1.81 |
200 |
0.0670 |
| 0.0568 |
1.9 |
210 |
0.0566 |
| 0.0434 |
1.99 |
220 |
0.0401 |
| 0.0365 |
2.08 |
230 |
0.0565 |
| 0.0375 |
2.18 |
240 |
0.0370 |
| 0.0271 |
2.27 |
250 |
0.0258 |
| 0.0252 |
2.36 |
260 |
0.0224 |
| 0.0217 |
2.45 |
270 |
0.0200 |
| 0.0141 |
2.54 |
280 |
0.0162 |
| 0.0216 |
2.63 |
290 |
0.0150 |
| 0.0187 |
2.72 |
300 |
0.0155 |
| 0.0154 |
2.81 |
310 |
0.0139 |
| 0.018 |
2.9 |
320 |
0.0145 |
| 0.0152 |
2.99 |
330 |
0.0140 |
Framework versions
- Transformers 4.36.0.dev0
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.14.0
