Hugging Face's logo

Digilidaahz
/
Freedream_1B_BaseModel

Text Generation
English
Freedream-1B
Pytorch
text-generation-inference
base-model
Model card Files
xet
 Community
1
Digilidaahz commited on
Commit
64052eb
·
verified ·
1 Parent(s): 6533279

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -55,9 +55,9 @@ Freedream is a ~1B parameter decoder-only Transformer language model trained fro
55
 
56
  ## Training:
57
  - Hardware: 1x RTX 5080 (16GB) + WSL2 (Ubuntu 24.04)
58
- - Precision: BF16
59
  - Optimizer: 8-bit AdamW
60
- - Gradient checkpointing: ON
61
  - Batch size: 1
62
  - Gradient accumulation: 16
63
  - Effective tokens / step: 65,536
 
55
 
56
  ## Training:
57
  - Hardware: 1x RTX 5080 (16GB) + WSL2 (Ubuntu 24.04)
58
+ - Precision: FP8 (except the 7B-tokens-seen version which is trained on bf16)
59
  - Optimizer: 8-bit AdamW
60
+ - Gradient checkpointing: OFF(except the 7B-tokens-seen version)
61
  - Batch size: 1
62
  - Gradient accumulation: 16
63
  - Effective tokens / step: 65,536