Buckets:
| license: apache-2.0 | |
| library_name: pytorch | |
| tags: | |
| - rwkv | |
| - rwkv-7 | |
| - language-model | |
| - pretraining | |
| # Winner-RWKV-7-285M-20260601 | |
| RWKV-7 checkpoint trained in `Jellyfish042/GatedDeltaNet-2`. | |
| - Architecture: RWKV-7 TimeMix, `rwkv7_time_1024x12` | |
| - Parameters printed by the training script: `219,611,136` | |
| - Pretraining data: globally shuffled FineWeb-Edu packed 100B-token run | |
| - Sequence length: `4096` | |
| - Global batch size: `1024` | |
| - Seed: `3407` | |
| - Checkpoint file: `pytorch_model.bin` | |
| This is a project checkpoint from `pretrain.py`, not a Hugging Face Transformers checkpoint. | |
| ## lm-eval-harness snapshot | |
| | Task | Metric | Value | | |
| |---|---:|---:| | |
| | `lambada_openai` | acc | `0.3728` | | |
| | `lambada_openai` | perplexity | `25.6649` | | |
| | `wikitext` | word_perplexity | `24.0226` | | |
| | `wikitext` | byte_perplexity | `1.8121` | | |
| | `wikitext` | bits_per_byte | `0.8577` | | |
Xet Storage Details
- Size:
- 878 Bytes
- Xet hash:
- 40d9a407271be2d0790e4d844d299ba0458497a07e71bb144a7c875250068fb1
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.