README.md

metadata

license: apache-2.0
library_name: pytorch
tags:
  - rwkv
  - rwkv-7
  - language-model
  - pretraining

Winner-RWKV-7-285M-20260601

RWKV-7 checkpoint trained in Jellyfish042/GatedDeltaNet-2.

• Architecture: RWKV-7 TimeMix, rwkv7_time_1024x12
• Parameters printed by the training script: 219,611,136
• Pretraining data: globally shuffled FineWeb-Edu packed 100B-token run
• Sequence length: 4096
• Global batch size: 1024
• Seed: 3407
• Checkpoint file: pytorch_model.bin

This is a project checkpoint from pretrain.py, not a Hugging Face Transformers checkpoint.

lm-eval-harness snapshot

Task	Metric	Value
lambada_openai	acc	0.3728
lambada_openai	perplexity	25.6649
wikitext	word_perplexity	24.0226
wikitext	byte_perplexity	1.8121
wikitext	bits_per_byte	0.8577