YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
MoE pretraining run
This repository is updated by a Kaggle notebook that resumes from the last saved state.
Status
- run_index:
0 - source_cursor:
0 - global_steps:
0 - completed_runs:
0 - finished_window:
False
Window policy
- rows selected this run:
10000 - raw rows scanned this run:
25000 - after each completed window, the notebook saves the full state and continues from the next raw cursor on the next trigger
Model shape
- layers:
6 - d_model:
512 - heads:
8 - experts:
100 - active experts per token:
12 - expert hidden size:
768
Recent stats
- phase:
bootstrap
Notes
- tokenizer: custom byte-level BPE
- resume: optimizer, scheduler, RNG, config, tokenizer, model weights, and cursor state are all saved
- Downloads last month
- 23
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support