Transformer-from-scratch / C1_Base_Model_config.json
Bejaine's picture
Upload folder using huggingface_hub
bb06882 verified
Raw History Blame Contribute Delete
342 Bytes
{
"run_name": "C1_Base_Model",
"d_model": 256,
"num_heads": 8,
"num_layers": 4,
"d_ff": 1024,
"dropout": 0.1,
"max_seq_len": 64,
"batch_size": 64,
"learning_rate": 0.0005,
"epochs": 100,
"tokenization": "subword",
"pos_type": "sinusoidal",
"attn_type": "mha",
"norm_type": "layernorm"
}