trilingua / config.json
aladewealth's picture
Upload Trilingua weights
6896879 verified
Raw History Blame Contribute Delete
271 Bytes
{
"vocab_size": 32768,
"max_seq_len": 1024,
"d_model": 640,
"n_layer": 10,
"n_head": 10,
"n_kv_head": 10,
"ffn_hidden": 1728,
"multiple_of": 64,
"rope_theta": 10000.0,
"norm_eps": 1e-05,
"dropout": 0.0,
"tie_embeddings": true,
"init_std": 0.02
}