ml-intern-explorers

/

nova-10b-dense

Like 0

Follow

ML intern explorers 5.95k

There isn't even enough credits currently to do it

ha ha no its not 20T

i made a naming mistake sorry

How much then?

its just gonna be 1-2T 20T is too much.

which context

it's in pretokenized chunks

2048 tokens.

its a bit short but i train in phases usally

bro you think you can train the next GPT 4 on 1.4T tokens?

This is for 1.4T tokens

in 2 days you could only:

8.6B tokens (8.8B with causal-aware attention).

Budget: 2 days × 3.2e15 FLOP/s ≈ 5.5e20 FLOPs, at ~6.5e10 FLOPs/token.

That's ~0.86 tokens/param — far below Chinchilla (20×). For a 2-day run on that node, a ~400M model at 8B tokens is much better matched.

no no! im not planning to use all. i only use a subset im well aware of the time constraints

hmm true

how many then?

i usually get data first so i dont have to worry about it later

If you have $316,800 then go and train on 1.4T tokens for 1 year

i plan the model after the dataa im planning a 3-10B ish parameter model ill test my pipeline first, then train in monthly shards

well i dont lol

your models are quite capable for their size i checked some out previously

I'd really reccomend like a smaller model on more tokens, then you could actually compete with other models. Thats what we do.

your profile says you have over 1900 TFLOPS of compute. Do you own 8 x H200s or is it cloud?

ok thanks for advice

@Bc-AI oops that was a error i once wanted to see how much with 8xH200 and forgot to remove, ty

@Banaxi-Tech can i join your beta testers organisation?

Sure!

@Banaxi-Tech @vovaRL You have inspired me to also try making a novel SLM. As my Larger one takes a looonng time to train i have decided to create a novel SLM combining elements from many sources. I will open source it too. Also @Banaxi-Tech I think you bananamind-2.1 unified concept is quite cool so i reutilised some concepts from it too :)

@Banaxi-Tech @vovaRL You have inspired me to also try making a novel SLM. As my Larger one takes a looonng time to train i have decided to create a novel SLM combining elements from many sources. I will open source it too. Also @Banaxi-Tech I think you bananamind-2.1 unified concept is quite cool so i reutilised some concepts from it too :)

🤗