Higher-Context
#17
by amarosnithe - opened
Congratulations to the development team, this is genuinely life-changing and you're clearly making history. The only criticism/suggestion I personally have is that there should be a bigger context-length (512 feels too small, decent for most tasks but not enough for bigger data/documents). If anyone can do a Laya that has like, 32k of context minimum, then you're literally asking for World Domination with this shit. The amount of potential with it is limitless.
Multilingual model supports up to 8k
I might be wrong but I feel like 8 is smaller than 32.. I think.