I want to research involving pre-training. do not delete
Moreover, if this were a complete fake, most people would likely either turn a blind eye or lose interest, resulting in very little engagement. This suggests that these model weights might contain something inconvenient for those currently resisting them so fiercely. In my view, this "imperfection" simply means these are model weights that were discarded because they were incomplete Pre-training though, depending on how they were trained, they might have had the potential to approach the performance of Claude (setting aside the issue of cost). The real question is at what stage they were abandoned. Unless someone performs a small-scale training run using the released dataset and examines the loss, I suspect that claims—seen elsewhere—that these are merely "random weights" are likely coming from suspicious bots. By the way, are there any Anthropic employees present here on Hugging Face?
Its token embedding layer is fine; it's a genuinely working embedding layer. But everything else is completely fake.
In short, weight training is worth doing.