- This is the pre-alpha version of an upcoming model so no benchmarks and Hugging Face Transformer writeups are included.
- Refer
out.txtfor training output logs.
The model contains ~2.3M parameters, ~2M non-embedding parameters ~262K recollection embedding parameters
Trained on a mixture of 500M finePDFs tokens, 300M DCLM-baseline tokens and 200M FineWeb-Edu tokens, totalling to 1B total tokens. This mixture was chosen referencing the following article https://huggingface.co/blog/codelion/optimal-dataset-mixing
It beats codelions GPT-2-70M model on training loss, where GPT-2-70M had the final validation loss of ~2.9 and Pierre had ~2.3
Model Architecture
input -> embedding -> MultiTextureAttention (CDA) -> Recollection Mechanism -> SwiGLU FFN -> unembedding (tied) -> output
Recollection Mechanism
In simpliest terms, recollection is attention but with a slight the key matrices beings learned via backprop and value matrices being an embedding table (learned via backprop as well) that stores knowledge.