Text Generation
English
  • This is the pre-alpha version of an upcoming model so no benchmarks and Hugging Face Transformer writeups are included.
  • Refer out.txt for training output logs.

The model contains ~2.3M parameters, ~2M non-embedding parameters ~262K recollection embedding parameters

Trained on a mixture of 500M finePDFs tokens, 300M DCLM-baseline tokens and 200M FineWeb-Edu tokens, totalling to 1B total tokens. This mixture was chosen referencing the following article https://huggingface.co/blog/codelion/optimal-dataset-mixing

It beats codelions GPT-2-70M model on training loss, where GPT-2-70M had the final validation loss of ~2.9 and Pierre had ~2.3

Model Architecture

input -> embedding -> MultiTextureAttention (CDA) -> Recollection Mechanism -> SwiGLU FFN -> unembedding (tied) -> output

Recollection Mechanism

In simpliest terms, recollection is attention but with a slight the key matrices beings learned via backprop and value matrices being an embedding table (learned via backprop as well) that stores knowledge.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train Srijan-Srivastava/Pierre

Collection including Srijan-Srivastava/Pierre