| --- |
| license: mit |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - GPT-4 |
| - GPT |
| --- |
|  |
| # GPT-Compact |
| GPT-Compact is a text generation model meant to answer the question: "How strong of a model can Google Colab create?" |
|
|
| ### Model Description |
| GPT-Compact has 16 heads, 24 layers, and 1280 embeddings, similar to GPT2-Large, with a context window of 16,384 tokens. This maximized GPU VRAM while balancing size and functionality. |
| It was trained on a diverse corpus of over 17 billion tokens. |
|
|
| ### Model Architecture |
| | | | |
| |---------------------------------|----:| |
| |Layers |24| |
| |Heads |16| |
| |Embeddings |1280| |
| |Context Window |16384 tokens| |
| |Tokenizer |cl100k_base| |
