GPT-Compact

GPT-Compact is a text generation model meant to answer the question: "How strong of a model can Google Colab create?"

Model Description

GPT-Compact has 16 heads, 24 layers, and 1280 embeddings, similar to GPT2-Large, with a context window of 16,384 tokens. This maximized GPU VRAM while balancing size and functionality. It was trained on a diverse corpus of over 17 billion tokens.

Model Architecture

Layers	24
Heads	16
Embeddings	1280
Context Window	16384 tokens
Tokenizer	cl100k_base