---
license: mit
language:
- en
pipeline_tag: text-generation
tags:
- GPT-4
- GPT
---
![6cex5h9ls0ug1](https://cdn-uploads.huggingface.co/production/uploads/64b7618e2f5a966b972e9978/c0k3vYt-5cq0dIVTdKNTi.png)
# GPT-Compact
GPT-Compact is a text generation model meant to answer the question: "How strong of a model can Google Colab create?"

### Model Description
GPT-Compact has 16 heads, 24 layers, and 1280 embeddings, similar to GPT2-Large, with a context window of 16,384 tokens. This maximized GPU VRAM while balancing size and functionality.
It was trained on a diverse corpus of over 17 billion tokens.

### Model Architecture
|                                |     |
|---------------------------------|----:|
|Layers                           |24|
|Heads                            |16|
|Embeddings                       |1280|
|Context Window                   |16384 tokens|
|Tokenizer                        |cl100k_base|