File size: 933 Bytes
9161284 b280b1e 9161284 4524d85 165f0d0 b280b1e 165f0d0 cf992ab b280b1e 23b9393 9925904 1efd71d 23b9393 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | ---
license: mit
language:
- en
pipeline_tag: text-generation
tags:
- GPT-4
- GPT
---
# GPT-Compact
GPT-Compact is a text generation model meant to answer the question: "How strong of a model can Google Colab create?"
### Model Description
GPT-Compact has 16 heads, 24 layers, and 1280 embeddings, similar to GPT2-Large, with a context window of 16,384 tokens. This maximized GPU VRAM while balancing size and functionality.
It was trained on a diverse corpus of over 17 billion tokens.
### Model Architecture
| | |
|---------------------------------|----:|
|Layers |24|
|Heads |16|
|Embeddings |1280|
|Context Window |16384 tokens|
|Tokenizer |cl100k_base| |