File size: 818 Bytes
e417a74 cf2ae50 e66fd3a cf2ae50 e66fd3a be52d09 e66fd3a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | ---
license: mit
language:
- en
tags:
- text-generation
- gpt
- from-scratch
- conversational
---
# Cortex 1 chat
A small GPT model for conversational chat, trained completely from scratch (custom PyTorch architecture, no `transformers` library involved).
- **Parameters**: 130m
- **Weights format**: fp16
- **Architecture**: Decoder-only Transformer (custom implementation)
- **Training**: from scratch, no pretrained weights used
> ⚠️ **Note:** the context window is only **128 tokens**, so the model will forget earlier parts of the conversation quickly.
## How to run
2. Install dependencies:
pip install -r requirements.txt
3. Run:
python chat.py
## Limitations
This is a small model, so it may lose coherence, mix up facts, or cut off mid-thought — expected behavior for this parameter count. |