---
license: mit
language:
- en
tags:
- text-generation
- gpt
- from-scratch
- conversational
---

# Cortex 1 chat


A small GPT model for conversational chat, trained completely from scratch (custom PyTorch architecture, no `transformers` library involved).

- **Parameters**: 130m
- **Weights format**: fp16
- **Architecture**: Decoder-only Transformer (custom implementation)
- **Training**: from scratch, no pretrained weights used

> ⚠️ **Note:** the context window is only **128 tokens**, so the model will forget earlier parts of the conversation quickly.

## How to run

2. Install dependencies:

pip install -r requirements.txt


3. Run:

python chat.py


## Limitations

This is a small model, so it may lose coherence, mix up facts, or cut off mid-thought — expected behavior for this parameter count.