metadata
license: mit
language:
- en
tags:
- text-generation
- gpt
- from-scratch
- conversational
Cortex 2 chat preview
A small GPT model for conversational chat, trained completely from scratch (custom PyTorch architecture, no transformers library involved).
⚠️ Note: this is a preview release and may communicate worse than Cortex 1 Chat.
- Parameters: 346m
- Weights format: fp16
- Architecture: Decoder-only Transformer (custom implementation)
- Training: from scratch, no pretrained weights used
⚠️ Note: the context window is only 256 tokens, so the model will forget earlier parts of the conversation quickly.
How to run
- Install dependencies:
pip install -r requirements.txt
- Run:
python chat.py
Limitations
This is a small model, so it may lose coherence, mix up facts, or cut off mid-thought — expected behavior for this parameter count.