Densy-330M: 15.0 Billion Token Foundation Model (Base)
⚡ Overview
Densy-330M is an open-source, pure dense causal transformer foundation model trained from scratch across 15.00 Billion tokens on a 4× NVIDIA H100 80GB SXM5 NVLink cluster.
This repository contains the full suite of pretraining checkpoints from Step 500 through the final completion at Step 3576 (15.0B tokens).
Model Architecture
- Parameters: 328,508,928 (328.5M)
- Architecture: 24 Pre-LayerNorm Transformer layers
- Hidden Dimension ($d_{\text{model}}$): 768
- Attention Heads: 12 heads (64-dim per head) with RoPE and SDPA
- MLP Dimension: 3,072 ($4 \times d_{\text{model}}$) with GeLU
- Context Length: 4,096 tokens
- Vocabulary Size: 206,464 tokens (CRAYON Turbo)
- Weight Tying: Input embedding tied to LM output head
Checkpoints in this Repository
step_500/(2.10B tokens)step_1000/(4.19B tokens)step_1500/(6.29B tokens)step_2000/(8.39B tokens)step_2500/(10.49B tokens)step_3000/(12.58B tokens)step_3500/(14.68B tokens)final/(15.00B tokens)
Downstream Aligned Models
- SFT Model: Phase-Technologies/densy-330m-sft
- Instruct/Chat Model (SFT + DPO): Phase-Technologies/Densy-330M-Chat
🚀 Loading Checkpoints
import torch
from huggingface_hub import hf_hub_download
# Download final checkpoint
weights_path = hf_hub_download(repo_id="Phase-Technologies/densy-330m-pretrain", filename="final/pytorch_model.bin")
config_path = hf_hub_download(repo_id="Phase-Technologies/densy-330m-pretrain", filename="final/model_config.json")
a note from the creator.....
honestly when we set out to build densy we wanted to prove that pure, classic dense transformer architecture still has unmatched elegance and power when pushed to its limits..... after wrestling with complex mixture-of-experts routing and hybrid state space layers in our previous models, densy was our return to first principles: no gate networks, no expert balancing losses, no quadratic memory tricks—just 24 deep, pure dense layers weight-tied to a massive 206k vocabulary running wide open on 4x nvidia h100 sxm5 gpus at over 580,000 tokens every single second.....
watching the loss drop from 12 down across 15 billion tokens in runpod's datacenter was an unbelievable feeling..... training independent frontier-style models with limited personal budgets is terrifying and exhilarating all at once..... every single dollar counted, every step mattered, and when little bugs cropped up in our loss shifts or dataloaders we had to diagnose them in real time on live silicon with the clock ticking.....
huge love and endless gratitude to the runpod team for building the infrastructure that lets independent developers spin up multi-h100 nodes with a single click and actually train real models from scratch..... this journey has been exhausting, beautiful, and deeply humbling..... to everyone following along, experimenting with our models, or building their own architectures in bedrooms and dorms: never let compute monopolies convince you that you can't train from scratch..... i hope you enjoy talking with densy..... thank you for believing in this crazy dream.....
— soham pal (phase technologies)