CodeSymb-1B-MI325X / README.md
NiftyOracleAI's picture
Update README.md
6041e5b verified
|
Raw History Blame Contribute Delete
7.22 kB
---
language:
- en
tags:
- code
- pretraining
- from-scratch
- amd
- rocm
- mi325x
- supermicro
pipeline_tag: text-generation
license: apache-2.0
---
# CodeSymb-1B (MI325X pretraining run)
**CodeSymb-1B is a 1.18B-parameter code model trained from scratch by [NiftyOracle AI](https://huggingface.co/NiftyOracleAI) on a Supermicro H14 8U system with 8× AMD Instinct MI325X, through the Supermicro JumpStart program.**
> **Status: work in progress.** This repository holds a **base (pretrained) checkpoint at 151.28B tokens** of a planned 300B, plus the prepared data for the next stages. It is **not instruction-tuned and not yet evaluated**. Do not use it in production.
![Run status](CodeSymb_MI325X_status.png)
## At a glance
| | |
| --- | --- |
| Parameters | 1.18B (decoder-only transformer) |
| Tokens trained | 151.28B of a planned 300B |
| Hardware | Supermicro H14 8U · 8× AMD Instinct MI325X (256 GB HBM3e each) · 2× AMD EPYC 9575F |
| Software | Ubuntu 24.04 · ROCm/HIP 6.3 · PyTorch 2.9.1+rocm6.3 |
| Throughput | 488–508K tokens/s (median by phase) · 34.6% median MFU, 35.5% best |
| Training loss | 5.33 at step 20 → 1.065 at 151.28B |
| Time | 89.7 h of training inside a ~111 h window (Nov 16–20, 2026) |
Full benchmark report (method, throughput by phase, restarts, data pipeline): [CodeSymb_MI325X_Benchmark_Report.pdf](CodeSymb_MI325X_Benchmark_Report.pdf)
## Architecture
| Setting | Value |
| --- | --- |
| Layers | 16 |
| Hidden size | 2,048 |
| Feed-forward | SwiGLU, 8,192 |
| Attention | 16 query heads, 4 key/value heads (grouped-query attention) |
| Normalization | RMSNorm (pre-norm) |
| Positions | RoPE, base 500,000 |
| Vocabulary | 50,000 SentencePiece tokens (padded to 50,048); input and output embeddings not tied |
| Context | 2,048 tokens in pretraining (4,096 planned for fine-tuning) |
## Training
- **Objective:** next-token prediction, cross-entropy.
- **Batch:** 48 sequences × 2,048 tokens per GPU, 8 GPUs, no gradient accumulation (786,432 tokens per step).
- **Precision:** BF16 autocast with FP32 master weights; fused AdamW.
- **Parallelism:** PyTorch DDP and torch.compile; GEMMs tuned with PyTorch TunableOp.
- **Data order:** each training window was seen once, except about 1B tokens repeated after two restarts.
### Data
| Phase | Tokens | Data |
| --- | --- | --- |
| 0 → 130.7B | 130.7B | Pretokenized mix of source code and web text |
| 130.7B → 151.28B (continuing) | 20.6B so far | 189.5B **never-seen** tokens, deduplicated against everything trained before, with **newlines and indentation kept** |
The pretraining mix draws on public datasets including The Stack v2 (17 programming languages), StarCoder GitHub issues and commits, FineWeb, FineWeb-Edu and OpenWebMath. About 64% of the formatted pool is code.
**Whitespace codec.** The tokenizer normalizes away newlines, tabs and space runs, which breaks code such as Python. From 130.7B tokens on, 7 token IDs that never occurred in the data carry fixed whitespace meanings (newline, tab, 1/2/4/8/16 spaces). The vocabulary is unchanged. Because the data changed at 130.7B, loss before and after that point is not directly comparable.
## Files
| Path | Contents |
| --- | --- |
| `checkpoints/codesymb1b_pre_010B.pt` … `_150B.pt` | Weights-only milestones every 10B tokens |
| `checkpoints/pretrain_latest.pt` | Full training state at 151.28B (weights, optimizer, data position) |
| `pretok_ws_clean/` | 189.5B-token formatted, deduplicated pretraining pool (uint16 shards + manifest) |
| `sft_cache_v4/` | 22.48B-token supervised fine-tuning set: 29.35M unique samples, 4,096 context |
| `sft_cache_20b/` | Earlier 20.5B-token SFT set (2,048 context, without the whitespace codec) |
| `backup/state_final.tgz` | Training code, launch scripts, logs and kernel-tuning results |
| `CodeSymb_MI325X_status.png`, `CodeSymb_MI325X_Benchmark_Report.pdf` | Status diagram and benchmark report |
### SFT set v4
Built from 26 public sources: code instruction data, reasoning traces (in a `<think>` block), math, general chat, and document-grounded question answering with citations (HotpotQA, SQuAD v2, MS MARCO), including examples that answer "the documents do not contain this". At most one sample is kept per prompt, and samples overlapping HumanEval or MBPP (13-word sequences) were removed, so those benchmarks stay clean.
| Category | Tokens | Share |
| --- | --- | --- |
| General chat | 8.65B | 38% |
| Code | 5.95B | 26% |
| Code with reasoning | 2.32B | 10% |
| General reasoning | 2.15B | 10% |
| Math with reasoning | 1.76B | 8% |
| Grounded QA (RAG) | 1.29B | 6% |
| Math | 0.37B | 2% |
## How to load
The checkpoints use the project's own PyTorch format (not `transformers`). The model code ships in `backup/state_final.tgz`. This example follows that code but has not yet been run outside the training environment; please open a discussion if it fails.
```python
import os, sys, tarfile, torch
from huggingface_hub import hf_hub_download
REPO = "NiftyOracleAI/CodeSymb-1B-MI325X"
tarfile.open(hf_hub_download(REPO, "backup/state_final.tgz")).extractall("codesymb_src")
sys.path.insert(0, "codesymb_src/codesymb")
os.environ["CODESYMB_WS_CODEC"] = "1" # keep newlines and indentation
import codesymb_common as C
model, meta, _ = C.build_from_checkpoint(hf_hub_download(REPO, "checkpoints/codesymb1b_pre_150B.pt"))
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()
tok = C.load_tokenizer() # SentencePiece model + whitespace codec
x = torch.tensor([tok.encode("def fibonacci(n):\n ")], device=device)
with torch.no_grad(), C.amp():
for _ in range(64): # greedy decoding, no KV cache
nxt = model(x)[0, -1].argmax()
x = torch.cat([x, nxt.view(1, 1)], dim=1)
print(tok.decode(x[0].tolist()))
```
## Limitations
- **Base model, mid-training.** It completes text; it does not follow instructions or chat.
- **Not evaluated.** No benchmark scores yet. HumanEval and MBPP results will be added after fine-tuning.
- **Small model.** At 1.18B parameters it will make mistakes, invent APIs, and write insecure code. Review everything it generates.
- **Data.** Trained on public web and code data, which contain errors, bias and licensed code. Each source dataset carries its own license and terms.
## Roadmap
1. Pretraining to 300B tokens (~3.5 days of 8× MI325X compute remaining).
2. Supervised fine-tuning on `sft_cache_v4` (22.48B tokens, 4,096 context).
3. Alignment with GRPO (verified-correctness rewards) and DPO, aiming for honest answers over agreeable ones.
4. Evaluation on HumanEval and MBPP, and a public technical report.
## Acknowledgements
Compute was provided by **Supermicro** through the **JumpStart Remote Access Program** on 8× **AMD Instinct MI325X**. Thank you to the JumpStart team for the access and support. Progress updates are posted in the [AMD Developer Community](https://devcommunity.amd.com/t/training-an-llm-from-scratch-on-8x-amd-instinct-mi325x-progress-update/1423).
## Contact
Built by Dev Malkar, founder of CodeSymb AI (NiftyOracle AI). Questions and feedback: open a discussion in the Community tab.