CodeSymb-1B-MI325X / README.md
NiftyOracleAI's picture
Update README.md
6041e5b verified
|
Raw History Blame Contribute Delete
7.22 kB
metadata
language:
  - en
tags:
  - code
  - pretraining
  - from-scratch
  - amd
  - rocm
  - mi325x
  - supermicro
pipeline_tag: text-generation
license: apache-2.0

CodeSymb-1B (MI325X pretraining run)

CodeSymb-1B is a 1.18B-parameter code model trained from scratch by NiftyOracle AI on a Supermicro H14 8U system with 8× AMD Instinct MI325X, through the Supermicro JumpStart program.

Status: work in progress. This repository holds a base (pretrained) checkpoint at 151.28B tokens of a planned 300B, plus the prepared data for the next stages. It is not instruction-tuned and not yet evaluated. Do not use it in production.

Run status

At a glance

Parameters 1.18B (decoder-only transformer)
Tokens trained 151.28B of a planned 300B
Hardware Supermicro H14 8U · 8× AMD Instinct MI325X (256 GB HBM3e each) · 2× AMD EPYC 9575F
Software Ubuntu 24.04 · ROCm/HIP 6.3 · PyTorch 2.9.1+rocm6.3
Throughput 488–508K tokens/s (median by phase) · 34.6% median MFU, 35.5% best
Training loss 5.33 at step 20 → 1.065 at 151.28B
Time 89.7 h of training inside a ~111 h window (Nov 16–20, 2026)

Full benchmark report (method, throughput by phase, restarts, data pipeline): CodeSymb_MI325X_Benchmark_Report.pdf

Architecture

Setting Value
Layers 16
Hidden size 2,048
Feed-forward SwiGLU, 8,192
Attention 16 query heads, 4 key/value heads (grouped-query attention)
Normalization RMSNorm (pre-norm)
Positions RoPE, base 500,000
Vocabulary 50,000 SentencePiece tokens (padded to 50,048); input and output embeddings not tied
Context 2,048 tokens in pretraining (4,096 planned for fine-tuning)

Training

  • Objective: next-token prediction, cross-entropy.
  • Batch: 48 sequences × 2,048 tokens per GPU, 8 GPUs, no gradient accumulation (786,432 tokens per step).
  • Precision: BF16 autocast with FP32 master weights; fused AdamW.
  • Parallelism: PyTorch DDP and torch.compile; GEMMs tuned with PyTorch TunableOp.
  • Data order: each training window was seen once, except about 1B tokens repeated after two restarts.

Data

Phase Tokens Data
0 → 130.7B 130.7B Pretokenized mix of source code and web text
130.7B → 151.28B (continuing) 20.6B so far 189.5B never-seen tokens, deduplicated against everything trained before, with newlines and indentation kept

The pretraining mix draws on public datasets including The Stack v2 (17 programming languages), StarCoder GitHub issues and commits, FineWeb, FineWeb-Edu and OpenWebMath. About 64% of the formatted pool is code.

Whitespace codec. The tokenizer normalizes away newlines, tabs and space runs, which breaks code such as Python. From 130.7B tokens on, 7 token IDs that never occurred in the data carry fixed whitespace meanings (newline, tab, 1/2/4/8/16 spaces). The vocabulary is unchanged. Because the data changed at 130.7B, loss before and after that point is not directly comparable.

Files

Path Contents
checkpoints/codesymb1b_pre_010B.pt … _150B.pt Weights-only milestones every 10B tokens
checkpoints/pretrain_latest.pt Full training state at 151.28B (weights, optimizer, data position)
pretok_ws_clean/ 189.5B-token formatted, deduplicated pretraining pool (uint16 shards + manifest)
sft_cache_v4/ 22.48B-token supervised fine-tuning set: 29.35M unique samples, 4,096 context
sft_cache_20b/ Earlier 20.5B-token SFT set (2,048 context, without the whitespace codec)
backup/state_final.tgz Training code, launch scripts, logs and kernel-tuning results
CodeSymb_MI325X_status.png, CodeSymb_MI325X_Benchmark_Report.pdf Status diagram and benchmark report

SFT set v4

Built from 26 public sources: code instruction data, reasoning traces (in a <think> block), math, general chat, and document-grounded question answering with citations (HotpotQA, SQuAD v2, MS MARCO), including examples that answer "the documents do not contain this". At most one sample is kept per prompt, and samples overlapping HumanEval or MBPP (13-word sequences) were removed, so those benchmarks stay clean.

Category Tokens Share
General chat 8.65B 38%
Code 5.95B 26%
Code with reasoning 2.32B 10%
General reasoning 2.15B 10%
Math with reasoning 1.76B 8%
Grounded QA (RAG) 1.29B 6%
Math 0.37B 2%

How to load

The checkpoints use the project's own PyTorch format (not transformers). The model code ships in backup/state_final.tgz. This example follows that code but has not yet been run outside the training environment; please open a discussion if it fails.

import os, sys, tarfile, torch
from huggingface_hub import hf_hub_download

REPO = "NiftyOracleAI/CodeSymb-1B-MI325X"
tarfile.open(hf_hub_download(REPO, "backup/state_final.tgz")).extractall("codesymb_src")
sys.path.insert(0, "codesymb_src/codesymb")
os.environ["CODESYMB_WS_CODEC"] = "1"          # keep newlines and indentation

import codesymb_common as C

model, meta, _ = C.build_from_checkpoint(hf_hub_download(REPO, "checkpoints/codesymb1b_pre_150B.pt"))
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()
tok = C.load_tokenizer()                       # SentencePiece model + whitespace codec

x = torch.tensor([tok.encode("def fibonacci(n):\n    ")], device=device)
with torch.no_grad(), C.amp():
    for _ in range(64):                        # greedy decoding, no KV cache
        nxt = model(x)[0, -1].argmax()
        x = torch.cat([x, nxt.view(1, 1)], dim=1)
print(tok.decode(x[0].tolist()))

Limitations

  • Base model, mid-training. It completes text; it does not follow instructions or chat.
  • Not evaluated. No benchmark scores yet. HumanEval and MBPP results will be added after fine-tuning.
  • Small model. At 1.18B parameters it will make mistakes, invent APIs, and write insecure code. Review everything it generates.
  • Data. Trained on public web and code data, which contain errors, bias and licensed code. Each source dataset carries its own license and terms.

Roadmap

  1. Pretraining to 300B tokens (~3.5 days of 8× MI325X compute remaining).
  2. Supervised fine-tuning on sft_cache_v4 (22.48B tokens, 4,096 context).
  3. Alignment with GRPO (verified-correctness rewards) and DPO, aiming for honest answers over agreeable ones.
  4. Evaluation on HumanEval and MBPP, and a public technical report.

Acknowledgements

Compute was provided by Supermicro through the JumpStart Remote Access Program on 8× AMD Instinct MI325X. Thank you to the JumpStart team for the access and support. Progress updates are posted in the AMD Developer Community.

Contact

Built by Dev Malkar, founder of CodeSymb AI (NiftyOracle AI). Questions and feedback: open a discussion in the Community tab.