ethanelkayam/scratch-780m-instruct-webllm

ScratchLM-780M-SFT (WebLLM, q4f16_1)

A 780M-parameter chat model trained from scratch as a personal learning project, packaged to run entirely in the browser with WebLLM 0.2.86 over WebGPU. This repository is a static file

Model

Architecture	decoder-only transformer that computes the stock Qwen3 graph (MLC qwen3)
Parameters	778,782,720 (tied input/output embedding)
Layers / width	40 layers, d_model 1280, SwiGLU 3584
Attention	grouped-query, 20 query heads / 5 KV heads, head_dim 64, per-head QK-RMSNorm before RoPE (θ = 10,000)
Context	4,096 tokens (trained and served)
Tokenizer	GPT-2 byte-level BPE (tiktoken gpt2 ids) + <|im_start|> 50257, <|im_end|> 50258. 50,259 tokens
Chat format	ChatML as trained, no BOS, no default system prompt, stops on <|im_end|> or <|endoftext|>

Training. Pretrained on 30.0B tokens of FineWeb-Edu sample-100BT (final validation loss 2.5422 on a 1,048,576-token held-out subset), then supervised fine-tuning for 3 epochs on TuluTalk (Tülu 3 SFT and SmolTalk conversations), with the loss on assistant tokens only: held-out validation loss 0.7677 per assistant token.

Evaluation of the unquantized checkpoint (lm-eval 0.4.13, zero-shot, full sets, float32): IFEval prompt-level strict 0.3771, instruction-level strict 0.4844 (mean of the two: 43.1). The pretrained base model scores 0.4760 HellaSwag acc_norm.

This build

q4f16_1: 4-bit group-quantized weights (group size 32, float16 scales) with float16 activations, compiled for WebGPU with MLC-LLM. On 256 held-out TuluTalk conversations (182,451 assistant tokens), the loss per assistant token goes from 0.7865 (unquantized) to 0.8173 (+3.9%, perplexity ×1.031), and the top-1 token agrees 93.2% of the time.

Limitations

States false facts with confidence, do not rely on it for any sort of advice, always cap max_tokens (model tends to run to limit with greedy decoding).