Echoic-Lite

Bilingual (Chinese/English) character-level language model family — custom Transformer trained from scratch by eyanchao.

No HuggingFace Transformers dependency — pure PyTorch with SDPA, GQA, QK-Norm, Z-loss.

Models

Version	Params	Dim	Layers	Heads	KV Heads	Status
v29	~1.0B	1408	32	32	8 (GQA)	🏃 Training
v28	~480M	1024	28	32	—	✅ Deployed
v27	~200M	768	20	24	—	📦 Archived
v26	~60M	512	14	16	—	📦 Archived

Architecture (v29)

Technique	Description
GQA	Grouped Query Attention (32Q / 8KV, ~30% VRAM saved)
QK-Norm	RMSNorm on Q/K projections — training stability
Z-loss	Logit L2 regularization — prevents explosion
SDPA	PyTorch fused scaled_dot_product_attention
RoPE + SwiGLU + RMSNorm	Llama-style architecture
Gradient checkpointing	Per-layer activation recompute
torch.compile	JIT with inductor disk cache
Layer-wise LR decay	Higher layers learn slower (0.8×)
Dynamic LR	Auto-half on loss spikes

Training Data Mix

Task	Weight	Sources
Math	40%	mathinstruct, school_math, math_large, math_sft
Code	20%	code_120k, code_large
Encyclopedia	25%	zh-wiki (5 partitions)
Stories	15%	TinyStories ZH/EN, story_instruct, story_punct
Identity	SFT	Custom dialogs × 50

Quick Start

import torch, os
from huggingface_hub import hf_hub_download

REPO = "eyanchao/echoic-lite"

# Tokenizer
tok = torch.load(
    hf_hub_download(REPO, "bilingual-tokenizer.pt"),
    map_location="cpu", weights_only=False
)
stoi, itos = tok.stoi, tok.itos

# v28 480M (dim=1024, layers=28, heads=32)
ckpt = torch.load(
    hf_hub_download(REPO, "v28-best-e15.pt"),
    map_location="cpu", weights_only=False
)
# See echoic_modal.py for full EchoicLM class definition

def encode(s): return [stoi.get(ord(c), 0) for c in s]
def decode(ids): return "".join(itos.get(k, "?") for k in ids)

prompt = "<|user|>你是谁<|assistant|>"
x = torch.tensor([encode(prompt)], dtype=torch.long)
out = model.generate(x, max_new_tokens=100, temperature=0.7, top_k=40)
print(decode(out[0].tolist()))

Inference API

POST https://eyanchao--echoic-generate.modal.run
Body: {"prompt": "从前有一个", "max_tokens": 200}

Files

File	Description
v28-best-e15.pt	v28 best (480M, deployed)
v28-final.pt	v28 final weights
v29-best-e*.pt	v29 checkpoints (1B, training)
bilingual-tokenizer.pt	Tokenizer

Data

eyanchao/echoic-data

Evolution

v26 60M → v27 200M → v28 480M → v29 1B → ...

Weight expansion from v26→v28. v29 trained from scratch with GQA architecture.