FLCMind

A small Llama-style language model family implemented from scratch in plain PyTorch — no transformers, no peft, no trl. It ships in two variants: a ~65M-parameter dense model and a ~200M-parameter MoE model (4 experts, Top-1 routing, ~65M active parameters per token).

FLCMind is an end-to-end LLM project built for learning: custom BPE tokenizer → pretraining → full-parameter SFT → OpenAI-compatible serving. This repository hosts both SFT checkpoints together with the model code and tokenizer, so the weights can be used without cloning anything else.

Highlights

• Llama-style architecture, written from scratch: RMSNorm, RoPE, GQA, SwiGLU, plus an optional Mixture-of-Experts FFN.
• Two variants: dense (65M params) and MoE (200M total, ~65M active per token, 4 experts Top-1).
• Tiny and cheap to run: inference works on CPU; training fits in ~6 GB of GPU memory.
• OpenAI-compatible API out of the box (/v1/chat/completions, streaming supported).
• Custom tokenizer (BPE, vocab size 6400) trained together with the model.

Which checkpoint should I use?

Checkpoint	Params	Size	Use it for
full_sft_768.pth	~65M (dense)	~138 MB	Chat / inference — recommended
moe_sft_768_moe.pth	~200M total, ~65M active	~407 MB	Chat, if you want to try the MoE variant

Both are SFT weights and can hold multi-turn conversations. A pretraining-only checkpoint is not included: it only continues text and produces incoherent replies in chat.

Important: not transformers-compatible

The checkpoints are plain PyTorch state_dicts and require the accompanying model_flcmind.py:

from model_flcmind import FLCMindConfig, FLCMindForCausalLM

config = FLCMindConfig(hidden_size=768, num_hidden_layers=8, use_moe=False)
model = FLCMindForCausalLM(config)
model.load_state_dict(torch.load("full_sft_768.pth", map_location="cpu"))

AutoModelForCausalLM.from_pretrained(...) will not work, and there is no trust_remote_code shortcut.

Quick start

The FLCMind scripts resolve paths relative to their own location, so the files must sit in the directories the project expects:

hf download FLCgeigeigei/flcmind --local-dir flcmind

cp flcmind/model_flcmind.py     <project>/model/
cp flcmind/tokenizer*.json      <project>/model/
cp flcmind/full_sft_768.pth     <project>/out/
cp flcmind/moe_sft_768_moe.pth  <project>/out/

Chat from the command line

# Dense — from the project root
python eval_llm.py --weight full_sft

# MoE — --use_moe must match how the weights were trained
python eval_llm.py --use_moe 1 --weight moe_sft

--weight is a prefix, not a filename. The script appends _768.pth for dense and _768_moe.pth for MoE, so moe_sft resolves to moe_sft_768_moe.pth.

OpenAI-compatible server

cd scripts
python serve_openai_api.py --weight full_sft             # Dense
python serve_openai_api.py --use_moe 1 --weight moe_sft  # MoE

Then call it like any OpenAI endpoint:

from openai import OpenAI
client = OpenAI(api_key="sk-any", base_url="http://localhost:8998/v1")
resp = client.chat.completions.create(
    model="flcmind-local:latest",
    messages=[{"role": "user", "content": "Hello"}],
    temperature=0.7,
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="")

Use a positive temperature (e.g. 0.7). temperature=0 triggers a divide-by-zero in the sampler and can leave the CUDA context corrupted until the server is restarted.

Architecture

Item	Dense	MoE
hidden_size	768	768
num_hidden_layers	8	8
Attention	GQA	GQA
FFN	SwiGLU	4 experts, Top-1
Total params	~65M	~200M
Active params / token	~65M	~65M
Vocab size	6400	6400

--hidden_size, --num_hidden_layers and --use_moe must match the weights at load time; otherwise you get shape-mismatch errors.

Training

Both checkpoints were trained on FLCgeigeigei/flcmind-datasets (~1.2 GB pretraining text, ~1.7 GB SFT conversations):

1. Pretraining — next-token prediction on the pretraining split.
2. SFT — full-parameter fine-tuning on multi-turn conversations, with the loss masked to assistant turns.

Limitations

• Very small model. Expect limited world knowledge, weak reasoning, and frequent factual errors. Do not rely on it for factual answers.
• Short context. Training used max_length of 512 (pretraining) and 1024 (SFT) tokens; long conversations degrade quickly.
• Chinese + English only, with a 6400-token vocabulary — other languages are effectively unsupported.
• Not instruction-safe by default. No RLHF or safety tuning was applied; outputs should be treated as raw model behavior.
• For learning and experimentation, not production use.

License

The weights and code are released for learning and research purposes. The training data comes from public sources; please also comply with the licenses of those original datasets.