| --- |
| license: apache-2.0 |
| language: |
| - id |
| library_name: transformers |
| tags: |
| - deepseek |
| - moe |
| - causal-lm |
| - chat |
| - cpu |
| - indonesian |
| datasets: |
| - neosantara/IndoTalkSFT |
| base_model: neosantara/wader-100m-base |
| pipeline_tag: text-generation |
| --- |
| |
| # lilo-100m — Lightweight Local Intelligence & Optimizer |
|
|
| A ~110M parameter language model implementing the DeepSeek-V4 architecture, fine-tuned for chat/instruction following in Indonesian. This is the **lilo** release of `neosantara/wader-100m`: same architecture and weights, re-packaged explicitly for **CPU-only, low-memory, low-footprint** use. |
|
|
| - **Base**: [neosantara/wader-100m-base](https://huggingface.co/neosantara/wader-100m-base) · pretrained on [neosantara/IndoTalk](https://huggingface.co/datasets/neosantara/IndoTalk) |
| - **Upstream SFT**: [neosantara/wader-100m](https://huggingface.co/neosantara/wader-100m) on [neosantara/IndoTalkSFT](https://huggingface.co/datasets/neosantara/IndoTalkSFT) |
| - **Trained from scratch** — no weights from DeepSeek-V4 were used. |
|
|
| ## Why lilo |
|
|
| | | wader-100m (original) | **lilo-100m** | |
| |---|---|---| |
| | Weights on disk | 421MB fp32 | **119MB** (int8) / **221MB** (fp16) | |
| | CPU-friendly runtime | bf16 NaN bugs, no KV cache | fp32/fp16 stable, KV cache, inference_mode | |
| | Runs on 2 vCPU / 2GB RAM VPS | untested | **measured** | |
| |
| ## Variants |
| |
| | Variant | File | Disk | PPL delta vs fp32 | Greedy outputs | |
| |---|---|---|---|---| |
| | `fp16` (default) | `model.fp16.safetensors` | 221MB | +0.08% | 5/5 identical | |
| | `int8` | `model.int8.safetensors` | 119MB | +0.32% | 5/5 identical | |
| |
| Notes: |
| - `int8` is symmetric group-wise quantization (group=64) on all `nn.Linear` layers + embeddings; norms/router/HC params stay fp32. Scales cost ~7MB. |
| - Both dequantize to fp32 compute on load — the **disk footprint is what shrinks**, because this CPU (Zen2, AVX2-only, no VNNI) does fp32 GEMM as fast as int8 anyway, and fp16 GEMM is *faster* still. |
| - fp16 is the default because it is both the closest to the original quality **and** the fastest on CPU. |
| |
| ## CPU speed (measured on 2 vCPU AMD EPYC 7K62 "Zen2", AVX2-only, box under load) |
| |
| | Variant | threads=1 | threads=2 | |
| |---|---|---| |
| | fp16 | **14–22 tok/s** | 11–12 tok/s | |
| | int8 | 9–19 tok/s | 5–8 tok/s | |
| |
| On this tiny model (320 hidden, 8 layers) thread sync overhead dominates — `--threads 1` is fastest. On 8+ core machines try 2–4. |
| |
| ## Memory safety (no OOM) |
| |
| Peak RSS measured on a 2 vCPU / 1.9GB RAM VPS: **~0.85GB (int8) / ~1.05GB (fp16)** including the Python/torch runtime. Rules baked into `run_cpu.sh`: |
|
|
| - `OMP_NUM_THREADS` bounded, `MALLOC_ARENA_MAX=1` (glibc per-thread arenas can silently eat GBs) |
| - `TMPDIR` forced to disk, never `/tmp` when `/tmp` is tmpfs |
| - prefill chunked at 256 tokens so attention never allocates a big `[S,S]` matrix |
|
|
| ## Usage |
|
|
| ```bash |
| git lfs install |
| git clone https://huggingface.co/hzlo0/lilo-100m |
| cd lilo-100m |
| pip install torch --index-url https://download.pytorch.org/whl/cpu |
| pip install transformers safetensors tokenizers accelerate |
| |
| bash run_cpu.sh # chat, default fp16 |
| CUDA_VISIBLE_DEVICES="" python chat_lilo.py --variant int8 --threads 1 "Halo, apa kabar?" |
| ``` |
|
|
| Chat template (DeepSeek-V4 style): |
|
|
| ``` |
| <|begin_of_sentence|><|User|>Halo<|Assistant|> |
| ``` |
|
|
| ```python |
| import torch |
| from lilo_common import load_lilo, generate_lilo, chat |
| |
| torch.set_num_threads(1) |
| tok, model = load_lilo(".", variant="fp16") # or "int8" |
| print(chat(tok, model, [{"role": "user", "content": "Halo, apa kabar?"}], |
| max_new_tokens=200, temperature=0.7, top_p=0.9)) |
| ``` |
|
|
| Requires `trust_remote_code`-free loading: the model code ships as plain files (`modeling_deepseek_v4.py`) loaded by `lilo_common.py`. (The same code also works via `AutoModelForCausalLM.from_pretrained(..., trust_remote_code=True)`.) |
|
|
| ## Architecture |
|
|
| | Component | Details | |
| |---|---| |
| | Parameters | ~110M (41M embeddings, 69M non-embedding) | |
| | Hidden size | 320 · Layers 8 | |
| | Attention | 8 heads, 1 KV head (MQA), MLA q_lora_rank=160, grouped low-rank output | |
| | MoE | 4 routed experts + 1 shared, top-2 routing | |
| | Hyper-Connections | hc_mult=4, Sinkhorn routing (replaces residual stream) | |
| | Vocab | 129,280 (DeepSeek-V4 tokenizer) · Context 2,048 | |
| |
| ## What was fixed for CPU in this release |
| |
| 1. **freqs_cis recomputed on demand** — original code left a buffer uninitialized after meta-device loading, corrupting RoPE beyond a few tokens. |
| 2. **KV cache honored** — original `forward` hardcoded `use_cache=False`, recomputing the whole sequence every decode step. This release caches and steps one token. |
| 3. **`return_dict` defaults to True** — transformers 5.x deprecated `config.use_return_dict`, which silently returned tuples. |
| 4. **Expert dtype bridge** — `w2(x.to(w2.weight.dtype))` silently cast activations to int8 under weight-only quantization. |
| 5. **bf16 NaN** — model runs fp32/fp16; no bf16 anywhere. |
| |
| ## Limitations |
| |
| - Tiny model: 110M params, most capacity in embeddings. Generations are often incoherent or factually wrong. |
| - Undertrained: 5K pretrain + 3K SFT steps (upstream). Production models train 100K+ steps on trillions of tokens. |
| |
| ## License |
| |
| Apache-2.0 (inherited from upstream `neosantara/wader-100m`). |
| |
