LFM3-1.7B-3D-Coder

A full fine-tune of LiquidAI/LFM3-1.7B-Base for one-shot Three.js scene generation, replicating the task of ccore/Qwen3.5-4B-3D-Coder on a hybrid (gated-conv + sliding-window + global attention) architecture. Part of an architecture-comparison portfolio: same dataset, same prompts, same verifier across LFM3, OLMoE and RWKV, to measure what architecture choice costs and buys for a narrow code-generation task on a single GPU. Independent experiment, not affiliated with the base-model or dataset authors.

Evaluation — honest numbers first

Programmatic verifier on the same stratified 60 validation prompts (6 per complexity level 1–10, seed 42), generation at temperature 0.6, top-p 0.95, top-k 20, max 16,384 new tokens. Three checks per output: complete HTML document, pinned Three.js import, and the extracted script passing node --check. Raw generations and scores are in this repo under eval/ (gen-lfm3-base.jsonl, gen-lfm3-sft.jsonl, scores-lfm3-base.json, scores-lfm3-sft.json).

Model Complete HTML Pinned import JS syntax pass
Gold (reference answers) 1.00 (60/60) 0.98 (59/60) 0.98 (59/60)
LFM3-1.7B-Base (zero-shot) 0.67 (40/60) 0.12 (7/60) 0.27 (16/60)
LFM3-1.7B-3D-Coder (this model) 0.27 (16/60) 0.27 (16/60) 0.05 (3/60)

SFT bought format adherence (pinned import 0.12 → 0.27) at the cost of completion and validity. Training was healthy (final train loss 0.56 over 492 steps); the failure is at generation time, and it is two independent failure modes, not one:

  1. Length / repetition loops — 44 of 60 outputs never finish. Median output is 50,363 characters; tails show variable-fragment loops (yarnRimOuterRimRimRim…, rimTopRimTop…) eating the 16k-token budget. Only 16/60 contain a complete </html>.
  2. Same-scope redeclaration — 13 of the 16 finished outputs still fail node --check. 11 of the 13 fail with the identical error class, Identifier 'X' has already been declared (camera, burnerRing, pivotCenter, notchGeometry, …): the model restarts a block and reuses the same names in the same scope. The other 2 are a Python-ism leaking into JS and a malformed call. This is the reference model's own disclosed duplicate-variable bug pattern, amplified — gold syntax is 0.98, this model is 0.05.

What is not a problem: 52/60 outputs open with a prose preamble before the ```html fence. That looks wrong, but the gold targets open the same way — the format was learned correctly, so do not "fix" it.

Two verifier metrics were dropped from this table after the gold run proved them broken, not after they looked bad: no_dup flags any reused variable name even across separate functions (normal code; gold scores 0.0 on it), and coverage was never wired up.

Greedy-decoding follow-up (same 60 prompts, temperature 0): greedy is worse, not better — 0 of 60 outputs contain a complete </html>; every run loops until the 16,384-token budget is exhausted (median 54,134 chars). Decoding is therefore not the rescue: the failure sits in the weights/generation behaviour, and a Phase 2 GRPO run with the verifier as reward — penalizing redeclaration specifically, not just completion — is the justified next step. A repetition_penalty=1.1 test was proposed but never actually launched. Track B (OLMoE, run unchanged) will show whether redeclaration is LFM3-specific or task-general. Greedy generations and scores are in eval/gen-lfm3-sft-greedy.jsonl and eval/scores-lfm3-sft-greedy.json.

Training

  • Data: ccore/ThreeJS-3D-Scenes (MIT), 3,933 train / 213 validation. Thinking traces were dropped: with traces, sequences run p50 ≈ 25,800 tokens, so the original 8,192-token recipe would have silently dropped almost everything. HTML-only targets at max_len 24,576 keep 3,929 of 3,933 examples.
  • Method: full fine-tune, 2 epochs, 492 steps, lr 1e-5 (cosine, 15 warmup steps), effective batch 16, bf16, gradient checkpointing, no packing (verified against training_args.bin in this repo).
  • Compute: molab marimo notebook, RTX PRO 6000 Blackwell (96 GB), ~3.1 h, $0. Checkpoints pushed to the Hub every epoch; checkpoint-492 is the final step.
  • Tracking: Weights & Biases project fde-3dcoder-portfolio, run lfm3-1.7b-3dcoder-sft.

Usage

System prompt used for training and eval — use only the Three.js core ES module from https://cdnjs.cloudflare.com/ajax/libs/three.js/0.160.0/three.module.min.js, no addons, implement controls inline:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("syaikhipin/LFM3-1.7B-3D-Coder", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "syaikhipin/LFM3-1.7B-3D-Coder", trust_remote_code=True,
    dtype=torch.bfloat16, device_map="cuda")

LFM3 ships custom code, so trust_remote_code=True and transformers ≥ 5.23 are required. Expect long, often unfinished outputs at the eval settings above; greedy decoding finishes 0 of 60 (see above).

GGUF builds (llama.cpp, verified loading and generating with llama-cli) are in gguf/: LFM3-1.7B-3D-Coder-Q8_0.gguf (1.85 GB) and LFM3-1.7B-3D-Coder-Q4_K_M.gguf (1.10 GB).

Limitations

  • One-shot scenes frequently contain duplicate variable declarations and runtime errors — see the eval table, not marketing copy.
  • Trained only on the 4,146-scene Three.js distribution; out-of-domain prompts degrade.
  • The LFM3 base was not pre-trained for code; it stays short and often valid (0.67 complete HTML zero-shot) but does not follow the task.
  • Demo: a static gallery Space with the completed evaluation scenes and live previews is at https://huggingface.co/spaces/syaikhipin/LFM3-3D-Coder-Demo (Gradio Spaces now require a paid plan, so the demo is a static page; live generation is not offered because HF Inference does not serve this custom-code model).
Downloads last month
217
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for syaikhipin/LFM3-1.7B-3D-Coder

Quantized
(21)
this model
Quantizations
1 model

Dataset used to train syaikhipin/LFM3-1.7B-3D-Coder