--- license: other license_name: lfm1.0 license_link: LICENSE base_model: LiquidAI/LFM3-1.7B tags: - lfm3 - reasoning - test-time-compute - inference-time-scaling - code-generation - llama.cpp - negative-results language: - en - ru --- # LFM3-1.7B + inference modules: a stronger 1.7B reasoning model without changing its weights **An experimental, unofficial release.** It is a research log and a working system around [LiquidAI/LFM3-1.7B](https://huggingface.co/LiquidAI/LFM3-1.7B) by Liquid AI, Inc., with the weights used in the experiments. Everything here is distributed under Liquid AI's [LFM Open License v1.0](LICENSE). We claim no ownership of the model. All gains below come from **inference-time modules**: the model's weights are unchanged. The original weights run through `llama-server` from llama.cpp (`weights/`). Every number is a **paired** comparison: same problems, same seeds, 95% CIs from paired bootstrap. Fast experiments ran on Q5_K_M; on the model's own reasoning traces it differs from bf16 by a KL of 0.008 nats/token. Hardware was one RTX 4080 Laptop (12 GB). The full report (in Russian, about 40 experiments, including everything that failed) is in [`research/REPORT.md`](research/REPORT.md). ## Quick start with the GGUF alone The largest single gain, answering at the thinking budget instead of being cut off mid-thought, can be reproduced with plain llama.cpp and no code from this repository. Use its built-in thinking budget: ```bash llama-server -m weights/LFM3-1.7B-Q5_K_M.gguf -ngl 99 -c 32768 -fa on \ --reasoning-budget 4608 \ --reasoning-budget-message "Time is up. I will stop thinking here and write the final answer now." ``` Keep `max_tokens` about 1.5k above the budget, so the answer has room after the reasoning is closed. For harder tasks, raise both. Paired check on 180 dev math problems (Q5_K_M, same seeds, 6144 tokens in total for both runs): | | Accuracy | No answer at all | |---|---|---| | bare llama-server | 42.8 | 78 of 180 | | + `--reasoning-budget 4608` | **62.8** | 0 of 180 | | Δ | **+20.0 [+14.4, +26.1]** | | Script and per-item results: `research/scripts/rbud_test.py`, `research/results/rbud/`. With a larger total budget fewer reasonings are cut off, so the gap is smaller (with our module at a 16k budget: AIME +4.2, MATH-500 +2.0, LiveCodeBench +14.9). This flag closes the reasoning abruptly; our module closes it at the last sentence end. It does not include the code, visual or voting modules below. For those, run the server. ## Important: the full set of improvements is not inside the GGUF A GGUF file holds only weights. Our gains come from **control logic around the model**, which a weight file cannot contain: - stopping the reasoning and reading the answer out at the budget; - running code on the examples and repairing it; - parallel reasoning with a vote; - rendering visual answers and repairing them. Attempts to bake this into the weights (`experimental/`) gave no significant gain. To test what was done, run **the GGUF together with the included server**: ```bash pip install -r requirements.txt # needs llama.cpp's llama-server on PATH (or LLAMA_BIN=); starts the model and the modules together python lfm3_server.py --start-backend weights/LFM3-1.7B-Q5_K_M.gguf --slots 8 # OpenAI-compatible endpoint: http://localhost:8090/v1/chat/completions, model "lfm3-deliberate" or "lfm3-deliberate-x4" ``` Any OpenAI-compatible client (Open WebUI, LM Studio as a client, `openai` in Python) can talk to it like a normal model. To see the difference side by side, run a bare llama-server on port 8080 next to `lfm3_server.py --backend http://localhost:8080`, then `python compare.py`. In our check with a 2048-token budget, the bare model returned no answer: it was still thinking when the budget ran out. The system returned an answer, though at such a small budget a wrong one. Use the default budget of 8192 or more. ## What works ### 1. Deliberation module: stop thinking at the right time A 1.7B reasoning model often already knows the answer but keeps thinking until the token budget runs out. A cut-off reasoning trace has no answer at all. The module watches the reasoning and handles this in two ways: - **Budget read-out (all tasks).** When the budget is reached, it closes the thinking at the last sentence end, exactly the way the model closes it itself, and reads the answer out. - **Temporal consensus (multiple choice only).** Every 2048 tokens it probes the answer on a side slot and stops once probes agree. The reasoning itself is never modified. | Test set (bf16, 16k budget) | n | Base | + module | Δ [95% CI] | |---|---|---|---|---| | AIME 2024+2025 (4 samples) | 60 | 52.50 | **56.67** | **+4.17** [+2.08, +6.67] | | MATH-500 | 500 | 94.00 | **96.00** | **+2.00** [+0.80, +3.20] | | MMLU-Pro (14 categories) | 1400 | 62.21 | **63.93** | **+1.71** [+0.79, +2.64] | | IFEval (prompt strict) | 541 | 94.27 | **95.01** | **+0.74** [+0.18, +1.48] | | GSM8K / MBPP+ / HumanEval+ / BBH | | | | ≈ 0 (no significant regressions) | | **Average of 8 sets** | | 81.55 | **82.71** | **+1.16** | ### 2. Code: test-and-repair on the examples from the statement The program is run on the public examples. On a failure the model sees expected vs actual output and fixes the program (up to 2 rounds). A repair is kept only if it passes strictly more examples. | LiveCodeBench v6 (175 problems, Jan–Apr 2025) | Accuracy | Δ | |---|---|---| | Base model | 32.6 | | | + deliberation module | 47.4 | +14.9 [+9.7, +20.6] | | + test-and-repair | **52.0** | +4.6 [+1.7, +8.0] | - **Held-out check.** On LiveCodeBench v5 (167 problems) with the modules frozen, the system goes from 34.1 to **56.9**, a gain of **+22.8 [+16.8, +29.3]**. - **Official harness.** An exact replica of the official LCB harness gives 32.0 → 52.0 on v6. ### 3. Parallel thinking ×4 + weighted vote (math, multiple choice) The system runs 4 independent reasonings in parallel and takes a weighted vote. Natural endings count 1, budget read-outs 0.25, and each vote is scaled by 1/√length. On the AIME test this gives **52.5 → 66.7**, at 4× the tokens but about the same wall-clock time thanks to batching. ### 4. Visual tasks (SVG, web pages, canvas, three.js) On 32 tasks each answer is rendered in headless Chrome. A run counts as clean if it parses, has no JavaScript errors, every resource loads, the page is not blank, and animations actually move. | | SVG | HTML | Canvas | three.js | All | |---|---|---|---|---|---| | Base model, 12k budget | 56 | 88 | 13 | 6 | 40.6 | | + deliberation module | 100 | 94 | 50 | 13 | 64.1 | | + render-check-repair + import-map fix | 100 | 94 | 63 | 19 | **68.8** | Quality is still far from a strong model. Blind-ish 1–5 ratings average 1.9 for the system against 4.7 for reference files written by Claude. ## What does not work (useful negative results) | Idea | Result | |---|---| | RL (GRPO-like, LoRA) and SFT on own verified solutions, 3 attempts | all within noise; for small models the signal is too sparse | | Steering vectors, layer looping (recurrent depth without training), layer contrast (DoLa-like), a trained "stop" head | no gain or harmful | | Latent thinking without training: Soft Thinking, Gumbel soft thinking, Mixture of Inputs, MTI | MoI +2.5 [−1.4, +6.1] (not significant), others ≤ 0 | | Teacher's rules / strategies / domain notes in the prompt; Wikipedia RAG | 0 to −4.2 | | Stress-testing module (brute force + generators + counterexamples) | LCB 52.0 → 52.0 at 1.75× tokens | | Code rescue (re-ask with a code block opened when the budget answer has no program) | −0.6 | | Teacher-written toolkits in the prompt ("LLM as Tool Makers"): three.js / canvas / UI kits, svgkit | clean 64.1 → 45.3% (three.js +19, others worse): a 1.7B model does not learn a new API from an in-context reference | | Vision critic with a second model (LFM3-VL-2B) | accurate descriptions, only cosmetic repairs; dropped (the system must use only LFM3-1.7B) | The lesson across all of these: for a model of this size, gains come from **inference compute plus precise, verifiable feedback**. That means not wasting the budget, running tests, rendering, and voting. Feeding it more text (rules, references, APIs) does not help. **Main untested reserve.** On LiveCodeBench v6, 112 of 175 problems hit the 16k thinking limit. About 20 failures are pseudo-code written at the limit. A paired budget curve up to 32k is prepared in `research/scripts/budget_view.py` but was not run. ## Weights in this repository | Path | What it is | |---|---| | `weights/LFM3-1.7B-Q5_K_M.gguf` | the original LFM3-1.7B converted to GGUF and quantized to Q5_K_M; used in all fast experiments (KL to bf16: 0.008 nats/token) | | `weights/LFM3-1.7B-Q8_0.gguf` | the same, Q8_0 (KL 0.0005; practically lossless) | | `weights/LFM3-1.7B-BF16.gguf` | the same, unquantized bf16 (used for the final test-set numbers) | | `tokenizer/` | tokenizer, chat template and config of the original model (the scripts take `--tokenizer tokenizer`) | | `experimental/` | **modified** checkpoints: 8 fine-tunes (bf16 transformers format and GGUF), 3 architecture edits and the embedding table for the latent-thinking runner; see [`experimental/README.md`](experimental/README.md). None beats the original with the modules. | The files in `weights/` are format conversions of the original weights, made with llama.cpp `convert_hf_to_gguf.py` and `llama-quantize` (build 12078). The values are unchanged except for quantization. The files in `experimental/` are modified versions. As required by section 4(b) of the license, their changes are listed in `experimental/README.md`. ## How to run ```bash # 1. the model in llama.cpp (any recent build with LFM3 support), several slots llama-server -m weights/LFM3-1.7B-Q5_K_M.gguf -ngl 99 -np 16 -c 327680 -fa on --port 8080 # 2a. command line python lfm3_deliberate.py --kind math "How many positive divisors does 2024 have?" python lfm3_deliberate.py --kind code " Write a Python 3 program ..." python lfm3_deliberate.py --kind visual "Draw a compass rose ... Answer with one complete SVG document ..." # 2b. or an OpenAI-compatible endpoint with automatic routing (math / choice / code / visual / free) python lfm3_server.py --backend http://localhost:8080 # POST http://localhost:8090/v1/chat/completions model: "lfm3-deliberate" (or "lfm3-deliberate-x4" for the vote) ``` The visual route needs `playwright` and an installed Chrome. `lfm3_deliberate.py` and `lfm3_server.py` load the tokenizer from the bundled `tokenizer/` folder. The research scripts in `research/scripts/` expect `--tokenizer ../tokenizer`. ## Repository layout What you need to run it: - `weights/` holds the GGUF files, and `tokenizer/` holds the original tokenizer and chat template. - `lfm3_server.py` is the OpenAI-compatible server with routing. With `--start-backend` it also starts llama-server. - `lfm3_deliberate.py` holds the modules: deliberation, test-and-repair, visual render-check-repair and parallel vote. - `runtime/` holds the helpers those two files import: answer extraction, the code sandbox, public tests and the headless-Chrome renderer. - `compare.py` asks the same question to the bare model and to the system. Research material, not needed to run the model, is in `research/`: - `REPORT.md` is the full report (Russian). - `scripts/` holds every experiment script: generation, graders, the official LCB harness replica, visual rendering and repair. - `cpp/soft_think.cpp` is the latent-thinking runner. - `kits/` holds the frozen teacher toolkits. - `data/` holds the task sets and Claude's reference solutions. - `results/` holds per-item grading of the main runs. Research scripts run from inside `research/` with `--tokenizer ../tokenizer`. `experimental/` holds the modified checkpoints (see above). `LICENSE` is Liquid AI's LFM Open License v1.0, which covers the whole repository. ## Notes - **License.** LFM3-1.7B is by Liquid AI, Inc. and licensed under the LFM Open License v1.0 (`LICENSE`). This repository redistributes it and its derivatives under the same license. Commercial use is limited by section 5 of the license (annual revenue threshold of $10M). - **Benchmark data.** LiveCodeBench, AIME, MATH-500, MMLU-Pro and the other sets come from their original sources and are not redistributed here. - **Credits.** The experiments, code and report were produced together with Claude Code (Claude Opus 5.5, Anthropic) acting as the research engineer.