HelioAI/LFM3-1.7B-inference-modules-experiments

LFM3-1.7B + inference modules: a stronger 1.7B reasoning model without changing its weights

An experimental, unofficial release. It is a research log and a working system around LiquidAI/LFM3-1.7B by Liquid AI, Inc., with the weights used in the experiments. Everything here is distributed under Liquid AI's LFM Open License v1.0. We claim no ownership of the model.

All gains below come from inference-time modules: the model's weights are unchanged. The original weights run through llama-server from llama.cpp (weights/).

Every number is a paired comparison: same problems, same seeds, 95% CIs from paired bootstrap. Fast experiments ran on Q5_K_M; on the model's own reasoning traces it differs from bf16 by a KL of 0.008 nats/token. Hardware was one RTX 4080 Laptop (12 GB).

The full report (in Russian, about 40 experiments, including everything that failed) is in research/REPORT.md.

Quick start with the GGUF alone

The largest single gain, answering at the thinking budget instead of being cut off mid-thought, can be reproduced with plain llama.cpp and no code from this repository. Use its built-in thinking budget:

llama-server -m weights/LFM3-1.7B-Q5_K_M.gguf -ngl 99 -c 32768 -fa on \
  --reasoning-budget 4608 \
  --reasoning-budget-message "Time is up. I will stop thinking here and write the final answer now."

Keep max_tokens about 1.5k above the budget, so the answer has room after the reasoning is closed. For harder tasks, raise both.

Paired check on 180 dev math problems (Q5_K_M, same seeds, 6144 tokens in total for both runs):

	Accuracy	No answer at all
bare llama-server	42.8	78 of 180
+ --reasoning-budget 4608	62.8	0 of 180
Δ	+20.0 [+14.4, +26.1]

Script and per-item results: research/scripts/rbud_test.py, research/results/rbud/. With a larger total budget fewer reasonings are cut off, so the gap is smaller (with our module at a 16k budget: AIME +4.2, MATH-500 +2.0, LiveCodeBench +14.9). This flag closes the reasoning abruptly; our module closes it at the last sentence end. It does not include the code, visual or voting modules below. For those, run the server.

Important: the full set of improvements is not inside the GGUF

A GGUF file holds only weights. Our gains come from control logic around the model, which a weight file cannot contain:

• stopping the reasoning and reading the answer out at the budget;
• running code on the examples and repairing it;
• parallel reasoning with a vote;
• rendering visual answers and repairing them.

Attempts to bake this into the weights (experimental/) gave no significant gain. To test what was done, run the GGUF together with the included server:

pip install -r requirements.txt
# needs llama.cpp's llama-server on PATH (or LLAMA_BIN=<folder>); starts the model and the modules together
python lfm3_server.py --start-backend weights/LFM3-1.7B-Q5_K_M.gguf --slots 8
# OpenAI-compatible endpoint: http://localhost:8090/v1/chat/completions, model "lfm3-deliberate" or "lfm3-deliberate-x4"

Any OpenAI-compatible client (Open WebUI, LM Studio as a client, openai in Python) can talk to it like a normal model.

To see the difference side by side, run a bare llama-server on port 8080 next to lfm3_server.py --backend http://localhost:8080, then python compare.py. In our check with a 2048-token budget, the bare model returned no answer: it was still thinking when the budget ran out. The system returned an answer, though at such a small budget a wrong one. Use the default budget of 8192 or more.

What works

1. Deliberation module: stop thinking at the right time

A 1.7B reasoning model often already knows the answer but keeps thinking until the token budget runs out. A cut-off reasoning trace has no answer at all. The module watches the reasoning and handles this in two ways:

• Budget read-out (all tasks). When the budget is reached, it closes the thinking at the last sentence end, exactly the way the model closes it itself, and reads the answer out.
• Temporal consensus (multiple choice only). Every 2048 tokens it probes the answer on a side slot and stops once probes agree.

The reasoning itself is never modified.

Test set (bf16, 16k budget)	n	Base	+ module	Δ [95% CI]
AIME 2024+2025 (4 samples)	60	52.50	56.67	+4.17 [+2.08, +6.67]
MATH-500	500	94.00	96.00	+2.00 [+0.80, +3.20]
MMLU-Pro (14 categories)	1400	62.21	63.93	+1.71 [+0.79, +2.64]
IFEval (prompt strict)	541	94.27	95.01	+0.74 [+0.18, +1.48]
GSM8K / MBPP+ / HumanEval+ / BBH				≈ 0 (no significant regressions)
Average of 8 sets		81.55	82.71	+1.16

2. Code: test-and-repair on the examples from the statement

The program is run on the public examples. On a failure the model sees expected vs actual output and fixes the program (up to 2 rounds). A repair is kept only if it passes strictly more examples.

LiveCodeBench v6 (175 problems, Jan–Apr 2025)	Accuracy	Δ
Base model	32.6
+ deliberation module	47.4	+14.9 [+9.7, +20.6]
+ test-and-repair	52.0	+4.6 [+1.7, +8.0]

• Held-out check. On LiveCodeBench v5 (167 problems) with the modules frozen, the system goes from 34.1 to 56.9, a gain of +22.8 [+16.8, +29.3].
• Official harness. An exact replica of the official LCB harness gives 32.0 → 52.0 on v6.

3. Parallel thinking ×4 + weighted vote (math, multiple choice)

The system runs 4 independent reasonings in parallel and takes a weighted vote. Natural endings count 1, budget read-outs 0.25, and each vote is scaled by 1/√length. On the AIME test this gives 52.5 → 66.7, at 4× the tokens but about the same wall-clock time thanks to batching.

4. Visual tasks (SVG, web pages, canvas, three.js)

On 32 tasks each answer is rendered in headless Chrome. A run counts as clean if it parses, has no JavaScript errors, every resource loads, the page is not blank, and animations actually move.

	SVG	HTML	Canvas	three.js	All
Base model, 12k budget	56	88	13	6	40.6
+ deliberation module	100	94	50	13	64.1
+ render-check-repair + import-map fix	100	94	63	19	68.8

Quality is still far from a strong model. Blind-ish 1–5 ratings average 1.9 for the system against 4.7 for reference files written by Claude.

What does not work (useful negative results)

Idea	Result
RL (GRPO-like, LoRA) and SFT on own verified solutions, 3 attempts	all within noise; for small models the signal is too sparse
Steering vectors, layer looping (recurrent depth without training), layer contrast (DoLa-like), a trained "stop" head	no gain or harmful
Latent thinking without training: Soft Thinking, Gumbel soft thinking, Mixture of Inputs, MTI	MoI +2.5 [−1.4, +6.1] (not significant), others ≤ 0
Teacher's rules / strategies / domain notes in the prompt; Wikipedia RAG	0 to −4.2
Stress-testing module (brute force + generators + counterexamples)	LCB 52.0 → 52.0 at 1.75× tokens
Code rescue (re-ask with a code block opened when the budget answer has no program)	−0.6
Teacher-written toolkits in the prompt ("LLM as Tool Makers"): three.js / canvas / UI kits, svgkit	clean 64.1 → 45.3% (three.js +19, others worse): a 1.7B model does not learn a new API from an in-context reference
Vision critic with a second model (LFM3-VL-2B)	accurate descriptions, only cosmetic repairs; dropped (the system must use only LFM3-1.7B)

The lesson across all of these: for a model of this size, gains come from inference compute plus precise, verifiable feedback. That means not wasting the budget, running tests, rendering, and voting. Feeding it more text (rules, references, APIs) does not help.

Main untested reserve. On LiveCodeBench v6, 112 of 175 problems hit the 16k thinking limit. About 20 failures are pseudo-code written at the limit. A paired budget curve up to 32k is prepared in research/scripts/budget_view.py but was not run.

Weights in this repository

Path	What it is
weights/LFM3-1.7B-Q5_K_M.gguf	the original LFM3-1.7B converted to GGUF and quantized to Q5_K_M; used in all fast experiments (KL to bf16: 0.008 nats/token)
weights/LFM3-1.7B-Q8_0.gguf	the same, Q8_0 (KL 0.0005; practically lossless)
weights/LFM3-1.7B-BF16.gguf	the same, unquantized bf16 (used for the final test-set numbers)
tokenizer/	tokenizer, chat template and config of the original model (the scripts take --tokenizer tokenizer)
experimental/	modified checkpoints: 8 fine-tunes (bf16 transformers format and GGUF), 3 architecture edits and the embedding table for the latent-thinking runner; see experimental/README.md. None beats the original with the modules.

The files in weights/ are format conversions of the original weights, made with llama.cpp convert_hf_to_gguf.py and llama-quantize (build 12078). The values are unchanged except for quantization. The files in experimental/ are modified versions. As required by section 4(b) of the license, their changes are listed in experimental/README.md.

How to run

# 1. the model in llama.cpp (any recent build with LFM3 support), several slots
llama-server -m weights/LFM3-1.7B-Q5_K_M.gguf -ngl 99 -np 16 -c 327680 -fa on --port 8080

# 2a. command line
python lfm3_deliberate.py --kind math "How many positive divisors does 2024 have?"
python lfm3_deliberate.py --kind code "<problem statement with examples> Write a Python 3 program ..."
python lfm3_deliberate.py --kind visual "Draw a compass rose ... Answer with one complete SVG document ..."

# 2b. or an OpenAI-compatible endpoint with automatic routing (math / choice / code / visual / free)
python lfm3_server.py --backend http://localhost:8080
#   POST http://localhost:8090/v1/chat/completions  model: "lfm3-deliberate" (or "lfm3-deliberate-x4" for the vote)

The visual route needs playwright and an installed Chrome. lfm3_deliberate.py and lfm3_server.py load the tokenizer from the bundled tokenizer/ folder. The research scripts in research/scripts/ expect --tokenizer ../tokenizer.

Repository layout

What you need to run it:

• weights/ holds the GGUF files, and tokenizer/ holds the original tokenizer and chat template.
• lfm3_server.py is the OpenAI-compatible server with routing. With --start-backend it also starts llama-server.
• lfm3_deliberate.py holds the modules: deliberation, test-and-repair, visual render-check-repair and parallel vote.
• runtime/ holds the helpers those two files import: answer extraction, the code sandbox, public tests and the headless-Chrome renderer.
• compare.py asks the same question to the bare model and to the system.

Research material, not needed to run the model, is in research/:

• REPORT.md is the full report (Russian).
• scripts/ holds every experiment script: generation, graders, the official LCB harness replica, visual rendering and repair.
• cpp/soft_think.cpp is the latent-thinking runner.
• kits/ holds the frozen teacher toolkits.
• data/ holds the task sets and Claude's reference solutions.
• results/ holds per-item grading of the main runs.

Research scripts run from inside research/ with --tokenizer ../tokenizer.

experimental/ holds the modified checkpoints (see above). LICENSE is Liquid AI's LFM Open License v1.0, which covers the whole repository.

Notes

• License. LFM3-1.7B is by Liquid AI, Inc. and licensed under the LFM Open License v1.0 (LICENSE). This repository redistributes it and its derivatives under the same license. Commercial use is limited by section 5 of the license (annual revenue threshold of $10M).
• Benchmark data. LiveCodeBench, AIME, MATH-500, MMLU-Pro and the other sets come from their original sources and are not redistributed here.
• Credits. The experiments, code and report were produced together with Claude Code (Claude Opus 5.5, Anthropic) acting as the research engineer.