runtime/README.md

Runtime for round2-1.7b-loop-preview

This directory contains the source patches needed to run the GGUF. It contains no LM Studio binaries and no official LM Studio runtime.

Patches

Apply both patches, in order, to llama.cpp commit ff0d396e3dda0f33e7ce5ad4267fce4f7690cfb1:

1. 01-lfm3-loop-prerequisites.patch (34 files) turns upstream into the custom runtime this model was built on. It contains:

   • llama.cpp PR #31306, Clover, by Maphist0 and contributors (MIT);
   • native SenseNova Loop support;
   • the earlier LFM3 Loop (mHC) and NVFP4 support;
   • llama.cpp PR #30306, Nemotron Dory, by its contributors (MIT).

   Patch 02 builds on the LFM3 loop code introduced here.

2. 02-lfm3-refine-loop.patch (6 files) adds the loop used by this model. It touches:

   • src/models/lfm3.cpp: the lfm3.loop_refine.* hyperparameters, the loop's tensors and its graph;
   • src/models/lfm-base.cpp, src/models/models.h: optional low-rank deltas on the short-convolution projections. Other models pass none, so their graphs are unchanged;
   • src/llama-arch.h, src/llama-arch.cpp: one new tensor type, LLM_TENSOR_LOOP_GATE (blk.%d.loop_gate), appended at the end of the enum so existing values do not move;
   • src/llama-context.cpp: the LFM3 loop node budget now also covers models whose routers start at a later layer.

Applying both patches to a fresh export of the pinned commit reproduces the validated runtime source exactly: no files differ.

The same source is published as a branch: renoirvieir4/llama.cpp, branch round2-1.7b-loop-preview. It contains these two patches as commits on top of ff0d396e3, plus a tested recipe and the script scripts/round2-loop-preview.sh. Cloning that branch replaces the git apply steps below.

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git checkout ff0d396e3dda0f33e7ce5ad4267fce4f7690cfb1
PATCH_DIR=/absolute/path/to/round2-1.7b-loop-preview/runtime
git apply "$PATCH_DIR/01-lfm3-loop-prerequisites.patch"
git apply "$PATCH_DIR/02-lfm3-refine-loop.patch"

Build and run

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DGGML_BACKEND_DL=ON -DGGML_NATIVE=OFF
cmake --build build --config Release --target llama-server -j
./build/bin/llama-server -m /absolute/path/to/round2-1.7b-loop-preview-BF16.gguf \
  -ngl 99 -fa on -c 65536 --temp 1.0 --top-k 20 --top-p 0.95 --repeat-penalty 1.1

The tested build:

• Toolchain: CUDA 12.8 and GCC 14, CUDA architecture 86 (RTX 3090), with -DGGML_BACKEND_DL=ON -DGGML_OPENMP=OFF.
• Why OpenMP was off: that machine's GCC 14 package shipped only a static libgomp. With a regular compiler installation, OpenMP can stay on.
• Result: it served the GGUF and answered a multiplication question correctly at 89 tokens/s.

Other toolchains may need different options.

What the GGUF contains

The GGUF uses the standard lfm3 architecture with a 40-layer execution plan:

• Plan: physical layers 0-20, then 5-20 again, then 21-23.
• Execution metadata: lfm3.block_count is 40, and the per-layer arrays (attention.head_count_kv, attention.sliding_window_pattern) are in execution order.
• Loop keys: lfm3.loop_refine.version = 1, begin = 5, end = 21, passes = 2, streams = 2, physical_layers = 24, delta_rank = 64.
• Base tensors: each physical layer's tensors keep their standard names under the execution index of its first visit, so layers 21-23 are stored as blk.37-blk.39.
• Extra-pass tensors (execution layers 21-36), all FP32:
  • blk.N.hc_attn_fn.weight {4096, 8}, blk.N.hc_attn_base.bias {8}, and the same for hc_ffn: the two-stream mHC routers;
  • blk.N.<projection>.delta_down {n_in, 64} and blk.N.<projection>.delta_up {64, n_out}, for attn_q, attn_k, attn_v, attn_output, shortconv.in_proj, shortconv.out_proj, ffn_gate, ffn_up and ffn_down;
  • blk.36.loop_gate.logits {2048} and blk.36.loop_gate.token {2048}.
• Numerics: the graph applies BF16 rounding at the same points as the BF16 PyTorch reference: the residual streams, the router's pre, post and mix weights, the stream mean, and the gate blend. Native logits then match the reference as closely as stock LFM3-1.7B's do.

LM Studio integration used for validation

The model was validated in LM Studio with a local runtime named llama.cpp-linux-x86_64-nvidia-cuda12-avx2@2.66.106.

• How it was made: it is a copy of LM Studio's own Linux x86_64 CUDA 12 AVX2 runtime 2.66.0 (llama.cpp ff0d396e3), where only libllama.so was replaced. Every other LM Studio file was kept as shipped: the ggml libraries, the CUDA libraries, the server and the bindings.

• The replacement library: it was built from the patched tree with -DGGML_CUDA=OFF -DGGML_BACKEND_DL=ON -DGGML_NATIVE=OFF -DBUILD_SHARED_LIBS=ON; ggml comes from LM Studio's own libraries. It was then adjusted as follows:

  patchelf --set-soname libllama.so --set-rpath '$ORIGIN' \
    --replace-needed libggml.so.0 libggml_llamacpp.so \
    --replace-needed libggml-base.so.0 libggml-base.so \
    /path/to/copied/runtime/libllama.so
  

• Registering it: the copied folder went next to the other runtimes in ~/.lmstudio/extensions/backends/, with a new version in its backend-manifest.json. It was then selected with lms runtime select llama.cpp-linux-x86_64-nvidia-cuda12-avx2@<version>.

• Load settings: full GPU offload, flash attention on, context 65,536, one parallel prediction, and a single GPU. That used about 5.2 GB of VRAM.

This is a version-specific integration, not a general drop-in for other LM Studio runtimes. Automatic runtime updates may switch away from it.

Checks

• Regressions: the CPU architecture suite (test-llama-archs: 175 tests over 162 architectures) gave identical results before and after patch 02. Greedy outputs were also token-for-token identical on models of the architectures patch 01 already supported. See ../validation/regression.json.
• Agreement with PyTorch: native logits match the PyTorch reference, and the loop's own effect is reproduced. See ../validation/native-vs-pytorch.json.