Blame for runtime/README.md

# Runtime for round2-1.7b-loop-preview

This directory contains the source patches needed to run the GGUF. It contains no LM Studio binaries and no official LM Studio runtime.

## Patches

Apply both patches, **in order**, to llama.cpp commit `ff0d396e3dda0f33e7ce5ad4267fce4f7690cfb1`:

1. **`01-lfm3-loop-prerequisites.patch`** (34 files) turns upstream into the custom runtime this model was built on. It contains:
   - [llama.cpp PR #31306](https://github.com/ggml-org/llama.cpp/pull/31306), Clover, by Maphist0 and contributors (MIT);
   - native SenseNova Loop support;
   - the earlier LFM3 Loop (mHC) and NVFP4 support;
   - [llama.cpp PR #30306](https://github.com/ggml-org/llama.cpp/pull/30306), Nemotron Dory, by its contributors (MIT).

   Patch 02 builds on the LFM3 loop code introduced here.
2. **`02-lfm3-refine-loop.patch`** (6 files) adds the loop used by this model. It touches:
   - `src/models/lfm3.cpp`: the `lfm3.loop_refine.*` hyperparameters, the loop's tensors and its graph;
   - `src/models/lfm-base.cpp`, `src/models/models.h`: optional low-rank deltas on the short-convolution projections. Other models pass none, so their graphs are unchanged;
   - `src/llama-arch.h`, `src/llama-arch.cpp`: one new tensor type, `LLM_TENSOR_LOOP_GATE` (`blk.%d.loop_gate`), appended at the end of the enum so existing values do not move;
   - `src/llama-context.cpp`: the LFM3 loop node budget now also covers models whose routers start at a later layer.

Applying both patches to a fresh export of the pinned commit reproduces the validated runtime source exactly: no files differ.

The same source is published as a branch: [renoirvieir4/llama.cpp, branch `round2-1.7b-loop-preview`](https://github.com/renoirvieir4/llama.cpp/tree/round2-1.7b-loop-preview). It contains these two patches as commits on top of `ff0d396e3`, plus a tested recipe and the script `scripts/round2-loop-preview.sh`. Cloning that branch replaces the `git apply` steps below.

```bash
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git checkout ff0d396e3dda0f33e7ce5ad4267fce4f7690cfb1
PATCH_DIR=/absolute/path/to/round2-1.7b-loop-preview/runtime
git apply "$PATCH_DIR/01-lfm3-loop-prerequisites.patch"
git apply "$PATCH_DIR/02-lfm3-refine-loop.patch"
```

## Build and run

```bash
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DGGML_BACKEND_DL=ON -DGGML_NATIVE=OFF
cmake --build build --config Release --target llama-server -j
./build/bin/llama-server -m /absolute/path/to/round2-1.7b-loop-preview-BF16.gguf \
  -ngl 99 -fa on -c 65536 --temp 1.0 --top-k 20 --top-p 0.95 --repeat-penalty 1.1
```

**The tested build:**
- **Toolchain:** CUDA 12.8 and GCC 14, CUDA architecture 86 (RTX 3090), with `-DGGML_BACKEND_DL=ON -DGGML_OPENMP=OFF`.
- **Why OpenMP was off:** that machine's GCC 14 package shipped only a static `libgomp`. With a regular compiler installation, OpenMP can stay on.
- **Result:** it served the GGUF and answered a multiplication question correctly at 89 tokens/s.

Other toolchains may need different options.

## What the GGUF contains

The GGUF uses the standard `lfm3` architecture with a 40-layer execution plan:
- **Plan:** physical layers 0-20, then 5-20 again, then 21-23.
- **Execution metadata:** `lfm3.block_count` is 40, and the per-layer arrays (`attention.head_count_kv`, `attention.sliding_window_pattern`) are in execution order.
- **Loop keys:** `lfm3.loop_refine.version` = 1, `begin` = 5, `end` = 21, `passes` = 2, `streams` = 2, `physical_layers` = 24, `delta_rank` = 64.
- **Base tensors:** each physical layer's tensors keep their standard names under the execution index of its first visit, so layers 21-23 are stored as `blk.37`-`blk.39`.
- **Extra-pass tensors (execution layers 21-36), all FP32:**
  - `blk.N.hc_attn_fn.weight` {4096, 8}, `blk.N.hc_attn_base.bias` {8}, and the same for `hc_ffn`: the two-stream mHC routers;
  - `blk.N.<projection>.delta_down` {n_in, 64} and `blk.N.<projection>.delta_up` {64, n_out}, for `attn_q`, `attn_k`, `attn_v`, `attn_output`, `shortconv.in_proj`, `shortconv.out_proj`, `ffn_gate`, `ffn_up` and `ffn_down`;
  - `blk.36.loop_gate.logits` {2048} and `blk.36.loop_gate.token` {2048}.
- **Numerics:** the graph applies BF16 rounding at the same points as the BF16 PyTorch reference: the residual streams, the router's pre, post and mix weights, the stream mean, and the gate blend. Native logits then match the reference as closely as stock LFM3-1.7B's do.

## LM Studio integration used for validation

The model was validated in **LM Studio** with a local runtime named `llama.cpp-linux-x86_64-nvidia-cuda12-avx2@2.66.106`.
- **How it was made:** it is a copy of LM Studio's own Linux x86_64 CUDA 12 AVX2 runtime 2.66.0 (llama.cpp `ff0d396e3`), where only `libllama.so` was replaced. Every other LM Studio file was kept as shipped: the ggml libraries, the CUDA libraries, the server and the bindings.
- **The replacement library:** it was built from the patched tree with `-DGGML_CUDA=OFF -DGGML_BACKEND_DL=ON -DGGML_NATIVE=OFF -DBUILD_SHARED_LIBS=ON`; ggml comes from LM Studio's own libraries. It was then adjusted as follows:

  ```bash
  patchelf --set-soname libllama.so --set-rpath '$ORIGIN' \
    --replace-needed libggml.so.0 libggml_llamacpp.so \
    --replace-needed libggml-base.so.0 libggml-base.so \
    /path/to/copied/runtime/libllama.so
  ```
- **Registering it:** the copied folder went next to the other runtimes in `~/.lmstudio/extensions/backends/`, with a new version in its `backend-manifest.json`. It was then selected with `lms runtime select llama.cpp-linux-x86_64-nvidia-cuda12-avx2@<version>`.
- **Load settings:** full GPU offload, flash attention on, context 65,536, one parallel prediction, and a single GPU. That used about 5.2 GB of VRAM.

This is a version-specific integration, not a general drop-in for other LM Studio runtimes. Automatic runtime updates may switch away from it.

## Checks

- **Regressions:** the CPU architecture suite (`test-llama-archs`: 175 tests over 162 architectures) gave identical results before and after patch 02. Greedy outputs were also token-for-token identical on models of the architectures patch 01 already supported. See `../validation/regression.json`.
- **Agreement with PyTorch:** native logits match the PyTorch reference, and the loop's own effect is reproduced. See `../validation/native-vs-pytorch.json`.