Instructions to use HelioAI/LFM3-1.7B-inference-modules-experiments with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use HelioAI/LFM3-1.7B-inference-modules-experiments with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M
# Run inference directly in the terminal:
llama cli -hf HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M
# Run inference directly in the terminal:
llama cli -hf HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M

Use Docker

docker model run hf.co/HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M

Llama new
LM Studio
Jan
Ollama
How to use HelioAI/LFM3-1.7B-inference-modules-experiments with Ollama:
```
ollama run hf.co/HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M
```
Unsloth Desktop

How to use HelioAI/LFM3-1.7B-inference-modules-experiments with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use HelioAI/LFM3-1.7B-inference-modules-experiments with Docker Model Runner:
```
docker model run hf.co/HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M
```

Lemonade

How to use HelioAI/LFM3-1.7B-inference-modules-experiments with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M

Run and chat with the model

lemonade run user.LFM3-1.7B-inference-modules-experiments-Q5_K_M

List all available models

lemonade list

Hermes Agent

How to use HelioAI/LFM3-1.7B-inference-modules-experiments with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use HelioAI/LFM3-1.7B-inference-modules-experiments with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "HelioAI/LFM3-1.7B-inference-modules-experiments:Q5_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

LFM3-1.7B-inference-modules-experiments / research /scripts /extract_cvec.py

HelioAI

LFM3-1.7B inference modules: experimental release

67ebcef verified 2 days ago

Raw History Blame Contribute Delete

4.25 kB

"""Steering direction 'hesitation' from the model's own reasoning traces (no labels, no external model).
At every paragraph break ("\n\n") inside <think>, the residual stream of each layer is recorded at the break token,
i.e. at the moment the model decides how to continue. Paragraphs that open with a hesitation/re-check marker
("Wait", "Alternatively", "Hmm", "Let me double-check", ...) form class R, all others class E.
direction[layer] = mean_R - mean_E. Saved as .pt (all layers) for building llama.cpp control vectors.
usage: extract_cvec.py --gen results/dev_orig/dev.jsonl --items data/train/dev.jsonl --out results/cvec/hes.pt
"""
import argparse, json, re, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import transformers.integrations.sdpa_attention as _sdpa

# Windows torch has no flash kernel: masked/GQA SDPA would fall back to the O(L^2)-memory math kernel.
# Expanding K/V instead keeps it on the memory-efficient kernel.
_sdpa.use_gqa_in_sdpa = lambda *a, **k: False

ap = argparse.ArgumentParser()
ap.add_argument("--gen", required=True)
ap.add_argument("--items", required=True)
ap.add_argument("--out", required=True)
ap.add_argument("--model", default="models/LFM3-1.7B")
ap.add_argument("--max-resp", type=int, default=200)
ap.add_argument("--max-len", type=int, default=8192)
args = ap.parse_args()

HES = re.compile(r"^(\*\*)?(wait|but wait|hmm|alternatively|actually|hold on|oh wait|however, wait|let me (double[- ]check|"
                 r"check (again|this|that)|verify|re-?examine|reconsider|think again|re-?check|re-?read|make sure)|"
                 r"but let me|let's (double[- ]check|verify|check again))", re.I)
tok = AutoTokenizer.from_pretrained(args.model)
model = AutoModelForCausalLM.from_pretrained(args.model, dtype=torch.bfloat16, attn_implementation="sdpa", trust_remote_code=True).cuda().eval()
L = model.config.num_hidden_layers
items = {json.loads(l)["id"]: json.loads(l) for l in open(args.items, encoding="utf-8")}
recs = [json.loads(l) for l in open(args.gen, encoding="utf-8")]
recs = [r for r in recs if items[r["id"]]["task"] == "math" and "tokens" in r][: args.max_resp]

cap = {}
hooks = [layer.register_forward_hook(lambda m, i, o, k=k: cap.__setitem__(k, (o[0] if isinstance(o, tuple) else o)[0]))
         for k, layer in enumerate(model.model.layers)]
sums = {c: torch.zeros(L, model.config.hidden_size, dtype=torch.float64) for c in "RE"}
cnt = {"R": 0, "E": 0}
norm = torch.zeros(L, dtype=torch.float64)
for n_r, r in enumerate(recs):
    p = tok.apply_chat_template([{"role": "user", "content": items[r["id"]]["prompt"]}], tokenize=True,
                                add_generation_prompt=True, return_dict=False)
    resp = r["tokens"]
    pieces = tok.convert_ids_to_tokens(resp)
    pieces = [tok.convert_tokens_to_string([x]) for x in pieces]
    text, pos, labels = "", [], []
    for i, s in enumerate(pieces):
        text += s
        if "</think>" in text:
            break
        if s.endswith("\n\n") and i + 1 < len(resp):
            nxt = tok.decode(resp[i + 1: i + 12]).lstrip()
            pos.append(len(p) + i)
            labels.append("R" if HES.match(nxt) else "E")
    ids = (p + resp)[: args.max_len]
    keep = [(q, c) for q, c in zip(pos, labels) if q < len(ids)]
    if not keep:
        continue
    with torch.no_grad():
        model(input_ids=torch.tensor([ids], device="cuda"), use_cache=False)
    qi = torch.tensor([q for q, _ in keep], device="cuda")
    isr = torch.tensor([c == "R" for _, c in keep])
    for k in range(L):
        h = cap[k][qi].double().cpu()
        sums["R"][k] += h[isr].sum(0)
        sums["E"][k] += h[~isr].sum(0)
        norm[k] += h.norm(dim=-1).sum()
    for _, c in keep:
        cnt[c] += 1
    if n_r % 20 == 0:
        print(n_r, cnt, flush=True)
mR, mE = sums["R"] / cnt["R"], sums["E"] / cnt["E"]
d = (mR - mE).float()
avg_norm = (norm / (cnt["R"] + cnt["E"])).float()
for k in range(L):
    print(f"layer {k:2d}  |dir| {d[k].norm():8.2f}  |h| {avg_norm[k]:8.2f}  ratio {d[k].norm() / avg_norm[k]:.3f}  "
          f"cos(mR,mE) {torch.nn.functional.cosine_similarity(mR[k], mE[k], 0):.3f}")
torch.save({"dir": d, "avg_norm": avg_norm, "count": cnt}, args.out)
print("saved", args.out, cnt)

	"""Steering direction 'hesitation' from the model's own reasoning traces (no labels, no external model).

	At every paragraph break ("\n\n") inside <think>, the residual stream of each layer is recorded at the break token,
	i.e. at the moment the model decides how to continue. Paragraphs that open with a hesitation/re-check marker
	("Wait", "Alternatively", "Hmm", "Let me double-check", ...) form class R, all others class E.
	direction[layer] = mean_R - mean_E. Saved as .pt (all layers) for building llama.cpp control vectors.

	usage: extract_cvec.py --gen results/dev_orig/dev.jsonl --items data/train/dev.jsonl --out results/cvec/hes.pt
	"""
	import argparse, json, re, torch
	from transformers import AutoModelForCausalLM, AutoTokenizer
	import transformers.integrations.sdpa_attention as _sdpa

	# Windows torch has no flash kernel: masked/GQA SDPA would fall back to the O(L^2)-memory math kernel.
	# Expanding K/V instead keeps it on the memory-efficient kernel.
	_sdpa.use_gqa_in_sdpa = lambda a, *k: False

	ap = argparse.ArgumentParser()
	ap.add_argument("--gen", required=True)
	ap.add_argument("--items", required=True)
	ap.add_argument("--out", required=True)
	ap.add_argument("--model", default="models/LFM3-1.7B")
	ap.add_argument("--max-resp", type=int, default=200)
	ap.add_argument("--max-len", type=int, default=8192)
	args = ap.parse_args()

	HES = re.compile(r"^(\\)?(wait\|but wait\|hmm\|alternatively\|actually\|hold on\|oh wait\|however, wait\|let me (double[- ]check\|"
	r"check (again\|this\|that)\|verify\|re-?examine\|reconsider\|think again\|re-?check\|re-?read\|make sure)\|"
	r"but let me\|let's (double[- ]check\|verify\|check again))", re.I)
	tok = AutoTokenizer.from_pretrained(args.model)
	model = AutoModelForCausalLM.from_pretrained(args.model, dtype=torch.bfloat16, attn_implementation="sdpa", trust_remote_code=True).cuda().eval()
	L = model.config.num_hidden_layers
	items = {json.loads(l)["id"]: json.loads(l) for l in open(args.items, encoding="utf-8")}
	recs = [json.loads(l) for l in open(args.gen, encoding="utf-8")]
	recs = [r for r in recs if items[r["id"]]["task"] == "math" and "tokens" in r][: args.max_resp]

	cap = {}
	hooks = [layer.register_forward_hook(lambda m, i, o, k=k: cap.__setitem__(k, (o[0] if isinstance(o, tuple) else o)[0]))
	for k, layer in enumerate(model.model.layers)]
	sums = {c: torch.zeros(L, model.config.hidden_size, dtype=torch.float64) for c in "RE"}
	cnt = {"R": 0, "E": 0}
	norm = torch.zeros(L, dtype=torch.float64)
	for n_r, r in enumerate(recs):
	p = tok.apply_chat_template([{"role": "user", "content": items[r["id"]]["prompt"]}], tokenize=True,
	add_generation_prompt=True, return_dict=False)
	resp = r["tokens"]
	pieces = tok.convert_ids_to_tokens(resp)
	pieces = [tok.convert_tokens_to_string([x]) for x in pieces]
	text, pos, labels = "", [], []
	for i, s in enumerate(pieces):
	text += s
	if "</think>" in text:
	break
	if s.endswith("\n\n") and i + 1 < len(resp):
	nxt = tok.decode(resp[i + 1: i + 12]).lstrip()
	pos.append(len(p) + i)
	labels.append("R" if HES.match(nxt) else "E")
	ids = (p + resp)[: args.max_len]
	keep = [(q, c) for q, c in zip(pos, labels) if q < len(ids)]
	if not keep:
	continue
	with torch.no_grad():
	model(input_ids=torch.tensor([ids], device="cuda"), use_cache=False)
	qi = torch.tensor([q for q, _ in keep], device="cuda")
	isr = torch.tensor([c == "R" for _, c in keep])
	for k in range(L):
	h = cap[k][qi].double().cpu()
	sums["R"][k] += h[isr].sum(0)
	sums["E"][k] += h[~isr].sum(0)
	norm[k] += h.norm(dim=-1).sum()
	for _, c in keep:
	cnt[c] += 1
	if n_r % 20 == 0:
	print(n_r, cnt, flush=True)
	mR, mE = sums["R"] / cnt["R"], sums["E"] / cnt["E"]
	d = (mR - mE).float()
	avg_norm = (norm / (cnt["R"] + cnt["E"])).float()
	for k in range(L):
	print(f"layer {k:2d} \|dir\| {d[k].norm():8.2f} \|h\| {avg_norm[k]:8.2f} ratio {d[k].norm() / avg_norm[k]:.3f} "
	f"cos(mR,mE) {torch.nn.functional.cosine_similarity(mR[k], mE[k], 0):.3f}")
	torch.save({"dir": d, "avg_norm": avg_norm, "count": cnt}, args.out)
	print("saved", args.out, cnt)