Instructions to use miutti/intel-mac-local-llm with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use miutti/intel-mac-local-llm with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL
# Run inference directly in the terminal:
llama cli -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL
# Run inference directly in the terminal:
llama cli -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf miutti/intel-mac-local-llm:UD-Q2_K_XL
# Run inference directly in the terminal:
./llama-cli -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf miutti/intel-mac-local-llm:UD-Q2_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Use Docker

docker model run hf.co/miutti/intel-mac-local-llm:UD-Q2_K_XL

LM Studio
Jan

vLLM

How to use miutti/intel-mac-local-llm with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "miutti/intel-mac-local-llm"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "miutti/intel-mac-local-llm",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/miutti/intel-mac-local-llm:UD-Q2_K_XL

Ollama
How to use miutti/intel-mac-local-llm with Ollama:
```
ollama run hf.co/miutti/intel-mac-local-llm:UD-Q2_K_XL
```
Unsloth Desktop

How to use miutti/intel-mac-local-llm with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "miutti/intel-mac-local-llm:UD-Q2_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use miutti/intel-mac-local-llm with Docker Model Runner:
```
docker model run hf.co/miutti/intel-mac-local-llm:UD-Q2_K_XL
```

Lemonade

How to use miutti/intel-mac-local-llm with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull miutti/intel-mac-local-llm:UD-Q2_K_XL

Run and chat with the model

lemonade run user.intel-mac-local-llm-UD-Q2_K_XL

List all available models

lemonade list

Hermes Agent

How to use miutti/intel-mac-local-llm with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default miutti/intel-mac-local-llm:UD-Q2_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use miutti/intel-mac-local-llm with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "miutti/intel-mac-local-llm:UD-Q2_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

intel-mac-local-llm / source /experiments /08_actaware /bench.py

miutti

Source: kernel, desktop app, tools (snapshot of the GitHub repo)

df41178 verified 21 days ago

Raw History Blame Contribute Delete

4.25 kB

import os
"""実験08: 活性化を意識した量子化（チャンネル重要度スケーリング）。
評議会が繰り返し指摘した本質的な問題:
  いまの量子化は「重みの誤差」を最小化しているが、本当に守りたいのは「出力の誤差」。
  よく使われる入力チャンネルの重みは大事、ほとんど使われないチャンネルは雑でよい。
対策は単純で、量子化の前に列ごとの物差しを変える:
    W' = W · diag(s),   x' = diag(s)^-1 · x     (W'x' = Wx なので出力は不変)
  s を活性化の大きさから決めると、大事な列が「大きく」なり、量子化が自然にそこを丁寧に扱う。
GPTQ と違い、使うのは**チャンネルごとのスケール1024個だけ**（ヘッセ行列100万個ではない）。
推定する量が桁違いに少ないので、過学習しにくいはず——というのが検証したい仮説。
"""
import json, os, sys, time
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.path.insert(0, os.path.join(ROOT, "lib"))
import numpy as np
import gguf, wcodec as C, rotate

BLOB = (os.environ.get("MODEL_BLOB") or "/path/to/localai/ollama-models/blobs/"
        "sha256-81fb60c7daa80fc1123380b98970b320ae233409f0f71a72ed7b9b0d62f40490")


def inner_rotate(A, B, seed=0):
    r = A.shape[1]
    d = rotate.signs(r, seed)
    s = 1.0 / np.sqrt(r)
    return rotate.fwht(A) * d * s, (rotate.fwht(B.T).T) * d[:, None] * s


def recipe(W, r, lr_bits, k, cb_bits, G, seed=0):
    """主力レシピ（内側回転あり）。"""
    out, inn = W.shape
    U, S, Vt = np.linalg.svd(W, full_matrices=False)
    A, B = inner_rotate(U[:, :r] * S[:r], Vt[:r], seed)
    Aq = C.rtn(A, lr_bits, min(G, A.shape[1]))[0]
    Bq = C.rtn(B, lr_bits, min(G, B.shape[1]))[0]
    Rh, res_bpw, _ = C.pvq(W - Aq @ Bq, k=k, cb_bits=cb_bits, G=G, seed=seed)
    lr_bpw = r * (out + inn) * lr_bits / (out * inn)
    ovh = 16 * (A.size / min(G, A.shape[1]) + B.size / min(G, B.shape[1])) / (out * inn)
    return (Aq @ Bq + Rh).astype(np.float32), lr_bpw + res_bpw + ovh


r_ = gguf.Reader(BLOB)
allrows = []
for TENSOR in ["v.blk.0.mlp.linear_fc1.weight", "blk.0.attn_qkv.weight"]:
    W0, tname = r_.load(TENSOR)
    W0 = np.ascontiguousarray(W0.astype(np.float32))
    out, inn = W0.shape
    if inn & (inn - 1):
        p = 1 << (inn.bit_length() - 1)
        W0 = np.ascontiguousarray(W0[:, :p]); inn = p

    def acts(ns, seed):
        rng = np.random.default_rng(seed)
        X = rng.standard_t(4, size=(inn, ns)).astype(np.float32)
        X[rng.choice(inn, max(1, inn // 100), replace=False)] *= 20.
        return X
    Xcal, Xev = acts(4096, 0), acts(128, 999)          # 較正用と評価用は別データ
    chan = np.sqrt((Xcal ** 2).mean(1))                # チャンネルごとの大きさ
    chan /= chan.mean()

    print(f"\n=== {TENSOR} {W0.shape} 元={tname} ===")
    print(f"{'設定':<40}{'bit/重み':>9}{'出力誤差':>10}{'改善':>9}")
    print("-" * 68)
    for r, lrb, k, cb in [(32, 4, 8, 4), (64, 4, 8, 4), (64, 4, 8, 8)]:
        base = None
        for alpha in (0.0, 0.25, 0.5, 0.75, 1.0):
            s = chan ** alpha
            s = s / np.exp(np.log(s).mean())            # 幾何平均を1に正規化
            W = W0 * s[None, :]                         # W' = W diag(s)
            Xe = Xev / s[:, None]                       # x' = diag(s)^-1 x
            Wh, bpw = recipe(W, r, lrb, k, cb, 128)
            _, ry = C.evaluate(W, Wh, X=Xe)
            if base is None:
                base = ry; mark = "(基準)"
            else:
                mark = f"{100*(base-ry)/base:+.1f}%"
            print(f"{f'ランク{r}+PVQコード{2**cb}  重要度^{alpha}':<40}"
                  f"{bpw:>9.3f}{ry:>10.4f}{mark:>9}")
            allrows.append(dict(tensor=TENSOR, rank=r, cb_bits=cb, alpha=alpha,
                                bpw=round(float(bpw), 4), 出力誤差=round(float(ry), 5)))

json.dump(allrows, open(os.path.join(os.path.dirname(os.path.abspath(__file__)),
                                     "results", "actaware.json"), "w"),
          ensure_ascii=False, indent=2)
print("\n保存完了")

	import os
	"""実験08: 活性化を意識した量子化（チャンネル重要度スケーリング）。

	評議会が繰り返し指摘した本質的な問題:
	いまの量子化は「重みの誤差」を最小化しているが、本当に守りたいのは「出力の誤差」。
	よく使われる入力チャンネルの重みは大事、ほとんど使われないチャンネルは雑でよい。

	対策は単純で、量子化の前に列ごとの物差しを変える:
	W' = W · diag(s), x' = diag(s)^-1 · x (W'x' = Wx なので出力は不変)
	s を活性化の大きさから決めると、大事な列が「大きく」なり、量子化が自然にそこを丁寧に扱う。

	GPTQ と違い、使うのはチャンネルごとのスケール1024個だけ（ヘッセ行列100万個ではない）。
	推定する量が桁違いに少ないので、過学習しにくいはず——というのが検証したい仮説。
	"""
	import json, os, sys, time
	ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
	sys.path.insert(0, os.path.join(ROOT, "lib"))
	import numpy as np
	import gguf, wcodec as C, rotate

	BLOB = (os.environ.get("MODEL_BLOB") or "/path/to/localai/ollama-models/blobs/"
	"sha256-81fb60c7daa80fc1123380b98970b320ae233409f0f71a72ed7b9b0d62f40490")


	def inner_rotate(A, B, seed=0):
	r = A.shape[1]
	d = rotate.signs(r, seed)
	s = 1.0 / np.sqrt(r)
	return rotate.fwht(A) * d * s, (rotate.fwht(B.T).T) * d[:, None] * s


	def recipe(W, r, lr_bits, k, cb_bits, G, seed=0):
	"""主力レシピ（内側回転あり）。"""
	out, inn = W.shape
	U, S, Vt = np.linalg.svd(W, full_matrices=False)
	A, B = inner_rotate(U[:, :r] * S[:r], Vt[:r], seed)
	Aq = C.rtn(A, lr_bits, min(G, A.shape[1]))[0]
	Bq = C.rtn(B, lr_bits, min(G, B.shape[1]))[0]
	Rh, res_bpw, _ = C.pvq(W - Aq @ Bq, k=k, cb_bits=cb_bits, G=G, seed=seed)
	lr_bpw = r * (out + inn) * lr_bits / (out * inn)
	ovh = 16 * (A.size / min(G, A.shape[1]) + B.size / min(G, B.shape[1])) / (out * inn)
	return (Aq @ Bq + Rh).astype(np.float32), lr_bpw + res_bpw + ovh


	r_ = gguf.Reader(BLOB)
	allrows = []
	for TENSOR in ["v.blk.0.mlp.linear_fc1.weight", "blk.0.attn_qkv.weight"]:
	W0, tname = r_.load(TENSOR)
	W0 = np.ascontiguousarray(W0.astype(np.float32))
	out, inn = W0.shape
	if inn & (inn - 1):
	p = 1 << (inn.bit_length() - 1)
	W0 = np.ascontiguousarray(W0[:, :p]); inn = p

	def acts(ns, seed):
	rng = np.random.default_rng(seed)
	X = rng.standard_t(4, size=(inn, ns)).astype(np.float32)
	X[rng.choice(inn, max(1, inn // 100), replace=False)] *= 20.
	return X
	Xcal, Xev = acts(4096, 0), acts(128, 999) # 較正用と評価用は別データ
	chan = np.sqrt((Xcal ** 2).mean(1)) # チャンネルごとの大きさ
	chan /= chan.mean()

	print(f"\n=== {TENSOR} {W0.shape} 元={tname} ===")
	print(f"{'設定':<40}{'bit/重み':>9}{'出力誤差':>10}{'改善':>9}")
	print("-" * 68)
	for r, lrb, k, cb in [(32, 4, 8, 4), (64, 4, 8, 4), (64, 4, 8, 8)]:
	base = None
	for alpha in (0.0, 0.25, 0.5, 0.75, 1.0):
	s = chan ** alpha
	s = s / np.exp(np.log(s).mean()) # 幾何平均を1に正規化
	W = W0 * s[None, :] # W' = W diag(s)
	Xe = Xev / s[:, None] # x' = diag(s)^-1 x
	Wh, bpw = recipe(W, r, lrb, k, cb, 128)
	_, ry = C.evaluate(W, Wh, X=Xe)
	if base is None:
	base = ry; mark = "(基準)"
	else:
	mark = f"{100*(base-ry)/base:+.1f}%"
	print(f"{f'ランク{r}+PVQコード{2**cb} 重要度^{alpha}':<40}"
	f"{bpw:>9.3f}{ry:>10.4f}{mark:>9}")
	allrows.append(dict(tensor=TENSOR, rank=r, cb_bits=cb, alpha=alpha,
	bpw=round(float(bpw), 4), 出力誤差=round(float(ry), 5)))

	json.dump(allrows, open(os.path.join(os.path.dirname(os.path.abspath(__file__)),
	"results", "actaware.json"), "w"),
	ensure_ascii=False, indent=2)
	print("\n保存完了")