Instructions to use miutti/intel-mac-local-llm with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use miutti/intel-mac-local-llm with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL
# Run inference directly in the terminal:
llama cli -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL
# Run inference directly in the terminal:
llama cli -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL
# Run inference directly in the terminal:
./build/bin/llama cli -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Use Docker

docker model run hf.co/miutti/intel-mac-local-llm:UD-Q2_K_XL

Llama new
LM Studio
Jan

vLLM

How to use miutti/intel-mac-local-llm with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "miutti/intel-mac-local-llm"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "miutti/intel-mac-local-llm",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/miutti/intel-mac-local-llm:UD-Q2_K_XL

Ollama
How to use miutti/intel-mac-local-llm with Ollama:
```
ollama run hf.co/miutti/intel-mac-local-llm:UD-Q2_K_XL
```
Unsloth Desktop

How to use miutti/intel-mac-local-llm with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "miutti/intel-mac-local-llm:UD-Q2_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use miutti/intel-mac-local-llm with Docker Model Runner:
```
docker model run hf.co/miutti/intel-mac-local-llm:UD-Q2_K_XL
```

Lemonade

How to use miutti/intel-mac-local-llm with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull miutti/intel-mac-local-llm:UD-Q2_K_XL

Run and chat with the model

lemonade run user.intel-mac-local-llm-UD-Q2_K_XL

List all available models

lemonade list

Hermes Agent

How to use miutti/intel-mac-local-llm with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default miutti/intel-mac-local-llm:UD-Q2_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use miutti/intel-mac-local-llm with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf miutti/intel-mac-local-llm:UD-Q2_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "miutti/intel-mac-local-llm:UD-Q2_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

intel-mac-local-llm / source /experiments /01_subbit /bench.py

miutti

Source: kernel, desktop app, tools (snapshot of the GitHub repo)

df41178 verified 29 days ago

Raw History Blame Contribute Delete

3 kB

import os
"""実モデルの重み行列で、1ビット未満まで含む各コーデックを比較する。
使い方:
  /usr/bin/python3 bench.py            # 既定のテンソル1枚
  /usr/bin/python3 bench.py --tensor blk.0.ffn_gate.weight
"""
import argparse, json, os, sys, time

ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
sys.path.insert(0, os.path.join(ROOT, "lib"))
import numpy as np
import gguf, wcodec as C

BLOB = (os.environ.get("MODEL_BLOB") or "/path/to/localai/ollama-models/blobs/"
        "sha256-81fb60c7daa80fc1123380b98970b320ae233409f0f71a72ed7b9b0d62f40490")


def build_codecs():
    return [
        ("基準8bit", lambda W: C.rtn(W, 8, 128)),
        ("基準4bit", lambda W: C.rtn(W, 4, 128)),
        ("基準2bit", lambda W: C.rtn(W, 2, 128)),
        ("1bit級", lambda W: C.ternary(W, 128)),
        ("1bit未満", lambda W: C.pvq(W, k=4, cb_bits=4, G=128)),   # 1.0bit
        ("1bit未満", lambda W: C.pvq(W, k=8, cb_bits=6, G=128)),   # 0.75bit
        ("1bit未満", lambda W: C.pvq(W, k=8, cb_bits=4, G=128)),   # 0.5bit
        ("1bit未満", lambda W: C.pvq(W, k=8, cb_bits=4, G=256)),   # 0.56→0.5bit
        ("1bit未満", lambda W: C.pvq(W, k=16, cb_bits=8, G=256)),  # 0.5bit
        ("1bit未満", lambda W: C.pvq(W, k=8, cb_bits=4, G=128, random_codebook=True)),
        ("ハイブリッド", lambda W: C.hybrid(W, 0.01, 8, G=128)),
        ("ハイブリッド", lambda W: C.hybrid(W, 0.03, 8, G=128)),
        ("ハイブリッド", lambda W: C.hybrid(
            W, 0.03, 8, G=128, sub=lambda M: C.pvq(M, k=16, cb_bits=8, G=256))),
    ]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--tensor", default="v.blk.0.mlp.linear_fc1.weight")
    ap.add_argument("--blob", default=BLOB)
    ap.add_argument("--out", default=os.path.join(os.path.dirname(os.path.abspath(__file__)), "results"))
    a = ap.parse_args()

    r = gguf.Reader(a.blob)
    W, tname = r.load(a.tensor)
    W = W.astype(np.float32)
    print(f"対象: {a.tensor}  {W.shape}  元の格納形式={tname}  std={W.std():.5f}")
    print(f"{'区分':<12}{'方式':<44}{'bit/重み':>9}{'重み誤差':>10}{'出力誤差':>10}{'秒':>7}")
    print("-" * 94)

    rows = []
    for cat, fn in build_codecs():
        t0 = time.time()
        Wh, bpw, label = fn(W)
        rw, ry = C.evaluate(W, Wh)
        dt = time.time() - t0
        print(f"{cat:<12}{label:<44}{bpw:>9.3f}{rw:>10.4f}{ry:>10.4f}{dt:>7.1f}")
        rows.append(dict(区分=cat, 方式=label, bpw=round(bpw, 4),
                         重み誤差=round(float(rw), 5), 出力誤差=round(float(ry), 5)))

    os.makedirs(a.out, exist_ok=True)
    p = os.path.join(a.out, a.tensor.replace("/", "_") + ".json")
    with open(p, "w") as f:
        json.dump(dict(tensor=a.tensor, shape=list(W.shape), stored_as=tname, rows=rows),
                  f, ensure_ascii=False, indent=2)
    print(f"\n保存: {p}")


if __name__ == "__main__":
    main()

	import os
	"""実モデルの重み行列で、1ビット未満まで含む各コーデックを比較する。

	使い方:
	/usr/bin/python3 bench.py # 既定のテンソル1枚
	/usr/bin/python3 bench.py --tensor blk.0.ffn_gate.weight
	"""
	import argparse, json, os, sys, time

	ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
	sys.path.insert(0, os.path.join(ROOT, "lib"))
	import numpy as np
	import gguf, wcodec as C

	BLOB = (os.environ.get("MODEL_BLOB") or "/path/to/localai/ollama-models/blobs/"
	"sha256-81fb60c7daa80fc1123380b98970b320ae233409f0f71a72ed7b9b0d62f40490")


	def build_codecs():
	return [
	("基準8bit", lambda W: C.rtn(W, 8, 128)),
	("基準4bit", lambda W: C.rtn(W, 4, 128)),
	("基準2bit", lambda W: C.rtn(W, 2, 128)),
	("1bit級", lambda W: C.ternary(W, 128)),
	("1bit未満", lambda W: C.pvq(W, k=4, cb_bits=4, G=128)), # 1.0bit
	("1bit未満", lambda W: C.pvq(W, k=8, cb_bits=6, G=128)), # 0.75bit
	("1bit未満", lambda W: C.pvq(W, k=8, cb_bits=4, G=128)), # 0.5bit
	("1bit未満", lambda W: C.pvq(W, k=8, cb_bits=4, G=256)), # 0.56→0.5bit
	("1bit未満", lambda W: C.pvq(W, k=16, cb_bits=8, G=256)), # 0.5bit
	("1bit未満", lambda W: C.pvq(W, k=8, cb_bits=4, G=128, random_codebook=True)),
	("ハイブリッド", lambda W: C.hybrid(W, 0.01, 8, G=128)),
	("ハイブリッド", lambda W: C.hybrid(W, 0.03, 8, G=128)),
	("ハイブリッド", lambda W: C.hybrid(
	W, 0.03, 8, G=128, sub=lambda M: C.pvq(M, k=16, cb_bits=8, G=256))),
	]


	def main():
	ap = argparse.ArgumentParser()
	ap.add_argument("--tensor", default="v.blk.0.mlp.linear_fc1.weight")
	ap.add_argument("--blob", default=BLOB)
	ap.add_argument("--out", default=os.path.join(os.path.dirname(os.path.abspath(__file__)), "results"))
	a = ap.parse_args()

	r = gguf.Reader(a.blob)
	W, tname = r.load(a.tensor)
	W = W.astype(np.float32)
	print(f"対象: {a.tensor} {W.shape} 元の格納形式={tname} std={W.std():.5f}")
	print(f"{'区分':<12}{'方式':<44}{'bit/重み':>9}{'重み誤差':>10}{'出力誤差':>10}{'秒':>7}")
	print("-" * 94)

	rows = []
	for cat, fn in build_codecs():
	t0 = time.time()
	Wh, bpw, label = fn(W)
	rw, ry = C.evaluate(W, Wh)
	dt = time.time() - t0
	print(f"{cat:<12}{label:<44}{bpw:>9.3f}{rw:>10.4f}{ry:>10.4f}{dt:>7.1f}")
	rows.append(dict(区分=cat, 方式=label, bpw=round(bpw, 4),
	重み誤差=round(float(rw), 5), 出力誤差=round(float(ry), 5)))

	os.makedirs(a.out, exist_ok=True)
	p = os.path.join(a.out, a.tensor.replace("/", "_") + ".json")
	with open(p, "w") as f:
	json.dump(dict(tensor=a.tensor, shape=list(W.shape), stored_as=tname, rows=rows),
	f, ensure_ascii=False, indent=2)
	print(f"\n保存: {p}")


	if __name__ == "__main__":
	main()