Instructions to use Harsha001/LFM3-1.7B-Base-ROCmFPX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Harsha001/LFM3-1.7B-Base-ROCmFPX with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M

Use Docker

docker model run hf.co/Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use Harsha001/LFM3-1.7B-Base-ROCmFPX with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Harsha001/LFM3-1.7B-Base-ROCmFPX"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Harsha001/LFM3-1.7B-Base-ROCmFPX",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M

Ollama
How to use Harsha001/LFM3-1.7B-Base-ROCmFPX with Ollama:
```
ollama run hf.co/Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M
```
Unsloth Desktop

How to use Harsha001/LFM3-1.7B-Base-ROCmFPX with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Harsha001/LFM3-1.7B-Base-ROCmFPX with Docker Model Runner:
```
docker model run hf.co/Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M
```

Lemonade

How to use Harsha001/LFM3-1.7B-Base-ROCmFPX with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M

Run and chat with the model

lemonade run user.LFM3-1.7B-Base-ROCmFPX-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Harsha001/LFM3-1.7B-Base-ROCmFPX with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Harsha001/LFM3-1.7B-Base-ROCmFPX with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Harsha001/LFM3-1.7B-Base-ROCmFPX:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

LFM3-1.7B-Base-ROCmFPX / README.md

Harsha001

Add README.md

192340a verified 6 days ago

preview code

Raw History Blame Contribute Delete

3.52 kB

---
license: other
license_name: lfm1.0
license_link: LICENSE
base_model: LiquidAI/LFM3-1.7B-Base
base_model_relation: quantized
pipeline_tag: text-generation
library_name: gguf
tags: [gguf, rocmfpx, amd, rocm, vulkan, strix-halo, quantized, liquid, lfm3, edge]
---
# LFM3-1.7B-Base, ROCmFPX GGUF

[LiquidAI/LFM3-1.7B-Base](https://huggingface.co/LiquidAI/LFM3-1.7B-Base), the pretrained base model of LFM3-1.7B, without chat tuning. It is meant for fine-tuning and plain text completion, not for chat, quantized for the
[ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) llama.cpp fork (low-bit weight types for AMD GPUs, Vulkan and
HIP/ROCm).

> The recommended 6-bit file needs [HarshaVardhan3002/ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX). Stock
> llama.cpp, LM Studio, Ollama and other ROCmFPX builds may not load it. The Q8_0 and Q4_K_M files use stock types and
> load in any llama.cpp build with LFM3 support. **These files were checked on the CPU and have not been tested on AMD
> hardware yet.**
>
> Not an official release of Liquid AI. All credit for the model goes to its authors.
## Files
| File | Size | Weights | KL divergence vs BF16 (lower is better) | Same top token as BF16 |
|---|---:|---|---:|---:|
| `LFM3-1.7B-Base-Q6_K_ROCMFPX.gguf` **(recommended)** | 1.43 GB | ROCmFPX 6-bit (`Q6_0_ROCMFPX` with tensor scales) | 0.0031 | 96.9% |
| `LFM3-1.7B-Base-Q8_0.gguf` | 1.85 GB | stock llama.cpp Q8_0 | 0.0006 | 98.7% |
| `LFM3-1.7B-Base-Q4_K_M.gguf` | 1.10 GB | stock llama.cpp Q4_K_M | 0.0267 | 92.0% |
| stock Q6_K, for comparison (not uploaded) | 1.43 GB | stock llama.cpp Q6_K | 0.0039 | 96.6% |
- **Measured on:** 40 chunks of 512 tokens of WikiText-2 (test split), against the BF16 GGUF, CPU.
- **The ROCmFPX 6-bit file beats stock Q6_K** at the same size. Its file name says `Q6_K_ROCMFPX` so that Hugging Face
  lists it as a 6-bit quant; the weights are the ROCmFPX `Q6_0_ROCMFPX` type, not a stock `Q6_K`.
- **No ROCmFPX 4-bit file:** our 4-bit types came out clearly worse than stock Q4_K_M on this model (KL divergence 1.5 to
  1.7 times higher), so we ship Q4_K_M instead.
- Every file passed a short greedy answer test before upload.

## How these were made

1. BF16 GGUF converted with the fork's `convert_hf_to_gguf.py`.
2. Importance matrix from the BF16 file over a general calibration text (`calibration_datav3`), 512-token chunks.
3. `llama-quantize --imatrix` from the BF16 file for every file, stock ones included. The 6-bit file was made with `LLAMA_ROCMFPX_TENSOR_SCALE=1`, which
   stores a power-of-two scale per weight tensor (the runtime applies it; no setting is needed to run the file), with
   `--token-embedding-type q6_K`.

## Run

```bash
git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build -j
build/bin/llama-completion -m LFM3-1.7B-Base-Q6_K_ROCMFPX.gguf -ngl 99 -p "The capital of France is" -n 64
```

This is a base model: give it text to continue, or fine-tune it. For chat use LFM3-1.7B instead.

LFM3 support in the fork comes from llama.cpp pull request #31252. LFM3-Omni and LFM3-Audio need further upstream
llama.cpp work and will follow later.

## License

[LFM Open License v1.0](LICENSE), same as the base model. Commercial use is allowed for companies below 10 million US
dollars of annual revenue; above that it is not covered by the license. Read `LICENSE` before you use these files.

	---
	license: other
	license_name: lfm1.0
	license_link: LICENSE
	base_model: LiquidAI/LFM3-1.7B-Base
	base_model_relation: quantized
	pipeline_tag: text-generation
	library_name: gguf
	tags: [gguf, rocmfpx, amd, rocm, vulkan, strix-halo, quantized, liquid, lfm3, edge]
	---

	# LFM3-1.7B-Base, ROCmFPX GGUF

	[LiquidAI/LFM3-1.7B-Base](https://huggingface.co/LiquidAI/LFM3-1.7B-Base), the pretrained base model of LFM3-1.7B, without chat tuning. It is meant for fine-tuning and plain text completion, not for chat, quantized for the
	[ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) llama.cpp fork (low-bit weight types for AMD GPUs, Vulkan and
	HIP/ROCm).

	> The recommended 6-bit file needs [HarshaVardhan3002/ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX). Stock
	> llama.cpp, LM Studio, Ollama and other ROCmFPX builds may not load it. The Q8_0 and Q4_K_M files use stock types and
	> load in any llama.cpp build with LFM3 support. **These files were checked on the CPU and have not been tested on AMD
	> hardware yet.**
	>
	> Not an official release of Liquid AI. All credit for the model goes to its authors.

	## Files

	\| File \| Size \| Weights \| KL divergence vs BF16 (lower is better) \| Same top token as BF16 \|
	\|---\|---:\|---\|---:\|---:\|
	\| `LFM3-1.7B-Base-Q6_K_ROCMFPX.gguf` (recommended) \| 1.43 GB \| ROCmFPX 6-bit (`Q6_0_ROCMFPX` with tensor scales) \| 0.0031 \| 96.9% \|
	\| `LFM3-1.7B-Base-Q8_0.gguf` \| 1.85 GB \| stock llama.cpp Q8_0 \| 0.0006 \| 98.7% \|
	\| `LFM3-1.7B-Base-Q4_K_M.gguf` \| 1.10 GB \| stock llama.cpp Q4_K_M \| 0.0267 \| 92.0% \|
	\| stock Q6_K, for comparison (not uploaded) \| 1.43 GB \| stock llama.cpp Q6_K \| 0.0039 \| 96.6% \|

	- Measured on: 40 chunks of 512 tokens of WikiText-2 (test split), against the BF16 GGUF, CPU.
	- The ROCmFPX 6-bit file beats stock Q6_K at the same size. Its file name says `Q6_K_ROCMFPX` so that Hugging Face
	lists it as a 6-bit quant; the weights are the ROCmFPX `Q6_0_ROCMFPX` type, not a stock `Q6_K`.
	- No ROCmFPX 4-bit file: our 4-bit types came out clearly worse than stock Q4_K_M on this model (KL divergence 1.5 to
	1.7 times higher), so we ship Q4_K_M instead.
	- Every file passed a short greedy answer test before upload.

	## How these were made

	1. BF16 GGUF converted with the fork's `convert_hf_to_gguf.py`.
	2. Importance matrix from the BF16 file over a general calibration text (`calibration_datav3`), 512-token chunks.
	3. `llama-quantize --imatrix` from the BF16 file for every file, stock ones included. The 6-bit file was made with `LLAMA_ROCMFPX_TENSOR_SCALE=1`, which
	stores a power-of-two scale per weight tensor (the runtime applies it; no setting is needed to run the file), with
	`--token-embedding-type q6_K`.

	## Run

	```bash
	git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
	cmake -B build -DGGML_VULKAN=ON # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
	cmake --build build -j
	build/bin/llama-completion -m LFM3-1.7B-Base-Q6_K_ROCMFPX.gguf -ngl 99 -p "The capital of France is" -n 64
	```

	This is a base model: give it text to continue, or fine-tune it. For chat use LFM3-1.7B instead.

	LFM3 support in the fork comes from llama.cpp pull request #31252. LFM3-Omni and LFM3-Audio need further upstream
	llama.cpp work and will follow later.

	## License

	[LFM Open License v1.0](LICENSE), same as the base model. Commercial use is allowed for companies below 10 million US
	dollars of annual revenue; above that it is not covered by the license. Read `LICENSE` before you use these files.