Text Generation
GGUF
vq3
quantization
llama.cpp
cpu
avx512-vnni
amx
Mixture of Experts
flint
nvfp4
conversational
Instructions to use HelloSun/flint_cpu_opt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use HelloSun/flint_cpu_opt with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL # Run inference directly in the terminal: ./build/bin/llama cli -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
Use Docker
docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
- Llama new
- LM Studio
- Jan
- vLLM
How to use HelloSun/flint_cpu_opt with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "HelloSun/flint_cpu_opt" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "HelloSun/flint_cpu_opt", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
- Ollama
How to use HelloSun/flint_cpu_opt with Ollama:
ollama run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
- Unsloth Desktop
- Pi
How to use HelloSun/flint_cpu_opt with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "HelloSun/flint_cpu_opt:UD-Q4_K_XL" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use HelloSun/flint_cpu_opt with Docker Model Runner:
docker model run hf.co/HelloSun/flint_cpu_opt:UD-Q4_K_XL
- Lemonade
How to use HelloSun/flint_cpu_opt with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull HelloSun/flint_cpu_opt:UD-Q4_K_XL
Run and chat with the model
lemonade run user.flint_cpu_opt-UD-Q4_K_XL
List all available models
lemonade list
- Hermes Agent
How to use HelloSun/flint_cpu_opt with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default HelloSun/flint_cpu_opt:UD-Q4_K_XL
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use HelloSun/flint_cpu_opt with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf HelloSun/flint_cpu_opt:UD-Q4_K_XL
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "HelloSun/flint_cpu_opt:UD-Q4_K_XL" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Download benchmarks/profile-prefill.txt from HelloSun/flint_cpu_opt: direct link, hf CLI and curl.
- Browser
- Download file 8.52 kB
-
https://huggingface.co/HelloSun/flint_cpu_opt/resolve/main/benchmarks/profile-prefill.txt
- Command line
-
hf download hf://HelloSun/flint_cpu_opt/benchmarks/profile-prefill.txt
-
curl -L -o profile-prefill.txt https://huggingface.co/HelloSun/flint_cpu_opt/resolve/main/benchmarks/profile-prefill.txt
8.52 kB
| | model | size | params | backend | threads | n_batch | n_ubatch | fa | lm | test | t/s | | |
| | ------------------------------ | ---------: | ---------: | ---------- | ------: | ------: | -------: | --: | ---------: | --------------: | -------------------: | | |
| | flint ?B Q4_K - Medium | 23.33 GiB | 39.79 B | CPU | 16 | 1024 | 1024 | 1 | mlock | pp2048 | 309.71 ± 0.00 | | |
| [GGML_PROF] total_us=13581874 | |
| [GGML_PROF] GET_ROWS f32 <q8_0 [ 2048, 1024, 1] 5.90 ms 0.04% calls=4 us/call=1476.0 | |
| [GGML_PROF] RMS_NORM f32 <f32 [ 2048, 1024, 1] 125.01 ms 0.92% calls=388 us/call=322.2 | |
| [GGML_PROF] MUL_MAT f32 <q8_0 [ 4096, 1024, 1] 732.99 ms 5.40% calls=168 us/call=4363.0 | |
| [GGML_PROF] RMS_NORM f32 <f32 [ 128, 32,1024] 139.90 ms 1.03% calls=196 us/call=713.8 | |
| [GGML_PROF] MUL_MAT f32 <q8_0 [ 512, 1024, 1] 245.74 ms 1.81% calls=336 us/call=731.4 | |
| [GGML_PROF] RMS_NORM f32 <f32 [ 128, 4,1024] 50.24 ms 0.37% calls=196 us/call=256.3 | |
| [GGML_PROF] SET_ROWS f16 <f32 [ 512, 2048, 1] 19.08 ms 0.14% calls=112 us/call=170.4 | |
| [GGML_PROF] FLASH_ATTN_EXT f32 <f32 [ 128, 32,1024] 3199.22 ms 23.56% calls=196 us/call=16322.6 | |
| [GGML_PROF] MUL_MAT f32 <q8_0 [ 2048, 1024, 1] 948.76 ms 6.99% calls=308 us/call=3080.4 | |
| [GGML_PROF] ADD f32 <f32 [ 2048, 1024, 1] 88.97 ms 0.66% calls=1488 us/call=59.8 | |
| [GGML_PROF] MUL_MAT f32 <q8_0 [ 6144, 1024, 1] 138.57 ms 1.02% calls=16 us/call=8660.9 | |
| [GGML_PROF] GLU f32 <f32 [ 6144, 1024, 1] 12.83 ms 0.09% calls=8 us/call=1603.5 | |
| [GGML_PROF] ROPE f32 <f32 [ 128, 32,1024] 42.53 ms 0.31% calls=140 us/call=303.8 | |
| [GGML_PROF] ROPE f32 <f32 [ 128, 4,1024] 10.87 ms 0.08% calls=140 us/call=77.6 | |
| [GGML_PROF] SET_ROWS f16 <f32 [ 512, 1792, 1] 3.89 ms 0.03% calls=280 us/call=13.9 | |
| [GGML_PROF] MUL_MAT f32 <bf16 [ 256, 1024, 1] 197.14 ms 1.45% calls=184 us/call=1071.4 | |
| [GGML_PROF] UNARY f32 <f32 [ 256, 1024, 1] 11.94 ms 0.09% calls=184 us/call=64.9 | |
| [GGML_PROF] ARGSORT i32 <f32 [ 256, 1024, 1] 150.49 ms 1.11% calls=184 us/call=817.9 | |
| [GGML_PROF] GET_ROWS f32 <f32 [ 1, 6,1024] 2.21 ms 0.02% calls=184 us/call=12.0 | |
| [GGML_PROF] SUM_ROWS f32 <f32 [ 1, 1024, 1] 1.02 ms 0.01% calls=184 us/call=5.6 | |
| [GGML_PROF] CLAMP f32 <f32 [ 1, 1024, 1] 1.94 ms 0.01% calls=184 us/call=10.5 | |
| [GGML_PROF] DIV f32 <f32 [ 6, 1024, 1] 0.65 ms 0.00% calls=184 us/call=3.5 | |
| [GGML_PROF] MUL_MAT_ID f32 <iq3_s [ 512, 6,1024] 111.13 ms 0.82% calls=8 us/call=13891.1 | |
| [GGML_PROF] GLU f32 <f32 [ 512, 6,1024] 41.05 ms 0.30% calls=184 us/call=223.1 | |
| [GGML_PROF] UNARY f32 <f32 [ 512, 6,1024] 14.72 ms 0.11% calls=184 us/call=80.0 | |
| [GGML_PROF] POOL_1D f32 <f32 [ 1, 6,1024] 717.08 ms 5.28% calls=184 us/call=3897.2 | |
| [GGML_PROF] SCALE f32 <f32 [ 1, 6,1024] 6.38 ms 0.05% calls=552 us/call=11.6 | |
| [GGML_PROF] LOG f32 <f32 [ 1, 6,1024] 1.32 ms 0.01% calls=184 us/call=7.2 | |
| [GGML_PROF] UNARY f32 <f32 [ 1, 6,1024] 1.79 ms 0.01% calls=368 us/call=4.9 | |
| [GGML_PROF] CLAMP f32 <f32 [ 1, 6,1024] 8.17 ms 0.06% calls=184 us/call=44.4 | |
| [GGML_PROF] DIV f32 <f32 [ 512, 6,1024] 48.76 ms 0.36% calls=184 us/call=265.0 | |
| [GGML_PROF] MUL_MAT_ID f32 <q5_K [ 2048, 6,1024] 1450.95 ms 10.68% calls=116 us/call=12508.2 | |
| [GGML_PROF] MUL f32 <f32 [ 2048, 6,1024] 388.11 ms 2.86% calls=368 us/call=1054.7 | |
| [GGML_PROF] MUL_MAT f32 <q8_0 [ 1024, 1024, 1] 292.28 ms 2.15% calls=264 us/call=1107.1 | |
| [GGML_PROF] GLU f32 <f32 [ 1024, 1024, 1] 9.64 ms 0.07% calls=184 us/call=52.4 | |
| [GGML_PROF] MUL_MAT_ID f32 <q5_K [ 512, 6,1024] 1415.40 ms 10.42% calls=144 us/call=9829.2 | |
| [GGML_PROF] MUL_MAT_ID f32 <q8_0 [ 2048, 6,1024] 105.06 ms 0.77% calls=4 us/call=26264.0 | |
| [GGML_PROF] MUL_MAT_ID f32 <q6_K [ 2048, 6,1024] 62.41 ms 0.46% calls=4 us/call=15601.8 | |
| [GGML_PROF] MUL_MAT f32 <q5_K [ 4096, 1024, 1] 140.26 ms 1.03% calls=28 us/call=5009.4 | |
| [GGML_PROF] MUL_MAT f32 <q5_K [ 512, 1024, 1] 48.46 ms 0.36% calls=56 us/call=865.3 | |
| [GGML_PROF] MUL_MAT f32 <q5_K [ 2048, 1024, 1] 215.82 ms 1.59% calls=76 us/call=2839.8 | |
| [GGML_PROF] MUL_MAT_ID f32 <iq4_xs [ 512, 6,1024] 1376.81 ms 10.14% calls=192 us/call=7170.9 | |
| [GGML_PROF] MUL_MAT_ID f32 <iq4_xs [ 2048, 6,1024] 631.64 ms 4.65% calls=60 us/call=10527.4 | |
| [GGML_PROF] MUL_MAT_ID f32 <q4_K [ 512, 6,1024] 194.15 ms 1.43% calls=24 us/call=8089.7 | |
| [GGML_PROF] MUL_MAT f32 <q5_K [ 1024, 1024, 1] 146.41 ms 1.08% calls=96 us/call=1525.1 | |
| [GGML_PROF] MUL_MAT f32 <iq4_xs [ 1024, 1024, 1] 10.24 ms 0.08% calls=8 us/call=1280.6 | |
| [GGML_PROF] MUL_MAT f32 <iq4_xs [ 2048, 1024, 1] 4.50 ms 0.03% calls=4 us/call=1126.0 | |
| [GGML_PROF] GET_ROWS f32 <f32 [ 2048, 1, 1] 0.02 ms 0.00% calls=8 us/call=2.1 | |
| [GGML_PROF] ADD f32 <f32 [ 2048, 1, 1] 0.05 ms 0.00% calls=32 us/call=1.5 | |
| [GGML_PROF] RMS_NORM f32 <f32 [ 2048, 1, 1] 0.04 ms 0.00% calls=8 us/call=4.5 | |
| [GGML_PROF] MUL_MAT f32 <bf16 [ 256, 1, 1] 0.07 ms 0.00% calls=4 us/call=16.8 | |
| [GGML_PROF] UNARY f32 <f32 [ 256, 1, 1] 0.01 ms 0.00% calls=4 us/call=2.5 | |
| [GGML_PROF] ARGSORT i32 <f32 [ 256, 1, 1] 0.06 ms 0.00% calls=4 us/call=13.8 | |
| [GGML_PROF] GET_ROWS f32 <f32 [ 1, 6, 1] 0.01 ms 0.00% calls=4 us/call=1.2 | |
| [GGML_PROF] SUM_ROWS f32 <f32 [ 1, 1, 1] 0.01 ms 0.00% calls=4 us/call=1.5 | |
| [GGML_PROF] CLAMP f32 <f32 [ 1, 1, 1] 0.00 ms 0.00% calls=4 us/call=1.0 | |
| [GGML_PROF] DIV f32 <f32 [ 6, 1, 1] 0.00 ms 0.00% calls=4 us/call=1.0 | |
| [GGML_PROF] MUL_MAT_ID f32 <iq4_xs [ 512, 6, 1] 0.25 ms 0.00% calls=8 us/call=30.9 | |
| [GGML_PROF] GLU f32 <f32 [ 512, 6, 1] 0.01 ms 0.00% calls=4 us/call=2.2 | |
| [GGML_PROF] UNARY f32 <f32 [ 512, 6, 1] 0.01 ms 0.00% calls=4 us/call=1.5 | |
| [GGML_PROF] POOL_1D f32 <f32 [ 1, 6, 1] 0.02 ms 0.00% calls=4 us/call=5.8 | |
| [GGML_PROF] SCALE f32 <f32 [ 1, 6, 1] 0.01 ms 0.00% calls=12 us/call=1.1 | |
| [GGML_PROF] LOG f32 <f32 [ 1, 6, 1] 0.01 ms 0.00% calls=4 us/call=1.8 | |
| [GGML_PROF] UNARY f32 <f32 [ 1, 6, 1] 0.01 ms 0.00% calls=8 us/call=1.4 | |
| [GGML_PROF] CLAMP f32 <f32 [ 1, 6, 1] 0.00 ms 0.00% calls=4 us/call=1.0 | |
| [GGML_PROF] DIV f32 <f32 [ 512, 6, 1] 0.01 ms 0.00% calls=4 us/call=1.2 | |
| [GGML_PROF] MUL_MAT_ID f32 <iq4_xs [ 2048, 6, 1] 0.14 ms 0.00% calls=4 us/call=35.0 | |
| [GGML_PROF] MUL f32 <f32 [ 2048, 6, 1] 0.03 ms 0.00% calls=8 us/call=3.5 | |
| [GGML_PROF] MUL_MAT f32 <q8_0 [ 1024, 1, 1] 0.15 ms 0.00% calls=8 us/call=18.6 | |
| [GGML_PROF] GLU f32 <f32 [ 1024, 1, 1] 0.01 ms 0.00% calls=4 us/call=2.0 | |
| [GGML_PROF] MUL_MAT f32 <q8_0 [ 2048, 1, 1] 0.07 ms 0.00% calls=4 us/call=17.5 | |
| [GGML_PROF] MUL_MAT f32 <q8_0 [150144, 1, 1] 8.43 ms 0.06% calls=4 us/call=2106.2 | |
| build: de5122d87 (1) | |