Instructions to use telosnex/fllama with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use telosnex/fllama with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf telosnex/fllama:UD-Q4_K_M # Run inference directly in the terminal: llama cli -hf telosnex/fllama:UD-Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf telosnex/fllama:UD-Q4_K_M # Run inference directly in the terminal: llama cli -hf telosnex/fllama:UD-Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf telosnex/fllama:UD-Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf telosnex/fllama:UD-Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf telosnex/fllama:UD-Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf telosnex/fllama:UD-Q4_K_M
Use Docker
docker model run hf.co/telosnex/fllama:UD-Q4_K_M
- LM Studio
- Jan
- Ollama
How to use telosnex/fllama with Ollama:
ollama run hf.co/telosnex/fllama:UD-Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use telosnex/fllama with Docker Model Runner:
docker model run hf.co/telosnex/fllama:UD-Q4_K_M
- Lemonade
How to use telosnex/fllama with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull telosnex/fllama:UD-Q4_K_M
Run and chat with the model
lemonade run user.fllama-UD-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Commit History
-
Upload 2 files
Verified signaturefb22ecb -
Upload Qwen3.8-27B-mmproj-F16.gguf
Verified signature877e9f5 -
Upload Qwen3.8-27B-Q4_K_M.gguf
Verified signature334fc67
-
Upload gemma-4-12B-it-qat-q4_0-assistant-Q4_0.gguf
Verified signature2eb1f63 -
Upload 2 files
Verified signature57db305
-
Delete mmproj-gemma-4-12b-it-qat-q4_0.gguf
Verified signature52281ae -
Upload gemma-4-12b-it-qat-q4_0-it-mmproj.gguf
Verified signature376bede -
Gemma 4 QAT assistant/draft models (12B/26Ba4B, 31B)
Verified signature03adbd4 -
Upload gemma-4-31B_q4_0-it.gguf
Verified signature0ab0b8e -
Gemma 4 QAT: 12B, 26B, 31B mmproj
Verified signature213fed1
-
Gemma 4 26B-A4B
Verified signature5f9a741 -
Gemma 4 31B
Verified signaturea762267 -
Gemma 4 E2B and E4B
Verified signature3f97149
-
Qwen 3.6 35B A3B
Verified signatureee27125
-
Qwen 3.5 mmprojs (via unsloth)
Verified signaturefca96b6
-
Qwen 3.5 35B 35B-A3B via Unsloth
Verified signature85e26cd -
Qwen 3.5 27B via Unsloth
Verified signatureac7a855 -
Qwen 3.5 0.8B, 2B, 4B, 9B via Unsloth
Verified signature996229d
-
Upload sharded TinyStories for testing sharded model DL
Verified signature4e5227d -
Upload sharded DeepSeek-R1 for testing sharded model DLs.
Verified signatured5fb402
-
Upload gpt-oss-120b-mxfp4-00001-of-00003.gguf
Verified signaturef75376e -
Upload 2 files
Verified signature8f4d5d1
-
Upload HuggingFaceTB_SmolLM3-3B-Q4_0.gguf
Verified signatureBartowski's q4 with template from: https://huggingface.co/ggml-org/SmolLM3-3B-GGUF. HF staff manually updated the template at ggml-org after release; there's an extra [:] on some arrays on the original template. That used to work, but, now llama.cpp is refusing to inference SmolLM3 with an error about a vector at row X column Y. I believe using this file will fix that.e14f222
-
Upload gpt-oss-20b-mxfp4.gguf
Verified signature00524bb
-
Upload 4 files
Verified signaturedbbd9e7
-
Upload deepseek-ai_DeepSeek-R1-0528-Qwen3-8B-Q4_0.gguf
Verified signature88eddaf -
Upload DeepSeek-R1-Distill-Qwen-1.5B-Q4_0.gguf
Verified signature5b28cc3 -
Upload Llama-3.2-1B-Instruct-Q4_0.gguf
Verified signature8f6a590 -
Upload HuggingFaceTB_SmolLM3-3B-Q4_0.gguf
Verified signature4af2dd3 -
Delete Phi-3.5-mini-instruct-IQ4_XS.gguf
Verified signature2905001 -
Upload microsoft_Phi-4-mini-instruct-Q4_0.gguf
Verified signature459ecff
-
Upload SmolLM3-Q4_K_M.gguf
Verified signaturevia gguf org's repo01f6122
-
Upload deepseek-ai_DeepSeek-R1-0528-Qwen3-8B-Q4_K_M.gguf
Verified signaturevia bartowskic3e4555
-
Rename mmproj-google_gemma-3-27b-it-f32 (1).gguf to google_gemma-3-27b-it-f32.mmproj.gguf
Verified signature03b7dbf -
Upload mmproj-google_gemma-3-27b-it-f32 (1).gguf
Verified signaturea54163a -
Upload google_gemma-3-27b-it-Q4_K_M.gguf
Verified signature22ec137
-
Rename mmproj-google_gemma-3-12b-it-f16.gguf to google_gemma-3-12b-it-f16.mmproj.gguf
Verified signature778346c -
Upload google_gemma-13-12b-it and mmproj
Verified signature3a08885
-
Upload microsoft_Phi-4-mini-instruct-Q4_K_M.gguf
Verified signatureacf95e7
-
Upload DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf
Verified signaturefa5d578
-
Upload Llama-3.2-1B-Instruct-Q4_K_M.gguf
Verified signature3abf94e -
Delete tinymistral-248m-sft-v4.q2_k.gguf
Verified signaturef6a4c4b -
Delete stablelm-zephyr-3b.Q4_K_M.gguf
Verified signaturedabb7ba -
Delete dolphin-2.9-llama3-8b-1m-Q4_K_M.gguf
Verified signature7856cc8 -
Delete Phi-3.1-mini-128k-instruct-Q4_K_M.gguf
Verified signature8838517
-
Upload Phi-3.5-mini-instruct-IQ4_XS.gguf
Verified signaturevia @bartowskic8c745c
-
Upload Meta-Llama-3.1-8B-Instruct-Q4_K_M-take2.gguf
Verified signatureFrom https://huggingface.co/lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main9e05861
-
Delete Phi-3-mini-128k-instruct.Q4_K_M.gguf
Verified signature36b3f0c -
Upload Phi-3.1-mini-128k-instruct-Q4_K_M.gguf
Verified signatureb654ceb
-
Upload DistilGPT2-TinyStories.IQ3_M.gguf
Verified signatureMistral Tiny crashes in ggml assert in latest llama.cpps. This is the smallest gguf I could find that is valid. The recommend demo GGUF creation can't provide output.e6f6e6a