Instructions to use VladHong/llama-cpp-K2-FAST2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use VladHong/llama-cpp-K2-FAST2 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf VladHong/llama-cpp-K2-FAST2 # Run inference directly in the terminal: llama cli -hf VladHong/llama-cpp-K2-FAST2
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf VladHong/llama-cpp-K2-FAST2 # Run inference directly in the terminal: llama cli -hf VladHong/llama-cpp-K2-FAST2
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf VladHong/llama-cpp-K2-FAST2 # Run inference directly in the terminal: ./build/bin/llama cli -hf VladHong/llama-cpp-K2-FAST2
Use Docker
docker model run hf.co/VladHong/llama-cpp-K2-FAST2
- Llama new
- LM Studio
- Jan
- Ollama
How to use VladHong/llama-cpp-K2-FAST2 with Ollama:
ollama run hf.co/VladHong/llama-cpp-K2-FAST2
- Unsloth Desktop
- Docker Model Runner
How to use VladHong/llama-cpp-K2-FAST2 with Docker Model Runner:
docker model run hf.co/VladHong/llama-cpp-K2-FAST2
- Lemonade
How to use VladHong/llama-cpp-K2-FAST2 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull VladHong/llama-cpp-K2-FAST2
Run and chat with the model
lemonade run user.llama-cpp-K2-FAST2-{{QUANT_TAG}}List all available models
lemonade list
- Atomic Chat
|
Download docs/multimodal/glmedge.md from VladHong/llama-cpp-K2-FAST2: direct link, hf CLI and curl.
- Browser
- Download file 1.52 kB
-
https://huggingface.co/VladHong/llama-cpp-K2-FAST2/resolve/main/docs/multimodal/glmedge.md
- Command line
-
hf download hf://VladHong/llama-cpp-K2-FAST2/docs/multimodal/glmedge.md
-
curl -L -o glmedge.md https://huggingface.co/VladHong/llama-cpp-K2-FAST2/resolve/main/docs/multimodal/glmedge.md
1.52 kB
| # GLMV-EDGE | |
| Currently this implementation supports [glm-edge-v-2b](https://huggingface.co/THUDM/glm-edge-v-2b) and [glm-edge-v-5b](https://huggingface.co/THUDM/glm-edge-v-5b). | |
| ## Usage | |
| Build the `llama-mtmd-cli` binary. | |
| After building, run: `./llama-mtmd-cli` to see the usage. For example: | |
| ```sh | |
| ./llama-mtmd-cli -m model_path/ggml-model-f16.gguf --mmproj model_path/mmproj-model-f16.gguf | |
| ``` | |
| **note**: A lower temperature like 0.1 is recommended for better quality. add `--temp 0.1` to the command to do so. | |
| **note**: For GPU offloading ensure to use the `-ngl` flag just like usual | |
| ## GGUF conversion | |
| 1. Clone a GLMV-EDGE model ([2B](https://huggingface.co/THUDM/glm-edge-v-2b) or [5B](https://huggingface.co/THUDM/glm-edge-v-5b)). For example: | |
| ```sh | |
| git clone https://huggingface.co/THUDM/glm-edge-v-5b or https://huggingface.co/THUDM/glm-edge-v-2b | |
| ``` | |
| 2. Use `glmedge-surgery.py` to split the GLMV-EDGE model to LLM and multimodel projector constituents: | |
| ```sh | |
| python ./tools/mtmd/glmedge-surgery.py -m ../model_path | |
| ``` | |
| 4. Use `glmedge-convert-image-encoder-to-gguf.py` to convert the GLMV-EDGE image encoder to GGUF: | |
| ```sh | |
| python ./tools/mtmd/glmedge-convert-image-encoder-to-gguf.py -m ../model_path --llava-projector ../model_path/glm.projector --output-dir ../model_path | |
| ``` | |
| 5. Use `examples/convert_hf_to_gguf.py` to convert the LLM part of GLMV-EDGE to GGUF: | |
| ```sh | |
| python convert_hf_to_gguf.py ../model_path | |
| ``` | |
| Now both the LLM part and the image encoder are in the `model_path` directory. | |