Instructions to use VladHong/llama-cpp-K2-FAST2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use VladHong/llama-cpp-K2-FAST2 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf VladHong/llama-cpp-K2-FAST2 # Run inference directly in the terminal: llama cli -hf VladHong/llama-cpp-K2-FAST2
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf VladHong/llama-cpp-K2-FAST2 # Run inference directly in the terminal: llama cli -hf VladHong/llama-cpp-K2-FAST2
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf VladHong/llama-cpp-K2-FAST2 # Run inference directly in the terminal: ./build/bin/llama cli -hf VladHong/llama-cpp-K2-FAST2
Use Docker
docker model run hf.co/VladHong/llama-cpp-K2-FAST2
- Llama new
- LM Studio
- Jan
- Ollama
How to use VladHong/llama-cpp-K2-FAST2 with Ollama:
ollama run hf.co/VladHong/llama-cpp-K2-FAST2
- Unsloth Desktop
- Docker Model Runner
How to use VladHong/llama-cpp-K2-FAST2 with Docker Model Runner:
docker model run hf.co/VladHong/llama-cpp-K2-FAST2
- Lemonade
How to use VladHong/llama-cpp-K2-FAST2 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull VladHong/llama-cpp-K2-FAST2
Run and chat with the model
lemonade run user.llama-cpp-K2-FAST2-{{QUANT_TAG}}List all available models
lemonade list
- Atomic Chat
Download tools/server/server-cors-proxy.h from VladHong/llama-cpp-K2-FAST2: direct link, hf CLI and curl.
- Browser
- Download file 2.56 kB
-
https://huggingface.co/VladHong/llama-cpp-K2-FAST2/resolve/main/tools/server/server-cors-proxy.h
- Command line
-
hf download hf://VladHong/llama-cpp-K2-FAST2/tools/server/server-cors-proxy.h
-
curl -L -o server-cors-proxy.h https://huggingface.co/VladHong/llama-cpp-K2-FAST2/resolve/main/tools/server/server-cors-proxy.h
2.56 kB
| static std::string proxy_header_to_lower(std::string header) { | |
| std::transform(header.begin(), header.end(), header.begin(), [](unsigned char c) { | |
| return std::tolower(c); | |
| }); | |
| return header; | |
| } | |
| static server_http_res_ptr proxy_request(const server_http_req & req, std::string method) { | |
| std::string target_url = req.get_param("url"); | |
| common_http_url parsed_url = common_http_parse_url(target_url); | |
| if (parsed_url.host.empty()) { | |
| throw std::runtime_error("invalid target URL: missing host"); | |
| } | |
| if (parsed_url.path.empty()) { | |
| parsed_url.path = "/"; | |
| } | |
| if (!parsed_url.password.empty()) { | |
| throw std::runtime_error("authentication in target URL is not supported"); | |
| } | |
| if (parsed_url.scheme != "http" && parsed_url.scheme != "https") { | |
| throw std::runtime_error("unsupported URL scheme in target URL: " + parsed_url.scheme); | |
| } | |
| SRV_INF("proxying %s request to %s://%s:%i%s\n", method.c_str(), parsed_url.scheme.c_str(), common_http_format_host(parsed_url.host).c_str(), parsed_url.port, parsed_url.path.c_str()); | |
| std::map<std::string, std::string> headers; | |
| const std::string proxy_header_prefix = "x-llama-server-proxy-header-"; | |
| for (auto [key, value] : req.headers) { | |
| const std::string lowered_key = proxy_header_to_lower(key); | |
| if (!string_starts_with(lowered_key, proxy_header_prefix)) { | |
| continue; | |
| } | |
| auto new_key = key.substr(proxy_header_prefix.size()); | |
| if (new_key.empty()) { | |
| continue; | |
| } | |
| headers[new_key] = value; | |
| } | |
| auto proxy = std::make_unique<server_http_proxy>( | |
| method, | |
| parsed_url.scheme, | |
| parsed_url.host, | |
| parsed_url.port, | |
| parsed_url.path, | |
| headers, | |
| req.body, | |
| req.files, | |
| req.should_stop, | |
| 600, // timeout_read (default to 10 minutes) | |
| 600 // timeout_write (default to 10 minutes) | |
| ); | |
| return proxy; | |
| } | |
| static server_http_context::handler_t proxy_handler_post = [](const server_http_req & req) -> server_http_res_ptr { | |
| return proxy_request(req, "POST"); | |
| }; | |
| static server_http_context::handler_t proxy_handler_get = [](const server_http_req & req) -> server_http_res_ptr { | |
| return proxy_request(req, "GET"); | |
| }; | |