Buckets:
| PROMPT CRITIC: SETTING UP A LOCAL LLM | |
| ===================================== | |
| Prompt Critic does not contain a language model. It talks to one that runs on your | |
| own computer, through the same kind of address ChatGPT-style programs use | |
| ("OpenAI-compatible"). Any of the three programs below gives you such an address. | |
| Pick one. | |
| What to expect: the model shares your graphics card with the image model. A small | |
| model (around 4B, file size 3 GB or so) is enough for fixing tags and is quick. If | |
| generating pictures gets slow or runs out of memory while the LLM is loaded, close | |
| the LLM program while you generate, or let it run on the CPU. | |
| A. LM STUDIO (no command line) | |
| ------------------------------- | |
| 1. Install LM Studio from lmstudio.ai and start it. | |
| 2. Use its search to download a chat model in GGUF format. Any recent "instruct" | |
| model works; start with a small one (4B to 8B, "Q4_K_M"). | |
| 3. Open the Developer page (older versions: "Local Server"), load the model there | |
| and start the server. It shows its address, normally http://localhost:1234 | |
| 4. In Krita: Settings > Dockers > Prompt Critic. In the docker's settings put | |
| Endpoint: http://localhost:1234/v1/chat/completions | |
| press the reload button next to "Model" and pick the model that appears. | |
| B. LLAMA.CPP (llama-server; the one with working picture input) | |
| ---------------------------------------------------------------- | |
| 1. Download llama.cpp for Windows from github.com/ggml-org/llama.cpp/releases | |
| (the zip with "cuda" in its name for an NVIDIA card) and extract it. | |
| 2. Download a model as a .gguf file (huggingface.co, search for the model name plus | |
| "GGUF"). For "Prompt vs canvas" also take the file with "mmproj" in its name from | |
| the same page. | |
| 3. Make a text file start-llm.bat next to llama-server.exe with one line | |
| (change the two paths): | |
| llama-server.exe -m "C:\path\to\model.gguf" --mmproj "C:\path\to\mmproj.gguf" -ngl 99 -c 16384 --port 8080 | |
| Leave out the --mmproj part if you have no such file. Double-click the .bat. | |
| -ngl 99 puts the model on the graphics card (-ngl 0 = CPU only). | |
| -c 16384 is how much text it can keep in mind; raise it if long sessions get cut. | |
| 4. In Prompt Critic the Endpoint is already right: | |
| http://localhost:8080/v1/chat/completions | |
| Press the reload button next to "Model". | |
| C. OLLAMA (simple, text only for most models) | |
| ---------------------------------------------- | |
| 1. Install Ollama from ollama.com. | |
| 2. In its window or a terminal: ollama pull <model name from ollama.com/library> | |
| 3. In Prompt Critic: | |
| Endpoint: http://localhost:11434/v1/chat/completions | |
| Press the reload button next to "Model". | |
| IF IT DOES NOT ANSWER | |
| --------------------- | |
| "Is the LLM server running?" The program from A, B or C is not started, or the | |
| address in Endpoint has another port number. | |
| The Model list stays empty Same thing; or no model is loaded in the program yet. | |
| Answers take minutes The model is too big for the free memory of the | |
| graphics card. Use a smaller one, or close it while | |
| you generate pictures. | |
| It writes a long "thinking" text Tick "Disable thinking" in the docker's settings. | |
| "Prompt vs canvas" says the model The model has no picture input. Use route B with the | |
| cannot see images mmproj file, or a vision model. | |
Xet Storage Details
- Size:
- 3.51 kB
- Xet hash:
- b8fd2ff599b333fc8fd79538e11ce65ba5314f9feda24e7ba56aa50d13cb1021
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.