cowpuncher's picture
download
raw
3.51 kB
PROMPT CRITIC: SETTING UP A LOCAL LLM
=====================================
Prompt Critic does not contain a language model. It talks to one that runs on your
own computer, through the same kind of address ChatGPT-style programs use
("OpenAI-compatible"). Any of the three programs below gives you such an address.
Pick one.
What to expect: the model shares your graphics card with the image model. A small
model (around 4B, file size 3 GB or so) is enough for fixing tags and is quick. If
generating pictures gets slow or runs out of memory while the LLM is loaded, close
the LLM program while you generate, or let it run on the CPU.
A. LM STUDIO (no command line)
-------------------------------
1. Install LM Studio from lmstudio.ai and start it.
2. Use its search to download a chat model in GGUF format. Any recent "instruct"
model works; start with a small one (4B to 8B, "Q4_K_M").
3. Open the Developer page (older versions: "Local Server"), load the model there
and start the server. It shows its address, normally http://localhost:1234
4. In Krita: Settings > Dockers > Prompt Critic. In the docker's settings put
Endpoint: http://localhost:1234/v1/chat/completions
press the reload button next to "Model" and pick the model that appears.
B. LLAMA.CPP (llama-server; the one with working picture input)
----------------------------------------------------------------
1. Download llama.cpp for Windows from github.com/ggml-org/llama.cpp/releases
(the zip with "cuda" in its name for an NVIDIA card) and extract it.
2. Download a model as a .gguf file (huggingface.co, search for the model name plus
"GGUF"). For "Prompt vs canvas" also take the file with "mmproj" in its name from
the same page.
3. Make a text file start-llm.bat next to llama-server.exe with one line
(change the two paths):
llama-server.exe -m "C:\path\to\model.gguf" --mmproj "C:\path\to\mmproj.gguf" -ngl 99 -c 16384 --port 8080
Leave out the --mmproj part if you have no such file. Double-click the .bat.
-ngl 99 puts the model on the graphics card (-ngl 0 = CPU only).
-c 16384 is how much text it can keep in mind; raise it if long sessions get cut.
4. In Prompt Critic the Endpoint is already right:
http://localhost:8080/v1/chat/completions
Press the reload button next to "Model".
C. OLLAMA (simple, text only for most models)
----------------------------------------------
1. Install Ollama from ollama.com.
2. In its window or a terminal: ollama pull <model name from ollama.com/library>
3. In Prompt Critic:
Endpoint: http://localhost:11434/v1/chat/completions
Press the reload button next to "Model".
IF IT DOES NOT ANSWER
---------------------
"Is the LLM server running?" The program from A, B or C is not started, or the
address in Endpoint has another port number.
The Model list stays empty Same thing; or no model is loaded in the program yet.
Answers take minutes The model is too big for the free memory of the
graphics card. Use a smaller one, or close it while
you generate pictures.
It writes a long "thinking" text Tick "Disable thinking" in the docker's settings.
"Prompt vs canvas" says the model The model has no picture input. Use route B with the
cannot see images mmproj file, or a vision model.

Xet Storage Details

Size:
3.51 kB
·
Xet hash:
b8fd2ff599b333fc8fd79538e11ce65ba5314f9feda24e7ba56aa50d13cb1021

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.