Instructions to use partum/terllm with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use partum/terllm with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="partum/terllm")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("partum/terllm", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use partum/terllm with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf partum/terllm:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf partum/terllm:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf partum/terllm:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf partum/terllm:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf partum/terllm:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf partum/terllm:Q4_K_M

Use Docker

docker model run hf.co/partum/terllm:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use partum/terllm with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "partum/terllm"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "partum/terllm",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/partum/terllm:Q4_K_M

SGLang

How to use partum/terllm with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "partum/terllm" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "partum/terllm",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "partum/terllm" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "partum/terllm",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use partum/terllm with Ollama:
```
ollama run hf.co/partum/terllm:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use partum/terllm with Docker Model Runner:
```
docker model run hf.co/partum/terllm:Q4_K_M
```

Lemonade

How to use partum/terllm with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull partum/terllm:Q4_K_M

Run and chat with the model

lemonade run user.terllm-Q4_K_M

List all available models

lemonade list

Atomic Chat

terllm / README.md

partum

terllm v2 and v3: SmolLM2-135M fine-tuned for shell command completion

65618fa verified 20 days ago

preview code

Raw History Blame Contribute Delete

3.55 kB

---
license: apache-2.0
base_model: HuggingFaceTB/SmolLM2-135M
library_name: transformers
pipeline_tag: text-generation
tags: [shell, terminal, autocomplete, command-line, gguf]
language: [en]
---
# terllm

Small local models that complete the shell command you are typing (fish / zsh-autosuggestions style ghost text), fine-tuned from
[SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M). Runs locally through llama.cpp (~25 ms per keystroke on CPU, Q4_K_M).
Code, data generator, zsh plugin and daemon: <https://github.com/partum55/terllm>.

## Prompt format (v3; v2 is the same without `<|fs|>`)

Plain text with added special tokens; the model continues the typed prefix and ends with `<|end|>`:

```
<|cwd|>~/proj<|ls|>README.md src<|git|>main -> origin/main | ?? x.py<|env|>VIRTUAL_ENV=.venv
<|hist|>$ git clone ...
$ cd proj<|cand|>git checkout main<|fs|>src/ scripts/<|cmd|>cd s
```

- `<|hist|>` last commands (with exit codes), `<|err|>`/`<|out|>` previous output (optional), `<|nl|>` natural-language request (optional)
- `<|cand|>` older history commands that extend the typed prefix (what zsh-autosuggestions would offer)
- `<|fs|>` (new in v3) up to 8 tab-completion candidates for the word being typed: files, directories, PATH commands. The model decides whether to
  copy one, write a new name (`mkdir`, branch names) or stop (command already complete)

The reference implementation of the prompt is `datagen/render.py` in the repository; use the daemon (`runtime/terllmd.py`) rather than building prompts by hand.
Show only the confident prefix of the completion (all tokens with probability >= 0.8); the policy is in `runtime/policy.py`.

## Models in this repo

| folder | what | files |
|---|---|---|
| `v3/` | current: adds the `<|fs|>` file/dir/command candidates | `terllm-v3-q4_k_m.gguf` (llama.cpp, 105 MB), `model.safetensors` + tokenizer (bf16) |
| `v2/` | previous: history candidates (`<|cand|>`), no `<|fs|>` — run the daemon with `--no-fs` | `terllm-v2-q4_k_m.gguf`, `model.safetensors` + tokenizer |

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("partum/terllm", subfolder="v3")
model = AutoModelForCausalLM.from_pretrained("partum/terllm", subfolder="v3")
```

## Evaluation (honest numbers)

| | v2 | v3 |
|---|---|---|
| chars saved on a real zsh history (150 commands, cwd unknown, threshold 0.8) | 15.8% | 18.4% (with `<|fs|>`) |
| exact match, synthetic path probe (fresh names) | 33% | 73.5% |
| exact match, synthetic val (no `<|fs|>` blocks) | 60.8% | 60-61% |

For reference, history-prefix matching (zsh-autosuggestions) saves 0.4% on the same commands. The real-history sample is small (about +-1-2 points) and context-poor;
it has not been validated on live usage yet. On command/tree shapes outside the training scenarios, path suggestions are noticeably weaker, and the model sometimes
ignores a single obvious candidate. Treat it as a suggestion engine: it never executes anything.

## Training data

Synthetic terminal sessions executed in a shell/git simulator (so exit codes, stderr and state are consistent), seed scenarios plus LLM-proposed sessions
(Qwen2.5-Coder, every command re-executed by the simulator), and text-to-command pairs derived from
[tldr-pages](https://github.com/tldr-pages/tldr) (CC BY 4.0, "tldr-pages team and contributors") and
[NL2Bash](https://github.com/TellinaTool/nl2bash) (MIT). Details and licenses: `data/SOURCES.md` in the repository. No personal shell history was used for training.

	---
	license: apache-2.0
	base_model: HuggingFaceTB/SmolLM2-135M
	library_name: transformers
	pipeline_tag: text-generation
	tags: [shell, terminal, autocomplete, command-line, gguf]
	language: [en]
	---

	# terllm

	Small local models that complete the shell command you are typing (fish / zsh-autosuggestions style ghost text), fine-tuned from
	[SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M). Runs locally through llama.cpp (~25 ms per keystroke on CPU, Q4_K_M).
	Code, data generator, zsh plugin and daemon: <https://github.com/partum55/terllm>.

	## Prompt format (v3; v2 is the same without `<\|fs\|>`)

	Plain text with added special tokens; the model continues the typed prefix and ends with `<\|end\|>`:

	```
	<\|cwd\|>~/proj<\|ls\|>README.md src<\|git\|>main -> origin/main \| ?? x.py<\|env\|>VIRTUAL_ENV=.venv
	<\|hist\|>$ git clone ...
	$ cd proj<\|cand\|>git checkout main<\|fs\|>src/ scripts/<\|cmd\|>cd s
	```

	- `<\|hist\|>` last commands (with exit codes), `<\|err\|>`/`<\|out\|>` previous output (optional), `<\|nl\|>` natural-language request (optional)
	- `<\|cand\|>` older history commands that extend the typed prefix (what zsh-autosuggestions would offer)
	- `<\|fs\|>` (new in v3) up to 8 tab-completion candidates for the word being typed: files, directories, PATH commands. The model decides whether to
	copy one, write a new name (`mkdir`, branch names) or stop (command already complete)

	The reference implementation of the prompt is `datagen/render.py` in the repository; use the daemon (`runtime/terllmd.py`) rather than building prompts by hand.
	Show only the confident prefix of the completion (all tokens with probability >= 0.8); the policy is in `runtime/policy.py`.

	## Models in this repo

	\| folder \| what \| files \|
	\|---\|---\|---\|
	\| `v3/` \| current: adds the `<\|fs\|>` file/dir/command candidates \| `terllm-v3-q4_k_m.gguf` (llama.cpp, 105 MB), `model.safetensors` + tokenizer (bf16) \|
	\| `v2/` \| previous: history candidates (`<\|cand\|>`), no `<\|fs\|>` — run the daemon with `--no-fs` \| `terllm-v2-q4_k_m.gguf`, `model.safetensors` + tokenizer \|

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer
	tok = AutoTokenizer.from_pretrained("partum/terllm", subfolder="v3")
	model = AutoModelForCausalLM.from_pretrained("partum/terllm", subfolder="v3")
	```

	## Evaluation (honest numbers)

	\| \| v2 \| v3 \|
	\|---\|---\|---\|
	\| chars saved on a real zsh history (150 commands, cwd unknown, threshold 0.8) \| 15.8% \| 18.4% (with `<\|fs\|>`) \|
	\| exact match, synthetic path probe (fresh names) \| 33% \| 73.5% \|
	\| exact match, synthetic val (no `<\|fs\|>` blocks) \| 60.8% \| 60-61% \|

	For reference, history-prefix matching (zsh-autosuggestions) saves 0.4% on the same commands. The real-history sample is small (about +-1-2 points) and context-poor;
	it has not been validated on live usage yet. On command/tree shapes outside the training scenarios, path suggestions are noticeably weaker, and the model sometimes
	ignores a single obvious candidate. Treat it as a suggestion engine: it never executes anything.

	## Training data

	Synthetic terminal sessions executed in a shell/git simulator (so exit codes, stderr and state are consistent), seed scenarios plus LLM-proposed sessions
	(Qwen2.5-Coder, every command re-executed by the simulator), and text-to-command pairs derived from
	[tldr-pages](https://github.com/tldr-pages/tldr) (CC BY 4.0, "tldr-pages team and contributors") and
	[NL2Bash](https://github.com/TellinaTool/nl2bash) (MIT). Details and licenses: `data/SOURCES.md` in the repository. No personal shell history was used for training.