How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Samzzzed/dolphin-mistral-7b:Q4_0# Run inference directly in the terminal:
llama cli -hf Samzzzed/dolphin-mistral-7b:Q4_0Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Samzzzed/dolphin-mistral-7b:Q4_0# Run inference directly in the terminal:
./llama-cli -hf Samzzzed/dolphin-mistral-7b:Q4_0Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Samzzzed/dolphin-mistral-7b:Q4_0# Run inference directly in the terminal:
./build/bin/llama-cli -hf Samzzzed/dolphin-mistral-7b:Q4_0Use Docker
docker model run hf.co/Samzzzed/dolphin-mistral-7b:Q4_0Quick Links
dolphin-mistral-7b (JARVIS mimic)
Original Dolphin-Mistral 7B v2.8 Q4_0 GGUF weights (same family as Ollama dolphin-mistral:7b used by the JARVIS FastAPI app), plus a FastAPI-aligned IMDb/MPAA multi-level system prompt.
Weights are not re-trained (no GPU LoRA here). Behavior is updated the same way the Python app does: ChatML template + rating ladder.
What changed vs stock Ollama
| Layer | Stock dolphin-mistral:7b |
This mimic |
|---|---|---|
| GGUF | Q4_0 7B, 32k ctx | Same v2.8 Q4_0 defaults |
| SYSTEM | "You are Dolphin, a helpful AI assistant." | FastAPI PROMPTS G / PG / PG-13 / R / NC-17 |
| Extra | none | Infer IMDb-style rating, emit IMDB_RATING:, then answer at that level. FastAPI rating is a ceiling. |
Use with Ollama
ollama create dolphin-mistral-imdb:7b -f Modelfile
ollama run dolphin-mistral-imdb:7b
Python helper: fastapi_imdb_prompts.py (build_ollama_payload).
Provenance
- GGUF:
bartowski/dolphin-2.8-mistral-7b-v02-GGUFfiledolphin-2.8-mistral-7b-v02-Q4_0.gguf - Upstream:
cognitivecomputations/dolphin-2.8-mistral-7b-v02(Apache-2.0) - Prompt ladder: JARVIS_APP FastAPI
main.pyPROMPTS/build_ollama_payload
- Downloads last month
- 62
Hardware compatibility
Log In to add your hardware
4-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for Samzzzed/dolphin-mistral-7b
Base model
mistral-community/Mistral-7B-v0.2 Finetuned
dphn/dolphin-2.8-mistral-7b-v02
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf Samzzzed/dolphin-mistral-7b:Q4_0# Run inference directly in the terminal: llama cli -hf Samzzzed/dolphin-mistral-7b:Q4_0