Instructions to use Rahspide/MNN-Pocket-0.3B-RU with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Rahspide/MNN-Pocket-0.3B-RU with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Rahspide/MNN-Pocket-0.3B-RU")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Rahspide/MNN-Pocket-0.3B-RU")
model = AutoModelForCausalLM.from_pretrained("Rahspide/MNN-Pocket-0.3B-RU", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Rahspide/MNN-Pocket-0.3B-RU with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M

Use Docker

docker model run hf.co/Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use Rahspide/MNN-Pocket-0.3B-RU with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Rahspide/MNN-Pocket-0.3B-RU"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Rahspide/MNN-Pocket-0.3B-RU",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M

SGLang

How to use Rahspide/MNN-Pocket-0.3B-RU with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Rahspide/MNN-Pocket-0.3B-RU" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Rahspide/MNN-Pocket-0.3B-RU",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Rahspide/MNN-Pocket-0.3B-RU" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Rahspide/MNN-Pocket-0.3B-RU",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use Rahspide/MNN-Pocket-0.3B-RU with Ollama:
```
ollama run hf.co/Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M
```
Unsloth Desktop

How to use Rahspide/MNN-Pocket-0.3B-RU with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Rahspide/MNN-Pocket-0.3B-RU with Docker Model Runner:
```
docker model run hf.co/Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M
```

Lemonade

How to use Rahspide/MNN-Pocket-0.3B-RU with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M

Run and chat with the model

lemonade run user.MNN-Pocket-0.3B-RU-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Rahspide/MNN-Pocket-0.3B-RU with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Rahspide/MNN-Pocket-0.3B-RU with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Rahspide commited on 1 day ago

Commit

45a2b69

verified ·

1 Parent(s): dca989a

fix: weights from ckpt step 36621 (previous export was corrupted)

Browse files

Files changed (9) hide show

added_tokens.json +28 -0
config.json +4 -7
generation_config.json +1 -1
merges.txt +0 -0
model.safetensors +2 -2
special_tokens_map.json +31 -0
tokenizer.json +4 -4
tokenizer_config.json +228 -4
vocab.json +0 -0

added_tokens.json ADDED Viewed

	@@ -0,0 +1,28 @@

+{
+  "</think>": 65535,
+  "</tool_call>": 65525,
+  "</tool_response>": 65533,
+  "<think>": 65534,
+  "<tool_call>": 65524,
+  "<tool_response>": 65532,
+  "<|box_end|>": 65516,
+  "<|box_start|>": 65515,
+  "<|endoftext|>": 65510,
+  "<|file_sep|>": 65531,
+  "<|fim_middle|>": 65527,
+  "<|fim_pad|>": 65529,
+  "<|fim_prefix|>": 65526,
+  "<|fim_suffix|>": 65528,
+  "<|im_end|>": 65512,
+  "<|im_start|>": 65511,
+  "<|image_pad|>": 65522,
+  "<|object_ref_end|>": 65514,
+  "<|object_ref_start|>": 65513,
+  "<|quad_end|>": 65518,
+  "<|quad_start|>": 65517,
+  "<|repo_name|>": 65530,
+  "<|video_pad|>": 65523,
+  "<|vision_end|>": 65520,
+  "<|vision_pad|>": 65521,
+  "<|vision_start|>": 65519
+}

config.json CHANGED Viewed

@@ -4,8 +4,7 @@
   ],
   "attention_bias": false,
   "attention_dropout": 0.0,
-  "bos_token_id": null,
-  "dtype": "float16",
   "eos_token_id": 65512,
   "head_dim": 64,
   "hidden_act": "silu",
@@ -50,13 +49,11 @@
   "num_key_value_heads": 2,
   "pad_token_id": 65510,
   "rms_norm_eps": 1e-06,
-  "rope_parameters": {
-    "rope_theta": 1000000,
-    "rope_type": "default"
-  },
   "sliding_window": null,
   "tie_word_embeddings": true,
-  "transformers_version": "5.22.1",
   "use_cache": true,
   "use_sliding_window": false,
   "vocab_size": 65536

   ],
   "attention_bias": false,
   "attention_dropout": 0.0,
+  "dtype": "float32",
   "eos_token_id": 65512,
   "head_dim": 64,
   "hidden_act": "silu",
   "num_key_value_heads": 2,
   "pad_token_id": 65510,
   "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
   "sliding_window": null,
   "tie_word_embeddings": true,
+  "transformers_version": "4.57.1",
   "use_cache": true,
   "use_sliding_window": false,
   "vocab_size": 65536

generation_config.json CHANGED Viewed

@@ -4,5 +4,5 @@
     65512
   ],
   "pad_token_id": 65510,
-  "transformers_version": "5.22.1"
 }

     65512
   ],
   "pad_token_id": 65510,
+  "transformers_version": "4.57.1"
 }

merges.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:302e83920e4f94da7264a86b8c99c0aecf937cdfb4dc4ec22489889d3df84170
-size 605696336

 version https://git-lfs.github.com/spec/v1
+oid sha256:b0c402579744b17ff8d4f33aeaf816c9d27f0027257c9ac68bbc3a9a9de1cf67
+size 1211358424

special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,31 @@

+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json CHANGED Viewed

@@ -255,7 +255,7 @@
       {
         "type": "ByteLevel",
         "add_prefix_space": false,
-        "trim_offsets": true,
         "use_regex": false
       }
     ]
@@ -268,9 +268,9 @@
   },
   "decoder": {
     "type": "ByteLevel",
-    "add_prefix_space": true,
-    "trim_offsets": true,
-    "use_regex": true
   },
   "model": {
     "type": "BPE",

       {
         "type": "ByteLevel",
         "add_prefix_space": false,
+        "trim_offsets": false,
         "use_regex": false
       }
     ]
   },
   "decoder": {
     "type": "ByteLevel",
+    "add_prefix_space": false,
+    "trim_offsets": false,
+    "use_regex": false
   },
   "model": {
     "type": "BPE",

tokenizer_config.json CHANGED Viewed

@@ -1,16 +1,240 @@
 {
   "add_prefix_space": false,
-  "backend": "tokenizers",
   "bos_token": null,
   "clean_up_tokenization_spaces": false,
   "eos_token": "<|im_end|>",
   "errors": "replace",
-  "is_local": false,
-  "local_files_only": false,
   "model_max_length": 1010000,
   "pad_token": "<|endoftext|>",
   "padding_side": "right",
   "split_special_tokens": false,
   "tokenizer_class": "Qwen2Tokenizer",
   "unk_token": null
-}

 {
+  "add_bos_token": false,
   "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "65510": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65511": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65512": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65513": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65514": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65515": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65516": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65517": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65518": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65519": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65520": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65521": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65522": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65523": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65524": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65525": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65526": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65527": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65528": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65529": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65530": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65531": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65532": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65533": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65534": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65535": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
   "bos_token": null,
   "clean_up_tokenization_spaces": false,
   "eos_token": "<|im_end|>",
   "errors": "replace",
+  "extra_special_tokens": {},
   "model_max_length": 1010000,
   "pad_token": "<|endoftext|>",
   "padding_side": "right",
   "split_special_tokens": false,
   "tokenizer_class": "Qwen2Tokenizer",
   "unk_token": null
+}

vocab.json ADDED Viewed

The diff for this file is too large to render. See raw diff