Instructions to use xquantize/userrepgen-4b-6bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use xquantize/userrepgen-4b-6bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("xquantize/userrepgen-4b-6bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use xquantize/userrepgen-4b-6bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "xquantize/userrepgen-4b-6bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "xquantize/userrepgen-4b-6bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use xquantize/userrepgen-4b-6bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "xquantize/userrepgen-4b-6bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "xquantize/userrepgen-4b-6bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "xquantize/userrepgen-4b-6bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Hermes Agent
How to use xquantize/userrepgen-4b-6bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "xquantize/userrepgen-4b-6bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default xquantize/userrepgen-4b-6bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use xquantize/userrepgen-4b-6bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "xquantize/userrepgen-4b-6bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "xquantize/userrepgen-4b-6bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
|
Download README.md from xquantize/userrepgen-4b-6bit: direct link, hf CLI and curl.
- Browser
- Download file 3.34 kB
-
https://huggingface.co/xquantize/userrepgen-4b-6bit/resolve/main/README.md
- Command line
-
hf download hf://xquantize/userrepgen-4b-6bit/README.md
-
curl -L -o README.md https://huggingface.co/xquantize/userrepgen-4b-6bit/resolve/main/README.md
3.34 kB
| license: mit | |
| base_model: microsoft/UserRepGen-4B | |
| base_model_relation: quantized | |
| library_name: mlx | |
| pipeline_tag: text-generation | |
| tags: | |
| - mlx | |
| - user-representation | |
| - user-simulation | |
| # userrepgen-4b-6bit (MLX) | |
| 6-bit MLX conversion of [microsoft/UserRepGen-4B](https://huggingface.co/microsoft/UserRepGen-4B) for Apple Silicon. Text-only (vision weights not included). **This is the recommended version**: it matched the original bf16 model in testing at a smaller size. An [8-bit version](https://huggingface.co/xquantize/userrepgen-4b-8bit) is also available. | |
| UserRepGen-4B reads a multi-turn user–assistant conversation and outputs a JSON description of the user: their `task` and their `user_behavior`. It is not a chatbot. | |
| ## Usage | |
| The model must be prompted with the included `representation_prompt.txt` template, with thinking disabled. | |
| ```python | |
| import json | |
| from mlx_lm import load, generate | |
| from huggingface_hub import hf_hub_download | |
| repo = "xquantize/userrepgen-4b-6bit" | |
| model, tokenizer = load(repo) | |
| template = open(hf_hub_download(repo, "representation_prompt.txt")).read().strip() | |
| conversation = [ | |
| {"role": "user", "content": "why does my python script say KeyError: 'name'"}, | |
| {"role": "assistant", "content": "The key 'name' isn't in the dictionary. Can you share the code?"}, | |
| {"role": "user", "content": "oh it's nested under data['user']['name']. thanks!!"}, | |
| ] | |
| history = "\n".join(f"[{m['role']}]: {m['content']}" for m in conversation) | |
| prompt = tokenizer.apply_chat_template( | |
| [{"role": "user", "content": template.format(conversation_history=history)}], | |
| add_generation_prompt=True, enable_thinking=False, tokenize=False, | |
| ) | |
| out = generate(model, tokenizer, prompt=prompt, max_tokens=1024) | |
| print(json.loads(out)) | |
| ``` | |
| ## Evaluation | |
| Tested against the original bf16 weights (also run in MLX) on 5 varied conversations (Excel help, Python debugging, travel planning, writing edits, health question), using greedy decoding as in Microsoft's reference script. | |
| | Version | Bits/weight | Valid JSON | Outputs with repeated sentences | | |
| |---|---|---|---| | |
| | bf16 (original) | 16 | 5/5 | 0/5 | | |
| | 8-bit | 8.5 | 5/5 | 1/5 | | |
| | **6-bit (this repo)** | **6.5** | **5/5** | **0/5** | | |
| | 4-bit, group size 32 | 5.0 | 4/5 | 1/5 | | |
| | 4-bit, mixed 4/6 | 4.7 | 3/5 | 2/5 | | |
| | 4-bit (plain) | 4.5 | 3/5 | 1/5 | | |
| **4-bit is not recommended for this model.** All three 4-bit variants tested broke on at least one conversation, falling into long repetition loops that produced invalid JSON. 6-bit and 8-bit both matched bf16; the difference between them is within noise for a test this small. | |
| ## Details | |
| - **Size:** ~3.2GB download; runs comfortably on Macs with 16 GB+ unified memory | |
| - **Max input:** the original model supports prompts up to 8,192 tokens | |
| - **Converted with:** mlx-lm 0.32.0 | |
| `mlx_lm.convert --hf-path microsoft/UserRepGen-4B -q --q-bits 6` | |
| ## Limitations | |
| The evaluation above is a small sanity check (5 conversations, greedy decoding), not a full benchmark. For the model's intended use, training details and known limitations, see the [original model card](https://huggingface.co/microsoft/UserRepGen-4B). | |
| ## Credits | |
| Original model by Microsoft, released under the MIT license. Converted with [mlx-lm](https://github.com/ml-explore/mlx-lm) by [xquantize](https://huggingface.co/xquantize). | |