userrepgen-4b-8bit / README.md
xquantize's picture
Upload README.md with huggingface_hub
9b11b8f verified
|
Raw History Blame Contribute Delete
3.5 kB
---
license: mit
base_model: microsoft/UserRepGen-4B
base_model_relation: quantized
library_name: mlx
pipeline_tag: text-generation
tags:
- mlx
- user-representation
- user-simulation
---
# userrepgen-4b-8bit (MLX)
8-bit MLX conversion of [microsoft/UserRepGen-4B](https://huggingface.co/microsoft/UserRepGen-4B) for Apple Silicon. Text-only (vision weights not included). A smaller [6-bit version](https://huggingface.co/xquantize/userrepgen-4b-6bit) performed just as well in testing and is the recommended choice for most users.
UserRepGen-4B reads a multi-turn user–assistant conversation and outputs a JSON description of the user: their `task` and their `user_behavior`. It is not a chatbot.
## Usage
The model must be prompted with the included `representation_prompt.txt` template, with thinking disabled.
```python
import json
from mlx_lm import load, generate
from huggingface_hub import hf_hub_download
repo = "xquantize/userrepgen-4b-8bit"
model, tokenizer = load(repo)
template = open(hf_hub_download(repo, "representation_prompt.txt")).read().strip()
conversation = [
{"role": "user", "content": "why does my python script say KeyError: 'name'"},
{"role": "assistant", "content": "The key 'name' isn't in the dictionary. Can you share the code?"},
{"role": "user", "content": "oh it's nested under data['user']['name']. thanks!!"},
]
history = "\n".join(f"[{m['role']}]: {m['content']}" for m in conversation)
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": template.format(conversation_history=history)}],
add_generation_prompt=True, enable_thinking=False, tokenize=False,
)
out = generate(model, tokenizer, prompt=prompt, max_tokens=1024)
print(json.loads(out))
```
## Evaluation
Tested against the original bf16 weights (also run in MLX) on 5 varied conversations (Excel help, Python debugging, travel planning, writing edits, health question), using greedy decoding as in Microsoft's reference script.
| Version | Bits/weight | Valid JSON | Outputs with repeated sentences |
|---|---|---|---|
| bf16 (original) | 16 | 5/5 | 0/5 |
| **8-bit (this repo)** | **8.5** | **5/5** | **1/5** |
| 6-bit | 6.5 | 5/5 | 0/5 |
| 4-bit, group size 32 | 5.0 | 4/5 | 1/5 |
| 4-bit, mixed 4/6 | 4.7 | 3/5 | 2/5 |
| 4-bit (plain) | 4.5 | 3/5 | 1/5 |
**4-bit is not recommended for this model.** All three 4-bit variants tested broke on at least one conversation, falling into long repetition loops that produced invalid JSON. 6-bit and 8-bit both matched bf16; the 8-bit's one repeated sentence (on the Excel example) still produced valid JSON, and the difference between 6-bit and 8-bit is within noise for a test this small.
## Details
- **Size:** ~4.2GB download; runs comfortably on Macs with 16 GB+ unified memory
- **Speed:** ~17 s for a full representation of an 8-turn conversation on an M3 Pro (18 GB)
- **Max input:** the original model supports prompts up to 8,192 tokens
- **Converted with:** mlx-lm 0.32.0
`mlx_lm.convert --hf-path microsoft/UserRepGen-4B -q --q-bits 8`
## Limitations
The evaluation above is a small sanity check (5 conversations, greedy decoding), not a full benchmark. For the model's intended use, training details and known limitations, see the [original model card](https://huggingface.co/microsoft/UserRepGen-4B).
## Credits
Original model by Microsoft, released under the MIT license. Converted with [mlx-lm](https://github.com/ml-explore/mlx-lm) by [xquantize](https://huggingface.co/xquantize).