File size: 3,342 Bytes
575683a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
---
license: mit
base_model: microsoft/UserRepGen-4B
base_model_relation: quantized
library_name: mlx
pipeline_tag: text-generation
tags:
- mlx
- user-representation
- user-simulation
---

# userrepgen-4b-6bit (MLX)

6-bit MLX conversion of [microsoft/UserRepGen-4B](https://huggingface.co/microsoft/UserRepGen-4B) for Apple Silicon. Text-only (vision weights not included). **This is the recommended version**: it matched the original bf16 model in testing at a smaller size. An [8-bit version](https://huggingface.co/xquantize/userrepgen-4b-8bit) is also available.

UserRepGen-4B reads a multi-turn user–assistant conversation and outputs a JSON description of the user: their `task` and their `user_behavior`. It is not a chatbot.

## Usage

The model must be prompted with the included `representation_prompt.txt` template, with thinking disabled.

```python
import json
from mlx_lm import load, generate
from huggingface_hub import hf_hub_download

repo = "xquantize/userrepgen-4b-6bit"
model, tokenizer = load(repo)
template = open(hf_hub_download(repo, "representation_prompt.txt")).read().strip()

conversation = [
    {"role": "user", "content": "why does my python script say KeyError: 'name'"},
    {"role": "assistant", "content": "The key 'name' isn't in the dictionary. Can you share the code?"},
    {"role": "user", "content": "oh it's nested under data['user']['name']. thanks!!"},
]
history = "\n".join(f"[{m['role']}]: {m['content']}" for m in conversation)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": template.format(conversation_history=history)}],
    add_generation_prompt=True, enable_thinking=False, tokenize=False,
)
out = generate(model, tokenizer, prompt=prompt, max_tokens=1024)
print(json.loads(out))
```

## Evaluation

Tested against the original bf16 weights (also run in MLX) on 5 varied conversations (Excel help, Python debugging, travel planning, writing edits, health question), using greedy decoding as in Microsoft's reference script.

| Version | Bits/weight | Valid JSON | Outputs with repeated sentences |
|---|---|---|---|
| bf16 (original) | 16 | 5/5 | 0/5 |
| 8-bit | 8.5 | 5/5 | 1/5 |
| **6-bit (this repo)** | **6.5** | **5/5** | **0/5** |
| 4-bit, group size 32 | 5.0 | 4/5 | 1/5 |
| 4-bit, mixed 4/6 | 4.7 | 3/5 | 2/5 |
| 4-bit (plain) | 4.5 | 3/5 | 1/5 |

**4-bit is not recommended for this model.** All three 4-bit variants tested broke on at least one conversation, falling into long repetition loops that produced invalid JSON. 6-bit and 8-bit both matched bf16; the difference between them is within noise for a test this small.

## Details

- **Size:** ~3.2GB download; runs comfortably on Macs with 16 GB+ unified memory
- **Max input:** the original model supports prompts up to 8,192 tokens
- **Converted with:** mlx-lm 0.32.0
  `mlx_lm.convert --hf-path microsoft/UserRepGen-4B -q --q-bits 6`

## Limitations

The evaluation above is a small sanity check (5 conversations, greedy decoding), not a full benchmark. For the model's intended use, training details and known limitations, see the [original model card](https://huggingface.co/microsoft/UserRepGen-4B).

## Credits

Original model by Microsoft, released under the MIT license. Converted with [mlx-lm](https://github.com/ml-explore/mlx-lm) by [xquantize](https://huggingface.co/xquantize).