xquantize commited on
Commit
48f7f3b
·
verified ·
1 Parent(s): c29119e

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +53 -6
README.md CHANGED
@@ -1,14 +1,61 @@
1
  ---
2
  license: mit
3
- language:
4
- - en
5
  base_model: microsoft/UserRepGen-4B
6
- pipeline_tag: text-generation
7
  library_name: mlx
 
8
  tags:
 
9
  - user-representation
10
  - user-simulation
11
- - simulation
12
- - multi-turn
13
- - mlx
14
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
 
 
3
  base_model: microsoft/UserRepGen-4B
4
+ base_model_relation: quantized
5
  library_name: mlx
6
+ pipeline_tag: text-generation
7
  tags:
8
+ - mlx
9
  - user-representation
10
  - user-simulation
 
 
 
11
  ---
12
+
13
+ # UserRepGen-4B-8bit (MLX)
14
+
15
+ 8-bit MLX conversion of [microsoft/UserRepGen-4B](https://huggingface.co/microsoft/UserRepGen-4B) for Apple Silicon. Text-only (vision weights not included).
16
+
17
+ UserRepGen-4B reads a multi-turn user–assistant conversation and outputs a JSON description of the user: their `task` and their `user_behavior`. It is not a chatbot.
18
+
19
+ ## Usage
20
+
21
+ The model must be prompted with the included `representation_prompt.txt` template, with thinking disabled.
22
+
23
+ ```python
24
+ import json
25
+ from mlx_lm import load, generate
26
+ from huggingface_hub import hf_hub_download
27
+
28
+ repo = "xquantize/UserRepGen-4B-8bit"
29
+ model, tokenizer = load(repo)
30
+ template = open(hf_hub_download(repo, "representation_prompt.txt")).read().strip()
31
+
32
+ conversation = [
33
+ {"role": "user", "content": "why does my python script say KeyError: 'name'"},
34
+ {"role": "assistant", "content": "The key 'name' isn't in the dictionary. Can you share the code?"},
35
+ {"role": "user", "content": "oh it's nested under data['user']['name']. thanks!!"},
36
+ ]
37
+ history = "\n".join(f"[{m['role']}]: {m['content']}" for m in conversation)
38
+
39
+ prompt = tokenizer.apply_chat_template(
40
+ [{"role": "user", "content": template.format(conversation_history=history)}],
41
+ add_generation_prompt=True, enable_thinking=False, tokenize=False,
42
+ )
43
+ out = generate(model, tokenizer, prompt=prompt, max_tokens=1024)
44
+ print(json.loads(out))
45
+ ```
46
+
47
+ ## Evaluation
48
+
49
+ Tested against the original bf16 weights (also run in MLX) on 5 varied conversations (Excel help, Python debugging, travel planning, writing edits, health question), using greedy decoding as in Microsoft's reference script.
50
+
51
+ | Version | Valid JSON | Outputs with repeated sentences |
52
+ |---|---|---|
53
+ | bf16 (original) | 5/5 | 0/5 |
54
+ | **8-bit (this repo)** | **5/5** | 1/5 |
55
+ | 4-bit (plain) | 3/5 | 1/5 |
56
+
57
+ The 8-bit closely matches bf16. In one case (the Excel example) it repeated a sentence near the end of `user_behavior`, but the output was still valid JSON. Plain 4-bit failed JSON validation on 2/5 conversations, so it is not published; better 4-bit variants are being tested.
58
+
59
+ ## Credits
60
+
61
+ Original model by Microsoft, released under the MIT license. Converted with [mlx-lm](https://github.com/ml-explore/mlx-lm) by [xquantize](https://huggingface.co/xquantize).