xquantize commited on
Commit
9b11b8f
·
verified ·
1 Parent(s): 5e067b4

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +10 -7
README.md CHANGED
@@ -12,7 +12,7 @@ tags:
12
 
13
  # userrepgen-4b-8bit (MLX)
14
 
15
- 8-bit MLX conversion of [microsoft/UserRepGen-4B](https://huggingface.co/microsoft/UserRepGen-4B) for Apple Silicon. Text-only (vision weights not included).
16
 
17
  UserRepGen-4B reads a multi-turn user–assistant conversation and outputs a JSON description of the user: their `task` and their `user_behavior`. It is not a chatbot.
18
 
@@ -48,13 +48,16 @@ print(json.loads(out))
48
 
49
  Tested against the original bf16 weights (also run in MLX) on 5 varied conversations (Excel help, Python debugging, travel planning, writing edits, health question), using greedy decoding as in Microsoft's reference script.
50
 
51
- | Version | Valid JSON | Outputs with repeated sentences |
52
- |---|---|---|
53
- | bf16 (original) | 5/5 | 0/5 |
54
- | **8-bit (this repo)** | **5/5** | 1/5 |
55
- | 4-bit (plain) | 3/5 | 1/5 |
 
 
 
56
 
57
- The 8-bit closely matches bf16. In one case (the Excel example) it repeated a sentence near the end of `user_behavior`, but the output was still valid JSON. Plain 4-bit failed JSON validation on 2/5 conversations, so it is not published; better 4-bit variants are being tested.
58
 
59
  ## Details
60
 
 
12
 
13
  # userrepgen-4b-8bit (MLX)
14
 
15
+ 8-bit MLX conversion of [microsoft/UserRepGen-4B](https://huggingface.co/microsoft/UserRepGen-4B) for Apple Silicon. Text-only (vision weights not included). A smaller [6-bit version](https://huggingface.co/xquantize/userrepgen-4b-6bit) performed just as well in testing and is the recommended choice for most users.
16
 
17
  UserRepGen-4B reads a multi-turn user–assistant conversation and outputs a JSON description of the user: their `task` and their `user_behavior`. It is not a chatbot.
18
 
 
48
 
49
  Tested against the original bf16 weights (also run in MLX) on 5 varied conversations (Excel help, Python debugging, travel planning, writing edits, health question), using greedy decoding as in Microsoft's reference script.
50
 
51
+ | Version | Bits/weight | Valid JSON | Outputs with repeated sentences |
52
+ |---|---|---|---|
53
+ | bf16 (original) | 16 | 5/5 | 0/5 |
54
+ | **8-bit (this repo)** | **8.5** | **5/5** | **1/5** |
55
+ | 6-bit | 6.5 | 5/5 | 0/5 |
56
+ | 4-bit, group size 32 | 5.0 | 4/5 | 1/5 |
57
+ | 4-bit, mixed 4/6 | 4.7 | 3/5 | 2/5 |
58
+ | 4-bit (plain) | 4.5 | 3/5 | 1/5 |
59
 
60
+ **4-bit is not recommended for this model.** All three 4-bit variants tested broke on at least one conversation, falling into long repetition loops that produced invalid JSON. 6-bit and 8-bit both matched bf16; the 8-bit's one repeated sentence (on the Excel example) still produced valid JSON, and the difference between 6-bit and 8-bit is within noise for a test this small.
61
 
62
  ## Details
63