xquantize commited on
Commit
5e067b4
·
verified ·
1 Parent(s): 48f7f3b

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +14 -2
README.md CHANGED
@@ -10,7 +10,7 @@ tags:
10
  - user-simulation
11
  ---
12
 
13
- # UserRepGen-4B-8bit (MLX)
14
 
15
  8-bit MLX conversion of [microsoft/UserRepGen-4B](https://huggingface.co/microsoft/UserRepGen-4B) for Apple Silicon. Text-only (vision weights not included).
16
 
@@ -25,7 +25,7 @@ import json
25
  from mlx_lm import load, generate
26
  from huggingface_hub import hf_hub_download
27
 
28
- repo = "xquantize/UserRepGen-4B-8bit"
29
  model, tokenizer = load(repo)
30
  template = open(hf_hub_download(repo, "representation_prompt.txt")).read().strip()
31
 
@@ -56,6 +56,18 @@ Tested against the original bf16 weights (also run in MLX) on 5 varied conversat
56
 
57
  The 8-bit closely matches bf16. In one case (the Excel example) it repeated a sentence near the end of `user_behavior`, but the output was still valid JSON. Plain 4-bit failed JSON validation on 2/5 conversations, so it is not published; better 4-bit variants are being tested.
58
 
 
 
 
 
 
 
 
 
 
 
 
 
59
  ## Credits
60
 
61
  Original model by Microsoft, released under the MIT license. Converted with [mlx-lm](https://github.com/ml-explore/mlx-lm) by [xquantize](https://huggingface.co/xquantize).
 
10
  - user-simulation
11
  ---
12
 
13
+ # userrepgen-4b-8bit (MLX)
14
 
15
  8-bit MLX conversion of [microsoft/UserRepGen-4B](https://huggingface.co/microsoft/UserRepGen-4B) for Apple Silicon. Text-only (vision weights not included).
16
 
 
25
  from mlx_lm import load, generate
26
  from huggingface_hub import hf_hub_download
27
 
28
+ repo = "xquantize/userrepgen-4b-8bit"
29
  model, tokenizer = load(repo)
30
  template = open(hf_hub_download(repo, "representation_prompt.txt")).read().strip()
31
 
 
56
 
57
  The 8-bit closely matches bf16. In one case (the Excel example) it repeated a sentence near the end of `user_behavior`, but the output was still valid JSON. Plain 4-bit failed JSON validation on 2/5 conversations, so it is not published; better 4-bit variants are being tested.
58
 
59
+ ## Details
60
+
61
+ - **Size:** ~4.2GB download; runs comfortably on Macs with 16 GB+ unified memory
62
+ - **Speed:** ~17 s for a full representation of an 8-turn conversation on an M3 Pro (18 GB)
63
+ - **Max input:** the original model supports prompts up to 8,192 tokens
64
+ - **Converted with:** mlx-lm 0.32.0
65
+ `mlx_lm.convert --hf-path microsoft/UserRepGen-4B -q --q-bits 8`
66
+
67
+ ## Limitations
68
+
69
+ The evaluation above is a small sanity check (5 conversations, greedy decoding), not a full benchmark. For the model's intended use, training details and known limitations, see the [original model card](https://huggingface.co/microsoft/UserRepGen-4B).
70
+
71
  ## Credits
72
 
73
  Original model by Microsoft, released under the MIT license. Converted with [mlx-lm](https://github.com/ml-explore/mlx-lm) by [xquantize](https://huggingface.co/xquantize).