--- license: mit base_model: microsoft/UserRepGen-4B base_model_relation: quantized library_name: mlx pipeline_tag: text-generation tags: - mlx - user-representation - user-simulation --- # userrepgen-4b-6bit (MLX) 6-bit MLX conversion of [microsoft/UserRepGen-4B](https://huggingface.co/microsoft/UserRepGen-4B) for Apple Silicon. Text-only (vision weights not included). **This is the recommended version**: it matched the original bf16 model in testing at a smaller size. An [8-bit version](https://huggingface.co/xquantize/userrepgen-4b-8bit) is also available. UserRepGen-4B reads a multi-turn user–assistant conversation and outputs a JSON description of the user: their `task` and their `user_behavior`. It is not a chatbot. ## Usage The model must be prompted with the included `representation_prompt.txt` template, with thinking disabled. ```python import json from mlx_lm import load, generate from huggingface_hub import hf_hub_download repo = "xquantize/userrepgen-4b-6bit" model, tokenizer = load(repo) template = open(hf_hub_download(repo, "representation_prompt.txt")).read().strip() conversation = [ {"role": "user", "content": "why does my python script say KeyError: 'name'"}, {"role": "assistant", "content": "The key 'name' isn't in the dictionary. Can you share the code?"}, {"role": "user", "content": "oh it's nested under data['user']['name']. thanks!!"}, ] history = "\n".join(f"[{m['role']}]: {m['content']}" for m in conversation) prompt = tokenizer.apply_chat_template( [{"role": "user", "content": template.format(conversation_history=history)}], add_generation_prompt=True, enable_thinking=False, tokenize=False, ) out = generate(model, tokenizer, prompt=prompt, max_tokens=1024) print(json.loads(out)) ``` ## Evaluation Tested against the original bf16 weights (also run in MLX) on 5 varied conversations (Excel help, Python debugging, travel planning, writing edits, health question), using greedy decoding as in Microsoft's reference script. | Version | Bits/weight | Valid JSON | Outputs with repeated sentences | |---|---|---|---| | bf16 (original) | 16 | 5/5 | 0/5 | | 8-bit | 8.5 | 5/5 | 1/5 | | **6-bit (this repo)** | **6.5** | **5/5** | **0/5** | | 4-bit, group size 32 | 5.0 | 4/5 | 1/5 | | 4-bit, mixed 4/6 | 4.7 | 3/5 | 2/5 | | 4-bit (plain) | 4.5 | 3/5 | 1/5 | **4-bit is not recommended for this model.** All three 4-bit variants tested broke on at least one conversation, falling into long repetition loops that produced invalid JSON. 6-bit and 8-bit both matched bf16; the difference between them is within noise for a test this small. ## Details - **Size:** ~3.2GB download; runs comfortably on Macs with 16 GB+ unified memory - **Max input:** the original model supports prompts up to 8,192 tokens - **Converted with:** mlx-lm 0.32.0 `mlx_lm.convert --hf-path microsoft/UserRepGen-4B -q --q-bits 6` ## Limitations The evaluation above is a small sanity check (5 conversations, greedy decoding), not a full benchmark. For the model's intended use, training details and known limitations, see the [original model card](https://huggingface.co/microsoft/UserRepGen-4B). ## Credits Original model by Microsoft, released under the MIT license. Converted with [mlx-lm](https://github.com/ml-explore/mlx-lm) by [xquantize](https://huggingface.co/xquantize).