mlxtorepo / CONVERSATION.md
jesusoctavioas's picture
v6: reusable MLX conversion pipeline (venv, in-script token, prompts, cleanup menu) + full project docs
5545d68
|
Raw History Blame Contribute Delete
3.3 kB

Conversation: mlx-my-repo β†’ local MLX conversion

Context

Working dir: /Users/jesusoctavioacostasantos/git

1. What is mlx-my-repo?

Explored /Users/jesusoctavioacostasantos/git/mlx-my-repo β€” it's the code for the Hugging Face Space mlx-community/mlx-my-repo.

  • A Gradio web app that converts HF models to MLX format and uploads the result to the user's own HF namespace (e.g. youruser/llama-mlx-Q4Bit).
  • app.py: login button β†’ HuggingfaceHubSearch model picker β†’ dropdown (FP16 / Q2–Q8) β†’ mlx_lm.convert() β†’ HfApi.upload_file() per file.
  • Writes a README card with base_model, mlx tag, and usage snippet.
  • Notable: runs mlx[cpu] on Linux (HF Spaces) β€” slow, no Apple silicon.
  • Background APScheduler job reboots the Space every 6 hours (21600 s) to free memory.
  • clear_hf_cache_space() wipes the HF cache after each run (Space disk quota).
  • demo.queue(default_concurrency_limit=1) β€” one conversion at a time.

2. How to implement it on your own HF repo

Two options discussed:

  1. Local conversion (recommended): mlx_lm.convert + huggingface-cli β€” much faster on a Mac since it uses Metal.
  2. Fork the Space: huggingface-cli repo create ... --type space --sdk gradio, copy app.py + requirements.txt + pngs, change the Space repo id in restart_space(), push.

3. Can it convert Qwen3.8-Flash-Next?

Yes. Verified via HF API:

  • Base model: Qwen/Qwen3.8-Flash-Next β€” MoE (99B-A5B), multimodal image-text-to-text, **180 GB** safetensors (131 shards).
  • Existing MLX conversions already on the Hub:
    • mlx-community/Qwen3.8-Flash-Next-oQ5e-mtp (7,950 dl)
    • mlx-community/Qwen3.8-Flash-Next-oQ6e-mtp, oQ8e-mtp
    • TensorFold/Qwen3.8-Flash-Next-MLX-4bit, and others
  • Recommendation: avoid FP16 (~200 GB+ weights); Q4 is the sweet spot.

4. Step-by-step local conversion plan

  1. Free up ~230 GB (180 GB download + ~50 GB Q4 output). ⚠️ At the time of conversation the Mac had only 45 GB free.
  2. pip install -U huggingface_hub hf-transfer "mlx-lm>=0.29.1" + hf auth login
  3. Repo: youruser/Qwen3.8-Flash-Next-mlx-q4bit
  4. hf download Qwen/Qwen3.8-Flash-Next --local-dir <workdir>/Qwen3.8-Flash-Next
  5. mlx_lm.convert --model <local-dir> --q-bits 4 --mlx-path <workdir>/...-4bit
  6. Optional sanity check: mlx_lm.generate --model ... --prompt "Say hello"
  7. Write README card (base_model, tags, usage snippet)
  8. hf upload youruser/Qwen3.8-Flash-Next-mlx-q4bit <converted-dir> --repo-type model
  9. Clean up the 180 GB original
  10. Verify on the Hub

5. Final decisions (this repo)

  • Folder: mlxtorepo (this directory)
  • 230 GB lives on the internal disk (~/mlxtorepo/work)
  • Repo is public; if the repo already exists β†’ just upload the files
  • Runs inside a venv (~/mlxtorepo/.venv); HF token prompted inside the script
  • v5 β€” REUSABLE: prompts for the model repo and quantization (2-8 or fp16) (also via --model / --quant flags). Fixed bug: mlx_lm.convert needs -q in addition to --q-bits to actually quantize (v4 saved full precision). Note: mlx-lm affine supports q2–q8 (no q1); fp16 available. Version banner (convert_and_upload.sh β€” version: 5) printed on every run.