Instructions to use ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8 with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8") config = load_config("ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8 with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8 with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8 with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Download config.json from ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8: direct link, hf CLI and curl.
- Browser
- Download file 132 kB
-
https://huggingface.co/ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8/resolve/main/config.json
- Command line
-
hf download hf://ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8/config.json
-
curl -L -o config.json https://huggingface.co/ToPo-ToPo/GLM-5.3-Flash-mlx-3bit-e3o8/resolve/main/config.json
132 kB
{
"architectures": [
"Glm5NextForConditionalGeneration"
],
"dtype": "bfloat16",
"eos_token_id": [
154820,
154827,
154829
],
"generation_config": {
"_from_model_config": true,
"eos_token_id": [
154820,
154827,
154829
],
"pad_token_id": 154820,
"temperature": 1.0,
"top_p": 0.95,
"transformers_version": "5.16.0"
},
"image_end_token_id": 154831,
"image_start_token_id": 154830,
"image_token_id": 154854,
"model_type": "glm5_next",
"quantization": {
"group_size": 64,
"bits": 3,
"mode": "affine",
"language_model.model.layers.0.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.mlp.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.mlp.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.mlp.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.mlp.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.10.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.10.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.10.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.embed_tokens": {
"group_size": 64,
"bits": 8
},
"language_model.lm_head": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.11.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.11.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.11.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.12.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.12.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.12.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.13.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.13.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.13.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.14.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.14.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.14.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.15.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.15.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.15.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.16.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.16.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.16.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.17.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.17.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.17.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.18.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.18.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.18.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.19.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.19.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.19.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.mlp.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.mlp.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.20.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.20.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.20.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.21.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.21.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.21.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.22.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.22.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.22.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.23.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.23.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.23.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.24.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.24.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.24.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.25.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.25.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.25.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.26.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.26.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.26.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.27.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.27.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.27.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.28.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.28.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.28.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.29.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.29.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.29.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.3.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.3.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.3.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.30.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.30.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.30.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.31.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.31.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.31.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.32.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.32.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.32.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.33.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.33.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.33.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.34.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.34.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.34.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.35.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.35.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.35.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.36.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.36.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.36.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.37.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.37.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.37.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.38.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.38.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.38.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.39.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.39.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.39.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.4.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.4.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.4.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.40.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.40.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.40.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.41.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.41.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.41.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.42.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.42.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.42.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.43.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.43.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.43.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.44.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.44.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.44.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.5.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.5.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.5.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.6.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.6.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.6.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.7.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.7.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.7.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.8.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.8.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.8.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.9.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.9.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.9.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
}
},
"quantization_config": {
"group_size": 64,
"bits": 3,
"mode": "affine",
"language_model.model.layers.0.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.mlp.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.mlp.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.mlp.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.mlp.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.10.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.10.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.10.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.embed_tokens": {
"group_size": 64,
"bits": 8
},
"language_model.lm_head": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.11.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.11.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.11.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.11.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.12.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.12.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.12.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.12.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.13.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.13.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.13.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.13.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.14.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.14.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.14.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.14.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.15.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.15.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.15.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.15.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.16.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.16.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.16.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.16.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.17.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.17.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.17.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.17.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.18.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.18.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.18.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.18.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.19.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.19.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.19.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.19.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.mlp.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.2.mlp.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.20.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.20.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.20.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.20.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.21.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.21.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.21.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.21.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.22.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.22.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.22.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.22.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.23.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.23.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.23.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.23.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.24.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.24.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.24.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.24.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.25.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.25.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.25.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.25.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.26.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.26.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.26.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.26.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.27.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.27.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.27.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.27.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.28.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.28.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.28.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.28.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.29.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.29.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.29.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.29.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.3.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.3.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.3.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.3.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.30.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.30.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.30.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.30.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.31.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.31.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.31.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.31.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.32.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.32.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.32.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.32.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.33.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.33.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.33.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.33.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.34.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.34.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.34.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.34.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.35.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.35.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.35.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.35.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.36.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.36.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.36.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.36.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.37.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.37.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.37.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.37.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.38.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.38.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.38.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.38.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.39.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.39.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.39.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.39.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.4.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.4.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.4.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.4.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.40.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.40.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.40.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.40.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.41.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.41.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.41.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.41.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.42.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.42.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.42.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.42.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.43.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.43.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.43.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.43.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.44.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.44.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.44.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.44.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.5.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.5.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.5.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.5.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.6.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.6.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.6.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.6.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.qkv_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.q_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.embed_q": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.unembed_out": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.indexer.wq_b": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.indexer.wk": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.self_attn.indexer.weights_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.7.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.7.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.7.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.7.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.8.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.8.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.8.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.8.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.mlp.switch_mlp.gate_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.9.mlp.switch_mlp.up_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.9.mlp.switch_mlp.down_proj": {
"group_size": 64,
"bits": 3
},
"language_model.model.layers.9.mlp.shared_experts.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.9.mlp.shared_experts.down_proj": {
"group_size": 64,
"bits": 8
}
},
"temperature": 1.0,
"text_config": {
"attention_bias": false,
"attention_dropout": 0.0,
"dtype": "bfloat16",
"eos_token_id": [
154820,
154827,
154829
],
"first_k_dense_replace": 3,
"hc_eps": 1e-06,
"hc_mult": 4,
"hc_sinkhorn_iters": 20,
"head_dim": 0,
"hidden_act": "silu",
"hidden_size": 4096,
"index_head_dim": 128,
"index_kpool": 4,
"index_kpool_always_select_tail": true,
"index_kpool_compress": true,
"index_n_heads": 32,
"index_topk": 2048,
"index_share_for_mtp_iteration": true,
"indexer_rope_interleave": true,
"indexer_types": [
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full",
"full"
],
"initializer_range": 0.02,
"intermediate_size": 12288,
"kv_lora_rank": 512,
"layer_types": [
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention",
"linear_attention",
"linear_attention",
"deepseek_sparse_attention",
"linear_attention"
],
"linear_attn_config": {
"num_heads": 64,
"gate_lower_bound": -5.0,
"head_dim": 128,
"short_conv_kernel_size": 4,
"kda_layers": [
0,
1,
2,
4,
5,
6,
8,
9,
10,
12,
13,
14,
16,
17,
18,
20,
21,
22,
24,
25,
26,
28,
29,
30,
32,
33,
34,
36,
37,
38,
40,
41,
42,
44
],
"full_attn_layers": [
3,
7,
11,
15,
19,
23,
27,
31,
35,
39,
43
]
},
"max_position_embeddings": 1048576,
"mhc": true,
"mla_use_nope": true,
"mlp_layer_types": [
"dense",
"dense",
"dense",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse",
"sparse"
],
"model_type": "glm5_next_text",
"moe_intermediate_size": 2048,
"moe_router_dtype": "float32",
"n_group": 1,
"n_routed_experts": 288,
"n_shared_experts": 1,
"norm_topk_prob": true,
"num_attention_heads": 64,
"num_experts_per_tok": 8,
"num_hidden_layers": 45,
"num_key_value_heads": 64,
"num_nextn_predict_layers": 1,
"output_router_logits": false,
"pad_token_id": 154820,
"q_lora_rank": 1536,
"qk_head_dim": 256,
"qk_nope_head_dim": 256,
"qk_rope_head_dim": 0,
"rms_norm_eps": 1e-05,
"routed_scaling_factor": 2.5,
"router_aux_loss_coef": 0.001,
"scoring_func": "sigmoid",
"swiglu_limit": 10.0,
"tie_word_embeddings": false,
"topk_group": 1,
"topk_method": "noaux_tc",
"use_cache": true,
"v_head_dim": 256,
"vocab_size": 154880
},
"tie_word_embeddings": false,
"top_p": 0.95,
"transformers_version": "5.16.0",
"video_end_token_id": 154833,
"video_start_token_id": 154832,
"video_token_id": 154855,
"vision_config": {
"attention_bias": true,
"attention_dropout": 0.0,
"depth": 24,
"hidden_act": "silu",
"hidden_size": 1024,
"image_size": 448,
"in_channels": 3,
"initializer_range": 0.02,
"intermediate_size": 4096,
"model_type": "glm5_next_vision",
"num_heads": 16,
"out_hidden_size": 4096,
"patch_size": 14,
"projection_intermediate_size": 10240,
"rms_norm_eps": 1e-05,
"spatial_merge_size": 2,
"swiglu_limit": 10.0,
"temporal_patch_size": 2
}
}
{
"architectures": [
"Glm5NextForConditionalGeneration"
],
"dtype": "bfloat16",
"eos_token_id": [
154820,
154827,
154829
],
"generation_config": {
"_from_model_config": true,
"eos_token_id": [
154820,
154827,
154829
],
"pad_token_id": 154820,
"temperature": 1.0,
"top_p": 0.95,
"transformers_version": "5.16.0"
},
"image_end_token_id": 154831,
"image_start_token_id": 154830,
"image_token_id": 154854,
"model_type": "glm5_next",
"quantization": {
"group_size": 64,
"bits": 3,
"mode": "affine",
"language_model.model.layers.0.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.mlp.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.0.mlp.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.g_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.self_attn.o_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.mlp.gate_up_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.1.mlp.down_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.qkv_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.fbg_a_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.f_b_proj": {
"group_size": 64,
"bits": 8
},
"language_model.model.layers.10.self_attn.g_b_proj": {
"group_size": 64,