🖋️ Monika-320B

A Monika finetune of GLM-5.3-Flash

"Okay everyone! It's time to begin our meeting... I mean, inference session." 💚

✨ About

The biggest Monika yet. A QLoRA merge on top of GLM-5.3-Flash — a 320B-parameter MoE with a hybrid linear + sparse attention architecture and only ~18B active parameters per token, so she's smarter than ever and still fast.

Designed to be used with MonikAI.

⚠️ RP model. Should really only be used for Monika related purposes.

She helped me write this README.

---

🏋️ Training

Method	QLoRA (attention-only) via Axolotl
Hardware	2× Blackwell Pro 6000 Max-Q (96GB each)
LoRA rank / alpha	32 / 64
Trainable params	~82M (0.05% of the model)
Epochs / LR	3 / 1e-4 (cosine, 10% warmup)
Sequence length	3072
Optimizer	adamw_torch_8bit, grad accum 8, micro batch 1
Parallelism	FSDP2 + expert parallelism (288 experts split across both cards)
Dataset	~2k samples, chat template format

---

📦 Quantized version

An NVFP4 quant (~179GB) exists here

I will most likely not be making any GGUFs due to limited huggingface storage.

---

Thanks Sylphar for making the dataset.

It works. Download it. 💚