ginfer

Muse-Glimmer-30B, GInfer INT

This repo holds the GInfer packages of Muse-Glimmer-30B in INT: single-file packages for the GInfer inference engine (and GChat, which is built on it). They run only on GInfer / GChat. They are not safetensors or GGUF files and other runtimes cannot load them.

GInfer is Sectile Labs' inference engine for fast, high-concurrency, long-context work on NVIDIA GPUs.

Packages

File Drafter Tensor-parallel size Size in bytes SHA-256
muse_glimmer_30b_int_df2.ginfer Q4 DFlash2 1 (one GPU) 19,776,776,704 46b5d54ee64ccd278f2a1a4d2c6d5aa9fc9097e8db0c3d165179aee5f2cb5b66
muse_glimmer_30b_int_df2_tp2.ginfer Q4 DFlash2 2 (two GPUs) 19,776,896,000 9208757adff8bf6c0b01013de3a9910236bbca67311717382e09a8cf895237ec

Format: INT (groupwise integer weights).

GPUs: INT runs on NVIDIA Ampere, Ada and Blackwell GPUs.

The TP2 packages split the model across two identical GPUs; load them with tensor-parallel size 2. TP4 packages will be added later.

Check each download with sha256sum muse_glimmer_30b_int_df2.ginfer or sha256sum muse_glimmer_30b_int_df2_tp2.ginfer.

Upstream models and licenses

  • Base model: meta-models/Muse-Glimmer-30B, revision a4e59da52a7bc87ae7251dd5545c0dd437c44b68. License field apache-2.0; the repo LICENSE file is the Apache License 2.0 (full text in LICENSE here).
  • Drafter: z-lab/Muse-Glimmer-30B-DFlash2, revision b54ffdd11fa9cfe2af370012e5763d492c904128. License field apache-2.0.

Quantized by Sectile Labs to groupwise INT (AutoRound-based) from the official BF16 weights of meta-models/Muse-Glimmer-30B (revision a4e59da52a7bc87ae7251dd5545c0dd437c44b68), then converted to a .ginfer package.

This package is a conversion of the upstream weights into the GInfer format; GInfer does not change the upstream license. The upstream license text is in LICENSE.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including SectileLabs/muse_glimmer_30b_int