Instructions to use Haverbex/Qwen3.8-27B-LittleBit-1bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Haverbex/Qwen3.8-27B-LittleBit-1bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Haverbex/Qwen3.8-27B-LittleBit-1bit")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Haverbex/Qwen3.8-27B-LittleBit-1bit")
model = AutoModelForMultimodalLM.from_pretrained("Haverbex/Qwen3.8-27B-LittleBit-1bit", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Haverbex/Qwen3.8-27B-LittleBit-1bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Haverbex/Qwen3.8-27B-LittleBit-1bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Haverbex/Qwen3.8-27B-LittleBit-1bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Haverbex/Qwen3.8-27B-LittleBit-1bit

SGLang

How to use Haverbex/Qwen3.8-27B-LittleBit-1bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Haverbex/Qwen3.8-27B-LittleBit-1bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Haverbex/Qwen3.8-27B-LittleBit-1bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Haverbex/Qwen3.8-27B-LittleBit-1bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Haverbex/Qwen3.8-27B-LittleBit-1bit",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Haverbex/Qwen3.8-27B-LittleBit-1bit with Docker Model Runner:
```
docker model run hf.co/Haverbex/Qwen3.8-27B-LittleBit-1bit
```

Qwen3.8-27B-LittleBit-1bit

현재 성능 보존 연구를 진행하고 있습니다. 이 프로젝트는 LittleBit 공식 연구가 아닙니다. LittleBit 논문과 공개 코드베이스를 바탕으로 독립적으로 재구성했습니다. 모델 성능과 성능 보존을 보장하지 않습니다.

Research on performance retention continues. This is an independent project. It is not official LittleBit research. We adapted the LittleBit papers and public code. We do not guarantee model performance or performance retention.

행동 백도어 부재는 검증되지 않았습니다. 모델 출력에 자동 실행 권한을 부여하지 마십시오.

Behavioral backdoor absence is not verified. Do not grant model outputs automatic action permissions.

Read the security review before use.

Model status

This is an experimental model from Haverbex.co. Performance research continues. The current checkpoint did not preserve the original task performance. The conversion used no QAT or recovery training. The new Mac lab does not change these model weights.

Item	Value
Source	orcarouter/Qwen3.8-27B-Uncensored
Architecture	Qwen3.5 conditional generation; 64 text layers
Compression scope	192 decoder MLP projections
Target MLP bits per weight	1
Other weights	BF16; MTP tensors retained
Stored weights	23.51 GB
Source weights	55.56 GB
Training steps	0

The name gives the target bits for the MLP projections. It does not give the average bits for the full model. The loader expands binary factors for inference. File size does not equal GPU memory use.

Measured results

Task	Items	Original accuracy	This model
ARC-Easy	128	99.22%	25.78%
ARC-Challenge	128	96.88%	32.81%
KMMLU	90	71.11%	17.78%

The mean task accuracy ratio is 28.29%. The paired bootstrap 95% interval is 23.12% to 34.17%. This ratio is a subset result. It is not a measure of all model abilities.

This model selected A in 283 of 346 items. The answer distribution shows a strong position bias.

WikiText-2 PPL changed from 8.73 to 3981.19. Lower PPL is better. Do not convert the PPL ratio into a retention percentage.

The tests use fixed item sets and the original chat template. Thinking is off. The evaluator compares next-token scores for valid answer letters. KMMLU uses two items from each of 45 subjects. These are not official leaderboard scores. The tests do not measure free-form generation, vision, tools, or safety behavior. See retention.json and evaluation_manifest.json for the protocol and results.

Mac mini research setup

Use the component lab on an Apple Silicon Mac. It selects MPS when available. The lab uses one projection at a time. It does not load the full 27B model. The local MPS smoke test passed on Apple M5 Pro with 48 GiB memory. The test used synthetic data. Full-model Mac inference and performance recovery remain unverified.

Install Python 3.12. Download the code with this command. The file filter excludes the model weights.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install huggingface-hub==1.33.0
hf download Haverbex/Qwen3.8-27B-LittleBit-1bit \
  --local-dir littlebit-lab \
  --include 'experiments/*' 'quantization/**' 'requirements-mac.txt' 'MAC_RESEARCH.md'
cd littlebit-lab
python -m pip install -r requirements-mac.txt
python -m experiments.mac_projection_lab doctor
python -m experiments.mac_projection_lab smoke \
  --config experiments/mac_config.json \
  --output outputs/mac-smoke

The default settings use target 1.0 bpw, 30 steps, and a batch size of 16. Edit mac_config.json to change the settings. Use a new output directory for each run. See MAC_RESEARCH.md for real projection inputs and limits.

Full checkpoint loader

The checkpoint requires the bundled custom loader. Standard AutoModel, GGUF, GPTQ, and AWQ loaders cannot read this format directly. The evaluation used an A100 80GB GPU.

Download all repository files before you use this example. Run the example from the full repository directory.

import torch
from quantization.qwen35 import load_qwen35_checkpoint

model = load_qwen35_checkpoint(
    '.', dtype=torch.bfloat16, device='cpu'
).eval()
model.to('cuda:0')

Keep mtp.safetensors. The autoregressive loader does not use the MTP module. The repository does not provide a packed inference kernel.

Sources and method

LittleBit uses training after initialization. LittleBit-2 improves the initialization with Joint-ITQ. This checkpoint uses residual factorization and 50 Joint-ITQ iterations. It has no recovery training. The Mac lab checks the packed file after reload, as required by the MIX-STQ measurement approach. It does not use the MIX-STQ IQ3 encoder. MIX-STQ accuracy results do not apply to this checkpoint.

Source revision: 8cb32d72080f6a47bf34dc5adf8067daa6c63a31. LittleBit reference commit: 933857ed1443b53fc43a875c2cf64249e3c56f0c. MIX-STQ reference commit: b4d3ea0ccdb2db0eace97a9887cf98e1580d48fe.

Licenses and writing

The source model card states Apache-2.0. See SOURCE_MODEL_CARD.md. The bundled LittleBit code uses CC BY-NC 4.0. See LICENSE-LittleBit.txt. The independent changes do not remove those license terms.

The English instructions follow ASD-STE100 writing principles. Full conformity has not been verified. ASD-STE100 applies to English. The Korean notice is a translation.