File size: 1,210 Bytes
9fa0cbc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | {
"service": "colibri",
"source": "https://github.com/JustVugg/colibri",
"description": "Colibri pure-C MoE inference engine serving OLMoE (1B active / 7B total) as a local OpenAI-compatible fast lane.",
"gateway": {
"host": "127.0.0.1",
"port": 18797,
"base_path": "/colibri"
},
"model": {
"family": "olmoe",
"model_id": "olmoe-1b-7b",
"dir": "/data/models/olmoe",
"repo": "",
"revision": "main",
"notes": [
"Colibri needs its OWN container format - not GGUF/AWQ/GPTQ/safetensors.",
"No pre-converted OLMoE container is published on the Hub (only GLM-5.2 / Hy3 variants).",
"Convert once offline with c/tools/convert_olmoe_merged.py (needs torch), push the",
"~7 GB int8 container to a Hub model repo, then set COLIBRI_MODEL_REPO to it.",
"Converting inside this Space is not viable: torch + the bf16 source checkpoint do not",
"fit alongside the other services on a 16 GB / ~50 GB free-tier box.",
"The conversion tools are copied to /opt/colibri-tools in the image when built with COLIBRI_BUILD=1."
]
},
"runtime": {
"threads": 2,
"kv_slots": 2,
"max_queue": 4,
"queue_timeout_sec": 600
}
}
|