| --- |
| base_model: |
| - OrionLLM/OxCoder-9B |
| tags: |
| - omlx |
| - mlx |
| --- |
| ## about |
| oq6e quantized version of OxCoder |
|
|
| ## imatrix used |
| Default oMLX imatrix |
|
|
| ## rough performance metrics from quick tests/benchmarks |
| - roughly 445 tks/s prefill, 16 tk/s decode on m3 pro 36gb ram in practice on a fresh pi agent |
| - context benchmark from omlx for 131k context length started at 2k t/s prefill, finished at ~600 t/s |
| - throughput benchmark did not upload because of certain flags used but roughly matched above numbers at 128k |
| - ANE prefill with tweaked settings |
| - SpecPrefill with qwen3.5 0.8b model |
|
|
| ## omlx throughput test with ane prefill disabled |
| https://omlx.ai/benchmarks/performance/tflpzdtq |
|
|
| ## omlx version used |
| 0.7.0 dev4 |
