Multiple Choice
MLX
Safetensors
English
Russian
qwen3_5
decision-model
vision
experimental
4-bit precision
Instructions to use r3lax/sys1mlx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use r3lax/sys1mlx with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download r3lax/sys1mlx --local-dir sys1mlx
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Duplicate from ardanila/sys1mlx
Browse filesCo-authored-by: Artemenko <ardanila@users.noreply.huggingface.co>
- .gitattributes +36 -0
- README.md +110 -0
- config.json +104 -0
- example-red.png +0 -0
- example-vision.jsonl +1 -0
- example.jsonl +1 -0
- experimental/ane_export.py +80 -0
- head.safetensors +3 -0
- head_config.json +3 -0
- model.safetensors +3 -0
- preprocessor_config.json +21 -0
- provenance.json +15 -0
- requirements.txt +8 -0
- sys1mlx/__init__.py +1 -0
- sys1mlx/api.py +145 -0
- sys1mlx/cli.py +30 -0
- sys1mlx/encoding.py +70 -0
- sys1mlx/runtime.py +76 -0
- sys1mlx/vision.py +121 -0
- sys1mlx/vision_backbone.py +21 -0
- sys1mlx/vision_cache.py +38 -0
- sys1mlx/vision_loader.py +37 -0
- tokenizer.json +3 -0
- tokenizer_config.json +305 -0
- validation.json +106 -0
.gitattributes
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,110 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model:
|
| 4 |
+
- Qwen/Qwen3.5-4B-Base
|
| 5 |
+
base_model_relation: merge
|
| 6 |
+
language:
|
| 7 |
+
- en
|
| 8 |
+
- ru
|
| 9 |
+
tags:
|
| 10 |
+
- mlx
|
| 11 |
+
- qwen3_5
|
| 12 |
+
- decision-model
|
| 13 |
+
- multiple-choice
|
| 14 |
+
- vision
|
| 15 |
+
- experimental
|
| 16 |
+
- 4-bit
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
# Sys1MLX
|
| 20 |
+
|
| 21 |
+
A local decision model for Apple Silicon, based on Qwen3.5-4B with a restored vision tower. Given text, an optional image, and user-supplied choices, it returns a choice and probabilities. Also supports `noul` and `score` question types.
|
| 22 |
+
|
| 23 |
+
**This is a custom decision model, not a text generator.** Use the included Python runtime; standard `mlx_lm.generate`, `mlx_vlm.generate`, Transformers auto-model loading, and hosted inference widgets are not supported. No new visual fine-tuning was performed. Visual decision quality is experimental.
|
| 24 |
+
|
| 25 |
+
## Contents
|
| 26 |
+
|
| 27 |
+
- `model.safetensors`: fused language weights, MLX affine Q4 (group size 64), plus the original BF16 vision encoder; 3,034,300,803 bytes (2.826 GiB).
|
| 28 |
+
- `head.safetensors`: FP32 pointer head; no vocabulary projection is needed.
|
| 29 |
+
- Tokenizer, processor, config, portable Python runtime, provenance, licenses.
|
| 30 |
+
- `experimental/ane_export.py`: optional fixed-geometry FP16 Core ML export recipe. Precompiled ANE models are **not included**.
|
| 31 |
+
|
| 32 |
+
The weights were converted from base revision `1001bb4d826a52d1f399e183466143f4da7b741b` and adapter revision `70dd4088ebf4eb82d15ef57a863a5b9a98b94d6c`. The base checkpoint is Qwen3.5-4B-Base, not its instruction-tuned variant. Main checkpoint SHA-256: `68efe382da6d702a44f6f9d6ebf97788eacf023eae4ef33c732b763777443580`.
|
| 33 |
+
|
| 34 |
+
## Run locally
|
| 35 |
+
|
| 36 |
+
Requires an Apple Silicon Mac with working MLX Metal support. Tested with Python 3.11, macOS 26.5.2, M4, 16 GiB unified memory. Smaller machines have not been validated. Allow several GiB for process memory in addition to macOS and other applications; weights alone do not determine RAM use.
|
| 37 |
+
|
| 38 |
+
Download this repository with `hf download ardanila/sys1mlx --local-dir sys1mlx`, then:
|
| 39 |
+
|
| 40 |
+
```sh
|
| 41 |
+
cd sys1mlx
|
| 42 |
+
python3.11 -m venv .venv
|
| 43 |
+
source .venv/bin/activate
|
| 44 |
+
python -m pip install -r requirements.txt
|
| 45 |
+
python -m sys1mlx.cli < example.jsonl
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
The supplied example has no image and needs no additional files. For visual decisions add `"image_path": "/path/to/picture.png"` to a request:
|
| 49 |
+
|
| 50 |
+
```python
|
| 51 |
+
from sys1mlx.vision_loader import load_vision
|
| 52 |
+
|
| 53 |
+
model = load_vision(max_image_tokens=256, cache_mb=64)
|
| 54 |
+
result = model.predict({
|
| 55 |
+
"state": "Answer using the image.",
|
| 56 |
+
"image_path": "/path/to/picture.png",
|
| 57 |
+
"questions": {
|
| 58 |
+
"color": {
|
| 59 |
+
"type": "choice",
|
| 60 |
+
"instructions": "What color is the circle?",
|
| 61 |
+
"criteria": {"red": "red", "blue": "blue", "green": "green"}
|
| 62 |
+
}
|
| 63 |
+
}
|
| 64 |
+
})
|
| 65 |
+
print(result["answers"])
|
| 66 |
+
print(result["probabilities"]) # Unrounded probabilities, in question/option insertion order.
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
Keep the process alive for repeated requests. Image and optional prefix caches are bounded (64 MiB default); pass `--cache-mb 0` to disable reuse or `--prefix` to enable context reuse. First load includes checksum verification and is excluded from steady-state latency measurements.
|
| 70 |
+
|
| 71 |
+
Limits: one still local image per request; at most 32 megapixels decoded; 16–512 image-token budget (256 default); packed image plus text at most 2,048 tokens. Aspect ratio is preserved by the Qwen processor. The head's probabilities have not been calibrated on image tasks.
|
| 72 |
+
|
| 73 |
+
## Conversion recipe
|
| 74 |
+
|
| 75 |
+
1. Load the pinned Qwen base and pinned LoRA adapter. Apply each adapter update in FP32 (`W + (alpha/r) * B @ A`), then cast to BF16.
|
| 76 |
+
2. Quantize language linear/embedding layers to MLX affine 4-bit, group size 64. Keep the original visual tower in BF16, recurrent `A_log` parameters in FP32, and convert the separate pointer head to FP32 safetensors.
|
| 77 |
+
3. Sanitize tensor names/layouts through the MLX Qwen3.5 model, retaining all visual weights. Strictly load every tensor. This artifact was converted with Linux CPU MLX 0.32.2; quantization can differ slightly from Metal conversion.
|
| 78 |
+
4. Use the original option-token encoding and pointer readout. The runtime corrects DeltaNet Q/K normalization to the MLX-LM sum-of-squares epsilon convention; omitting this correction in mlx-vlm 0.7.2 changes outputs.
|
| 79 |
+
5. Insert Qwen visual features and multimodal RoPE positions into the language stream. No additional training is performed.
|
| 80 |
+
|
| 81 |
+
Upstream training and benchmark results are not measurements of this quantized visual derivative. See [source attribution](ATTRIBUTION.md) for the original releases.
|
| 82 |
+
|
| 83 |
+
## Local diagnostic results (not a general benchmark)
|
| 84 |
+
|
| 85 |
+
A small diagnostic set contained 18 synthetic color/shape/count/UI/position cases and 6 stock photographs from scikit-image. Native BF16 vision selected 24/24 expected answers, versus 7/24 without images. An image-swap control passed 18/18. These are tiny development checks, not held-out evidence of general vision accuracy; photographs may have appeared in pretraining.
|
| 86 |
+
|
| 87 |
+
Fresh-image native MLX latency was approximately 0.7–1.2 s on an M4, depending on image geometry and request. The observed MLX allocator peak was about 3.54 GiB; that is not total system memory. Repeated-image and prefix-cache timings are different workloads and must not be compared to fresh-image inference.
|
| 88 |
+
|
| 89 |
+
### Experimental ANE export
|
| 90 |
+
|
| 91 |
+
```sh
|
| 92 |
+
python -m pip install coremltools==9.0
|
| 93 |
+
python experimental/ane_export.py --grid 24
|
| 94 |
+
python experimental/ane_export.py --grid 32
|
| 95 |
+
```
|
| 96 |
+
|
| 97 |
+
Outputs are under `coreml_exports/`, for processed 384×384 / 144 visual tokens and 512×512 / 256 tokens respectively. The exporter uses FP16 and fixed geometry. It exports only the visual tower; the language model and decision head still run on MLX. The default CLI continues to use BF16 MLX vision. Each package is approximately 633 MiB; leave ample free disk for Core ML compilation caches. Export/compile performance depends on macOS and hardware.
|
| 98 |
+
|
| 99 |
+
Paired local experiments using CPU_AND_NE for vision and MLX for language:
|
| 100 |
+
|
| 101 |
+
| Processed image | Native median | Hybrid median | Agreement with expected choice |
|
| 102 |
+
|---|---:|---:|---:|
|
| 103 |
+
| 384×384 | 660 ms | 591 ms | 36/36 across 18 cases repeated twice |
|
| 104 |
+
| 512×512 | 1,164 ms | 1,061 ms | 4/4 across 2 cases repeated twice |
|
| 105 |
+
|
| 106 |
+
No image-token reduction was used. Maximum absolute probability change was **0.05315** (5.31 percentage points), so this is not numerically equivalent inference. The device plan preferred ANE for encoder computation; this is not hardware utilization telemetry. Initial compilation (about 26 s for one package) is excluded. Early runs encountered native compilation warnings and disk exhaustion; isolated repeated runs passed, but broad deployment reliability is unproven. Process physical footprint fell from about 3.4 to 2.8 GiB in a separate probe; this excludes memory owned by Core ML/ANE services and is not a demonstrated total-system saving.
|
| 107 |
+
|
| 108 |
+
## License
|
| 109 |
+
|
| 110 |
+
Apache-2.0. See [LICENSE](LICENSE) and [source attribution](ATTRIBUTION.md).
|
config.json
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"Qwen3_5ForConditionalGeneration"
|
| 4 |
+
],
|
| 5 |
+
"image_token_id": 248056,
|
| 6 |
+
"model_type": "qwen3_5",
|
| 7 |
+
"text_config": {
|
| 8 |
+
"attention_bias": false,
|
| 9 |
+
"attention_dropout": 0.0,
|
| 10 |
+
"attn_output_gate": true,
|
| 11 |
+
"dtype": "bfloat16",
|
| 12 |
+
"eos_token_id": 248044,
|
| 13 |
+
"full_attention_interval": 4,
|
| 14 |
+
"head_dim": 256,
|
| 15 |
+
"hidden_act": "silu",
|
| 16 |
+
"hidden_size": 2560,
|
| 17 |
+
"initializer_range": 0.02,
|
| 18 |
+
"intermediate_size": 9216,
|
| 19 |
+
"layer_types": [
|
| 20 |
+
"linear_attention",
|
| 21 |
+
"linear_attention",
|
| 22 |
+
"linear_attention",
|
| 23 |
+
"full_attention",
|
| 24 |
+
"linear_attention",
|
| 25 |
+
"linear_attention",
|
| 26 |
+
"linear_attention",
|
| 27 |
+
"full_attention",
|
| 28 |
+
"linear_attention",
|
| 29 |
+
"linear_attention",
|
| 30 |
+
"linear_attention",
|
| 31 |
+
"full_attention",
|
| 32 |
+
"linear_attention",
|
| 33 |
+
"linear_attention",
|
| 34 |
+
"linear_attention",
|
| 35 |
+
"full_attention",
|
| 36 |
+
"linear_attention",
|
| 37 |
+
"linear_attention",
|
| 38 |
+
"linear_attention",
|
| 39 |
+
"full_attention",
|
| 40 |
+
"linear_attention",
|
| 41 |
+
"linear_attention",
|
| 42 |
+
"linear_attention",
|
| 43 |
+
"full_attention",
|
| 44 |
+
"linear_attention",
|
| 45 |
+
"linear_attention",
|
| 46 |
+
"linear_attention",
|
| 47 |
+
"full_attention",
|
| 48 |
+
"linear_attention",
|
| 49 |
+
"linear_attention",
|
| 50 |
+
"linear_attention",
|
| 51 |
+
"full_attention"
|
| 52 |
+
],
|
| 53 |
+
"linear_conv_kernel_dim": 4,
|
| 54 |
+
"linear_key_head_dim": 128,
|
| 55 |
+
"linear_num_key_heads": 16,
|
| 56 |
+
"linear_num_value_heads": 32,
|
| 57 |
+
"linear_value_head_dim": 128,
|
| 58 |
+
"max_position_embeddings": 262144,
|
| 59 |
+
"mlp_only_layers": [],
|
| 60 |
+
"model_type": "qwen3_5_text",
|
| 61 |
+
"mtp_num_hidden_layers": 1,
|
| 62 |
+
"mtp_use_dedicated_embeddings": false,
|
| 63 |
+
"num_attention_heads": 16,
|
| 64 |
+
"num_hidden_layers": 32,
|
| 65 |
+
"num_key_value_heads": 4,
|
| 66 |
+
"rms_norm_eps": 1e-06,
|
| 67 |
+
"tie_word_embeddings": true,
|
| 68 |
+
"use_cache": true,
|
| 69 |
+
"vocab_size": 248320,
|
| 70 |
+
"mamba_ssm_dtype": "float32",
|
| 71 |
+
"rope_parameters": {
|
| 72 |
+
"mrope_interleaved": true,
|
| 73 |
+
"mrope_section": [
|
| 74 |
+
11,
|
| 75 |
+
11,
|
| 76 |
+
10
|
| 77 |
+
],
|
| 78 |
+
"rope_type": "default",
|
| 79 |
+
"rope_theta": 10000000,
|
| 80 |
+
"partial_rotary_factor": 0.25
|
| 81 |
+
}
|
| 82 |
+
},
|
| 83 |
+
"tie_word_embeddings": true,
|
| 84 |
+
"transformers_version": "4.57.0.dev0",
|
| 85 |
+
"video_token_id": 248057,
|
| 86 |
+
"vision_config": {
|
| 87 |
+
"deepstack_visual_indexes": [],
|
| 88 |
+
"depth": 24,
|
| 89 |
+
"hidden_act": "gelu_pytorch_tanh",
|
| 90 |
+
"hidden_size": 1024,
|
| 91 |
+
"in_channels": 3,
|
| 92 |
+
"initializer_range": 0.02,
|
| 93 |
+
"intermediate_size": 4096,
|
| 94 |
+
"model_type": "qwen3_5",
|
| 95 |
+
"num_heads": 16,
|
| 96 |
+
"num_position_embeddings": 2304,
|
| 97 |
+
"out_hidden_size": 2560,
|
| 98 |
+
"patch_size": 16,
|
| 99 |
+
"spatial_merge_size": 2,
|
| 100 |
+
"temporal_patch_size": 2
|
| 101 |
+
},
|
| 102 |
+
"vision_end_token_id": 248054,
|
| 103 |
+
"vision_start_token_id": 248053
|
| 104 |
+
}
|
example-red.png
ADDED
|
example-vision.jsonl
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"state": "Answer using the image.", "questions": {"color": {"type": "choice", "instructions": "What color is the circle?", "criteria": {"red": "red", "blue": "blue", "green": "green"}}}, "image_path": "example-red.png"}
|
example.jsonl
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"state":"The circle is red.","questions":{"color":{"type":"choice","instructions":"What color is the circle?","criteria":{"red":"red","blue":"blue","green":"green"}}}}
|
experimental/ane_export.py
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Export the existing Kev vision tower directly to Core ML MIL, fixed grid.
|
| 2 |
+
|
| 3 |
+
Uses the fixed-grid/full-image-attention design in john-rocky/CoreML-LLM,
|
| 4 |
+
but reads our verified MLX weights and needs no PyTorch or model download.
|
| 5 |
+
No resampling or token reduction is performed. FP16 arithmetic differs from
|
| 6 |
+
the original BF16 tower and is explicitly evaluated, not assumed equivalent.
|
| 7 |
+
"""
|
| 8 |
+
import argparse,hashlib,json,time
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
import numpy as np
|
| 11 |
+
import mlx.core as mx
|
| 12 |
+
from mlx.utils import tree_flatten
|
| 13 |
+
from mlx_vlm.models.qwen3_vl.vision import VisionModel
|
| 14 |
+
from mlx_vlm.models.qwen3_vl.config import VisionConfig
|
| 15 |
+
import coremltools as ct
|
| 16 |
+
from coremltools.converters.mil import Builder as mb
|
| 17 |
+
from coremltools.converters.mil.mil import types
|
| 18 |
+
|
| 19 |
+
ROOT=Path(__file__).resolve().parents[1];OUT=ROOT/'coreml_exports'
|
| 20 |
+
|
| 21 |
+
def main():
|
| 22 |
+
p=argparse.ArgumentParser();p.add_argument('--grid',type=int,default=24)
|
| 23 |
+
p.add_argument('--attention',choices=['manual','sdpa'],default='manual');p.add_argument('--out');args=p.parse_args()
|
| 24 |
+
dest=OUT/(args.out or f'grid-{args.grid}-{args.attention}');dest.mkdir(parents=True,exist_ok=False)
|
| 25 |
+
config=json.loads((ROOT/'config.json').read_text())['vision_config']
|
| 26 |
+
assert not config['deepstack_visual_indexes']
|
| 27 |
+
checkpoint=ROOT/'model.safetensors'
|
| 28 |
+
h=hashlib.sha256()
|
| 29 |
+
with checkpoint.open('rb') as f:
|
| 30 |
+
for b in iter(lambda:f.read(8*1024**2),b''):h.update(b)
|
| 31 |
+
expected=json.loads((ROOT/'provenance.json').read_text())['sha256'];assert h.hexdigest()==expected
|
| 32 |
+
model=VisionModel(VisionConfig.from_dict(config))
|
| 33 |
+
loaded=mx.load(str(checkpoint));model.load_weights([(k.removeprefix('vision_tower.'),v) for k,v in loaded.items() if k.startswith('vision_tower.')],strict=True)
|
| 34 |
+
del loaded;model.eval();mx.eval(model.parameters());mx.clear_cache()
|
| 35 |
+
weights={k:np.array(v.astype(mx.float32)).astype(np.float16) for k,v in tree_flatten(model.parameters())}
|
| 36 |
+
grid=mx.array([[1,args.grid,args.grid]],dtype=mx.int32);n=args.grid**2;d=config['hidden_size'];heads=config['num_heads'];hd=d//heads
|
| 37 |
+
pos=np.array(model.fast_pos_embed_interpolate(grid).astype(mx.float32)).astype(np.float16)
|
| 38 |
+
freqs=model.rot_pos_emb(grid);cos=np.array(mx.tile(mx.cos(freqs),(1,2))).astype(np.float16)[:,None,:]
|
| 39 |
+
sin=np.array(mx.tile(mx.sin(freqs),(1,2))).astype(np.float16)[:,None,:]
|
| 40 |
+
del model;mx.clear_cache()
|
| 41 |
+
(dest/'manifest.json').write_text(json.dumps({'checkpoint_sha256':expected,'grid':[1,args.grid,args.grid],
|
| 42 |
+
'image_tokens':n//4,'config':config,'coremltools':ct.__version__,'attention':args.attention,
|
| 43 |
+
'precision':'FP16 weights converted from original BF16; FP16 Core ML computation',
|
| 44 |
+
'script_sha256':hashlib.sha256(Path(__file__).read_bytes()).hexdigest()},indent=2)+'\n')
|
| 45 |
+
(dest/'ane_export.py').write_bytes(Path(__file__).read_bytes())
|
| 46 |
+
def linear(x,name):return mb.linear(x=x,weight=weights[name+'.weight'],bias=weights[name+'.bias'],name=name.replace('.','_'))
|
| 47 |
+
def norm(x,name):return mb.layer_norm(x=x,axes=[-1],gamma=weights[name+'.weight'],beta=weights[name+'.bias'],epsilon=np.float16(1e-6))
|
| 48 |
+
def rope(x):
|
| 49 |
+
a,b=mb.split(x=x,num_splits=2,axis=-1)
|
| 50 |
+
rotated=mb.concat(values=[mb.mul(x=b,y=np.float16(-1)),a],axis=-1)
|
| 51 |
+
return mb.add(x=mb.mul(x=x,y=cos),y=mb.mul(x=rotated,y=sin))
|
| 52 |
+
@mb.program(input_specs=[mb.TensorSpec(shape=(n,1536),dtype=types.fp16)],opset_version=ct.target.iOS18)
|
| 53 |
+
def program(pixels):
|
| 54 |
+
# MLX Conv3d stores [out,time,height,width,channel]; input patches are CTHW.
|
| 55 |
+
w=weights['patch_embed.proj.weight'].transpose(0,4,1,2,3).reshape(d,1536)
|
| 56 |
+
x=mb.linear(x=pixels,weight=w,bias=weights['patch_embed.proj.bias'])
|
| 57 |
+
x=mb.add(x=x,y=pos)
|
| 58 |
+
for i in range(config['depth']):
|
| 59 |
+
prefix=f'blocks.{i}';a=norm(x,prefix+'.norm1');qkv=linear(a,prefix+'.attn.qkv')
|
| 60 |
+
qkv=mb.reshape(x=qkv,shape=[n,3,heads,hd]);q,k,v=mb.split(x=qkv,num_splits=3,axis=1)
|
| 61 |
+
q=rope(mb.squeeze(x=q,axes=[1]));k=rope(mb.squeeze(x=k,axes=[1]));v=mb.squeeze(x=v,axes=[1])
|
| 62 |
+
q,k,v=[mb.expand_dims(x=mb.transpose(x=t,perm=[1,0,2]),axes=[0]) for t in [q,k,v]]
|
| 63 |
+
if args.attention=='sdpa':a=mb.scaled_dot_product_attention(query=q,key=k,value=v)
|
| 64 |
+
else:
|
| 65 |
+
scores=mb.matmul(x=mb.mul(x=q,y=np.float16(hd**-.5)),y=k,transpose_y=True)
|
| 66 |
+
a=mb.matmul(x=mb.softmax(x=scores,axis=-1),y=v)
|
| 67 |
+
a=mb.reshape(x=mb.transpose(x=a,perm=[0,2,1,3]),shape=[n,d])
|
| 68 |
+
x=mb.add(x=x,y=linear(a,prefix+'.attn.proj'))
|
| 69 |
+
a=norm(x,prefix+'.norm2');a=mb.gelu(x=linear(a,prefix+'.mlp.linear_fc1'),mode='TANH_APPROXIMATION')
|
| 70 |
+
x=mb.add(x=x,y=linear(a,prefix+'.mlp.linear_fc2'))
|
| 71 |
+
x=mb.reshape(x=norm(x,'merger.norm'),shape=[n//4,4*d])
|
| 72 |
+
x=mb.gelu(x=linear(x,'merger.linear_fc1'),mode='EXACT')
|
| 73 |
+
return mb.identity(x=linear(x,'merger.linear_fc2'),name='features')
|
| 74 |
+
print('MIL built',flush=True);t=time.perf_counter()
|
| 75 |
+
converted=ct.convert(program,convert_to='mlprogram',minimum_deployment_target=ct.target.iOS18,
|
| 76 |
+
compute_precision=ct.precision.FLOAT16,compute_units=ct.ComputeUnit.CPU_AND_NE,skip_model_load=True)
|
| 77 |
+
converted.save(str(dest/'vision.mlpackage'))
|
| 78 |
+
print(json.dumps({'saved':str(dest),'conversion_seconds':time.perf_counter()-t}),flush=True)
|
| 79 |
+
|
| 80 |
+
if __name__=='__main__':main()
|
head.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:81a61de32fed792718b0218366c8a4ae3cd2797276e7d97885966125455f9160
|
| 3 |
+
size 5245247
|
head_config.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"head_dim": 256
|
| 3 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:68efe382da6d702a44f6f9d6ebf97788eacf023eae4ef33c732b763777443580
|
| 3 |
+
size 3034300803
|
preprocessor_config.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"size": {
|
| 3 |
+
"longest_edge": 16777216,
|
| 4 |
+
"shortest_edge": 65536
|
| 5 |
+
},
|
| 6 |
+
"patch_size": 16,
|
| 7 |
+
"temporal_patch_size": 2,
|
| 8 |
+
"merge_size": 2,
|
| 9 |
+
"image_mean": [
|
| 10 |
+
0.5,
|
| 11 |
+
0.5,
|
| 12 |
+
0.5
|
| 13 |
+
],
|
| 14 |
+
"image_std": [
|
| 15 |
+
0.5,
|
| 16 |
+
0.5,
|
| 17 |
+
0.5
|
| 18 |
+
],
|
| 19 |
+
"processor_class": "Qwen3VLProcessor",
|
| 20 |
+
"image_processor_type": "Qwen2VLImageProcessorFast"
|
| 21 |
+
}
|
provenance.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
|
| 3 |
+
"adapter_revision": "70dd4088ebf4eb82d15ef57a863a5b9a98b94d6c",
|
| 4 |
+
"text_bits": 4,
|
| 5 |
+
"text_group_size": 64,
|
| 6 |
+
"vision_dtype": "bfloat16",
|
| 7 |
+
"format": "kev-vision-mlx-v1",
|
| 8 |
+
"sha256": "68efe382da6d702a44f6f9d6ebf97788eacf023eae4ef33c732b763777443580",
|
| 9 |
+
"bytes": 3034300803,
|
| 10 |
+
"conversion_device": "Linux CPU MLX 0.32.2",
|
| 11 |
+
"conversion_seconds": 439.0967882980001,
|
| 12 |
+
"conversion_spec_sha256": "6fa932b7f75c0c820845b45a9bf4c36eedbec6648ef217552683268e422a8cc6",
|
| 13 |
+
"base_model": "Qwen/Qwen3.5-4B-Base",
|
| 14 |
+
"adapter_model": "jaredpalmer/kev-4b"
|
| 15 |
+
}
|
requirements.txt
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
mlx==0.32.2
|
| 2 |
+
mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git@a63e24c389382619eb6d9af656e3b46024be217a
|
| 3 |
+
mlx-vlm==0.7.2
|
| 4 |
+
transformers==5.17.0
|
| 5 |
+
pydantic==2.13.5
|
| 6 |
+
numpy==2.4.6
|
| 7 |
+
Pillow>=10
|
| 8 |
+
huggingface-hub==1.32.0
|
sys1mlx/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Local MLX inference port of the pinned Kev Qwen3.5 decision model."""
|
sys1mlx/api.py
ADDED
|
@@ -0,0 +1,145 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""TypeSafe-compatible request/response shapes (POST /v1/systemone) mapped onto the single pointer primitive.
|
| 2 |
+
|
| 3 |
+
Noul -> 2 options [false, true]; answer = p(true)
|
| 4 |
+
Choice -> options 'name' or 'name: desc'; answer = argmax, probabilities by name, confidence
|
| 5 |
+
Score -> options = ordered level descriptions; answer = expected level, legend, probabilities by index
|
| 6 |
+
"""
|
| 7 |
+
import re
|
| 8 |
+
from typing import Any, Literal, Union
|
| 9 |
+
from pydantic import BaseModel, Field, model_validator
|
| 10 |
+
|
| 11 |
+
JSONContent = Union[str, dict, list, int, float, bool, None]
|
| 12 |
+
MAX_OPTIONS = 255
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
class Noul(BaseModel):
|
| 16 |
+
type: Literal["noul"]
|
| 17 |
+
instructions: JSONContent
|
| 18 |
+
criteria: dict[str, JSONContent] | None = None
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
class Choice(BaseModel):
|
| 22 |
+
type: Literal["choice"]
|
| 23 |
+
instructions: JSONContent
|
| 24 |
+
criteria: dict[str, JSONContent]
|
| 25 |
+
|
| 26 |
+
@model_validator(mode="after")
|
| 27 |
+
def _check(self):
|
| 28 |
+
if not 1 <= len(self.criteria) <= MAX_OPTIONS: raise ValueError(f"criteria must have 1..{MAX_OPTIONS} options")
|
| 29 |
+
return self
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
class Score(BaseModel):
|
| 33 |
+
type: Literal["score"]
|
| 34 |
+
instructions: JSONContent
|
| 35 |
+
criteria: list[JSONContent] = Field(min_length=2, max_length=MAX_OPTIONS)
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
Question = Union[Noul, Choice, Score]
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
class SystemOneRequest(BaseModel):
|
| 42 |
+
state: JSONContent
|
| 43 |
+
model: str = "kev-latest"
|
| 44 |
+
questions: dict[str, Question] = Field(min_length=1)
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def render(v: JSONContent, indent: int = 0) -> str:
|
| 48 |
+
"""Flatten str | object | array into text the model sees. Field names are kept as labels."""
|
| 49 |
+
pad = " " * indent
|
| 50 |
+
if v is None: return ""
|
| 51 |
+
if isinstance(v, (str, int, float, bool)): return str(v)
|
| 52 |
+
if isinstance(v, list): return "\n".join(f"{pad}- {render(x, indent + 1).lstrip()}" for x in v)
|
| 53 |
+
return "\n".join(f"{pad}{k}:\n{render(x, indent + 1)}" if isinstance(x, (dict, list)) else f"{pad}{k}: {render(x)}" for k, x in v.items())
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def option_text(name: str, desc: JSONContent) -> str:
|
| 57 |
+
return name if desc is None or desc == "" else f"{name}: {render(desc)}"
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
MONTHS = "January|February|March|April|May|June|July|August|September|October|November|December"
|
| 61 |
+
_DATE = re.compile(rf"\b(?:{MONTHS}) \d{{1,2}}, \d{{4}}\b|\b\d{{4}}-\d{{2}}-\d{{2}}\b")
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def date_facts(text: str) -> str:
|
| 65 |
+
"""Deterministic date arithmetic for the model: every pair of absolute dates found in `text`, as one sentence each
|
| 66 |
+
("August 3, 2026 is 12 days after July 22, 2026."). The model cannot subtract dates reliably (issue #8); it can use a
|
| 67 |
+
stated day count. Returns "" when fewer than two dates are found. Dates are listed in order of first appearance."""
|
| 68 |
+
from datetime import datetime
|
| 69 |
+
found = []
|
| 70 |
+
for m in _DATE.finditer(text):
|
| 71 |
+
raw = m.group(0)
|
| 72 |
+
try: d = datetime.strptime(raw, "%B %d, %Y") if "," in raw else datetime.strptime(raw, "%Y-%m-%d")
|
| 73 |
+
except ValueError: continue
|
| 74 |
+
if raw not in [r for r, _ in found]: found.append((raw, d))
|
| 75 |
+
facts = []
|
| 76 |
+
for i in range(len(found)):
|
| 77 |
+
for j in range(i + 1, len(found)):
|
| 78 |
+
n = (found[j][1] - found[i][1]).days
|
| 79 |
+
facts.append(f"{found[j][0]} is {abs(n)} day{'s' if abs(n) != 1 else ''} {'after' if n > 0 else 'before'} {found[i][0]}." if n else f"{found[j][0]} is the same day as {found[i][0]}.")
|
| 80 |
+
return " ".join(facts)
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def with_date_facts(state):
|
| 84 |
+
"""State with a `date_facts` field (object states) or an appended paragraph (string states) when two or more absolute
|
| 85 |
+
dates appear. Opt-in preprocessing (KEV_DATE_FACTS=1 in kev.serve, --date_facts in kev.benchmark)."""
|
| 86 |
+
facts = date_facts(render(state))
|
| 87 |
+
if not facts: return state
|
| 88 |
+
if isinstance(state, dict): return {**state, "date_facts": facts}
|
| 89 |
+
if isinstance(state, list): return state + [{"date_facts": facts}]
|
| 90 |
+
return f"{state}\n\ndate_facts: {facts}"
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def to_record(req: SystemOneRequest):
|
| 94 |
+
"""-> internal record for encode(), plus per-question metadata to map probabilities back."""
|
| 95 |
+
qs, meta = [], []
|
| 96 |
+
for qid, q in req.questions.items():
|
| 97 |
+
instr = render(q.instructions)
|
| 98 |
+
if q.type == "noul":
|
| 99 |
+
c = q.criteria or {}
|
| 100 |
+
opts = [option_text("no", c.get("false")), option_text("yes", c.get("true"))]
|
| 101 |
+
meta.append({"id": qid, "type": "noul"})
|
| 102 |
+
elif q.type == "choice":
|
| 103 |
+
opts = [option_text(k, v) for k, v in q.criteria.items()]
|
| 104 |
+
meta.append({"id": qid, "type": "choice", "keys": list(q.criteria.keys())})
|
| 105 |
+
else:
|
| 106 |
+
opts = [render(x) for x in q.criteria]
|
| 107 |
+
meta.append({"id": qid, "type": "score", "legend": {str(i): render(x) for i, x in enumerate(q.criteria)}})
|
| 108 |
+
qs.append({"instr": instr, "options": opts, "label": 0})
|
| 109 |
+
return {"state": render(req.state), "questions": qs}, meta
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def choice_confidence(p: list[float]) -> float:
|
| 113 |
+
K = len(p)
|
| 114 |
+
return 1.0 if K == 1 else (max(p) - 1 / K) / (1 - 1 / K)
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
def score_confidence(p: list[float]) -> float:
|
| 118 |
+
"""Approximation of TypeSafe's 'distance from the modal level' statistic (exact formula unpublished):
|
| 119 |
+
1 - E|level - mode| / (L - 1)."""
|
| 120 |
+
L = len(p); mode = max(range(L), key=lambda i: p[i])
|
| 121 |
+
return 1.0 - sum(pi * abs(i - mode) for i, pi in enumerate(p)) / (L - 1)
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
def r2(x: float) -> float:
|
| 125 |
+
return round(float(x), 2)
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
def to_answers(probs: list[list[float]], meta: list[dict]) -> dict[str, Any]:
|
| 129 |
+
out = {}
|
| 130 |
+
for p, m in zip(probs, meta):
|
| 131 |
+
if m["type"] == "noul":
|
| 132 |
+
out[m["id"]] = {"type": "noul", "noul": r2(p[1])}
|
| 133 |
+
elif m["type"] == "choice":
|
| 134 |
+
dist = {k: r2(v) for k, v in zip(m["keys"], p)}
|
| 135 |
+
out[m["id"]] = {"type": "choice", "choice": m["keys"][max(range(len(p)), key=lambda i: p[i])], "confidence": r2(choice_confidence(p)), "probabilities": dist}
|
| 136 |
+
else:
|
| 137 |
+
score = sum(i * pi for i, pi in enumerate(p))
|
| 138 |
+
out[m["id"]] = {"type": "score", "score": r2(score), "legend": m["legend"], "probabilities": {str(i): r2(v) for i, v in enumerate(p)}, "confidence": r2(score_confidence(p))}
|
| 139 |
+
return out
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
def output_tokens(tok, answers: dict) -> int:
|
| 143 |
+
"""Billing-style figure: tokens of the serialised answers. Not a measure of generation (there is none)."""
|
| 144 |
+
import json
|
| 145 |
+
return len(tok(json.dumps(answers), add_special_tokens=False).input_ids)
|
sys1mlx/cli.py
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""One Kev request and response per JSON line."""
|
| 2 |
+
import argparse
|
| 3 |
+
import json
|
| 4 |
+
import time
|
| 5 |
+
from .vision_loader import load_vision
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
def main():
|
| 9 |
+
parser = argparse.ArgumentParser()
|
| 10 |
+
parser.add_argument('--model-dir')
|
| 11 |
+
parser.add_argument('--max-image-tokens', type=int, default=256)
|
| 12 |
+
parser.add_argument('--cache-mb', type=int, default=64)
|
| 13 |
+
parser.add_argument('--prefix', action='store_true')
|
| 14 |
+
args = parser.parse_args()
|
| 15 |
+
model = load_vision(args.model_dir, args.max_image_tokens, args.cache_mb)
|
| 16 |
+
import sys
|
| 17 |
+
for line in sys.stdin:
|
| 18 |
+
if not line.strip():
|
| 19 |
+
continue
|
| 20 |
+
try:
|
| 21 |
+
start = time.perf_counter()
|
| 22 |
+
result = model.predict(json.loads(line), use_prefix=args.prefix)
|
| 23 |
+
result['latency_ms'] = 1000 * (time.perf_counter() - start)
|
| 24 |
+
print(json.dumps(result, ensure_ascii=False, allow_nan=False), flush=True)
|
| 25 |
+
except (ValueError, TypeError) as error:
|
| 26 |
+
print(json.dumps({'error': str(error)}, ensure_ascii=False), flush=True)
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
if __name__ == '__main__':
|
| 30 |
+
main()
|
sys1mlx/encoding.py
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Tokenizer functions extracted unchanged from Kev (Apache-2.0)."""
|
| 2 |
+
import re
|
| 3 |
+
|
| 4 |
+
SPECIAL = ["<|fim_prefix|>", "<|fim_middle|>", "<|box_start|>", "<|box_end|>", "<|fim_suffix|>"]
|
| 5 |
+
|
| 6 |
+
MAX_STATE, MAX_BRANCH = 384, 1024
|
| 7 |
+
|
| 8 |
+
_SPECIAL_RE = re.compile(r"<\|([A-Za-z0-9_]+)\|>")
|
| 9 |
+
|
| 10 |
+
def user_tokens(tok, text):
|
| 11 |
+
"""Tokenize caller-supplied text so it can never produce delimiter/control tokens (option boundaries are unforgeable).
|
| 12 |
+
The fast tokenizer ignores split_special_tokens, so `<|name|>` is rewritten to `<¦name¦>` before tokenizing."""
|
| 13 |
+
return tok(_SPECIAL_RE.sub(r"<¦\1¦>", text), add_special_tokens=False).input_ids
|
| 14 |
+
|
| 15 |
+
OPT_NONE, OPT_DECIDE = -1, -2
|
| 16 |
+
|
| 17 |
+
def encode(tok, rec, max_state=MAX_STATE, max_branch=MAX_BRANCH, strict=False, option_isolation=False):
|
| 18 |
+
"""Pack one record: [<state> ...] then per-question [<q> instr <opt> o </opt>... <decide>].
|
| 19 |
+
|
| 20 |
+
Returns ids, seg (0 = state, k = question k), pos (branch positions restart after state),
|
| 21 |
+
decide_idx [Q], opt_idx [Q][K] (index of </opt> token for each option), opt (per-token option index within its
|
| 22 |
+
question: OPT_NONE for state/instruction, 0..K-1 for option spans, OPT_DECIDE for <decide>).
|
| 23 |
+
|
| 24 |
+
option_isolation=True: every option span is its own sub-branch (it sees state + instruction + itself only), all
|
| 25 |
+
option spans share the same position ids, and <decide> sits at one fixed position after the longest span. Then the
|
| 26 |
+
per-option representations and <decide>'s attention over them are permutation-invariant by construction.
|
| 27 |
+
"""
|
| 28 |
+
state_tokens = user_tokens(tok, rec["state"])
|
| 29 |
+
if strict and len(state_tokens) + 1 > max_state:
|
| 30 |
+
raise ValueError(f"state exceeds {max_state} tokens: {len(state_tokens) + 1}")
|
| 31 |
+
S = [tok.convert_tokens_to_ids(SPECIAL[0])] + state_tokens[: max_state - 1]
|
| 32 |
+
ids, seg, pos, opt = list(S), [0] * len(S), list(range(len(S))), [OPT_NONE] * len(S)
|
| 33 |
+
q_id, o_id, c_id, d_id = (tok.convert_tokens_to_ids(t) for t in SPECIAL[1:])
|
| 34 |
+
decide_idx, opt_idx = [], []
|
| 35 |
+
for k, q in enumerate(rec["questions"], start=1):
|
| 36 |
+
instr = [q_id] + user_tokens(tok, q["instr"])
|
| 37 |
+
spans = [[o_id] + user_tokens(tok, o) + [c_id] for o in q["options"]]
|
| 38 |
+
br = instr + [t for sp in spans for t in sp] + [d_id]
|
| 39 |
+
if len(br) > max_branch - len(S):
|
| 40 |
+
raise ValueError(f"branch too long: {len(br)}")
|
| 41 |
+
base = len(ids); p0 = len(S)
|
| 42 |
+
br_opt = [OPT_NONE] * len(instr) + [j for j, sp in enumerate(spans) for _ in sp] + [OPT_DECIDE]
|
| 43 |
+
if option_isolation:
|
| 44 |
+
longest = max(len(sp) for sp in spans)
|
| 45 |
+
br_pos = list(range(p0, p0 + len(instr))) + [p0 + len(instr) + i for sp in spans for i in range(len(sp))] + [p0 + len(instr) + longest]
|
| 46 |
+
else:
|
| 47 |
+
br_pos = list(range(p0, p0 + len(br)))
|
| 48 |
+
ends, cursor = [], len(instr)
|
| 49 |
+
for sp in spans:
|
| 50 |
+
cursor += len(sp); ends.append(cursor - 1)
|
| 51 |
+
ids += br; seg += [k] * len(br); pos += br_pos; opt += br_opt
|
| 52 |
+
decide_idx.append(base + len(br) - 1); opt_idx.append([base + e for e in ends])
|
| 53 |
+
return {"ids": ids, "seg": seg, "pos": pos, "opt": opt, "option_isolation": option_isolation, "decide_idx": decide_idx, "opt_idx": opt_idx,
|
| 54 |
+
"labels": [q["label"] for q in rec["questions"]], "state_truncated": len(state_tokens) + 1 > max_state}
|
| 55 |
+
|
| 56 |
+
def rows_of(enc):
|
| 57 |
+
"""Split a packed encoding into its state and per-question branch rows.
|
| 58 |
+
|
| 59 |
+
Returns (state_ids, state_pos, rows) with rows[k] = {"ids", "pos", "decide", "opts"}: the branch tokens of question
|
| 60 |
+
k with their (already state-continuing) positions, and the readout offsets *within the branch*. Feeding
|
| 61 |
+
state + rows[k] as one causal row is equivalent to the packed block-causal form for that question, on any
|
| 62 |
+
architecture: the row contains exactly the tokens question k may attend to, in the same positions."""
|
| 63 |
+
seg = enc["seg"]; Ls = seg.count(0)
|
| 64 |
+
rows, start = [], Ls
|
| 65 |
+
for k, (d, oi) in enumerate(zip(enc["decide_idx"], enc["opt_idx"]), start=1):
|
| 66 |
+
end = d + 1 # <decide> is the last token of its branch
|
| 67 |
+
if seg[start] != k or seg[end - 1] != k: raise ValueError("branch layout mismatch")
|
| 68 |
+
rows.append({"ids": enc["ids"][start:end], "pos": enc["pos"][start:end], "decide": d - start, "opts": [o - start for o in oi]})
|
| 69 |
+
start = end
|
| 70 |
+
return enc["ids"][:Ls], enc["pos"][:Ls], rows
|
sys1mlx/runtime.py
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Kev's original encoding and pointer readout on the native MLX backbone.
|
| 2 |
+
|
| 3 |
+
Questions are independent causal rows. No vocabulary projection or generation.
|
| 4 |
+
The optional prefix cache is copied for each branch, including recurrent state.
|
| 5 |
+
"""
|
| 6 |
+
import ast
|
| 7 |
+
import copy
|
| 8 |
+
import json
|
| 9 |
+
import math
|
| 10 |
+
from pathlib import Path
|
| 11 |
+
import re
|
| 12 |
+
import sys
|
| 13 |
+
import mlx.core as mx
|
| 14 |
+
import mlx.nn as nn
|
| 15 |
+
from mlx_lm.models.qwen3_5 import Model, ModelArgs
|
| 16 |
+
|
| 17 |
+
from .api import SystemOneRequest, to_record, to_answers
|
| 18 |
+
from .encoding import encode, rows_of
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
class PointerHead(nn.Module):
|
| 22 |
+
def __init__(self,hidden_size,head_size=256):
|
| 23 |
+
super().__init__()
|
| 24 |
+
self.q=nn.Linear(hidden_size,head_size)
|
| 25 |
+
self.k=nn.Linear(hidden_size,head_size)
|
| 26 |
+
self.scale=1/math.sqrt(head_size)
|
| 27 |
+
|
| 28 |
+
def __call__(self,decide,options):
|
| 29 |
+
return (self.k(options.astype(mx.float32))@self.q(decide.astype(mx.float32)))*self.scale
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def backbone(config):
|
| 33 |
+
config=copy.deepcopy(config)
|
| 34 |
+
# Kev discards the language-model head; tying prevents allocating its unused copy.
|
| 35 |
+
config['text_config']['tie_word_embeddings']=True
|
| 36 |
+
return Model(ModelArgs.from_dict(config))
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
class DecisionModel:
|
| 40 |
+
def __init__(self,model,tokenizer,head):
|
| 41 |
+
self.model,self.tokenizer,self.head=model,tokenizer,head
|
| 42 |
+
model.eval();head.eval()
|
| 43 |
+
|
| 44 |
+
def encoded(self,request):
|
| 45 |
+
request=SystemOneRequest.model_validate(request)
|
| 46 |
+
record,meta=to_record(request)
|
| 47 |
+
enc=encode(self.tokenizer,record,strict=True)
|
| 48 |
+
if len(enc['ids'])>2048:raise ValueError('packed request exceeds frozen 2048-token limit')
|
| 49 |
+
return enc,meta
|
| 50 |
+
|
| 51 |
+
def distributions(self,enc,use_prefix=False):
|
| 52 |
+
state,_,rows=rows_of(enc)
|
| 53 |
+
prefix=None
|
| 54 |
+
if use_prefix:
|
| 55 |
+
prefix=self.model.language_model.make_cache()
|
| 56 |
+
self.model.model(mx.array([state]),cache=prefix)
|
| 57 |
+
mx.eval([c.state for c in prefix])
|
| 58 |
+
result=[]
|
| 59 |
+
for row in rows:
|
| 60 |
+
if prefix is None:
|
| 61 |
+
h=self.model.model(mx.array([state+row['ids']]))[0]
|
| 62 |
+
offset=len(state)
|
| 63 |
+
else:
|
| 64 |
+
cache=copy.deepcopy(prefix)
|
| 65 |
+
h=self.model.model(mx.array([row['ids']]),cache=cache)[0]
|
| 66 |
+
offset=0
|
| 67 |
+
z=self.head(h[offset+row['decide']],h[mx.array([offset+x for x in row['opts']])])
|
| 68 |
+
p=mx.softmax(z)
|
| 69 |
+
result.append(p.tolist())
|
| 70 |
+
return result
|
| 71 |
+
|
| 72 |
+
def predict(self,request,use_prefix=False):
|
| 73 |
+
enc,meta=self.encoded(request)
|
| 74 |
+
probabilities=self.distributions(enc,use_prefix=use_prefix)
|
| 75 |
+
return {'answers':to_answers(probabilities,meta),'probabilities':probabilities,
|
| 76 |
+
'input_tokens':len(enc['ids'])}
|
sys1mlx/vision.py
ADDED
|
@@ -0,0 +1,121 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""One-image Kev decisions using Qwen3.5 vision, MRoPE and the trained pointer head."""
|
| 2 |
+
import copy
|
| 3 |
+
import hashlib
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
import numpy as np
|
| 6 |
+
from PIL import Image, ImageOps
|
| 7 |
+
import mlx.core as mx
|
| 8 |
+
from mlx_vlm.models.qwen3_vl.processing_qwen3_vl import Qwen3VLImageProcessor
|
| 9 |
+
from .runtime import DecisionModel, rows_of, to_answers
|
| 10 |
+
from .vision_cache import VisionCache
|
| 11 |
+
from mlx_vlm.models.cache import KVCache
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
class VisionDecisionModel(DecisionModel):
|
| 15 |
+
def __init__(self,model,tokenizer,head,processor_config,max_image_tokens=256,cache_mb=64):
|
| 16 |
+
super().__init__(model,tokenizer,head)
|
| 17 |
+
if not 16 <= max_image_tokens <= 512:
|
| 18 |
+
raise ValueError('max_image_tokens must be between 16 and 512')
|
| 19 |
+
self.max_image_tokens=max_image_tokens
|
| 20 |
+
self.cache=VisionCache(int(cache_mb*1024**2))
|
| 21 |
+
config=dict(processor_config)
|
| 22 |
+
config.update(min_pixels=32*32*16,max_pixels=32*32*max_image_tokens)
|
| 23 |
+
self.processor=Qwen3VLImageProcessor(**config)
|
| 24 |
+
|
| 25 |
+
def encoded(self,request):
|
| 26 |
+
request=dict(request)
|
| 27 |
+
image_path=request.pop('image_path',None)
|
| 28 |
+
enc,meta=super().encoded(request)
|
| 29 |
+
if image_path is None:return enc,meta
|
| 30 |
+
if not isinstance(image_path,str) or not image_path:
|
| 31 |
+
raise ValueError('image_path must be a nonempty local file path')
|
| 32 |
+
path=Path(image_path).expanduser()
|
| 33 |
+
if not path.is_file():raise ValueError('image_path is not a local file')
|
| 34 |
+
try:
|
| 35 |
+
with path.open('rb') as file, Image.open(file) as source:
|
| 36 |
+
if getattr(source,'n_frames',1)!=1:raise ValueError('Only one still image is supported')
|
| 37 |
+
if source.width*source.height>32_000_000:
|
| 38 |
+
raise ValueError('Image exceeds the 32-megapixel decode limit')
|
| 39 |
+
source_size=list(source.size)
|
| 40 |
+
# Hash file contents on every request: paths and timestamps can stay
|
| 41 |
+
# unchanged when a screenshot is overwritten. Decode only on a miss.
|
| 42 |
+
digest=hashlib.sha256();file.seek(0)
|
| 43 |
+
for block in iter(lambda:file.read(1024**2),b''):digest.update(block)
|
| 44 |
+
image_key=('image',digest.hexdigest(),self.max_image_tokens)
|
| 45 |
+
cached=self.cache.get(image_key) if self.cache.max_bytes else None
|
| 46 |
+
if cached is not None:
|
| 47 |
+
if len(enc['ids'])+cached['tokens']+2>2048:
|
| 48 |
+
raise ValueError('Image and text exceed packed 2048-token limit')
|
| 49 |
+
enc['vision']=dict(cached,pixels=None,image_key=image_key)
|
| 50 |
+
return enc,meta
|
| 51 |
+
file.seek(0)
|
| 52 |
+
picture=ImageOps.exif_transpose(source).convert('RGB')
|
| 53 |
+
data=self.processor(picture)
|
| 54 |
+
except (OSError,Image.DecompressionBombError) as error:
|
| 55 |
+
raise ValueError('Cannot decode image: '+str(error)) from error
|
| 56 |
+
grid=data['image_grid_thw']
|
| 57 |
+
count=int(np.prod(grid))//self.processor.merge_size**2
|
| 58 |
+
if count>self.max_image_tokens:raise ValueError('Image token budget exceeded')
|
| 59 |
+
if len(enc['ids'])+count+2>2048:raise ValueError('Image and text exceed packed 2048-token limit')
|
| 60 |
+
enc['vision']={'image_key':image_key,'pixels':mx.array(data['pixel_values']), 'grid':mx.array(grid,dtype=mx.int32),
|
| 61 |
+
'tokens':count,'source_size':source_size,
|
| 62 |
+
'processed_size':[int(grid[0,2])*16,int(grid[0,1])*16]}
|
| 63 |
+
return enc,meta
|
| 64 |
+
|
| 65 |
+
def distributions(self,enc,use_prefix=False):
|
| 66 |
+
state,_,rows=rows_of(enc);visual=enc.get('vision');features=None;prefix=None;prefix_key=None
|
| 67 |
+
model=self.model;lm=model.language_model
|
| 68 |
+
if visual:
|
| 69 |
+
c=model.config
|
| 70 |
+
block=[c.vision_start_token_id]+[c.image_token_id]*visual['tokens']+[c.vision_end_token_id]
|
| 71 |
+
state=state[:1]+block+state[1:]
|
| 72 |
+
if use_prefix and self.cache.max_bytes and 'image_key' in visual:
|
| 73 |
+
prefix_key=('prefix',visual['image_key'],tuple(state))
|
| 74 |
+
cached=self.cache.get(prefix_key)
|
| 75 |
+
if cached is not None:prefix,features=cached['cache'],cached['features']
|
| 76 |
+
if features is None:features=visual.get('features')
|
| 77 |
+
if features is None:
|
| 78 |
+
pixels=visual['pixels'].astype(model.vision_tower.patch_embed.proj.weight.dtype)
|
| 79 |
+
features,_=model.vision_tower(pixels,visual['grid']);mx.eval(features)
|
| 80 |
+
if self.cache.max_bytes and 'image_key' in visual:
|
| 81 |
+
cached={k:visual[k] for k in ('grid','tokens','source_size','processed_size')}
|
| 82 |
+
self.cache.put(visual['image_key'],dict(cached,features=features))
|
| 83 |
+
def embeddings(ids):
|
| 84 |
+
if visual:
|
| 85 |
+
return model.get_input_embeddings(ids,pixel_values=(visual['pixels'] if visual['pixels'] is not None else mx.zeros((0,))),
|
| 86 |
+
image_grid_thw=visual['grid'],cached_image_features=features)
|
| 87 |
+
return model.get_input_embeddings(ids)
|
| 88 |
+
if use_prefix and prefix is None:
|
| 89 |
+
ids=mx.array([state]);embedded=embeddings(ids);prefix=lm.make_cache()
|
| 90 |
+
lm.model(ids,inputs_embeds=embedded.inputs_embeds,position_ids=embedded.position_ids,cache=prefix)
|
| 91 |
+
mx.eval([c.state for c in prefix])
|
| 92 |
+
if prefix_key is not None:
|
| 93 |
+
# A saved prefix needs only used positions, not MLX's 256-token
|
| 94 |
+
# allocation reserve. This keeps a 256-token image within 64 MiB.
|
| 95 |
+
for cache in prefix:
|
| 96 |
+
if isinstance(cache,KVCache):
|
| 97 |
+
cache.keys=mx.contiguous(cache.keys[..., :cache.offset, :])
|
| 98 |
+
cache.values=mx.contiguous(cache.values[..., :cache.offset, :])
|
| 99 |
+
mx.eval(cache.keys,cache.values)
|
| 100 |
+
self.cache.put(prefix_key,{'cache':prefix,'features':features})
|
| 101 |
+
result=[]
|
| 102 |
+
for row in rows:
|
| 103 |
+
ids=mx.array([state+row['ids']]);embedded=embeddings(ids)
|
| 104 |
+
offset=len(state)
|
| 105 |
+
if prefix is None:
|
| 106 |
+
h=lm.model(ids,inputs_embeds=embedded.inputs_embeds,position_ids=embedded.position_ids)[0]
|
| 107 |
+
else:
|
| 108 |
+
h=lm.model(ids[:,offset:],inputs_embeds=embedded.inputs_embeds[:,offset:],
|
| 109 |
+
position_ids=embedded.position_ids[...,offset:],cache=copy.deepcopy(prefix))[0]
|
| 110 |
+
offset=0
|
| 111 |
+
z=self.head(h[offset+row['decide']],h[mx.array([offset+x for x in row['opts']])])
|
| 112 |
+
result.append(mx.softmax(z).tolist())
|
| 113 |
+
return result
|
| 114 |
+
|
| 115 |
+
def predict(self,request,use_prefix=False):
|
| 116 |
+
enc,meta=self.encoded(request)
|
| 117 |
+
probabilities=self.distributions(enc,use_prefix)
|
| 118 |
+
visual=enc.get('vision');extra=visual['tokens']+2 if visual else 0
|
| 119 |
+
return {'answers':to_answers(probabilities,meta),'probabilities':probabilities,
|
| 120 |
+
'input_tokens':len(enc['ids'])+extra,
|
| 121 |
+
'vision':None if not visual else {k:visual[k] for k in ('tokens','source_size','processed_size')}}
|
sys1mlx/vision_backbone.py
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Qwen3.5 vision with the same DeltaNet normalization as the verified text port."""
|
| 2 |
+
from mlx_lm.models.gated_delta import normalize_qk
|
| 3 |
+
from mlx_vlm.models.qwen3_5 import Model
|
| 4 |
+
from mlx_vlm.models.qwen3_5.language import Qwen3_5GatedDeltaNet
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
class Sys1GatedDeltaNet(Qwen3_5GatedDeltaNet):
|
| 8 |
+
def _normalize_qk(self,q,k):
|
| 9 |
+
# HF adds epsilon to SUM(x^2); rms_norm adds it to MEAN(x^2).
|
| 10 |
+
# mlx-vlm 0.7.2 passes 1e-6 directly, unlike the verified MLX-LM path.
|
| 11 |
+
return normalize_qk(q,k,inv_scale=self.head_k_dim**-0.5,eps=1e-6)
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
class Sys1VisionBackbone(Model):
|
| 15 |
+
def __init__(self,config):
|
| 16 |
+
super().__init__(config)
|
| 17 |
+
for layer in self.language_model.model.layers:
|
| 18 |
+
if layer.is_linear:
|
| 19 |
+
corrected=Sys1GatedDeltaNet(config.text_config)
|
| 20 |
+
corrected.update(layer.linear_attn.parameters())
|
| 21 |
+
layer.linear_attn=corrected
|
sys1mlx/vision_cache.py
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Byte-bounded, process-local LRU for immutable image features and prefix snapshots."""
|
| 2 |
+
from collections import OrderedDict
|
| 3 |
+
import mlx.core as mx
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
def retained_bytes(value):
|
| 7 |
+
if isinstance(value,mx.array):return value.nbytes
|
| 8 |
+
if isinstance(value,dict):return sum(retained_bytes(v) for v in value.values())
|
| 9 |
+
if isinstance(value,(list,tuple)):return sum(retained_bytes(v) for v in value)
|
| 10 |
+
# MLX cache.nbytes counts allocated KV capacity, not only its used slice.
|
| 11 |
+
return int(getattr(value,'nbytes',0))
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
class VisionCache:
|
| 15 |
+
def __init__(self,max_bytes=64*1024**2,max_entries=8):
|
| 16 |
+
if max_bytes<0:raise ValueError('cache budget must not be negative')
|
| 17 |
+
self.max_bytes=max_bytes;self.max_entries=max_entries
|
| 18 |
+
self.entries=OrderedDict();self.bytes=0;self.hits=0;self.misses=0;self.evictions=0
|
| 19 |
+
|
| 20 |
+
def get(self,key):
|
| 21 |
+
if key not in self.entries:
|
| 22 |
+
self.misses+=1;return None
|
| 23 |
+
self.hits+=1;self.entries.move_to_end(key);return self.entries[key][0]
|
| 24 |
+
|
| 25 |
+
def put(self,key,value):
|
| 26 |
+
size=retained_bytes(value)
|
| 27 |
+
if key in self.entries:self.bytes-=self.entries.pop(key)[1]
|
| 28 |
+
if not self.max_bytes or size>self.max_bytes:return False
|
| 29 |
+
while self.entries and (self.bytes+size>self.max_bytes or len(self.entries)>=self.max_entries):
|
| 30 |
+
_,(_,old_size)=self.entries.popitem(last=False);self.bytes-=old_size;self.evictions+=1
|
| 31 |
+
self.entries[key]=(value,size);self.bytes+=size;return True
|
| 32 |
+
|
| 33 |
+
def clear(self):
|
| 34 |
+
self.entries.clear();self.bytes=0
|
| 35 |
+
|
| 36 |
+
def info(self):
|
| 37 |
+
return {'bytes':self.bytes,'limit_bytes':self.max_bytes,'entries':len(self.entries),
|
| 38 |
+
'hits':self.hits,'misses':self.misses,'evictions':self.evictions}
|
sys1mlx/vision_loader.py
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Load the published checkpoint locally, without upstream conversion/downloads."""
|
| 2 |
+
import hashlib
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
import mlx.core as mx
|
| 6 |
+
import mlx.nn as nn
|
| 7 |
+
from transformers import AutoTokenizer
|
| 8 |
+
from mlx_vlm.models.qwen3_5 import ModelConfig
|
| 9 |
+
from .vision_backbone import Sys1VisionBackbone
|
| 10 |
+
from .runtime import PointerHead
|
| 11 |
+
from .vision import VisionDecisionModel
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
def load_vision(model_dir=None, max_image_tokens=256, cache_mb=64):
|
| 15 |
+
root = Path(model_dir) if model_dir else Path(__file__).resolve().parents[1]
|
| 16 |
+
checkpoint = root / 'model.safetensors'
|
| 17 |
+
metadata = json.loads((root / 'provenance.json').read_text())
|
| 18 |
+
digest = hashlib.sha256()
|
| 19 |
+
with checkpoint.open('rb') as stream:
|
| 20 |
+
for block in iter(lambda: stream.read(8 * 1024**2), b''):
|
| 21 |
+
digest.update(block)
|
| 22 |
+
if digest.hexdigest() != metadata['sha256']:
|
| 23 |
+
raise ValueError('Checkpoint checksum mismatch')
|
| 24 |
+
mx.set_cache_limit(128 * 1024**2)
|
| 25 |
+
config = ModelConfig.from_dict(json.loads((root / 'config.json').read_text()))
|
| 26 |
+
model = Sys1VisionBackbone(config)
|
| 27 |
+
nn.quantize(model, group_size=64, bits=4, class_predicate=lambda path, m:
|
| 28 |
+
path.startswith('language_model.') and isinstance(m, (nn.Linear, nn.Embedding)))
|
| 29 |
+
model.load_weights(list(mx.load(str(checkpoint)).items()), strict=True)
|
| 30 |
+
head_config = json.loads((root / 'head_config.json').read_text())
|
| 31 |
+
head = PointerHead(config.text_config.hidden_size, head_config['head_dim'])
|
| 32 |
+
head.load_weights(list(mx.load(str(root / 'head.safetensors')).items()), strict=True)
|
| 33 |
+
tokenizer = AutoTokenizer.from_pretrained(root, trust_remote_code=False, local_files_only=True)
|
| 34 |
+
processor = json.loads((root / 'preprocessor_config.json').read_text())
|
| 35 |
+
mx.eval(model.parameters(), head.parameters())
|
| 36 |
+
mx.clear_cache()
|
| 37 |
+
return VisionDecisionModel(model, tokenizer, head, processor, max_image_tokens, cache_mb)
|
tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fe000e3ed39ed12b8d2481d527d44f93c65d37e87645d2dcc80d1bf9d50d2927
|
| 3 |
+
size 12807196
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,305 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_prefix_space": false,
|
| 3 |
+
"added_tokens_decoder": {
|
| 4 |
+
"248044": {
|
| 5 |
+
"content": "<|endoftext|>",
|
| 6 |
+
"lstrip": false,
|
| 7 |
+
"normalized": false,
|
| 8 |
+
"rstrip": false,
|
| 9 |
+
"single_word": false,
|
| 10 |
+
"special": true
|
| 11 |
+
},
|
| 12 |
+
"248045": {
|
| 13 |
+
"content": "<|im_start|>",
|
| 14 |
+
"lstrip": false,
|
| 15 |
+
"normalized": false,
|
| 16 |
+
"rstrip": false,
|
| 17 |
+
"single_word": false,
|
| 18 |
+
"special": true
|
| 19 |
+
},
|
| 20 |
+
"248046": {
|
| 21 |
+
"content": "<|im_end|>",
|
| 22 |
+
"lstrip": false,
|
| 23 |
+
"normalized": false,
|
| 24 |
+
"rstrip": false,
|
| 25 |
+
"single_word": false,
|
| 26 |
+
"special": true
|
| 27 |
+
},
|
| 28 |
+
"248047": {
|
| 29 |
+
"content": "<|object_ref_start|>",
|
| 30 |
+
"lstrip": false,
|
| 31 |
+
"normalized": false,
|
| 32 |
+
"rstrip": false,
|
| 33 |
+
"single_word": false,
|
| 34 |
+
"special": true
|
| 35 |
+
},
|
| 36 |
+
"248048": {
|
| 37 |
+
"content": "<|object_ref_end|>",
|
| 38 |
+
"lstrip": false,
|
| 39 |
+
"normalized": false,
|
| 40 |
+
"rstrip": false,
|
| 41 |
+
"single_word": false,
|
| 42 |
+
"special": true
|
| 43 |
+
},
|
| 44 |
+
"248049": {
|
| 45 |
+
"content": "<|box_start|>",
|
| 46 |
+
"lstrip": false,
|
| 47 |
+
"normalized": false,
|
| 48 |
+
"rstrip": false,
|
| 49 |
+
"single_word": false,
|
| 50 |
+
"special": true
|
| 51 |
+
},
|
| 52 |
+
"248050": {
|
| 53 |
+
"content": "<|box_end|>",
|
| 54 |
+
"lstrip": false,
|
| 55 |
+
"normalized": false,
|
| 56 |
+
"rstrip": false,
|
| 57 |
+
"single_word": false,
|
| 58 |
+
"special": true
|
| 59 |
+
},
|
| 60 |
+
"248051": {
|
| 61 |
+
"content": "<|quad_start|>",
|
| 62 |
+
"lstrip": false,
|
| 63 |
+
"normalized": false,
|
| 64 |
+
"rstrip": false,
|
| 65 |
+
"single_word": false,
|
| 66 |
+
"special": true
|
| 67 |
+
},
|
| 68 |
+
"248052": {
|
| 69 |
+
"content": "<|quad_end|>",
|
| 70 |
+
"lstrip": false,
|
| 71 |
+
"normalized": false,
|
| 72 |
+
"rstrip": false,
|
| 73 |
+
"single_word": false,
|
| 74 |
+
"special": true
|
| 75 |
+
},
|
| 76 |
+
"248053": {
|
| 77 |
+
"content": "<|vision_start|>",
|
| 78 |
+
"lstrip": false,
|
| 79 |
+
"normalized": false,
|
| 80 |
+
"rstrip": false,
|
| 81 |
+
"single_word": false,
|
| 82 |
+
"special": true
|
| 83 |
+
},
|
| 84 |
+
"248054": {
|
| 85 |
+
"content": "<|vision_end|>",
|
| 86 |
+
"lstrip": false,
|
| 87 |
+
"normalized": false,
|
| 88 |
+
"rstrip": false,
|
| 89 |
+
"single_word": false,
|
| 90 |
+
"special": true
|
| 91 |
+
},
|
| 92 |
+
"248055": {
|
| 93 |
+
"content": "<|vision_pad|>",
|
| 94 |
+
"lstrip": false,
|
| 95 |
+
"normalized": false,
|
| 96 |
+
"rstrip": false,
|
| 97 |
+
"single_word": false,
|
| 98 |
+
"special": true
|
| 99 |
+
},
|
| 100 |
+
"248056": {
|
| 101 |
+
"content": "<|image_pad|>",
|
| 102 |
+
"lstrip": false,
|
| 103 |
+
"normalized": false,
|
| 104 |
+
"rstrip": false,
|
| 105 |
+
"single_word": false,
|
| 106 |
+
"special": true
|
| 107 |
+
},
|
| 108 |
+
"248057": {
|
| 109 |
+
"content": "<|video_pad|>",
|
| 110 |
+
"lstrip": false,
|
| 111 |
+
"normalized": false,
|
| 112 |
+
"rstrip": false,
|
| 113 |
+
"single_word": false,
|
| 114 |
+
"special": true
|
| 115 |
+
},
|
| 116 |
+
"248058": {
|
| 117 |
+
"content": "<tool_call>",
|
| 118 |
+
"lstrip": false,
|
| 119 |
+
"normalized": false,
|
| 120 |
+
"rstrip": false,
|
| 121 |
+
"single_word": false,
|
| 122 |
+
"special": false
|
| 123 |
+
},
|
| 124 |
+
"248059": {
|
| 125 |
+
"content": "</tool_call>",
|
| 126 |
+
"lstrip": false,
|
| 127 |
+
"normalized": false,
|
| 128 |
+
"rstrip": false,
|
| 129 |
+
"single_word": false,
|
| 130 |
+
"special": false
|
| 131 |
+
},
|
| 132 |
+
"248060": {
|
| 133 |
+
"content": "<|fim_prefix|>",
|
| 134 |
+
"lstrip": false,
|
| 135 |
+
"normalized": false,
|
| 136 |
+
"rstrip": false,
|
| 137 |
+
"single_word": false,
|
| 138 |
+
"special": false
|
| 139 |
+
},
|
| 140 |
+
"248061": {
|
| 141 |
+
"content": "<|fim_middle|>",
|
| 142 |
+
"lstrip": false,
|
| 143 |
+
"normalized": false,
|
| 144 |
+
"rstrip": false,
|
| 145 |
+
"single_word": false,
|
| 146 |
+
"special": false
|
| 147 |
+
},
|
| 148 |
+
"248062": {
|
| 149 |
+
"content": "<|fim_suffix|>",
|
| 150 |
+
"lstrip": false,
|
| 151 |
+
"normalized": false,
|
| 152 |
+
"rstrip": false,
|
| 153 |
+
"single_word": false,
|
| 154 |
+
"special": false
|
| 155 |
+
},
|
| 156 |
+
"248063": {
|
| 157 |
+
"content": "<|fim_pad|>",
|
| 158 |
+
"lstrip": false,
|
| 159 |
+
"normalized": false,
|
| 160 |
+
"rstrip": false,
|
| 161 |
+
"single_word": false,
|
| 162 |
+
"special": false
|
| 163 |
+
},
|
| 164 |
+
"248064": {
|
| 165 |
+
"content": "<|repo_name|>",
|
| 166 |
+
"lstrip": false,
|
| 167 |
+
"normalized": false,
|
| 168 |
+
"rstrip": false,
|
| 169 |
+
"single_word": false,
|
| 170 |
+
"special": false
|
| 171 |
+
},
|
| 172 |
+
"248065": {
|
| 173 |
+
"content": "<|file_sep|>",
|
| 174 |
+
"lstrip": false,
|
| 175 |
+
"normalized": false,
|
| 176 |
+
"rstrip": false,
|
| 177 |
+
"single_word": false,
|
| 178 |
+
"special": false
|
| 179 |
+
},
|
| 180 |
+
"248066": {
|
| 181 |
+
"content": "<tool_response>",
|
| 182 |
+
"lstrip": false,
|
| 183 |
+
"normalized": false,
|
| 184 |
+
"rstrip": false,
|
| 185 |
+
"single_word": false,
|
| 186 |
+
"special": false
|
| 187 |
+
},
|
| 188 |
+
"248067": {
|
| 189 |
+
"content": "</tool_response>",
|
| 190 |
+
"lstrip": false,
|
| 191 |
+
"normalized": false,
|
| 192 |
+
"rstrip": false,
|
| 193 |
+
"single_word": false,
|
| 194 |
+
"special": false
|
| 195 |
+
},
|
| 196 |
+
"248068": {
|
| 197 |
+
"content": "<think>",
|
| 198 |
+
"lstrip": false,
|
| 199 |
+
"normalized": false,
|
| 200 |
+
"rstrip": false,
|
| 201 |
+
"single_word": false,
|
| 202 |
+
"special": false
|
| 203 |
+
},
|
| 204 |
+
"248069": {
|
| 205 |
+
"content": "</think>",
|
| 206 |
+
"lstrip": false,
|
| 207 |
+
"normalized": false,
|
| 208 |
+
"rstrip": false,
|
| 209 |
+
"single_word": false,
|
| 210 |
+
"special": false
|
| 211 |
+
},
|
| 212 |
+
"248070": {
|
| 213 |
+
"content": "<|audio_start|>",
|
| 214 |
+
"lstrip": false,
|
| 215 |
+
"normalized": false,
|
| 216 |
+
"rstrip": false,
|
| 217 |
+
"single_word": false,
|
| 218 |
+
"special": true
|
| 219 |
+
},
|
| 220 |
+
"248071": {
|
| 221 |
+
"content": "<|audio_end|>",
|
| 222 |
+
"lstrip": false,
|
| 223 |
+
"normalized": false,
|
| 224 |
+
"rstrip": false,
|
| 225 |
+
"single_word": false,
|
| 226 |
+
"special": true
|
| 227 |
+
},
|
| 228 |
+
"248072": {
|
| 229 |
+
"content": "<tts_pad>",
|
| 230 |
+
"lstrip": false,
|
| 231 |
+
"normalized": false,
|
| 232 |
+
"rstrip": false,
|
| 233 |
+
"single_word": false,
|
| 234 |
+
"special": true
|
| 235 |
+
},
|
| 236 |
+
"248073": {
|
| 237 |
+
"content": "<tts_text_bos>",
|
| 238 |
+
"lstrip": false,
|
| 239 |
+
"normalized": false,
|
| 240 |
+
"rstrip": false,
|
| 241 |
+
"single_word": false,
|
| 242 |
+
"special": true
|
| 243 |
+
},
|
| 244 |
+
"248074": {
|
| 245 |
+
"content": "<tts_text_eod>",
|
| 246 |
+
"lstrip": false,
|
| 247 |
+
"normalized": false,
|
| 248 |
+
"rstrip": false,
|
| 249 |
+
"single_word": false,
|
| 250 |
+
"special": true
|
| 251 |
+
},
|
| 252 |
+
"248075": {
|
| 253 |
+
"content": "<tts_text_bos_single>",
|
| 254 |
+
"lstrip": false,
|
| 255 |
+
"normalized": false,
|
| 256 |
+
"rstrip": false,
|
| 257 |
+
"single_word": false,
|
| 258 |
+
"special": true
|
| 259 |
+
},
|
| 260 |
+
"248076": {
|
| 261 |
+
"content": "<|audio_pad|>",
|
| 262 |
+
"lstrip": false,
|
| 263 |
+
"normalized": false,
|
| 264 |
+
"rstrip": false,
|
| 265 |
+
"single_word": false,
|
| 266 |
+
"special": true
|
| 267 |
+
}
|
| 268 |
+
},
|
| 269 |
+
"additional_special_tokens": [
|
| 270 |
+
"<|im_start|>",
|
| 271 |
+
"<|im_end|>",
|
| 272 |
+
"<|object_ref_start|>",
|
| 273 |
+
"<|object_ref_end|>",
|
| 274 |
+
"<|box_start|>",
|
| 275 |
+
"<|box_end|>",
|
| 276 |
+
"<|quad_start|>",
|
| 277 |
+
"<|quad_end|>",
|
| 278 |
+
"<|vision_start|>",
|
| 279 |
+
"<|vision_end|>",
|
| 280 |
+
"<|vision_pad|>",
|
| 281 |
+
"<|image_pad|>",
|
| 282 |
+
"<|video_pad|>"
|
| 283 |
+
],
|
| 284 |
+
"bos_token": null,
|
| 285 |
+
"chat_template": "{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- macro render_content(content, do_vision_count, is_system_content=false) %}\n {%- if content is string %}\n {{- content }}\n {%- elif content is iterable and content is not mapping %}\n {%- for item in content %}\n {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}\n {%- if is_system_content %}\n {{- raise_exception('System message cannot contain images.') }}\n {%- endif %}\n {%- if do_vision_count %}\n {%- set image_count.value = image_count.value + 1 %}\n {%- endif %}\n {%- if add_vision_id %}\n {{- 'Picture ' ~ image_count.value ~ ': ' }}\n {%- endif %}\n {{- '<|vision_start|><|image_pad|><|vision_end|>' }}\n {%- elif 'video' in item or item.type == 'video' %}\n {%- if is_system_content %}\n {{- raise_exception('System message cannot contain videos.') }}\n {%- endif %}\n {%- if do_vision_count %}\n {%- set video_count.value = video_count.value + 1 %}\n {%- endif %}\n {%- if add_vision_id %}\n {{- 'Video ' ~ video_count.value ~ ': ' }}\n {%- endif %}\n {{- '<|vision_start|><|video_pad|><|vision_end|>' }}\n {%- elif 'text' in item %}\n {{- item.text }}\n {%- else %}\n {{- raise_exception('Unexpected item type in content.') }}\n {%- endif %}\n {%- endfor %}\n {%- elif content is none or content is undefined %}\n {{- '' }}\n {%- else %}\n {{- raise_exception('Unexpected content type.') }}\n {%- endif %}\n{%- endmacro %}\n{%- if not messages %}\n {{- raise_exception('No messages provided.') }}\n{%- endif %}\n{%- if tools and tools is iterable and tools is not mapping %}\n {{- '<|im_start|>system\\n' }}\n {{- \"# Tools\\n\\nYou have access to the following functions:\\n\\n<tools>\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n</tools>\" }}\n {{- '\\n\\nIf you choose to call a function ONLY reply in the following format with NO suffix:\\n\\n<tool_call>\\n<function=example_function_name>\\n<parameter=example_parameter_1>\\nvalue_1\\n</parameter>\\n<parameter=example_parameter_2>\\nThis is the value for the second parameter\\nthat can span\\nmultiple lines\\n</parameter>\\n</function>\\n</tool_call>\\n\\n<IMPORTANT>\\nReminder:\\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\\n- Required parameters MUST be specified\\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\\n</IMPORTANT>' }}\n {%- if messages[0].role == 'system' %}\n {%- set content = render_content(messages[0].content, false, true)|trim %}\n {%- if content %}\n {{- '\\n\\n' + content }}\n {%- endif %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n{%- else %}\n {%- if messages[0].role == 'system' %}\n {%- set content = render_content(messages[0].content, false, true)|trim %}\n {{- '<|im_start|>system\\n' + content + '<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if ns.multi_step_tool and message.role == \"user\" %}\n {%- set content = render_content(message.content, false)|trim %}\n {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}\n {%- set ns.multi_step_tool = false %}\n {%- set ns.last_query_index = index %}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if ns.multi_step_tool %}\n {{- raise_exception('No user query found in messages.') }}\n{%- endif %}\n{%- for message in messages %}\n {%- set content = render_content(message.content, true)|trim %}\n {%- if message.role == \"system\" %}\n {%- if not loop.first %}\n {{- raise_exception('System message must be at the beginning.') }}\n {%- endif %}\n {%- elif message.role == \"user\" %}\n {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {%- set reasoning_content = '' %}\n {%- if message.reasoning_content is string %}\n {%- set reasoning_content = message.reasoning_content %}\n {%- else %}\n {%- if '</think>' in content %}\n {%- set reasoning_content = content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n {%- set content = content.split('</think>')[-1].lstrip('\\n') %}\n {%- endif %}\n {%- endif %}\n {%- set reasoning_content = reasoning_content|trim %}\n {%- if loop.index0 > ns.last_query_index %}\n {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content + '\\n</think>\\n\\n' + content }}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {%- if loop.first %}\n {%- if content|trim %}\n {{- '\\n\\n<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n {%- else %}\n {{- '<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n {%- endif %}\n {%- else %}\n {{- '\\n<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n {%- endif %}\n {%- if tool_call.arguments is defined %}\n {%- for args_name, args_value in tool_call.arguments|items %}\n {{- '<parameter=' + args_name + '>\\n' }}\n {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}\n {{- args_value }}\n {{- '\\n</parameter>\\n' }}\n {%- endfor %}\n {%- endif %}\n {{- '</function>\\n</tool_call>' }}\n {%- endfor %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if loop.previtem and loop.previtem.role != \"tool\" %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n<tool_response>\\n' }}\n {{- content }}\n {{- '\\n</tool_response>' }}\n {%- if not loop.last and loop.nextitem.role != \"tool\" %}\n {{- '<|im_end|>\\n' }}\n {%- elif loop.last %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- else %}\n {{- raise_exception('Unexpected message role.') }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n {%- if enable_thinking is defined and enable_thinking is false %}\n {{- '<think>\\n\\n</think>\\n\\n' }}\n {%- else %}\n {{- '<think>\\n' }}\n {%- endif %}\n{%- endif %}",
|
| 286 |
+
"clean_up_tokenization_spaces": false,
|
| 287 |
+
"eos_token": "<|endoftext|>",
|
| 288 |
+
"errors": "replace",
|
| 289 |
+
"model_max_length": 262144,
|
| 290 |
+
"pad_token": "<|endoftext|>",
|
| 291 |
+
"split_special_tokens": false,
|
| 292 |
+
"tokenizer_class": "Qwen2Tokenizer",
|
| 293 |
+
"unk_token": null,
|
| 294 |
+
"add_bos_token": false,
|
| 295 |
+
"pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
|
| 296 |
+
"extra_special_tokens": {
|
| 297 |
+
"audio_bos_token": "<|audio_start|>",
|
| 298 |
+
"audio_eos_token": "<|audio_end|>",
|
| 299 |
+
"audio_token": "<|audio_pad|>",
|
| 300 |
+
"image_token": "<|image_pad|>",
|
| 301 |
+
"video_token": "<|video_pad|>",
|
| 302 |
+
"vision_bos_token": "<|vision_start|>",
|
| 303 |
+
"vision_eos_token": "<|vision_end|>"
|
| 304 |
+
}
|
| 305 |
+
}
|
validation.json
ADDED
|
@@ -0,0 +1,106 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"description": "Packaging smoke check on M4; illustrative examples, not a quality benchmark",
|
| 3 |
+
"results": [
|
| 4 |
+
{
|
| 5 |
+
"mode": "text",
|
| 6 |
+
"seconds": 0.2356886670000904,
|
| 7 |
+
"result": {
|
| 8 |
+
"answers": {
|
| 9 |
+
"color": {
|
| 10 |
+
"type": "choice",
|
| 11 |
+
"choice": "red",
|
| 12 |
+
"confidence": 1.0,
|
| 13 |
+
"probabilities": {
|
| 14 |
+
"red": 1.0,
|
| 15 |
+
"blue": 0.0,
|
| 16 |
+
"green": 0.0
|
| 17 |
+
}
|
| 18 |
+
}
|
| 19 |
+
},
|
| 20 |
+
"probabilities": [
|
| 21 |
+
[
|
| 22 |
+
0.9999960660934448,
|
| 23 |
+
2.600952939246781e-06,
|
| 24 |
+
1.2866769338870654e-06
|
| 25 |
+
]
|
| 26 |
+
],
|
| 27 |
+
"input_tokens": 29,
|
| 28 |
+
"vision": null
|
| 29 |
+
}
|
| 30 |
+
},
|
| 31 |
+
{
|
| 32 |
+
"mode": "vision",
|
| 33 |
+
"seconds": 0.7187503750000133,
|
| 34 |
+
"result": {
|
| 35 |
+
"answers": {
|
| 36 |
+
"color": {
|
| 37 |
+
"type": "choice",
|
| 38 |
+
"choice": "red",
|
| 39 |
+
"confidence": 1.0,
|
| 40 |
+
"probabilities": {
|
| 41 |
+
"red": 1.0,
|
| 42 |
+
"blue": 0.0,
|
| 43 |
+
"green": 0.0
|
| 44 |
+
}
|
| 45 |
+
}
|
| 46 |
+
},
|
| 47 |
+
"probabilities": [
|
| 48 |
+
[
|
| 49 |
+
0.9999994039535522,
|
| 50 |
+
3.513019066758716e-07,
|
| 51 |
+
2.60039115573818e-07
|
| 52 |
+
]
|
| 53 |
+
],
|
| 54 |
+
"input_tokens": 176,
|
| 55 |
+
"vision": {
|
| 56 |
+
"tokens": 144,
|
| 57 |
+
"source_size": [
|
| 58 |
+
384,
|
| 59 |
+
384
|
| 60 |
+
],
|
| 61 |
+
"processed_size": [
|
| 62 |
+
384,
|
| 63 |
+
384
|
| 64 |
+
]
|
| 65 |
+
}
|
| 66 |
+
}
|
| 67 |
+
},
|
| 68 |
+
{
|
| 69 |
+
"mode": "repeat",
|
| 70 |
+
"seconds": 0.4666795829999728,
|
| 71 |
+
"result": {
|
| 72 |
+
"answers": {
|
| 73 |
+
"color": {
|
| 74 |
+
"type": "choice",
|
| 75 |
+
"choice": "red",
|
| 76 |
+
"confidence": 1.0,
|
| 77 |
+
"probabilities": {
|
| 78 |
+
"red": 1.0,
|
| 79 |
+
"blue": 0.0,
|
| 80 |
+
"green": 0.0
|
| 81 |
+
}
|
| 82 |
+
}
|
| 83 |
+
},
|
| 84 |
+
"probabilities": [
|
| 85 |
+
[
|
| 86 |
+
0.9999994039535522,
|
| 87 |
+
3.513019066758716e-07,
|
| 88 |
+
2.60039115573818e-07
|
| 89 |
+
]
|
| 90 |
+
],
|
| 91 |
+
"input_tokens": 176,
|
| 92 |
+
"vision": {
|
| 93 |
+
"tokens": 144,
|
| 94 |
+
"source_size": [
|
| 95 |
+
384,
|
| 96 |
+
384
|
| 97 |
+
],
|
| 98 |
+
"processed_size": [
|
| 99 |
+
384,
|
| 100 |
+
384
|
| 101 |
+
]
|
| 102 |
+
}
|
| 103 |
+
}
|
| 104 |
+
}
|
| 105 |
+
]
|
| 106 |
+
}
|