File size: 3,527 Bytes
f2e1b8b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | {
"format": "6bit",
"strict_reload": true,
"finite_prefill_decode_logits": true,
"cache_offsets_correct": true,
"source_sidecars_identical": true,
"native_thinking_template_preserved": true,
"temperature": 0.0,
"max_tokens": 512,
"smoke_tests": [
{
"task": "arithmetic",
"prompt": "What is 17 multiplied by 23? Give the answer.",
"text": "<think>We need answer simple multiplication. Need maybe just answer. 17*23=391. Ensure no extra? User asks \"What is 17 multiplied by 23? Give the answer.\" So final maybe 391.</think>391",
"answer": "391",
"checks": {
"expected_391": true
},
"generated_tokens": 48,
"seconds": 0.5709626669995487
},
{
"task": "retrieval",
"prompt": "The passcode is 941731. Repeat the passcode exactly.",
"text": "<think>The user is asking me to repeat a passcode exactly. This is a simple request. I should just repeat it back exactly as given.\n\nThe passcode is 941731.\n\nI should repeat it exactly. No need for extra commentary unless helpful. The instruction says \"Repeat the passcode exactly.\" So I should just output the passcode.</think>941731",
"answer": "941731",
"checks": {
"expected_passcode": true
},
"generated_tokens": 70,
"seconds": 0.7855458341073245
},
{
"task": "code",
"prompt": "Write a Python function add(a, b) that returns a + b.",
"text": "<think>We need to write a Python function add(a, b) that returns a + b. This is straightforward. The user likely expects a simple implementation.\n\nWe should provide the code, maybe with a brief explanation.\n\nThe instruction: \"Write a Python function add(a, b) that returns a + b.\"\n\nSo we can just output:\n\ndef add(a, b):\n return a + b\n\nWe might also include a docstring or example usage, but the core is simple.\n\nSince the user asked to write the function, we should just provide the code. Possibly in a code block.\n\nLet's respond with the function definition and maybe a quick test.</think>Here's a simple Python function that adds two numbers:\n\n```python\ndef add(a, b):\n \"\"\"Return the sum of a and b.\"\"\"\n return a + b\n```\n\nYou can use it like this:\n\n```python\nresult = add(3, 5) # result will be 8\n```",
"answer": "Here's a simple Python function that adds two numbers:\n\n```python\ndef add(a, b):\n \"\"\"Return the sum of a and b.\"\"\"\n return a + b\n```\n\nYou can use it like this:\n\n```python\nresult = add(3, 5) # result will be 8\n```",
"checks": {
"add_function": true
},
"generated_tokens": 196,
"seconds": 2.3982239169999957
}
],
"all_smoke_checks_pass": true,
"device": {
"device_name": "Apple M5 Max",
"max_recommended_working_set_size": 115448725504,
"memory_size": 137438953472,
"architecture": "applegpu_g17s",
"max_buffer_length": 86586540032,
"resource_limit": 499000
},
"mlx_version": "0.32.4",
"mlx_vlm_version": "0.7.10",
"converter_revision": "6d9ba328abbd6129c7461eda86280aa9fbfd9986",
"python": "3.12.15",
"peak_active_gb": 32.405607624,
"seconds": 8.617207458009943,
"swap_before": "vm.swapusage: total = 10240.00M used = 8736.50M free = 1503.50M (encrypted)",
"swap_after": "vm.swapusage: total = 10240.00M used = 8736.50M free = 1503.50M (encrypted)",
"limits": "Three synthetic smoke prompts and finite/cache checks; not a quality benchmark or proof of parity to BF16/Transformers."
} |