RoderickQiu commited on
Commit
ea7b4df
·
verified ·
1 Parent(s): 9d06938

Add development-suite evaluation results for the 8-bit file

Browse files
Files changed (1) hide show
  1. README.md +28 -0
README.md CHANGED
@@ -10,6 +10,34 @@ tags:
10
  - typed-decisions
11
  - 8-bit
12
  - apple-silicon
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  ---
14
 
15
  # Kev-4B, merged and quantized to 8 bits for MLX
 
10
  - typed-decisions
11
  - 8-bit
12
  - apple-silicon
13
+ model-index:
14
+ - name: kev-4b-mlx-8bit
15
+ results:
16
+ - task: { type: text-classification, name: typed decision, 8-bit MLX }
17
+ dataset: { type: mixed, name: "transfer-v4 development (656 questions; 8-bit, same Mac as bf16)" }
18
+ metrics:
19
+ - { type: accuracy, value: 0.817 }
20
+ - { type: brier_score, value: 0.243 }
21
+ - task: { type: text-classification, name: typed decision, 8-bit MLX }
22
+ dataset: { type: mixed, name: "decision-v7 development (1,264 questions; 8-bit, same Mac as bf16)" }
23
+ metrics:
24
+ - { type: accuracy, value: 0.873 }
25
+ - { type: expected_calibration_error, value: 0.010, name: "ECE, as served" }
26
+ - task: { type: text-classification, name: typed decision, 8-bit MLX }
27
+ dataset: { type: mixed, name: "hard-v1 development (1,083 questions; 8-bit, same Mac as bf16)" }
28
+ metrics:
29
+ - { type: accuracy, value: 0.786 }
30
+ - { type: brier_score, value: 0.313 }
31
+ - task: { type: text-classification, name: typed decision, 8-bit MLX }
32
+ dataset: { type: mixed, name: "devtools-v1 development (1,074 questions; 8-bit, same Mac as bf16)" }
33
+ metrics:
34
+ - { type: accuracy, value: 0.746 }
35
+ - { type: brier_score, value: 0.357 }
36
+ - task: { type: text-classification, name: typed decision, 8-bit MLX }
37
+ dataset: { type: mixed, name: "documents-v1 development (920 questions; 8-bit, same Mac as bf16)" }
38
+ metrics:
39
+ - { type: accuracy, value: 0.896 }
40
+ - { type: brier_score, value: 0.173 }
41
  ---
42
 
43
  # Kev-4B, merged and quantized to 8 bits for MLX