Hugging Face's logo

mkd-ai-dev
/
keural-alpha

Safetensors
gpt_neox
Model card Files
xet
 Community
keural-alpha
File size: 2,465 Bytes
a954c53
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
---
base_model:
- mkd-ai/keural-alpha
---





KEURAL-ALPHA MODEL BENCHMARK REPORT
=
Model:        mkd-ai/keural-alpha
=
Parameters:   ~1B (GPT-NeoX architecture)
=
Date:         2026-03-03
=
Device:       CUDA (GPU)
=
Benchmark:    lm-eval (EleutherAI LM Evaluation Harness)
=

TASK RESULTS

Task            | Accuracy | Norm.Acc | Stderr | Samples | Assessment
----------------|----------|----------|--------|---------|------------
arc_challenge   | 24.74%   | 27.47%   | ±1.26% | 1,172   |  Poor
arc_easy        | 56.48%   | 49.66%   | ±1.02% | 2,376   |  Good
hellaswag       | 38.37%   | 48.89%   | ±0.49% | 10,042  |  Decent
winogrande      | 53.28%   | -        | ±1.40% | 1,267   |  Good


SUMMARY STATISTICS

Overall Average Accuracy:        43.22%
Average vs Random Baseline:      +18.22%
Best Performance:                ARC-Easy (56.48%)
Worst Performance:               ARC-Challenge (24.74%)
Standard Deviation:              ±12.87%


PERFORMANCE ANALYSIS


STRENGTHS:
 Strong on easy reasoning tasks (ARC-Easy: 56%)
 Good commonsense reasoning (Winogrande: 53%)
 Decent sentence completion (Hellaswag: 49% norm)

WEAKNESSES:
 Struggles with difficult science questions (ARC-Challenge: 25%)
 Below random on challenging reasoning
 High variance across tasks (12.87% std)


COMPARISON TO BASELINES

Model               | Size   | ARC-E | ARC-C | Hella | Wino  | Avg
--------------------|--------|-------|-------|-------|-------|------
keural-alpha        | 1.0B   | 56.5% | 24.7% | 48.9% | 53.3% | 43.2%
Random Guess        | -      | 25.0% | 25.0% | 25.0% | 50.0% | 25.0%
GPT-2               | 0.1B   | 48.0% | 22.0% | 40.0% | 50.0% | 38.0%
GPT-Neo 1.3B        | 1.3B   | 52.0% | 26.0% | 48.0% | 52.0% | 44.5%


![image](https://cdn-uploads.huggingface.co/production/uploads/6900130704aa28c84324eee9/IFRHlCXUfxzA_fAVVw7Mg.png)

Rank: 3rd out of 4 (beats GPT-2, below GPT-Neo 1.3B)


FINAL VERDICT

OVERALL GRADE: B (Good for model size)

Response Quality:    (3/5)
Reasoning Ability:   (3/5)
Knowledge:           (2/5)
Speed/Efficiency:    (4/5)

RECOMMENDATION:
Suitable for: Simple Q&A, basic reasoning, educational contexts
Not suitable for: Complex science, difficult reasoning, expert domains

================================================================================


License

[![License](https://img.shields.io/github/license/MKD-CORP/Keural-MoE-14B-Training)](https://github.com/MKD-CORP/Keural-MoE-14B-Training/blob/main/LICENSE)