Hugging Face's logo

OiQ
/
daa-tokenizers

Arabic
Latin
tokenizer
moroccan-darija
arabic
bpe
unigram
wordpiece
bbpe
benchmark
Model card Files
xet
 Community
Ouaill commited on
Commit
07b7716
·
verified ·
1 Parent(s): b57219a

Upload results/dataset_stats.json with huggingface_hub

Browse files
Files changed (1) hide show
  1. results/dataset_stats.json +142 -0
results/dataset_stats.json ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "dataset": "DODa",
4
+ "repo": "atlasia/DODa",
5
+ "text_col": "darija",
6
+ "total_rows": 87741,
7
+ "total_processed": 87741,
8
+ "ar": {
9
+ "count": 0,
10
+ "min": 0,
11
+ "max": 0,
12
+ "mean": 0.0,
13
+ "median": 0.0
14
+ },
15
+ "az": {
16
+ "count": 87739,
17
+ "min": 2,
18
+ "max": 155,
19
+ "mean": 23.96,
20
+ "median": 21.0
21
+ },
22
+ "mi": {
23
+ "count": 2,
24
+ "min": 11,
25
+ "max": 12,
26
+ "mean": 11.5,
27
+ "median": 11.5
28
+ },
29
+ "ALL": {
30
+ "count": 87741,
31
+ "min": 2,
32
+ "max": 155,
33
+ "mean": 23.96,
34
+ "median": 21.0
35
+ }
36
+ },
37
+ {
38
+ "dataset": "Darija-Wiki",
39
+ "repo": "atlasia/Moroccan-Darija-Wiki-Dataset",
40
+ "text_col": "content",
41
+ "total_rows": 10044,
42
+ "total_processed": 10044,
43
+ "ar": {
44
+ "count": 9570,
45
+ "min": 31,
46
+ "max": 43065,
47
+ "mean": 761.42,
48
+ "median": 506.0
49
+ },
50
+ "az": {
51
+ "count": 1,
52
+ "min": 3616,
53
+ "max": 3616,
54
+ "mean": 3616.0,
55
+ "median": 3616.0
56
+ },
57
+ "mi": {
58
+ "count": 473,
59
+ "min": 66,
60
+ "max": 95926,
61
+ "mean": 2631.33,
62
+ "median": 914.0
63
+ },
64
+ "ALL": {
65
+ "count": 10044,
66
+ "min": 31,
67
+ "max": 95926,
68
+ "mean": 849.76,
69
+ "median": 508.0
70
+ }
71
+ },
72
+ {
73
+ "dataset": "Atlaset",
74
+ "repo": "atlasia/Atlaset",
75
+ "text_col": "text",
76
+ "total_rows": 1170963,
77
+ "total_processed": 1170963,
78
+ "ar": {
79
+ "count": 1082406,
80
+ "min": 3,
81
+ "max": 429082,
82
+ "mean": 783.54,
83
+ "median": 94.5
84
+ },
85
+ "az": {
86
+ "count": 9897,
87
+ "min": 3,
88
+ "max": 228575,
89
+ "mean": 204.23,
90
+ "median": 29.0
91
+ },
92
+ "mi": {
93
+ "count": 78660,
94
+ "min": 4,
95
+ "max": 96263,
96
+ "mean": 421.6,
97
+ "median": 115.0
98
+ },
99
+ "ALL": {
100
+ "count": 1170963,
101
+ "min": 3,
102
+ "max": 429082,
103
+ "mean": 754.33,
104
+ "median": 86.0
105
+ }
106
+ },
107
+ {
108
+ "dataset": "Ours (daa-pairs)",
109
+ "repo": "OiQ/daa-pairs",
110
+ "text_col": "arabic",
111
+ "total_rows": 112814,
112
+ "total_processed": 112814,
113
+ "ar": {
114
+ "count": 84293,
115
+ "min": 19,
116
+ "max": 153,
117
+ "mean": 65.97,
118
+ "median": 65.0
119
+ },
120
+ "az": {
121
+ "count": 43,
122
+ "min": 48,
123
+ "max": 94,
124
+ "mean": 69.49,
125
+ "median": 66.0
126
+ },
127
+ "mi": {
128
+ "count": 28478,
129
+ "min": 27,
130
+ "max": 143,
131
+ "mean": 68.94,
132
+ "median": 68.0
133
+ },
134
+ "ALL": {
135
+ "count": 112814,
136
+ "min": 19,
137
+ "max": 153,
138
+ "mean": 66.72,
139
+ "median": 66.0
140
+ }
141
+ }
142
+ ]