Lamapi commited on
Commit
d14d984
·
verified ·
1 Parent(s): 977aad6

Publish Next-Ask model

Browse files
Files changed (9) hide show
  1. README.md +68 -0
  2. ask_core.py +115 -0
  3. config.json +12 -0
  4. evaluation.json +509 -0
  5. inference.py +52 -0
  6. model.json +1 -0
  7. nlrt.py +226 -0
  8. preview.png +0 -0
  9. selftest.json +1 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-4.0
3
+ library_name: numpy
4
+ tags:
5
+ - active-learning
6
+ - tabular-classification
7
+ - next-lab
8
+ ---
9
+
10
+ # Next Ask
11
+
12
+ Next Ask decides **which unlabeled example to send to a person next** when labels are expensive. It ranks the pool by
13
+ classifier uncertainty, shortlists the 40 least certain items, then picks items that are both uncertain and far from
14
+ everything already labeled. A small softmax classifier retrains after each batch.
15
+
16
+ **Space:** https://huggingface.co/spaces/thelamapi/Next-Ask — label the digits the selector asks for; the classifier retrains in your browser.
17
+
18
+ ## Results (10 seeds, 360 held-out test digits, batches of 10, budget 200)
19
+
20
+ | Strategy | Area under learning curve | Final accuracy | Labels to reach 90% (median) |
21
+ |---|---:|---:|---:|
22
+ | Uncertainty + diversity (Next Ask) | 0.8897 ± 0.0066 | 0.9583 | 70 (10/10 seeds) |
23
+ | Uncertainty only | 0.8837 ± 0.0135 | 0.9569 | 70 (10/10 seeds) |
24
+ | Random | 0.8508 ± 0.0115 | 0.9283 | 115 (10/10 seeds) |
25
+
26
+ Mean test accuracy at selected label counts:
27
+
28
+ | Labels | Random | Uncertainty | Uncertainty + diversity |
29
+ |---:|---:|---:|---:|
30
+ | 20 | 0.6508 | 0.6608 | 0.7036 |
31
+ | 50 | 0.8386 | 0.8650 | 0.8736 |
32
+ | 100 | 0.8864 | 0.9294 | 0.9314 |
33
+ | 150 | 0.9131 | 0.9447 | 0.9492 |
34
+ | 200 | 0.9283 | 0.9569 | 0.9583 |
35
+
36
+ Paired by seed (same initial labels): hybrid − random AULC = +0.0388 ± 0.0113
37
+ (hybrid better in 10/10 seeds); hybrid − uncertainty = +0.0060 ± 0.0104
38
+ (6/10 seeds).
39
+
40
+ ![Learning curves](preview.png)
41
+
42
+ ## Method
43
+ * Classifier: multinomial logistic regression on the 64 pixel intensities, full-batch gradient descent from zero weights
44
+ (steps=150, lr=1.0, l2=0.001, chosen on a 200-example validation split, never on test).
45
+ * Uncertainty: margin between the two most probable classes.
46
+ * Diversity: farthest-first selection on the uncertain shortlist, score = (1 − margin) × distance to the nearest labeled or already chosen example.
47
+ * The browser runs a line-by-line JavaScript port. Node parity on 4 fixed cases: max abs difference 1.1102230246251565e-16.
48
+
49
+ ## Usage
50
+ ```python
51
+ from inference import load, predict, next_queries
52
+ spec = load() # model.json, NumPy only
53
+ probs = predict(spec, pixels) # pixels: (N, 64) ints 0..16
54
+ picks = next_queries(spec, pool_pixels, labeled_idx, labels, strategy="hybrid", batch=5)
55
+ ```
56
+ `python inference.py --selftest` reloads the model and checks stored outputs.
57
+
58
+ ## Data
59
+ Optical Recognition of Handwritten Digits (8x8) — UCI ML Repository id 80 via sklearn.datasets.load_digits, license CC BY 4.0. Split: 1237 pool /
60
+ 200 validation / 360 test, stratified, seed 73.
61
+
62
+ ## Limitations
63
+ * 8×8 digits are an easy, balanced benchmark; gains from active learning are usually smaller on noisy, imbalanced real data.
64
+ * The simulation assumes a perfect annotator (dataset labels). The Space lets people label, but no human study was run.
65
+ * Final model in `model.json` is the seed-0 hybrid run at the full budget, kept for demonstration (17 of 360 test digits wrong).
66
+ * Hardware for the recorded run: NVIDIA L4. Profile: full.
67
+
68
+ Files: `model.json` (weights), `ask_core.py` (selectors and training), `inference.py`, `nlrt.py`, `config.json`, `evaluation.json` (all numbers, seeds, failures, tests).
ask_core.py ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Next Ask: pool-based active learning with a label budget.
2
+
3
+ Classifier: multinomial logistic regression trained by full-batch gradient descent
4
+ from zero weights (deterministic, mirrored exactly in web/core.js).
5
+ Strategies:
6
+ random - uniform sample from the unlabeled pool (Mulberry32 PRNG)
7
+ uncertainty - smallest top-2 probability margin
8
+ hybrid - uncertainty + diversity: shortlist the 4*B most uncertain points, then
9
+ greedily pick the point maximising (1 - margin) * distance to everything
10
+ already labeled or picked (farthest-first on the shortlist)
11
+ """
12
+ import numpy as np
13
+
14
+ from nlrt import Mulberry32
15
+
16
+ STRATEGIES = ("random", "uncertainty", "hybrid")
17
+
18
+
19
+ def features(pixels):
20
+ x = np.asarray(pixels, dtype=np.float64) / 16.0
21
+ return np.hstack([x, np.ones((x.shape[0], 1))])
22
+
23
+
24
+ def train(X, y, n_classes=10, steps=150, lr=0.5, l2=1e-3):
25
+ n, d = X.shape
26
+ W = np.zeros((d, n_classes))
27
+ Y = np.zeros((n, n_classes))
28
+ Y[np.arange(n), y] = 1.0
29
+ reg = np.ones((d, 1))
30
+ reg[-1] = 0.0
31
+ for _ in range(steps):
32
+ Z = X @ W
33
+ Z -= Z.max(axis=1, keepdims=True)
34
+ P = np.exp(Z)
35
+ P /= P.sum(axis=1, keepdims=True)
36
+ G = X.T @ (P - Y) / n + l2 * reg * W
37
+ W -= lr * G
38
+ return W
39
+
40
+
41
+ def proba(W, X):
42
+ Z = X @ W
43
+ Z -= Z.max(axis=1, keepdims=True)
44
+ P = np.exp(Z)
45
+ return P / P.sum(axis=1, keepdims=True)
46
+
47
+
48
+ def margins(P):
49
+ s = np.sort(P, axis=1)
50
+ return s[:, -1] - s[:, -2]
51
+
52
+
53
+ def select(strategy, W, X, labeled, B, seed):
54
+ """Return B pool indices (ordered) to request labels for next."""
55
+ labeled_set = set(int(i) for i in labeled)
56
+ unl = [i for i in range(X.shape[0]) if i not in labeled_set]
57
+ if strategy == "random":
58
+ rng = Mulberry32(seed)
59
+ arr = list(unl)
60
+ picks = []
61
+ for k in range(min(B, len(arr))):
62
+ j = k + rng.randint(len(arr) - k)
63
+ arr[k], arr[j] = arr[j], arr[k]
64
+ picks.append(arr[k])
65
+ return picks
66
+ m = margins(proba(W, X[unl]))
67
+ order = sorted(range(len(unl)), key=lambda k: (m[k], unl[k]))
68
+ if strategy == "uncertainty":
69
+ return [unl[k] for k in order[:B]]
70
+ if strategy != "hybrid":
71
+ raise ValueError(strategy)
72
+ short = order[:4 * B]
73
+ feats = X[:, :-1]
74
+ anchors = feats[sorted(labeled_set)] if labeled_set else np.zeros((0, feats.shape[1]))
75
+ cand = feats[[unl[k] for k in short]]
76
+ if len(anchors):
77
+ dmin = ((cand[:, None, :] - anchors[None, :, :]) ** 2).sum(-1).min(axis=1)
78
+ else:
79
+ dmin = np.full(len(short), np.inf)
80
+ dmin = np.sqrt(dmin)
81
+ picks, used = [], np.zeros(len(short), dtype=bool)
82
+ for _ in range(min(B, len(short))):
83
+ score = np.where(used, -np.inf, (1.0 - m[short]) * dmin)
84
+ k = int(np.argmax(score))
85
+ used[k] = True
86
+ picks.append(unl[short[k]])
87
+ dnew = np.sqrt(((cand - cand[k]) ** 2).sum(-1))
88
+ dmin = np.minimum(dmin, dnew)
89
+ return picks
90
+
91
+
92
+ def initial_set(n_pool, n_init, seed):
93
+ rng = Mulberry32(seed * 7919 + 1)
94
+ arr = list(range(n_pool))
95
+ for k in range(n_init):
96
+ j = k + rng.randint(n_pool - k)
97
+ arr[k], arr[j] = arr[j], arr[k]
98
+ return arr[:n_init]
99
+
100
+
101
+ def simulate(strategy, Xp, yp, Xt, yt, seed, hyper, n_init=10, B=10, budget=200):
102
+ labeled = initial_set(len(Xp), n_init, seed)
103
+ curve = []
104
+ W = None
105
+ rnd = 0
106
+ while True:
107
+ W = train(Xp[labeled], yp[labeled], **hyper)
108
+ acc = float((proba(W, Xt).argmax(1) == yt).mean())
109
+ curve.append((len(labeled), acc))
110
+ if len(labeled) >= budget:
111
+ break
112
+ picks = select(strategy, W, Xp, labeled, min(B, budget - len(labeled)), seed * 100003 + rnd)
113
+ labeled = labeled + picks
114
+ rnd += 1
115
+ return curve, labeled, W
config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "softmax-regression",
3
+ "input": "8x8 digit, ints 0..16",
4
+ "num_classes": 10,
5
+ "selector": "hybrid uncertainty+diversity",
6
+ "shortlist_factor": 4,
7
+ "hyper": {
8
+ "steps": 150,
9
+ "lr": 1.0,
10
+ "l2": 0.001
11
+ }
12
+ }
evaluation.json ADDED
@@ -0,0 +1,509 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "project": "Next Ask",
3
+ "profile": "full",
4
+ "seed": 73,
5
+ "device": {
6
+ "python": "3.13.16",
7
+ "platform": "Linux-6.6.159+-x86_64-with-glibc2.39",
8
+ "cpu_count": 12,
9
+ "numpy": "2.4.6",
10
+ "torch": "2.14.1+cu130",
11
+ "cuda": true,
12
+ "gpu": "NVIDIA L4",
13
+ "colab": true
14
+ },
15
+ "data": {
16
+ "name": "Optical Recognition of Handwritten Digits (8x8)",
17
+ "source": "UCI ML Repository id 80 via sklearn.datasets.load_digits",
18
+ "license": "CC BY 4.0",
19
+ "sklearn_version": "1.6.1",
20
+ "n_total": 1797,
21
+ "split": {
22
+ "pool": 1237,
23
+ "validation": 200,
24
+ "test": 360,
25
+ "method": "stratified train_test_split, random_state=73; test only for final reporting"
26
+ }
27
+ },
28
+ "classifier": {
29
+ "type": "multinomial logistic regression, full-batch GD from zeros",
30
+ "hyper": {
31
+ "steps": 150,
32
+ "lr": 1.0,
33
+ "l2": 0.001
34
+ },
35
+ "hyper_selection": {
36
+ "split": "validation",
37
+ "grid": [
38
+ {
39
+ "hyper": {
40
+ "steps": 100,
41
+ "lr": 0.5,
42
+ "l2": 0.001
43
+ },
44
+ "val_accuracy": 0.7816666666666666
45
+ },
46
+ {
47
+ "hyper": {
48
+ "steps": 100,
49
+ "lr": 0.5,
50
+ "l2": 0.01
51
+ },
52
+ "val_accuracy": 0.775
53
+ },
54
+ {
55
+ "hyper": {
56
+ "steps": 100,
57
+ "lr": 1.0,
58
+ "l2": 0.001
59
+ },
60
+ "val_accuracy": 0.7966666666666667
61
+ },
62
+ {
63
+ "hyper": {
64
+ "steps": 100,
65
+ "lr": 1.0,
66
+ "l2": 0.01
67
+ },
68
+ "val_accuracy": 0.79
69
+ },
70
+ {
71
+ "hyper": {
72
+ "steps": 150,
73
+ "lr": 0.5,
74
+ "l2": 0.001
75
+ },
76
+ "val_accuracy": 0.7949999999999999
77
+ },
78
+ {
79
+ "hyper": {
80
+ "steps": 150,
81
+ "lr": 0.5,
82
+ "l2": 0.01
83
+ },
84
+ "val_accuracy": 0.7833333333333333
85
+ },
86
+ {
87
+ "hyper": {
88
+ "steps": 150,
89
+ "lr": 1.0,
90
+ "l2": 0.001
91
+ },
92
+ "val_accuracy": 0.8083333333333335
93
+ },
94
+ {
95
+ "hyper": {
96
+ "steps": 150,
97
+ "lr": 1.0,
98
+ "l2": 0.01
99
+ },
100
+ "val_accuracy": 0.7949999999999999
101
+ }
102
+ ]
103
+ }
104
+ },
105
+ "protocol": {
106
+ "seeds": [
107
+ 0,
108
+ 1,
109
+ 2,
110
+ 3,
111
+ 4,
112
+ 5,
113
+ 6,
114
+ 7,
115
+ 8,
116
+ 9
117
+ ],
118
+ "initial_labels": 10,
119
+ "batch": 10,
120
+ "budget": 200,
121
+ "pairing": "all strategies share the same initial labeled set per seed"
122
+ },
123
+ "label_axis": [
124
+ 10,
125
+ 20,
126
+ 30,
127
+ 40,
128
+ 50,
129
+ 60,
130
+ 70,
131
+ 80,
132
+ 90,
133
+ 100,
134
+ 110,
135
+ 120,
136
+ 130,
137
+ 140,
138
+ 150,
139
+ 160,
140
+ 170,
141
+ 180,
142
+ 190,
143
+ 200
144
+ ],
145
+ "results": {
146
+ "random": {
147
+ "mean_curve": [
148
+ 0.4972222222222221,
149
+ 0.6508333333333334,
150
+ 0.7319444444444445,
151
+ 0.8058333333333334,
152
+ 0.8386111111111111,
153
+ 0.8577777777777778,
154
+ 0.875,
155
+ 0.8733333333333334,
156
+ 0.8822222222222222,
157
+ 0.8863888888888889,
158
+ 0.891388888888889,
159
+ 0.8952777777777777,
160
+ 0.9038888888888887,
161
+ 0.9038888888888892,
162
+ 0.9130555555555556,
163
+ 0.9152777777777779,
164
+ 0.9175000000000001,
165
+ 0.9233333333333335,
166
+ 0.9252777777777779,
167
+ 0.9283333333333335
168
+ ],
169
+ "std_curve": [
170
+ 0.059004080210452226,
171
+ 0.06322175844391503,
172
+ 0.04262327799642462,
173
+ 0.04171201236225172,
174
+ 0.03069287326295477,
175
+ 0.020927564668708192,
176
+ 0.024438130497691987,
177
+ 0.025645975305741767,
178
+ 0.02351122663277648,
179
+ 0.019580870609530135,
180
+ 0.01654819000956276,
181
+ 0.016622627000419093,
182
+ 0.016536528962374692,
183
+ 0.015020562038439754,
184
+ 0.010974859036188004,
185
+ 0.012187450538044603,
186
+ 0.015518308581626014,
187
+ 0.013217085837078126,
188
+ 0.010573814617041271,
189
+ 0.009922539499717643
190
+ ],
191
+ "aulc_mean": 0.8508194444444446,
192
+ "aulc_std": 0.011463921196411408,
193
+ "final_accuracy_mean": 0.9283333333333335,
194
+ "labels_to_90pct_per_seed": [
195
+ 120,
196
+ 90,
197
+ 130,
198
+ 110,
199
+ 80,
200
+ 100,
201
+ 170,
202
+ 70,
203
+ 130,
204
+ 150
205
+ ]
206
+ },
207
+ "uncertainty": {
208
+ "mean_curve": [
209
+ 0.4972222222222221,
210
+ 0.6608333333333334,
211
+ 0.7691666666666667,
212
+ 0.836111111111111,
213
+ 0.865,
214
+ 0.8869444444444443,
215
+ 0.9080555555555556,
216
+ 0.9172222222222223,
217
+ 0.9241666666666666,
218
+ 0.9294444444444444,
219
+ 0.9358333333333333,
220
+ 0.9411111111111111,
221
+ 0.9413888888888889,
222
+ 0.943611111111111,
223
+ 0.9447222222222222,
224
+ 0.9497222222222221,
225
+ 0.9533333333333334,
226
+ 0.9541666666666666,
227
+ 0.9588888888888889,
228
+ 0.9569444444444445
229
+ ],
230
+ "std_curve": [
231
+ 0.059004080210452226,
232
+ 0.04963524360177361,
233
+ 0.06437556184580423,
234
+ 0.04897089082453114,
235
+ 0.032231799343022866,
236
+ 0.02629515516564194,
237
+ 0.012695994313362711,
238
+ 0.010815512407739881,
239
+ 0.01061750825200912,
240
+ 0.011466537466972383,
241
+ 0.009972957261675673,
242
+ 0.007222222222222197,
243
+ 0.008093779046851896,
244
+ 0.007762882562490039,
245
+ 0.010277777777777776,
246
+ 0.006854979266251711,
247
+ 0.006781419786518746,
248
+ 0.006364688465216471,
249
+ 0.004937885787397574,
250
+ 0.006242279181734529
251
+ ],
252
+ "aulc_mean": 0.8836944444444444,
253
+ "aulc_std": 0.013546217184144036,
254
+ "final_accuracy_mean": 0.9569444444444443,
255
+ "labels_to_90pct_per_seed": [
256
+ 60,
257
+ 70,
258
+ 70,
259
+ 70,
260
+ 60,
261
+ 80,
262
+ 60,
263
+ 80,
264
+ 70,
265
+ 60
266
+ ]
267
+ },
268
+ "hybrid": {
269
+ "mean_curve": [
270
+ 0.4972222222222221,
271
+ 0.7036111111111111,
272
+ 0.7972222222222223,
273
+ 0.8483333333333333,
274
+ 0.8736111111111111,
275
+ 0.8933333333333333,
276
+ 0.9052777777777778,
277
+ 0.9175000000000001,
278
+ 0.9244444444444445,
279
+ 0.931388888888889,
280
+ 0.9349999999999999,
281
+ 0.9375,
282
+ 0.9422222222222221,
283
+ 0.9480555555555554,
284
+ 0.9491666666666665,
285
+ 0.9533333333333334,
286
+ 0.9574999999999999,
287
+ 0.9605555555555554,
288
+ 0.9594444444444443,
289
+ 0.9583333333333333
290
+ ],
291
+ "std_curve": [
292
+ 0.059004080210452226,
293
+ 0.04441057737427031,
294
+ 0.027943947422131746,
295
+ 0.03218388523991164,
296
+ 0.011197580205970207,
297
+ 0.006712803318663668,
298
+ 0.012877029984999185,
299
+ 0.013031893450822117,
300
+ 0.007535922203472533,
301
+ 0.008962369891612037,
302
+ 0.009312808119022332,
303
+ 0.011266277661485132,
304
+ 0.009362388636862604,
305
+ 0.007458734212276422,
306
+ 0.006579288490181691,
307
+ 0.006060395619242084,
308
+ 0.006579288490181691,
309
+ 0.0049378857873975624,
310
+ 0.0033333333333333435,
311
+ 0.005555555555555575
312
+ ],
313
+ "aulc_mean": 0.8896527777777777,
314
+ "aulc_std": 0.0066435507806675955,
315
+ "final_accuracy_mean": 0.9583333333333333,
316
+ "labels_to_90pct_per_seed": [
317
+ 60,
318
+ 70,
319
+ 60,
320
+ 70,
321
+ 70,
322
+ 80,
323
+ 80,
324
+ 40,
325
+ 80,
326
+ 80
327
+ ]
328
+ }
329
+ },
330
+ "paired_aulc": {
331
+ "hybrid_minus_random": {
332
+ "aulc_diff_mean": 0.038833333333333366,
333
+ "aulc_diff_std": 0.011321139954816477,
334
+ "hybrid_wins": 10,
335
+ "seeds": 10
336
+ },
337
+ "hybrid_minus_uncertainty": {
338
+ "aulc_diff_mean": 0.005958333333333366,
339
+ "aulc_diff_std": 0.010409525329840541,
340
+ "hybrid_wins": 6,
341
+ "seeds": 10
342
+ }
343
+ },
344
+ "failure_examples": {
345
+ "model": "hybrid seed 0 at full budget",
346
+ "n_wrong": 17,
347
+ "first": [
348
+ {
349
+ "test_index": 648,
350
+ "true": 2,
351
+ "predicted": 3,
352
+ "confidence": 0.4906110093411642
353
+ },
354
+ {
355
+ "test_index": 1149,
356
+ "true": 8,
357
+ "predicted": 1,
358
+ "confidence": 0.279524654801159
359
+ },
360
+ {
361
+ "test_index": 1264,
362
+ "true": 1,
363
+ "predicted": 8,
364
+ "confidence": 0.4017349208333158
365
+ },
366
+ {
367
+ "test_index": 901,
368
+ "true": 9,
369
+ "predicted": 8,
370
+ "confidence": 0.39549887000865813
371
+ },
372
+ {
373
+ "test_index": 1595,
374
+ "true": 7,
375
+ "predicted": 4,
376
+ "confidence": 0.3269425499797302
377
+ },
378
+ {
379
+ "test_index": 498,
380
+ "true": 7,
381
+ "predicted": 9,
382
+ "confidence": 0.3155363201045011
383
+ },
384
+ {
385
+ "test_index": 492,
386
+ "true": 6,
387
+ "predicted": 8,
388
+ "confidence": 0.4289399503435557
389
+ },
390
+ {
391
+ "test_index": 746,
392
+ "true": 4,
393
+ "predicted": 7,
394
+ "confidence": 0.513914544250967
395
+ },
396
+ {
397
+ "test_index": 447,
398
+ "true": 7,
399
+ "predicted": 5,
400
+ "confidence": 0.4440512466278117
401
+ },
402
+ {
403
+ "test_index": 457,
404
+ "true": 5,
405
+ "predicted": 9,
406
+ "confidence": 0.5211896895367181
407
+ },
408
+ {
409
+ "test_index": 905,
410
+ "true": 8,
411
+ "predicted": 1,
412
+ "confidence": 0.5462623831942506
413
+ },
414
+ {
415
+ "test_index": 1095,
416
+ "true": 4,
417
+ "predicted": 9,
418
+ "confidence": 0.38364309630663607
419
+ },
420
+ {
421
+ "test_index": 857,
422
+ "true": 7,
423
+ "predicted": 3,
424
+ "confidence": 0.5475465941519988
425
+ },
426
+ {
427
+ "test_index": 719,
428
+ "true": 9,
429
+ "predicted": 1,
430
+ "confidence": 0.5353181203934337
431
+ },
432
+ {
433
+ "test_index": 1384,
434
+ "true": 4,
435
+ "predicted": 6,
436
+ "confidence": 0.4296230000789044
437
+ },
438
+ {
439
+ "test_index": 769,
440
+ "true": 8,
441
+ "predicted": 2,
442
+ "confidence": 0.3883439380665226
443
+ },
444
+ {
445
+ "test_index": 794,
446
+ "true": 8,
447
+ "predicted": 1,
448
+ "confidence": 0.4823884399122324
449
+ }
450
+ ]
451
+ },
452
+ "tests": {
453
+ "reload": {
454
+ "returncode": 0,
455
+ "seconds": 0.18,
456
+ "process": "fresh python -E subprocess, cwd = temporary copy of the model folder",
457
+ "passed": true,
458
+ "max_abs_diff": 5.220997367771929e-08,
459
+ "picks": [
460
+ 96,
461
+ 36,
462
+ 26,
463
+ 7,
464
+ 105
465
+ ],
466
+ "expected_picks": [
467
+ 96,
468
+ 36,
469
+ 26,
470
+ 7,
471
+ 105
472
+ ],
473
+ "n_samples": 16
474
+ },
475
+ "node_parity": {
476
+ "cases": 4,
477
+ "failed": 0,
478
+ "maxDiff": 1.1102230246251565e-16,
479
+ "results": [
480
+ {
481
+ "name": "hybrid-10",
482
+ "maxDiff": 1.1102230246251565e-16
483
+ },
484
+ {
485
+ "name": "uncertainty-25",
486
+ "maxDiff": 5.551115123125783e-17
487
+ },
488
+ {
489
+ "name": "random-40",
490
+ "maxDiff": 1.1102230246251565e-16
491
+ },
492
+ {
493
+ "name": "hybrid-60",
494
+ "maxDiff": 4.163336342344337e-17
495
+ }
496
+ ],
497
+ "ms": 114,
498
+ "runtime": "node v22.23.4",
499
+ "status": "passed"
500
+ },
501
+ "browser": {
502
+ "status": "skipped",
503
+ "reason": "playwright not installed"
504
+ }
505
+ },
506
+ "not_run": [
507
+ "Human annotation study: the Space accepts user labels, but no user study was run."
508
+ ]
509
+ }
inference.py ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Next Ask inference: predict digits and choose the next examples to label.
2
+
3
+ Usage:
4
+ python inference.py --selftest
5
+ from inference import load, predict, next_queries
6
+ """
7
+ import json
8
+ import os
9
+ import sys
10
+
11
+ HERE = os.path.dirname(os.path.abspath(__file__))
12
+ sys.path.insert(0, HERE)
13
+
14
+ import numpy as np # noqa: E402
15
+
16
+ import ask_core as ak # noqa: E402
17
+ from nlrt import load_model # noqa: E402
18
+
19
+
20
+ def load(path=os.path.join(HERE, "model.json")):
21
+ return load_model(path)
22
+
23
+
24
+ def predict(spec, pixels):
25
+ """pixels: (N, 64) ints 0..16 (8x8 digits). Returns class probabilities (N, 10)."""
26
+ return ak.proba(spec["tensors"]["W"], ak.features(pixels))
27
+
28
+
29
+ def next_queries(spec, pool_pixels, labeled, labels, strategy="hybrid", batch=5, seed=0):
30
+ """Retrain on the labeled pool items and return the next `batch` pool indices to label."""
31
+ X = ak.features(pool_pixels)
32
+ W = ak.train(X[labeled], np.asarray(labels), **spec["meta"]["hyper"])
33
+ return ak.select(strategy, W, X, labeled, batch, seed)
34
+
35
+
36
+ def _selftest():
37
+ spec = load()
38
+ with open(os.path.join(HERE, "selftest.json"), encoding="utf-8") as f:
39
+ t = json.load(f)
40
+ P = predict(spec, t["pixels"])
41
+ diff = float(np.abs(P - np.asarray(t["expected_proba"])).max())
42
+ picks = next_queries(spec, t["pool_pixels"], t["labeled"], t["labels"], t["strategy"], t["batch"], t["seed"])
43
+ ok = diff < 1e-6 and picks == t["expected_picks"]
44
+ print(json.dumps({"passed": bool(ok), "max_abs_diff": diff, "picks": picks,
45
+ "expected_picks": t["expected_picks"], "n_samples": len(t["pixels"])}))
46
+ return ok
47
+
48
+
49
+ if __name__ == "__main__":
50
+ if "--selftest" in sys.argv:
51
+ sys.exit(0 if _selftest() else 1)
52
+ print(__doc__)
model.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"format":"nlrt-1","meta":{"name":"Next Ask","hyper":{"steps":150,"lr":1.0,"l2":0.001},"classes":[0,1,2,3,4,5,6,7,8,9],"trained_on":"200 pool labels chosen by the hybrid selector (seed 0)"},"nets":{},"tensors":{"W":{"shape":[65,10],"data":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKZLEb2iGbC968HNPGL8Xj44/Ee9uFEsPgXsM72a98A7iFKevQeLAL6B7Dk+c9EpvgL3iD6BKTm+D+mXvvNCjz95RRe/Tm9iPhV0+L6cLI+9UP20PpLM6b2OzUw+LzLQPpzKkr/F9oO9yXozvtxkkj60/Mq+Hq0mP3dOcz2hwpu/Mw//vSJklT8JoAO/pguuPuzf7b024ZU+VmwfPvS3ML1P1ai+0v4fP5uex75b78g+sLt4vyoWcT90HN6+kKoFPx6sJb5SIUe+3x3FvsPgcb13ala+vBllPVh53L7Ft1o/dntxvkxDLT9cazW/pgPiPmRl0Lx8i0C9N2x7vLXjtryLDLY+s7NYvXM0ibx5QHo8Bn+5vbPOzb0Gbxa7hGoJvMldObxd2F28sWCBu+/IjDso4d+7lc3uujy2Sz29DMG7ocEUvpFNM79keeI+QnVKPmGgpb0EZTQ+pyLivhJM3zyatdI9BP7XPtezFT5/2JO/qwbgOzo3Bj/QyBm/65s6PymNBb9ofnc+8E3HPvMAdj5ZHoI+jW6KvEOQILoSpQo+S7wEv/o1oj7Csdc99ECfPr5Trb54IX6+4396vYSpAT9mIxM/nS2UPpZoJL80LNi7y9sIv9wuST+uAlu/9TJnvY1w0z5HyPS85VOSPIP9BT8Vjjy/I7Bjv12wZL9DMqM+KOZBPyXNBD/e/Om99cUvvyKcJ751mQo/x+xWvi+/or07e1++wKHaPufP4z1d9Mg+HdrTvHZ87LyEjh28kXavvHoCBL0nulS9RkoyvI1ZZz5x2yO96Iqnu5Jh/LlzLlC7kO2fu5AYULuaVIm6y9w6u5OnILsUiwG6psqhPNIqnLpPqBU+Jx6RvpUAE7xyVTe+V+XcPWomTT4CDqu+Ru7Hvl0Uhj4dqPU+Ro4IP29YgrxoSAu/NeWZv1wKhj51fUw/0s5oPmHoV7/g/UY+4wkXP+64BL6Cgsw/4n5uv6tfjr8FchQ/50jHPbNypT7/+I+/4zhgPnRt9j7xhm+/dF/HP43ipz4sDQg/HKaSPrw4vb9VE3m/v0ZcPsJ+xLtG5fI+TSIQP/zCrz1VjqE+iy2Mvpulgz0DCta/GqCuv/09uT63d9A+zZzBP+jcyD4gNde+2GicPLETvL3zbU290FFVv19Xr76MsAs/b5DWPs1NuT7o7Z+7uLocvHddKrupiva7iYgTPTfezbth/Ky7vQgXPYQxo7wz84m88TnDuS9VGrxWjSi6mUn5uZQRujwgO5S7ByjAusKuv7puwlG7HcCWui9N8T5plrq9N3vVvkUEkr6ppfI+u/eBPueJ3LvLPYK+8+Ztvrp3sz3Te68+sYdPPqPhwb8WYsW/uX6OP7E8kj+sG70+Katuvt/JJb+dV0Q/kCR0vwUaMD+g3c6/RtpAvlP1mz0jBSu8qhLJPgkBJr8UYa0/U99nP6Fq3r8l8D4/cEgTvryiWD+z4eA9SbnbPeb2Ab/jxBw9AinnvP3kET/rb2Q99oqDPj3HCL1DB4C/cUMnPqY7jr5eL/++61EcPwoxM71MVEQ/ZvTvPtoy8L5blmc8EJ8hv5bMbD9OJL2+fVjPvnKMRT/Icgi/U+BpPjJANrpVyv+6USvWuWFqrrlgU+M81niRuv+GQboN5Ua8E2OIu0iNy7sAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAW8MoPyr4hr119G2+FlHVvrxnGz+NXhW+/0g3PgS37z6k6/u+toYPvzlZ+j6O/48+v750vzirXL+/0VE/nw5FPjPOUj80AJ0+oJrlvtBYJb80sGq/OFtKvpmavr4yDjE+0dmMPa34fr6+AiA+3y0GvFp1ij8QQoc+S1e/v6f0CD4OZVy8EHUAPyRvET8gMgC+iAFLuwaiFT/f7hM/j7o6v/YVpj0mXZG+/5k3v0hh9r2ufsM+fk2ePuLLPT7ORS0/hQxHv0p/hz7ruOc+qtUov8cLzb5Ht3w9blYTP8v9aD5/3Q08gNvRPgUwZb9EomA+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAByWYU+gXoyv77pnD4W5sG9jGpyPyxg177s3TW+Cga7PZ+vOD7xc8q+qDBIP7jn0L5Ad8s+xzBLv8Bbfj4d84m/Q3S2PxRRkjw6bSQ/U/Cdv2uCmr79zoY+wfOhP0WEjb8pDKA/Ee2TvyiIuT53byI/NfHPPss3z7/XFxW/8HsNP2Zgp746M++8Mr2ZP75pyL5M/T8+ewxWPpvsST61PoK/AZ7HPskvWL/+Oa6/PhJRPwn/PD7N5nc+pLrZPne5ZL2+0MY+I8Q9vue+MT7Eq02/nfESv3Wigz9Owh2+wLhbPpHe4D6QjSu+WccKvsjdrbytTeK6q7bJvMWCEDw+KKW86/wAvMRrdrt/8po9ApEru+6Ml7x28qq7AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADlUkb2uJcC+7Dc9Pry21rzcq7Q+uPK7PZDaUL73S8q9eMWQPhRHDL7kT/E+vSg4vsVm3z5wGJK9bOqZvsW3X74JbIA+qzK3viz9Iz/KxCu/LgTNPViglT5GEZE/LGZCv6MSsz00vZ++q0PYPoi9mz7Ddhm/ojorv4emiz4C5zo/4750Pw5saj6N2HI+1roEvQG2p77S7kq/BjMgv8mBJb/En8k+cRBMvjmGQj8Mwag+SsQRv12TpT2HADY/q+qfv11SnT4GdRC/gtKYvus1hb3N7Q0/xE5UPws9FL8Asse+KYvZPkGcx75OB72+V+uQPtXVu7z/AtQ+jILqPUwsP75TDAy9MaFXvYITqb1UXK28TlcPvsh7PTwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA89tXvBlWqb0ftSo9cV1bPqaqQLzRnfU9jbo0vW9Fbr0GjyC9a8IBvmQd6Txd1Vi/qDjwPsu/7z58mw6+q6mRPyp1Jr/lZKY+iwZvvxS0ED4/oUc+/kkWv+fiOD5vIgI/xliEv9dkUz8nv3a+XePtvgCpHz+PJtq7iynMva8jHz7BeNQ+aCTxPY/kWb5M6Lq+cZ1FPj/Mjb/Qmo0/cu9OvvIel76iRy8/a/aBPxyH+T2wFEG/604Av9qsgT/dRYa/vHj4vVWJ6b2W92G+pKq1Pvu4mj+oSwY9aHW+vpcnwL7eUtI9He+Ivm1pE78TuOU9fIomvWFR1j5ItNw+UI6FvlXnK71pUvy9i5v1vUcDBL3oUUa+LyIYvb5gYz0kWpO+LqdOOwMlST0wXAk/prtDPrXztL7PBya7akZ5vhfHUz0="}}}
nlrt.py ADDED
@@ -0,0 +1,226 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """nlrt: tiny portable runtime shared by Next Lab models.
2
+
3
+ The same layer list runs in NumPy (this file) and in the browser (nlrt.js).
4
+ Weights are stored as base64 little-endian float32 inside model.json.
5
+ Only NumPy is required to load and run a model.
6
+ """
7
+ import base64
8
+ import json
9
+ import math
10
+
11
+ import numpy as np
12
+
13
+ __all__ = ["b64e", "b64d", "forward", "load_model", "Mulberry32", "export_sequential"]
14
+
15
+
16
+ def b64e(a):
17
+ return base64.b64encode(np.ascontiguousarray(a, dtype="<f4").tobytes()).decode("ascii")
18
+
19
+
20
+ def b64d(s, shape=None):
21
+ a = np.frombuffer(base64.b64decode(s), dtype="<f4").astype(np.float64)
22
+ return a.reshape(shape) if shape is not None else a
23
+
24
+
25
+ def load_model(path):
26
+ with open(path, "r", encoding="utf-8") as f:
27
+ spec = json.load(f)
28
+ for net in spec.get("nets", {}).values():
29
+ _decode(net)
30
+ for k, v in list(spec.get("tensors", {}).items()):
31
+ spec["tensors"][k] = b64d(v["data"], v["shape"])
32
+ return spec
33
+
34
+
35
+ def _decode(layers):
36
+ for L in layers:
37
+ for key in ("w", "b"):
38
+ if key in L and isinstance(L[key], str):
39
+ L[key] = b64d(L[key], L.get(key + "_shape"))
40
+
41
+
42
+ class Mulberry32:
43
+ """Deterministic PRNG with an identical JavaScript twin in nlrt.js."""
44
+
45
+ def __init__(self, seed):
46
+ self.a = seed & 0xFFFFFFFF
47
+
48
+ def random(self):
49
+ self.a = (self.a + 0x6D2B79F5) & 0xFFFFFFFF
50
+ t = self.a
51
+ t = ((t ^ (t >> 15)) * (t | 1)) & 0xFFFFFFFF
52
+ t ^= (t + (((t ^ (t >> 7)) * (t | 61)) & 0xFFFFFFFF)) & 0xFFFFFFFF
53
+ return ((t ^ (t >> 14)) & 0xFFFFFFFF) / 4294967296.0
54
+
55
+ def uniform(self, lo, hi):
56
+ return lo + (hi - lo) * self.random()
57
+
58
+ def randint(self, n):
59
+ return int(math.floor(self.random() * n))
60
+
61
+ def normal(self):
62
+ u = max(self.random(), 1e-12)
63
+ v = self.random()
64
+ return math.sqrt(-2.0 * math.log(u)) * math.cos(2.0 * math.pi * v)
65
+
66
+
67
+ def _pair(v):
68
+ return list(v) if isinstance(v, (list, tuple)) else [v, v]
69
+
70
+
71
+ def _conv2d(x, w, b, stride, pad):
72
+ n, c, h, wd = x.shape
73
+ o, _, kh, kw = w.shape
74
+ sh, sw = stride
75
+ ph, pw = pad
76
+ xp = np.pad(x, ((0, 0), (0, 0), (ph, ph), (pw, pw)))
77
+ oh = (h + 2 * ph - kh) // sh + 1
78
+ ow = (wd + 2 * pw - kw) // sw + 1
79
+ cols = np.empty((n, c, kh, kw, oh, ow), dtype=x.dtype)
80
+ for i in range(kh):
81
+ for j in range(kw):
82
+ cols[:, :, i, j] = xp[:, :, i:i + sh * oh:sh, j:j + sw * ow:sw]
83
+ out = np.tensordot(cols, w, axes=([1, 2, 3], [1, 2, 3])) # n, oh, ow, o
84
+ out = out.transpose(0, 3, 1, 2)
85
+ if b is not None:
86
+ out = out + b[None, :, None, None]
87
+ return out
88
+
89
+
90
+ def _maxpool(x, k):
91
+ kh, kw = k
92
+ n, c, h, w = x.shape
93
+ oh, ow = h // kh, w // kw
94
+ x = x[:, :, :oh * kh, :ow * kw].reshape(n, c, oh, kh, ow, kw)
95
+ return x.max(axis=(3, 5))
96
+
97
+
98
+ def forward(layers, x):
99
+ """Run a layer list on a batch. x: (N, D) or (N, C, H, W) or (N, C, L)."""
100
+ x = np.asarray(x, dtype=np.float64)
101
+ for L in layers:
102
+ t = L["type"]
103
+ if t == "linear":
104
+ x = x @ L["w"].T + L["b"]
105
+ elif t == "conv2d":
106
+ squeeze = x.ndim == 3
107
+ if squeeze:
108
+ x = x[:, :, None, :]
109
+ x = _conv2d(x, L["w"], L["b"], _pair(L["stride"]), _pair(L["pad"]))
110
+ if squeeze:
111
+ x = x[:, :, 0, :]
112
+ elif t == "relu":
113
+ x = np.maximum(x, 0)
114
+ elif t == "tanh":
115
+ x = np.tanh(x)
116
+ elif t == "sigmoid":
117
+ x = 1 / (1 + np.exp(-x))
118
+ elif t == "maxpool":
119
+ squeeze = x.ndim == 3
120
+ if squeeze:
121
+ x = x[:, :, None, :]
122
+ x = _maxpool(x, _pair(L["k"]))
123
+ if squeeze:
124
+ x = x[:, :, 0, :]
125
+ elif t == "gap":
126
+ x = x.reshape(x.shape[0], x.shape[1], -1).mean(axis=2)
127
+ elif t == "flatten":
128
+ x = x.reshape(x.shape[0], -1)
129
+ elif t == "upsample":
130
+ x = x.repeat(L["s"], axis=-1)
131
+ if x.ndim == 4:
132
+ x = x.repeat(L["s"], axis=-2)
133
+ elif t == "l2norm":
134
+ x = x / np.maximum(np.linalg.norm(x, axis=1, keepdims=True), 1e-12)
135
+ elif t == "reshape":
136
+ x = x.reshape([x.shape[0]] + list(L["shape"]))
137
+ else:
138
+ raise ValueError("unknown layer " + t)
139
+ return x
140
+
141
+
142
+ def export_sequential(seq):
143
+ """Convert a torch nn.Sequential into a portable layer list (BatchNorm folded)."""
144
+ import torch.nn as nn
145
+
146
+ mods = list(seq.modules())[1:] if not isinstance(seq, (list, tuple)) else list(seq)
147
+ mods = [m for m in mods if not isinstance(m, nn.Sequential)]
148
+ layers = []
149
+
150
+ def tensor(t):
151
+ return t.detach().cpu().double().numpy()
152
+
153
+ for m in mods:
154
+ if isinstance(m, nn.Linear):
155
+ b = tensor(m.bias) if m.bias is not None else np.zeros(m.out_features)
156
+ layers.append({"type": "linear", "w": tensor(m.weight), "b": b})
157
+ elif isinstance(m, (nn.Conv2d, nn.Conv1d)):
158
+ w = tensor(m.weight)
159
+ b = tensor(m.bias) if m.bias is not None else np.zeros(w.shape[0])
160
+ if isinstance(m, nn.Conv1d):
161
+ w = w[:, :, None, :]
162
+ stride, pad = [1, m.stride[0]], [0, m.padding[0]]
163
+ else:
164
+ stride, pad = list(m.stride), list(m.padding)
165
+ layers.append({"type": "conv2d", "w": w, "b": b, "stride": stride, "pad": pad})
166
+ elif isinstance(m, (nn.BatchNorm1d, nn.BatchNorm2d)):
167
+ prev = layers[-1]
168
+ assert prev["type"] in ("conv2d", "linear"), "BatchNorm must follow conv/linear"
169
+ scale = tensor(m.weight) / np.sqrt(tensor(m.running_var) + m.eps)
170
+ shift = tensor(m.bias) - tensor(m.running_mean) * scale
171
+ if prev["type"] == "conv2d":
172
+ prev["w"] = prev["w"] * scale[:, None, None, None]
173
+ else:
174
+ prev["w"] = prev["w"] * scale[:, None]
175
+ prev["b"] = prev["b"] * scale + shift
176
+ elif isinstance(m, nn.ReLU):
177
+ layers.append({"type": "relu"})
178
+ elif isinstance(m, nn.Tanh):
179
+ layers.append({"type": "tanh"})
180
+ elif isinstance(m, nn.Sigmoid):
181
+ layers.append({"type": "sigmoid"})
182
+ elif isinstance(m, nn.MaxPool2d):
183
+ layers.append({"type": "maxpool", "k": _pair(m.kernel_size)})
184
+ elif isinstance(m, nn.MaxPool1d):
185
+ k = m.kernel_size if isinstance(m, int) else m.kernel_size
186
+ layers.append({"type": "maxpool", "k": [1, k if isinstance(k, int) else k[0]]})
187
+ elif isinstance(m, (nn.AdaptiveAvgPool2d, nn.AdaptiveAvgPool1d)):
188
+ layers.append({"type": "gap"})
189
+ elif isinstance(m, nn.Flatten):
190
+ layers.append({"type": "flatten"})
191
+ elif isinstance(m, nn.Upsample):
192
+ layers.append({"type": "upsample", "s": int(m.scale_factor)})
193
+ elif isinstance(m, nn.Dropout):
194
+ continue
195
+ elif m.__class__.__name__ == "L2Norm":
196
+ layers.append({"type": "l2norm"})
197
+ elif m.__class__.__name__ == "Reshape":
198
+ layers.append({"type": "reshape", "shape": list(m.shape)})
199
+ else:
200
+ raise ValueError("cannot export " + m.__class__.__name__)
201
+ return layers
202
+
203
+
204
+ def encode_layers(layers):
205
+ out = []
206
+ for L in layers:
207
+ L2 = dict(L)
208
+ for key in ("w", "b"):
209
+ if key in L2:
210
+ arr = np.asarray(L2[key])
211
+ L2[key + "_shape"] = list(arr.shape)
212
+ L2[key] = b64e(arr)
213
+ out.append(L2)
214
+ return out
215
+
216
+
217
+ def save_model(path, nets=None, tensors=None, meta=None):
218
+ spec = {"format": "nlrt-1", "meta": meta or {}, "nets": {}, "tensors": {}}
219
+ for name, layers in (nets or {}).items():
220
+ spec["nets"][name] = encode_layers(layers)
221
+ for name, arr in (tensors or {}).items():
222
+ arr = np.asarray(arr)
223
+ spec["tensors"][name] = {"shape": list(arr.shape), "data": b64e(arr)}
224
+ with open(path, "w", encoding="utf-8") as f:
225
+ json.dump(spec, f, separators=(",", ":"))
226
+ return spec
preview.png ADDED
selftest.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"pixels": [[0, 1, 9, 15, 13, 4, 0, 0, 0, 8, 12, 4, 8, 8, 0, 0, 0, 9, 11, 0, 5, 16, 3, 0, 0, 4, 13, 8, 16, 16, 6, 0, 0, 0, 4, 8, 2, 11, 9, 0, 0, 0, 0, 0, 0, 10, 8, 0, 0, 0, 7, 0, 3, 14, 6, 0, 0, 0, 7, 16, 14, 8, 0, 0], [0, 0, 0, 2, 9, 13, 6, 0, 0, 0, 11, 15, 8, 9, 10, 0, 0, 3, 16, 10, 4, 13, 5, 0, 0, 1, 15, 16, 15, 15, 7, 0, 0, 0, 0, 0, 1, 14, 1, 0, 0, 0, 0, 0, 11, 6, 0, 0, 0, 0, 0, 1, 13, 0, 0, 0, 0, 0, 0, 5, 4, 0, 0, 0], [0, 0, 12, 16, 15, 6, 0, 0, 0, 0, 15, 12, 7, 15, 1, 0, 0, 1, 15, 15, 7, 16, 4, 0, 0, 1, 12, 16, 16, 14, 1, 0, 0, 0, 0, 4, 10, 13, 0, 0, 0, 0, 0, 0, 1, 15, 3, 0, 0, 0, 3, 0, 2, 16, 6, 0, 0, 0, 13, 16, 16, 15, 1, 0], [0, 0, 2, 15, 16, 15, 0, 0, 0, 0, 12, 9, 11, 12, 0, 0, 0, 5, 15, 0, 13, 7, 0, 0, 0, 5, 6, 3, 14, 5, 2, 0, 0, 0, 0, 9, 16, 16, 9, 0, 0, 0, 7, 16, 9, 2, 0, 0, 0, 0, 1, 15, 3, 0, 0, 0, 0, 0, 3, 16, 0, 0, 0, 0], [0, 0, 0, 9, 15, 12, 1, 0, 0, 1, 11, 12, 5, 15, 4, 0, 0, 6, 14, 0, 0, 13, 7, 0, 0, 5, 16, 12, 12, 16, 4, 0, 0, 0, 3, 8, 14, 8, 0, 0, 0, 0, 0, 2, 15, 1, 0, 0, 0, 0, 0, 9, 10, 0, 0, 0, 0, 0, 0, 10, 9, 0, 0, 0], [0, 0, 14, 16, 13, 9, 1, 0, 0, 0, 12, 10, 9, 16, 3, 0, 0, 0, 9, 14, 13, 13, 1, 0, 0, 0, 3, 13, 16, 6, 0, 0, 0, 0, 5, 16, 12, 9, 0, 0, 0, 0, 13, 8, 14, 6, 0, 0, 0, 1, 16, 11, 15, 1, 0, 0, 0, 1, 13, 16, 6, 0, 0, 0], [0, 0, 1, 13, 15, 8, 0, 0, 0, 0, 11, 14, 8, 15, 0, 0, 0, 2, 16, 3, 0, 13, 2, 0, 0, 5, 15, 0, 0, 10, 5, 0, 0, 3, 10, 0, 0, 10, 5, 0, 0, 3, 13, 0, 1, 15, 3, 0, 0, 0, 12, 10, 11, 11, 0, 0, 0, 0, 1, 12, 11, 4, 0, 0], [0, 0, 0, 2, 13, 0, 0, 0, 0, 0, 0, 8, 15, 0, 0, 0, 0, 0, 5, 16, 5, 2, 0, 0, 0, 0, 15, 12, 1, 16, 4, 0, 0, 4, 16, 2, 9, 16, 8, 0, 0, 0, 10, 14, 16, 16, 4, 0, 0, 0, 0, 0, 13, 8, 0, 0, 0, 0, 0, 0, 13, 6, 0, 0], [0, 0, 0, 11, 5, 3, 11, 0, 0, 0, 7, 14, 2, 12, 9, 0, 0, 2, 15, 6, 3, 16, 5, 0, 0, 7, 16, 8, 13, 16, 13, 0, 0, 7, 16, 16, 16, 7, 1, 0, 0, 0, 4, 10, 13, 0, 0, 0, 0, 0, 0, 12, 6, 0, 0, 0, 0, 0, 0, 12, 0, 0, 0, 0], [0, 0, 0, 6, 15, 1, 0, 0, 0, 0, 5, 16, 10, 0, 8, 6, 0, 2, 16, 11, 0, 9, 16, 6, 0, 8, 16, 14, 14, 16, 13, 1, 0, 6, 12, 12, 12, 16, 3, 0, 0, 0, 0, 0, 13, 11, 0, 0, 0, 0, 0, 6, 16, 5, 0, 0, 0, 0, 0, 10, 14, 0, 0, 0], [0, 0, 0, 12, 13, 5, 0, 0, 0, 0, 0, 11, 16, 9, 0, 0, 0, 0, 3, 15, 16, 6, 0, 0, 0, 7, 15, 16, 16, 2, 0, 0, 0, 0, 1, 16, 16, 3, 0, 0, 0, 0, 1, 16, 16, 6, 0, 0, 0, 0, 1, 16, 16, 6, 0, 0, 0, 0, 0, 11, 16, 10, 0, 0], [0, 1, 12, 16, 16, 16, 4, 0, 0, 4, 16, 10, 4, 1, 1, 0, 0, 6, 13, 0, 0, 0, 0, 0, 0, 6, 16, 12, 5, 0, 0, 0, 0, 7, 11, 11, 15, 0, 0, 0, 0, 0, 0, 4, 16, 2, 0, 0, 0, 0, 2, 13, 10, 0, 0, 0, 0, 2, 16, 11, 0, 0, 0, 0], [0, 0, 15, 13, 1, 0, 0, 0, 0, 0, 14, 14, 4, 0, 0, 0, 0, 0, 1, 4, 12, 0, 0, 0, 0, 0, 0, 6, 12, 0, 0, 0, 0, 0, 0, 11, 10, 0, 0, 0, 0, 0, 1, 16, 8, 6, 5, 0, 0, 0, 13, 16, 16, 16, 14, 0, 0, 0, 10, 13, 10, 6, 2, 0], [0, 2, 9, 15, 16, 15, 2, 0, 0, 11, 11, 5, 9, 16, 3, 0, 0, 0, 0, 0, 9, 12, 0, 0, 0, 0, 0, 0, 6, 15, 1, 0, 0, 0, 0, 0, 0, 14, 9, 0, 0, 0, 0, 0, 0, 12, 12, 0, 0, 0, 5, 3, 6, 15, 7, 0, 0, 0, 12, 16, 15, 9, 1, 0], [0, 0, 7, 16, 16, 16, 10, 0, 0, 0, 10, 10, 5, 12, 16, 2, 0, 0, 0, 0, 7, 15, 6, 0, 0, 0, 1, 13, 16, 13, 0, 0, 0, 0, 0, 7, 12, 16, 6, 0, 0, 0, 0, 0, 2, 16, 6, 0, 0, 0, 2, 9, 11, 14, 1, 0, 0, 0, 5, 16, 15, 5, 0, 0], [0, 0, 0, 6, 15, 1, 0, 0, 0, 0, 3, 16, 9, 15, 3, 0, 0, 1, 15, 7, 5, 15, 0, 0, 0, 9, 16, 4, 11, 14, 10, 0, 0, 9, 16, 16, 16, 16, 9, 0, 0, 0, 2, 4, 16, 2, 0, 0, 0, 0, 0, 6, 14, 0, 0, 0, 0, 0, 0, 7, 10, 0, 0, 0]], "expected_proba": [[0.009301860370823182, 0.0003377928139933008, 0.0013839765150211144, 0.035602431854951155, 0.0006540874242048349, 0.0298196515318514, 0.00030498365783880013, 0.002123026723148042, 0.0157471246614007, 0.9047250644467675], [0.0028240724909350413, 0.036274817711693774, 7.113303381339093e-05, 0.0021464604344428526, 0.021200474653877432, 0.035493768210747303, 0.00025769072432768104, 0.01721777719617439, 0.008813059846323192, 0.875700745697665], [0.001450546197480699, 0.0033330393772091132, 0.0003767377322893228, 0.03406754889211714, 0.00013686932658937866, 0.02673137349280285, 0.0003405598212421597, 0.00034304103088633167, 0.05409465973631347, 0.8791256243930696], [0.002456998036268147, 0.017088970615153423, 0.03098230091564292, 0.05043930626270735, 0.023020664088033185, 0.02105146048694959, 0.0013188887367387983, 0.8021728147949646, 0.027510691788954794, 0.023957904274586998], [0.005961087363735631, 0.02539113601005707, 0.0016723656954290016, 0.014925400823868442, 0.04609702421531616, 0.04983481221609488, 0.0010214437067805305, 0.11776247787037508, 0.18729862154423216, 0.5500356305541111], [0.0036178901258497737, 0.13965795437893436, 0.03407488023093344, 0.1094509111299805, 0.015705093163603884, 0.01616728470147477, 0.0015910387986742872, 0.07429357431694378, 0.5575549144426457, 0.04788645871095949], [0.9558311689922097, 0.000350861111374977, 0.001714676000150265, 0.001575889366323491, 0.0017953067028007195, 0.012447555272226256, 0.005171154010356051, 0.0031824488448161403, 0.007408871212754265, 0.010522068486988131], [0.004102211765226119, 0.03730815623155579, 9.517112409106778e-05, 0.00014381765576289233, 0.8921925760738441, 0.0018476737819977532, 0.05701265822319445, 0.003118883231690147, 0.003639289749421567, 0.000539562163216065], [0.003950151887914675, 0.02797338356414346, 0.0005714729115313496, 0.00024462419251279936, 0.4939780272879793, 0.0055931038757240144, 0.0018089761550852705, 0.32177752895436784, 0.022839548891113247, 0.12126318227962812], [0.010186311533662503, 0.02210852119538656, 0.00010902514023930408, 0.0019010228070838427, 0.6349683828202733, 0.04641132352401945, 0.007504429702659099, 0.026086893400481124, 0.03729591367358075, 0.21342817620261423], [2.651930390592233e-05, 0.9147050706658323, 0.00398747533058666, 0.004028625847954033, 0.02483290525380609, 0.0003179868384166654, 0.0020945260728950474, 0.0008751432820669269, 0.047174117869075986, 0.001957629535460297], [0.0006835852606436577, 0.0005739255292792072, 0.00025826974800563333, 0.0029260931404641857, 0.0036102321290476897, 0.976605518949828, 0.0006731554847526922, 0.011887050975316796, 0.0011593832422737288, 0.001622785540388273], [0.0004906855305762449, 0.013753489423693048, 0.8605707619141295, 0.09657820316649421, 0.0014748377475428775, 0.0051216381578545345, 0.005253629229684178, 0.0009549514501056732, 0.015497810221385008, 0.0003039931585348891], [0.010427952563331538, 0.00011871210196458524, 0.0182959473891222, 0.8990303597589187, 1.5288170799436145e-05, 0.010284646644514795, 7.441671710524454e-05, 0.005471423603045739, 0.0022312423705374194, 0.05405001068066063], [0.000449093652761953, 0.000517288297889674, 0.001246857447342939, 0.8714550800760784, 5.2803402101887623e-05, 0.006948368075363695, 9.348230834924801e-05, 0.014335285052146129, 0.018846692363411292, 0.08605504932455481], [0.00668916842699502, 0.027883023983182962, 0.00030022711826186154, 0.0008570347012656974, 0.8182107702296748, 0.00433620481802077, 0.0014941325944485921, 0.08331770102292452, 0.008437188422566565, 0.04847454868265904]], "pool_pixels": [[0, 0, 8, 13, 11, 1, 0, 0, 0, 4, 15, 5, 12, 6, 0, 0, 0, 0, 0, 0, 10, 4, 0, 0, 0, 0, 0, 6, 16, 4, 0, 0, 0, 0, 0, 5, 13, 15, 3, 0, 0, 0, 1, 0, 0, 7, 12, 0, 0, 1, 15, 8, 7, 12, 12, 0, 0, 0, 6, 14, 16, 15, 5, 0], [0, 0, 0, 1, 11, 0, 0, 0, 0, 0, 0, 7, 8, 0, 0, 0, 0, 0, 1, 13, 6, 2, 2, 0, 0, 0, 7, 15, 0, 9, 8, 0, 0, 5, 16, 10, 0, 16, 6, 0, 0, 4, 15, 16, 13, 16, 1, 0, 0, 0, 0, 3, 15, 10, 0, 0, 0, 0, 0, 2, 16, 4, 0, 0], [0, 0, 10, 10, 14, 16, 14, 0, 0, 0, 14, 8, 4, 0, 0, 0, 0, 0, 16, 0, 6, 11, 5, 0, 0, 3, 16, 14, 10, 10, 9, 0, 0, 3, 14, 5, 0, 9, 8, 0, 0, 0, 0, 0, 6, 13, 0, 0, 0, 0, 3, 9, 13, 3, 0, 0, 0, 0, 8, 13, 1, 0, 0, 0], [0, 0, 10, 16, 15, 0, 0, 0, 0, 4, 14, 8, 16, 1, 0, 0, 0, 0, 1, 4, 16, 0, 2, 0, 0, 0, 3, 11, 16, 16, 13, 0, 0, 0, 12, 16, 11, 7, 2, 0, 0, 0, 6, 16, 0, 0, 0, 0, 0, 0, 7, 15, 0, 0, 0, 0, 0, 0, 12, 11, 0, 0, 0, 0], [0, 0, 0, 7, 15, 0, 0, 0, 0, 0, 6, 15, 8, 0, 0, 0, 0, 0, 13, 9, 0, 0, 0, 0, 0, 2, 16, 5, 4, 1, 0, 0, 0, 5, 16, 16, 16, 12, 3, 0, 0, 1, 15, 4, 1, 8, 12, 0, 0, 0, 8, 14, 5, 5, 15, 0, 0, 0, 0, 6, 16, 16, 11, 0], [0, 0, 9, 14, 14, 3, 0, 0, 0, 4, 13, 1, 4, 11, 0, 0, 0, 1, 14, 1, 11, 7, 6, 0, 0, 0, 15, 13, 9, 12, 3, 0, 0, 0, 1, 14, 14, 0, 0, 0, 0, 0, 5, 14, 12, 0, 0, 0, 0, 0, 12, 1, 9, 5, 0, 0, 0, 0, 11, 12, 13, 3, 0, 0], [0, 0, 0, 9, 11, 0, 0, 0, 0, 0, 5, 14, 3, 2, 0, 0, 0, 0, 15, 2, 1, 14, 3, 0, 0, 5, 13, 0, 13, 8, 1, 0, 0, 8, 13, 3, 16, 14, 6, 0, 0, 6, 15, 16, 13, 3, 0, 0, 0, 0, 0, 9, 11, 0, 0, 0, 0, 0, 0, 11, 7, 0, 0, 0], [0, 0, 0, 0, 9, 16, 4, 0, 0, 0, 0, 5, 15, 16, 2, 0, 0, 3, 12, 16, 16, 14, 0, 0, 0, 10, 16, 15, 16, 15, 0, 0, 0, 1, 4, 0, 16, 13, 0, 0, 0, 0, 0, 0, 15, 13, 0, 0, 0, 0, 0, 0, 12, 16, 12, 0, 0, 0, 0, 0, 12, 16, 6, 0], [0, 0, 2, 15, 16, 9, 0, 0, 0, 0, 3, 13, 11, 16, 0, 0, 0, 0, 0, 2, 13, 12, 0, 0, 0, 0, 0, 9, 16, 11, 0, 0, 0, 3, 3, 1, 6, 15, 8, 0, 0, 11, 13, 0, 0, 10, 12, 0, 0, 3, 16, 12, 7, 16, 8, 0, 0, 0, 3, 15, 16, 10, 0, 0], [0, 2, 12, 13, 16, 15, 1, 0, 0, 8, 16, 14, 11, 7, 0, 0, 0, 8, 16, 7, 0, 0, 0, 0, 0, 7, 16, 16, 11, 1, 0, 0, 0, 0, 2, 6, 15, 9, 0, 0, 0, 0, 0, 0, 9, 15, 0, 0, 0, 0, 1, 7, 16, 11, 0, 0, 0, 1, 16, 16, 13, 1, 0, 0], [0, 0, 0, 2, 14, 0, 0, 0, 0, 0, 0, 10, 9, 0, 0, 0, 0, 0, 1, 15, 8, 11, 0, 0, 0, 0, 12, 7, 12, 8, 0, 0, 0, 5, 16, 12, 15, 14, 5, 0, 0, 8, 13, 9, 16, 13, 3, 0, 0, 0, 0, 0, 16, 4, 0, 0, 0, 0, 0, 0, 16, 6, 0, 0], [0, 0, 2, 15, 6, 0, 0, 0, 0, 0, 11, 16, 4, 0, 0, 0, 0, 3, 16, 7, 0, 0, 0, 0, 0, 4, 16, 6, 4, 1, 0, 0, 0, 6, 16, 16, 14, 16, 3, 0, 0, 2, 14, 9, 0, 11, 9, 0, 0, 0, 10, 14, 8, 15, 5, 0, 0, 0, 3, 13, 16, 8, 0, 0], [0, 1, 10, 13, 9, 2, 0, 0, 0, 2, 12, 4, 12, 10, 0, 0, 0, 0, 14, 5, 11, 11, 0, 0, 0, 0, 2, 8, 8, 15, 2, 0, 0, 0, 0, 0, 0, 12, 5, 0, 0, 0, 0, 0, 0, 4, 12, 0, 0, 0, 4, 1, 0, 8, 11, 0, 0, 0, 11, 16, 16, 14, 3, 0], [0, 0, 7, 13, 16, 11, 0, 0, 0, 3, 16, 5, 4, 14, 2, 0, 0, 8, 11, 1, 4, 15, 2, 0, 0, 3, 12, 14, 16, 8, 0, 0, 0, 0, 8, 16, 15, 1, 0, 0, 0, 1, 15, 5, 11, 12, 0, 0, 0, 3, 16, 5, 7, 16, 1, 0, 0, 0, 5, 14, 16, 15, 2, 0], [0, 0, 11, 12, 2, 0, 0, 0, 0, 0, 15, 8, 13, 2, 0, 0, 0, 0, 7, 11, 9, 9, 0, 0, 0, 4, 12, 12, 16, 7, 0, 0, 0, 2, 10, 12, 9, 12, 1, 0, 0, 0, 0, 0, 0, 7, 11, 0, 0, 0, 0, 0, 0, 0, 16, 2, 0, 0, 6, 12, 12, 13, 11, 0], [0, 0, 0, 4, 9, 0, 0, 0, 0, 0, 0, 12, 11, 0, 0, 0, 0, 0, 4, 13, 16, 4, 0, 0, 0, 0, 12, 6, 14, 4, 0, 0, 0, 4, 16, 9, 15, 13, 3, 0, 0, 4, 12, 12, 16, 14, 6, 0, 0, 0, 0, 0, 16, 4, 0, 0, 0, 0, 0, 2, 16, 3, 0, 0], [0, 0, 7, 15, 16, 10, 0, 0, 0, 0, 14, 9, 10, 16, 1, 0, 0, 0, 2, 5, 15, 14, 0, 0, 0, 0, 0, 11, 16, 5, 0, 0, 0, 0, 0, 2, 16, 8, 0, 0, 0, 0, 0, 0, 10, 13, 0, 0, 0, 0, 11, 9, 15, 16, 1, 0, 0, 0, 8, 16, 16, 12, 0, 0], [0, 0, 9, 16, 16, 12, 0, 0, 0, 2, 16, 8, 9, 16, 0, 0, 0, 1, 8, 0, 13, 14, 0, 0, 0, 0, 0, 13, 16, 5, 0, 0, 0, 0, 0, 8, 14, 15, 5, 0, 0, 0, 0, 0, 0, 9, 14, 0, 0, 0, 9, 6, 0, 11, 15, 0, 0, 0, 8, 16, 16, 16, 6, 0], [0, 0, 6, 13, 16, 10, 0, 0, 0, 4, 13, 5, 4, 16, 0, 0, 0, 0, 0, 1, 10, 8, 0, 0, 0, 0, 0, 12, 16, 3, 0, 0, 0, 0, 0, 3, 7, 15, 1, 0, 0, 0, 0, 0, 0, 9, 8, 0, 0, 0, 8, 6, 3, 11, 7, 0, 0, 0, 4, 14, 16, 11, 1, 0], [0, 0, 1, 12, 16, 14, 2, 0, 0, 0, 13, 11, 3, 16, 5, 0, 0, 4, 14, 0, 0, 15, 6, 0, 0, 6, 12, 8, 13, 16, 5, 0, 0, 0, 9, 12, 4, 10, 8, 0, 0, 0, 3, 0, 0, 11, 5, 0, 0, 0, 16, 14, 5, 15, 4, 0, 0, 0, 3, 12, 16, 11, 1, 0], [0, 0, 7, 16, 16, 10, 0, 0, 0, 0, 14, 13, 7, 4, 0, 0, 0, 5, 16, 6, 0, 0, 0, 0, 0, 14, 15, 8, 6, 1, 0, 0, 0, 4, 12, 12, 16, 13, 2, 0, 0, 0, 0, 0, 1, 15, 6, 0, 0, 0, 5, 6, 6, 16, 4, 0, 0, 0, 7, 16, 16, 11, 1, 0], [0, 1, 9, 16, 13, 7, 0, 0, 0, 7, 14, 4, 10, 12, 0, 0, 0, 6, 15, 9, 16, 11, 0, 0, 0, 0, 9, 11, 7, 14, 0, 0, 0, 0, 0, 0, 0, 15, 2, 0, 0, 0, 0, 0, 0, 11, 6, 0, 0, 3, 13, 8, 5, 14, 5, 0, 0, 0, 9, 14, 13, 10, 1, 0], [0, 0, 2, 13, 13, 1, 0, 0, 0, 0, 9, 13, 5, 0, 0, 0, 0, 0, 13, 5, 0, 0, 0, 0, 0, 0, 15, 2, 0, 0, 0, 0, 0, 0, 15, 10, 9, 9, 2, 0, 0, 0, 16, 11, 8, 11, 12, 0, 0, 1, 14, 11, 1, 4, 13, 0, 0, 0, 3, 11, 16, 15, 4, 0], [0, 1, 14, 16, 16, 11, 2, 0, 0, 0, 14, 9, 2, 10, 11, 0, 0, 0, 5, 16, 5, 14, 5, 0, 0, 0, 0, 12, 16, 16, 5, 0, 0, 0, 7, 14, 14, 8, 0, 0, 0, 1, 14, 3, 6, 11, 0, 0, 0, 4, 8, 0, 8, 11, 0, 0, 0, 2, 13, 12, 15, 2, 0, 0], [0, 0, 2, 10, 14, 10, 0, 0, 0, 1, 15, 9, 9, 16, 1, 0, 0, 7, 9, 0, 9, 12, 0, 0, 0, 7, 7, 3, 15, 15, 0, 0, 0, 2, 15, 15, 7, 16, 1, 0, 0, 0, 1, 2, 0, 9, 4, 0, 0, 0, 5, 13, 4, 8, 9, 0, 0, 0, 1, 10, 15, 16, 6, 0], [0, 0, 4, 14, 5, 0, 0, 0, 0, 0, 13, 14, 0, 0, 0, 0, 0, 2, 16, 10, 0, 0, 0, 0, 0, 4, 16, 7, 0, 0, 0, 0, 0, 6, 16, 16, 15, 4, 0, 0, 0, 4, 16, 9, 4, 16, 2, 0, 0, 1, 15, 13, 6, 16, 11, 0, 0, 0, 4, 13, 16, 15, 5, 0], [0, 0, 0, 14, 14, 9, 0, 0, 0, 0, 4, 16, 16, 10, 0, 0, 0, 0, 13, 16, 15, 2, 0, 0, 0, 1, 15, 16, 11, 0, 0, 0, 0, 7, 16, 16, 5, 0, 0, 0, 0, 3, 16, 16, 7, 0, 0, 0, 0, 0, 16, 16, 8, 0, 0, 0, 0, 0, 3, 12, 12, 0, 0, 0], [0, 0, 0, 3, 14, 1, 0, 0, 0, 0, 0, 13, 12, 1, 0, 0, 0, 0, 7, 16, 5, 3, 0, 0, 0, 3, 15, 11, 5, 16, 2, 0, 0, 5, 16, 11, 11, 16, 6, 0, 0, 0, 6, 12, 16, 13, 3, 0, 0, 0, 0, 1, 15, 7, 0, 0, 0, 0, 0, 2, 16, 7, 0, 0], [0, 0, 1, 14, 16, 8, 0, 0, 0, 0, 10, 16, 11, 4, 0, 0, 0, 0, 1, 11, 1, 0, 0, 0, 0, 1, 12, 3, 0, 0, 0, 0, 0, 2, 16, 14, 13, 8, 1, 0, 0, 3, 16, 16, 13, 16, 8, 0, 0, 0, 12, 16, 7, 15, 12, 0, 0, 0, 1, 13, 16, 16, 8, 0], [0, 0, 3, 12, 16, 16, 15, 0, 0, 0, 9, 10, 7, 12, 14, 0, 0, 0, 0, 0, 2, 15, 6, 0, 0, 0, 0, 0, 11, 13, 0, 0, 0, 1, 9, 9, 16, 11, 1, 0, 0, 13, 16, 16, 16, 16, 4, 0, 0, 0, 1, 16, 7, 0, 0, 0, 0, 0, 6, 16, 2, 0, 0, 0], [0, 0, 7, 8, 13, 16, 15, 1, 0, 0, 7, 7, 4, 11, 12, 0, 0, 0, 0, 0, 8, 13, 1, 0, 0, 4, 8, 8, 15, 15, 6, 0, 0, 2, 11, 15, 15, 4, 0, 0, 0, 0, 0, 16, 5, 0, 0, 0, 0, 0, 9, 15, 1, 0, 0, 0, 0, 0, 13, 5, 0, 0, 0, 0], [0, 0, 7, 15, 14, 6, 0, 0, 0, 5, 16, 5, 10, 16, 4, 0, 0, 6, 15, 2, 10, 14, 1, 0, 0, 1, 13, 16, 14, 1, 0, 0, 0, 0, 10, 13, 15, 8, 0, 0, 0, 0, 15, 2, 3, 15, 6, 0, 0, 0, 15, 3, 8, 15, 6, 0, 0, 0, 6, 16, 11, 4, 0, 0], [0, 0, 14, 10, 0, 0, 0, 0, 0, 0, 15, 13, 0, 0, 0, 0, 0, 11, 16, 16, 2, 0, 0, 0, 0, 3, 10, 16, 5, 0, 0, 0, 0, 0, 0, 14, 10, 0, 0, 0, 0, 0, 0, 10, 14, 0, 0, 0, 0, 0, 9, 14, 16, 11, 6, 0, 0, 0, 12, 16, 16, 16, 16, 9], [0, 0, 0, 7, 16, 16, 11, 0, 0, 0, 6, 16, 16, 16, 16, 0, 0, 0, 11, 16, 16, 16, 9, 0, 0, 0, 2, 9, 11, 14, 10, 0, 0, 0, 0, 0, 0, 10, 6, 0, 0, 0, 0, 0, 4, 11, 1, 0, 0, 0, 0, 2, 14, 2, 0, 0, 0, 0, 0, 11, 3, 0, 0, 0], [0, 1, 8, 12, 15, 14, 4, 0, 0, 3, 11, 8, 8, 12, 12, 0, 0, 0, 0, 0, 2, 13, 7, 0, 0, 0, 0, 2, 15, 12, 1, 0, 0, 0, 0, 0, 13, 5, 0, 0, 0, 0, 0, 0, 9, 13, 0, 0, 0, 0, 7, 8, 14, 15, 0, 0, 0, 0, 14, 15, 11, 2, 0, 0], [0, 3, 15, 15, 3, 0, 0, 0, 0, 8, 14, 12, 10, 0, 0, 0, 0, 5, 11, 6, 14, 0, 0, 0, 0, 0, 0, 7, 14, 0, 0, 0, 0, 0, 0, 10, 12, 0, 0, 0, 0, 0, 0, 15, 9, 0, 0, 0, 0, 1, 11, 16, 12, 8, 5, 0, 0, 5, 16, 16, 16, 16, 16, 0], [0, 0, 0, 1, 16, 3, 0, 0, 0, 0, 0, 10, 11, 0, 0, 0, 0, 0, 1, 14, 3, 0, 0, 0, 0, 0, 7, 12, 0, 3, 9, 0, 0, 0, 14, 2, 0, 10, 7, 0, 0, 6, 13, 5, 11, 14, 1, 0, 0, 11, 16, 16, 16, 9, 0, 0, 0, 1, 2, 1, 14, 1, 0, 0], [0, 0, 0, 6, 16, 6, 0, 0, 0, 0, 0, 13, 16, 10, 0, 0, 0, 0, 9, 16, 16, 6, 0, 0, 0, 3, 16, 16, 16, 4, 0, 0, 0, 5, 16, 16, 16, 6, 0, 0, 0, 0, 0, 9, 16, 10, 0, 0, 0, 0, 0, 8, 16, 15, 0, 0, 0, 0, 0, 4, 13, 11, 2, 0], [0, 0, 0, 10, 11, 0, 0, 0, 0, 0, 9, 16, 6, 0, 0, 0, 0, 0, 15, 13, 0, 0, 0, 0, 0, 0, 14, 10, 0, 0, 0, 0, 0, 1, 15, 12, 8, 2, 0, 0, 0, 0, 12, 16, 16, 16, 10, 1, 0, 0, 7, 16, 12, 12, 16, 4, 0, 0, 0, 9, 15, 12, 5, 0], [0, 0, 0, 7, 15, 1, 0, 0, 0, 0, 0, 15, 16, 1, 0, 0, 0, 0, 9, 16, 16, 2, 0, 0, 0, 2, 16, 16, 16, 0, 0, 0, 0, 10, 16, 16, 16, 16, 8, 0, 0, 8, 15, 15, 14, 8, 5, 0, 0, 0, 0, 11, 12, 0, 0, 0, 0, 0, 0, 6, 15, 1, 0, 0], [0, 0, 7, 10, 16, 9, 0, 0, 0, 0, 15, 16, 13, 15, 1, 0, 0, 0, 10, 13, 10, 16, 2, 0, 0, 0, 1, 16, 16, 11, 0, 0, 0, 0, 8, 16, 16, 5, 0, 0, 0, 0, 15, 8, 14, 7, 0, 0, 0, 0, 16, 7, 16, 4, 0, 0, 0, 0, 9, 16, 14, 0, 0, 0], [0, 0, 0, 10, 13, 0, 0, 0, 0, 0, 0, 13, 16, 5, 0, 0, 0, 0, 0, 16, 16, 4, 0, 0, 0, 0, 3, 16, 16, 7, 0, 0, 0, 0, 7, 16, 16, 9, 0, 0, 0, 0, 9, 16, 16, 10, 0, 0, 0, 0, 10, 16, 16, 14, 0, 0, 0, 0, 1, 5, 7, 15, 8, 0], [0, 0, 1, 9, 15, 1, 0, 0, 0, 0, 9, 14, 4, 0, 0, 0, 0, 0, 16, 3, 0, 0, 0, 0, 0, 6, 13, 0, 0, 0, 0, 0, 0, 4, 14, 12, 16, 13, 3, 0, 0, 2, 15, 13, 4, 3, 13, 0, 0, 0, 9, 8, 2, 4, 16, 1, 0, 0, 0, 9, 12, 12, 8, 0], [0, 0, 9, 16, 16, 13, 2, 0, 0, 2, 15, 2, 3, 3, 0, 0, 0, 7, 9, 0, 1, 4, 0, 0, 0, 8, 12, 7, 13, 14, 7, 0, 0, 6, 16, 8, 0, 5, 8, 0, 0, 1, 3, 0, 0, 9, 6, 0, 0, 0, 3, 4, 1, 15, 0, 0, 0, 0, 7, 16, 12, 7, 0, 0], [0, 0, 5, 12, 10, 4, 0, 0, 0, 0, 5, 16, 16, 16, 3, 0, 0, 0, 0, 16, 16, 16, 0, 0, 0, 0, 3, 16, 16, 13, 0, 0, 0, 0, 4, 16, 16, 12, 0, 0, 0, 0, 8, 16, 16, 8, 0, 0, 0, 0, 10, 16, 16, 7, 0, 0, 0, 0, 8, 12, 12, 4, 0, 0], [0, 0, 3, 10, 16, 16, 4, 0, 0, 0, 0, 0, 1, 14, 7, 0, 0, 0, 0, 0, 2, 15, 4, 0, 0, 0, 4, 4, 12, 15, 5, 0, 0, 1, 15, 16, 16, 9, 4, 0, 0, 0, 2, 11, 13, 0, 0, 0, 0, 0, 1, 16, 5, 0, 0, 0, 0, 0, 3, 12, 0, 0, 0, 0], [0, 0, 0, 13, 15, 2, 0, 0, 0, 0, 4, 16, 11, 1, 0, 0, 0, 0, 13, 15, 1, 0, 0, 0, 0, 0, 15, 9, 0, 0, 0, 0, 0, 4, 16, 14, 15, 8, 0, 0, 0, 1, 16, 15, 8, 13, 9, 0, 0, 0, 11, 10, 0, 11, 16, 0, 0, 0, 2, 14, 16, 16, 13, 0], [0, 0, 1, 12, 16, 16, 12, 0, 0, 0, 7, 16, 11, 12, 9, 0, 0, 0, 6, 7, 0, 14, 4, 0, 0, 0, 0, 0, 6, 16, 3, 0, 0, 0, 0, 10, 16, 16, 9, 0, 0, 0, 0, 8, 15, 5, 0, 0, 0, 0, 0, 8, 10, 0, 0, 0, 0, 0, 0, 16, 5, 0, 0, 0], [0, 0, 2, 10, 16, 10, 0, 0, 0, 0, 14, 9, 6, 16, 16, 0, 0, 0, 16, 6, 5, 14, 11, 0, 0, 0, 5, 14, 14, 16, 6, 0, 0, 0, 0, 0, 1, 16, 3, 0, 0, 0, 3, 1, 4, 16, 3, 0, 0, 2, 15, 13, 11, 13, 1, 0, 0, 0, 3, 12, 13, 4, 0, 0], [0, 0, 9, 12, 14, 2, 0, 0, 0, 0, 12, 6, 4, 0, 0, 0, 0, 0, 12, 1, 3, 0, 0, 0, 0, 0, 9, 16, 16, 12, 0, 0, 0, 0, 4, 4, 0, 12, 6, 0, 0, 0, 0, 0, 0, 4, 12, 0, 0, 0, 9, 7, 4, 10, 11, 0, 0, 0, 9, 14, 16, 14, 5, 0], [0, 0, 4, 11, 15, 8, 0, 0, 0, 0, 13, 16, 11, 13, 7, 0, 0, 3, 16, 12, 0, 4, 8, 0, 0, 6, 16, 5, 0, 4, 8, 0, 0, 7, 9, 0, 0, 9, 7, 0, 0, 4, 10, 0, 2, 15, 2, 0, 0, 1, 16, 12, 14, 10, 0, 0, 0, 0, 4, 14, 14, 1, 0, 0], [0, 0, 10, 16, 14, 5, 0, 0, 0, 2, 16, 16, 8, 0, 0, 0, 0, 0, 9, 16, 16, 5, 0, 0, 0, 0, 7, 16, 16, 3, 0, 0, 0, 0, 14, 14, 13, 11, 0, 0, 0, 5, 16, 1, 6, 15, 0, 0, 0, 7, 14, 9, 13, 15, 1, 0, 0, 1, 11, 16, 15, 6, 0, 0], [0, 2, 15, 13, 1, 0, 0, 0, 0, 13, 12, 14, 7, 0, 0, 0, 0, 8, 6, 6, 13, 0, 0, 0, 0, 0, 0, 5, 15, 0, 0, 0, 0, 0, 0, 11, 10, 0, 0, 0, 0, 0, 2, 16, 5, 0, 0, 0, 0, 0, 11, 16, 13, 14, 12, 0, 0, 3, 16, 14, 8, 8, 7, 0], [0, 0, 7, 11, 15, 9, 0, 0, 0, 0, 15, 15, 4, 11, 4, 0, 0, 3, 11, 5, 0, 2, 10, 0, 0, 7, 8, 0, 0, 3, 8, 0, 0, 6, 8, 0, 0, 4, 8, 0, 0, 5, 8, 0, 0, 8, 5, 0, 0, 1, 12, 2, 1, 13, 0, 0, 0, 0, 5, 16, 14, 3, 0, 0], [0, 0, 5, 13, 13, 5, 0, 0, 0, 0, 16, 16, 10, 15, 3, 0, 0, 5, 16, 2, 1, 8, 4, 0, 0, 4, 13, 0, 0, 4, 8, 0, 0, 8, 12, 0, 0, 6, 7, 0, 0, 5, 15, 0, 0, 7, 7, 0, 0, 0, 16, 8, 5, 15, 3, 0, 0, 0, 5, 14, 15, 9, 0, 0], [0, 0, 0, 12, 8, 0, 0, 0, 0, 0, 6, 16, 3, 12, 4, 0, 0, 1, 16, 5, 8, 14, 0, 0, 0, 9, 15, 0, 13, 10, 2, 0, 0, 10, 15, 12, 16, 16, 9, 0, 0, 6, 16, 16, 15, 9, 1, 0, 0, 0, 0, 14, 5, 0, 0, 0, 0, 0, 0, 15, 0, 0, 0, 0], [0, 0, 13, 16, 15, 4, 0, 0, 0, 0, 9, 8, 13, 16, 3, 0, 0, 0, 0, 0, 13, 16, 7, 0, 0, 0, 0, 1, 16, 12, 0, 0, 0, 0, 0, 0, 15, 10, 0, 0, 0, 0, 0, 0, 8, 15, 0, 0, 0, 0, 3, 6, 15, 16, 7, 0, 0, 0, 15, 16, 16, 11, 1, 0], [0, 0, 6, 16, 16, 16, 12, 0, 0, 0, 13, 10, 8, 16, 5, 0, 0, 1, 15, 1, 9, 12, 0, 0, 0, 0, 4, 0, 13, 7, 0, 0, 0, 0, 10, 16, 16, 16, 9, 0, 0, 0, 7, 14, 12, 8, 3, 0, 0, 0, 3, 15, 5, 0, 0, 0, 0, 0, 8, 15, 0, 0, 0, 0], [0, 0, 3, 16, 12, 1, 0, 0, 0, 0, 3, 16, 16, 5, 0, 0, 0, 0, 2, 16, 16, 5, 0, 0, 0, 0, 0, 16, 16, 5, 0, 0, 0, 0, 4, 16, 16, 2, 0, 0, 0, 0, 4, 16, 14, 0, 0, 0, 0, 0, 6, 16, 14, 0, 0, 0, 0, 0, 2, 16, 14, 0, 0, 0], [0, 2, 10, 14, 16, 12, 0, 0, 0, 10, 12, 7, 10, 12, 0, 0, 0, 0, 0, 0, 13, 10, 0, 0, 0, 0, 0, 3, 16, 3, 0, 0, 0, 0, 0, 13, 10, 0, 0, 0, 0, 0, 5, 16, 1, 0, 0, 0, 0, 0, 14, 15, 12, 16, 10, 0, 0, 1, 16, 13, 9, 3, 0, 0], [0, 0, 2, 12, 9, 0, 0, 0, 0, 0, 12, 10, 1, 0, 0, 0, 0, 4, 14, 0, 0, 0, 0, 0, 0, 8, 9, 0, 0, 0, 0, 0, 0, 8, 9, 5, 11, 8, 0, 0, 0, 4, 16, 14, 6, 12, 5, 0, 0, 0, 13, 7, 0, 10, 8, 0, 0, 0, 3, 14, 16, 16, 5, 0], [0, 1, 10, 12, 15, 11, 0, 0, 0, 8, 16, 13, 9, 4, 0, 0, 0, 5, 15, 1, 0, 0, 0, 0, 0, 8, 10, 0, 0, 0, 0, 0, 0, 2, 14, 16, 7, 0, 0, 0, 0, 0, 1, 4, 13, 7, 0, 0, 0, 0, 0, 6, 11, 11, 0, 0, 0, 0, 9, 16, 14, 2, 0, 0], [0, 0, 7, 12, 10, 0, 0, 0, 0, 3, 16, 16, 16, 9, 1, 0, 0, 0, 8, 16, 16, 11, 1, 0, 0, 0, 10, 16, 16, 0, 0, 0, 0, 3, 16, 14, 16, 4, 0, 0, 0, 4, 13, 0, 7, 15, 0, 0, 0, 4, 14, 2, 2, 16, 0, 0, 0, 0, 6, 11, 10, 5, 0, 0], [0, 0, 1, 6, 12, 16, 9, 0, 0, 0, 10, 15, 10, 13, 9, 0, 0, 0, 2, 1, 0, 14, 2, 0, 0, 0, 0, 6, 12, 16, 15, 0, 0, 0, 5, 16, 16, 14, 7, 0, 0, 0, 3, 6, 15, 0, 0, 0, 0, 0, 0, 7, 9, 0, 0, 0, 0, 0, 0, 11, 2, 0, 0, 0], [0, 0, 2, 10, 12, 2, 0, 0, 0, 0, 11, 16, 13, 10, 0, 0, 0, 0, 9, 7, 2, 15, 0, 0, 0, 0, 3, 2, 3, 15, 0, 0, 0, 0, 0, 0, 10, 10, 0, 0, 0, 0, 2, 7, 16, 7, 0, 0, 0, 0, 9, 16, 16, 16, 16, 3, 0, 0, 3, 13, 9, 8, 4, 0], [0, 0, 3, 12, 12, 2, 0, 0, 0, 0, 11, 10, 7, 14, 2, 0, 0, 0, 11, 1, 0, 8, 4, 0, 0, 2, 14, 2, 0, 5, 7, 0, 0, 8, 9, 0, 0, 6, 8, 0, 0, 3, 13, 0, 0, 12, 7, 0, 0, 0, 15, 6, 11, 12, 0, 0, 0, 0, 4, 15, 11, 1, 0, 0], [0, 0, 8, 16, 15, 6, 0, 0, 0, 5, 14, 4, 4, 15, 0, 0, 0, 6, 13, 0, 1, 15, 2, 0, 0, 1, 11, 11, 13, 10, 0, 0, 0, 0, 1, 16, 16, 3, 0, 0, 0, 0, 12, 9, 5, 13, 2, 0, 0, 0, 16, 2, 1, 13, 8, 0, 0, 0, 8, 15, 16, 14, 1, 0], [0, 0, 1, 10, 11, 2, 0, 0, 0, 0, 12, 12, 8, 15, 4, 0, 0, 6, 11, 0, 1, 12, 7, 0, 0, 4, 16, 7, 15, 12, 0, 0, 0, 0, 5, 15, 15, 3, 0, 0, 0, 0, 0, 14, 7, 14, 0, 0, 0, 0, 4, 12, 7, 10, 0, 0, 0, 0, 1, 16, 16, 3, 0, 0], [0, 0, 2, 12, 4, 0, 0, 0, 0, 1, 12, 16, 16, 3, 0, 0, 0, 7, 16, 6, 4, 13, 0, 0, 0, 8, 16, 6, 0, 13, 5, 0, 0, 1, 16, 5, 0, 7, 9, 0, 0, 0, 16, 8, 0, 8, 12, 0, 0, 0, 13, 14, 14, 16, 10, 0, 0, 0, 4, 14, 15, 7, 0, 0], [0, 0, 3, 8, 12, 15, 16, 2, 0, 0, 12, 14, 10, 13, 15, 0, 0, 0, 1, 1, 2, 14, 6, 0, 0, 0, 2, 8, 13, 16, 8, 0, 0, 0, 9, 16, 16, 10, 5, 0, 0, 0, 1, 8, 12, 1, 0, 0, 0, 0, 0, 14, 8, 0, 0, 0, 0, 0, 2, 16, 5, 0, 0, 0], [0, 0, 3, 16, 12, 2, 0, 0, 0, 0, 6, 16, 16, 3, 0, 0, 0, 0, 9, 16, 16, 0, 0, 0, 0, 0, 9, 16, 16, 1, 0, 0, 0, 0, 9, 16, 16, 0, 0, 0, 0, 0, 9, 16, 13, 0, 0, 0, 0, 0, 7, 16, 13, 0, 0, 0, 0, 0, 3, 14, 16, 3, 0, 0], [0, 0, 5, 12, 12, 8, 1, 0, 0, 0, 10, 16, 16, 15, 0, 0, 0, 0, 11, 16, 16, 8, 0, 0, 0, 4, 16, 16, 16, 4, 0, 0, 0, 3, 16, 16, 10, 0, 0, 0, 0, 0, 13, 16, 16, 3, 0, 0, 0, 0, 13, 16, 16, 0, 0, 0, 0, 0, 2, 10, 12, 0, 0, 0], [0, 0, 4, 14, 14, 2, 0, 0, 0, 5, 16, 11, 7, 10, 0, 0, 0, 4, 16, 4, 0, 11, 1, 0, 0, 7, 14, 1, 0, 7, 5, 0, 0, 4, 10, 0, 0, 7, 7, 0, 0, 1, 12, 0, 0, 12, 3, 0, 0, 0, 9, 6, 6, 13, 0, 0, 0, 0, 0, 10, 14, 4, 0, 0], [0, 0, 4, 13, 15, 6, 0, 0, 0, 0, 15, 11, 2, 14, 2, 0, 0, 3, 14, 1, 0, 12, 4, 0, 0, 5, 12, 0, 0, 9, 5, 0, 0, 5, 5, 0, 0, 12, 2, 0, 0, 4, 9, 0, 2, 13, 2, 0, 0, 0, 13, 2, 14, 7, 0, 0, 0, 0, 5, 16, 7, 0, 0, 0], [0, 0, 10, 13, 9, 1, 0, 0, 0, 2, 16, 7, 10, 8, 0, 0, 0, 0, 12, 12, 7, 11, 0, 0, 0, 3, 16, 16, 16, 7, 0, 0, 0, 0, 5, 8, 12, 10, 1, 0, 0, 0, 0, 0, 0, 11, 7, 0, 0, 0, 0, 0, 0, 3, 15, 0, 0, 0, 11, 16, 16, 16, 8, 0], [0, 2, 13, 13, 11, 9, 0, 0, 0, 10, 16, 16, 16, 15, 10, 0, 0, 11, 16, 9, 0, 0, 0, 0, 0, 3, 15, 16, 8, 0, 0, 0, 0, 0, 2, 11, 14, 0, 0, 0, 0, 0, 0, 8, 16, 0, 0, 0, 0, 0, 1, 11, 11, 0, 0, 0, 0, 1, 16, 15, 4, 0, 0, 0], [0, 0, 9, 16, 7, 0, 0, 0, 0, 0, 14, 13, 16, 2, 0, 0, 0, 0, 7, 9, 15, 8, 0, 0, 0, 0, 0, 1, 13, 9, 0, 0, 0, 0, 0, 0, 14, 7, 0, 0, 0, 0, 0, 4, 16, 5, 0, 0, 0, 0, 7, 16, 16, 8, 6, 0, 0, 0, 9, 15, 12, 16, 16, 9], [0, 0, 1, 11, 15, 0, 0, 0, 0, 0, 11, 15, 5, 0, 0, 0, 0, 3, 15, 1, 0, 0, 0, 0, 0, 5, 12, 0, 0, 0, 0, 0, 0, 8, 15, 15, 16, 14, 3, 0, 0, 2, 16, 11, 2, 7, 12, 0, 0, 0, 14, 11, 4, 9, 13, 0, 0, 0, 2, 11, 16, 15, 6, 0], [0, 0, 1, 13, 14, 3, 0, 0, 0, 0, 8, 16, 13, 2, 0, 0, 0, 2, 16, 16, 3, 0, 0, 0, 0, 3, 16, 12, 1, 0, 0, 0, 0, 5, 16, 14, 5, 0, 0, 0, 0, 3, 16, 16, 16, 16, 6, 0, 0, 1, 14, 16, 16, 16, 12, 0, 0, 0, 3, 12, 15, 14, 7, 0], [0, 0, 3, 12, 15, 11, 2, 0, 0, 0, 11, 13, 7, 13, 8, 0, 0, 7, 15, 1, 5, 15, 3, 0, 0, 1, 12, 16, 16, 5, 0, 0, 0, 0, 0, 13, 15, 15, 2, 0, 0, 0, 2, 13, 0, 10, 5, 0, 0, 0, 4, 11, 4, 11, 6, 0, 0, 0, 2, 13, 16, 12, 0, 0], [0, 0, 9, 16, 13, 1, 0, 0, 0, 8, 16, 8, 11, 8, 0, 0, 0, 11, 10, 0, 8, 10, 0, 0, 0, 1, 5, 0, 11, 11, 0, 0, 0, 0, 0, 0, 13, 9, 0, 0, 0, 0, 0, 4, 16, 2, 0, 0, 0, 0, 4, 14, 15, 7, 4, 1, 0, 0, 8, 16, 16, 16, 16, 12], [0, 0, 8, 16, 16, 13, 0, 0, 0, 0, 10, 11, 9, 16, 2, 0, 0, 0, 0, 4, 16, 12, 0, 0, 0, 0, 2, 16, 15, 1, 0, 0, 0, 0, 2, 15, 11, 0, 0, 0, 0, 0, 0, 4, 16, 3, 0, 0, 0, 3, 12, 2, 14, 4, 0, 0, 0, 0, 9, 16, 16, 5, 0, 0], [0, 0, 11, 16, 16, 11, 0, 0, 0, 0, 10, 16, 7, 2, 0, 0, 0, 1, 16, 9, 0, 0, 0, 0, 0, 0, 13, 16, 8, 0, 0, 0, 0, 0, 0, 4, 15, 4, 0, 0, 0, 0, 0, 0, 14, 8, 0, 0, 0, 0, 3, 8, 16, 4, 0, 0, 0, 0, 14, 16, 9, 0, 0, 0], [0, 2, 9, 16, 13, 13, 2, 0, 0, 11, 11, 4, 2, 10, 4, 0, 0, 6, 12, 2, 4, 12, 0, 0, 0, 0, 6, 14, 13, 2, 0, 0, 0, 0, 1, 14, 12, 0, 0, 0, 0, 0, 8, 7, 13, 0, 0, 0, 0, 0, 12, 5, 12, 0, 0, 0, 0, 0, 13, 14, 3, 0, 0, 0], [0, 0, 2, 12, 8, 7, 6, 2, 0, 0, 9, 16, 15, 16, 16, 5, 0, 0, 13, 11, 0, 10, 14, 0, 0, 0, 11, 3, 2, 15, 4, 0, 0, 0, 0, 0, 11, 9, 0, 0, 0, 0, 0, 8, 16, 4, 0, 0, 0, 0, 0, 15, 12, 0, 0, 0, 0, 0, 4, 16, 2, 0, 0, 0], [0, 0, 2, 12, 9, 0, 0, 0, 0, 0, 1, 15, 7, 9, 0, 0, 0, 0, 0, 11, 1, 14, 1, 0, 0, 0, 10, 8, 11, 13, 1, 0, 0, 0, 9, 14, 16, 9, 0, 0, 0, 0, 0, 0, 1, 10, 3, 0, 0, 0, 0, 0, 0, 3, 14, 2, 0, 0, 5, 14, 13, 14, 10, 1], [0, 4, 15, 14, 11, 2, 0, 0, 0, 7, 10, 1, 11, 11, 0, 0, 0, 1, 13, 4, 13, 16, 2, 0, 0, 0, 3, 11, 7, 9, 6, 0, 0, 0, 0, 0, 0, 5, 8, 0, 0, 0, 0, 0, 0, 5, 8, 0, 0, 3, 12, 0, 3, 13, 5, 0, 0, 2, 13, 16, 16, 8, 0, 0], [0, 0, 6, 16, 16, 8, 0, 0, 0, 2, 16, 8, 9, 16, 3, 0, 0, 8, 16, 1, 0, 9, 9, 0, 0, 9, 12, 0, 0, 8, 12, 0, 0, 10, 12, 0, 0, 8, 10, 0, 0, 8, 13, 0, 0, 9, 8, 0, 0, 2, 16, 8, 6, 15, 3, 0, 0, 0, 8, 16, 15, 8, 0, 0], [0, 1, 13, 16, 16, 12, 0, 0, 0, 3, 12, 6, 11, 14, 0, 0, 0, 0, 0, 0, 9, 12, 0, 0, 0, 1, 9, 15, 16, 16, 9, 0, 0, 2, 12, 15, 14, 5, 5, 0, 0, 0, 5, 15, 1, 0, 0, 0, 0, 0, 11, 9, 0, 0, 0, 0, 0, 0, 15, 3, 0, 0, 0, 0], [0, 0, 4, 15, 16, 6, 0, 0, 0, 0, 13, 11, 11, 15, 0, 0, 0, 0, 15, 13, 15, 16, 7, 0, 0, 0, 7, 16, 16, 11, 2, 0, 0, 0, 5, 15, 16, 2, 0, 0, 0, 0, 16, 9, 12, 11, 0, 0, 0, 2, 16, 6, 8, 16, 0, 0, 0, 0, 7, 14, 13, 8, 0, 0], [0, 0, 3, 12, 16, 10, 0, 0, 0, 2, 14, 12, 12, 12, 0, 0, 0, 5, 10, 0, 10, 11, 0, 0, 0, 0, 0, 1, 14, 9, 2, 0, 0, 0, 8, 16, 16, 16, 10, 0, 0, 0, 6, 16, 13, 7, 0, 0, 0, 0, 0, 16, 5, 0, 0, 0, 0, 0, 5, 13, 0, 0, 0, 0], [0, 0, 2, 16, 5, 0, 0, 0, 0, 0, 4, 16, 10, 0, 0, 0, 0, 0, 5, 16, 7, 0, 0, 0, 0, 0, 8, 16, 9, 0, 0, 0, 0, 0, 15, 16, 14, 2, 0, 0, 0, 0, 7, 8, 14, 10, 0, 0, 0, 0, 12, 15, 14, 16, 14, 9, 0, 0, 2, 10, 13, 16, 10, 3], [0, 0, 10, 16, 16, 8, 0, 0, 0, 5, 16, 6, 7, 14, 0, 0, 0, 3, 16, 3, 13, 9, 0, 0, 0, 0, 13, 15, 9, 0, 0, 0, 0, 0, 10, 16, 1, 0, 0, 0, 0, 0, 16, 15, 6, 0, 0, 0, 0, 1, 15, 12, 11, 0, 0, 0, 0, 0, 5, 16, 10, 0, 0, 0], [0, 2, 13, 16, 16, 11, 0, 0, 0, 10, 11, 4, 12, 12, 0, 0, 0, 1, 1, 4, 14, 8, 0, 0, 0, 0, 2, 16, 16, 8, 0, 0, 0, 0, 0, 7, 9, 16, 8, 0, 0, 0, 0, 0, 0, 10, 12, 0, 0, 0, 5, 9, 10, 16, 9, 0, 0, 0, 15, 16, 13, 7, 0, 0], [0, 1, 13, 16, 15, 5, 0, 0, 0, 4, 16, 7, 14, 12, 0, 0, 0, 3, 12, 2, 11, 10, 0, 0, 0, 0, 0, 0, 14, 8, 0, 0, 0, 0, 0, 3, 16, 4, 0, 0, 0, 0, 1, 11, 13, 0, 0, 0, 0, 0, 9, 16, 14, 16, 7, 0, 0, 1, 16, 16, 15, 12, 5, 0], [0, 0, 2, 12, 12, 2, 0, 0, 0, 0, 10, 16, 16, 9, 0, 0, 0, 3, 16, 16, 8, 15, 3, 0, 0, 6, 16, 6, 0, 13, 8, 0, 0, 8, 16, 4, 0, 15, 8, 0, 0, 5, 16, 8, 12, 16, 6, 0, 0, 0, 15, 16, 16, 15, 2, 0, 0, 0, 3, 13, 12, 3, 0, 0], [0, 5, 15, 12, 12, 12, 4, 0, 0, 10, 14, 12, 12, 9, 7, 0, 0, 12, 11, 0, 0, 0, 0, 0, 0, 5, 15, 10, 0, 0, 0, 0, 0, 0, 3, 16, 3, 0, 0, 0, 0, 0, 0, 15, 4, 0, 0, 0, 0, 1, 7, 16, 4, 0, 0, 0, 0, 4, 15, 12, 0, 0, 0, 0], [0, 0, 5, 16, 15, 5, 0, 0, 0, 0, 8, 13, 9, 15, 0, 0, 0, 0, 0, 4, 0, 15, 5, 0, 0, 0, 0, 0, 0, 12, 6, 0, 0, 0, 0, 0, 0, 15, 4, 0, 0, 0, 11, 10, 10, 15, 0, 0, 0, 0, 16, 16, 16, 15, 13, 4, 0, 0, 7, 16, 13, 10, 8, 3], [0, 0, 9, 13, 16, 16, 4, 0, 0, 2, 12, 5, 4, 14, 4, 0, 0, 0, 0, 0, 5, 13, 0, 0, 0, 0, 0, 3, 13, 12, 5, 0, 0, 0, 0, 13, 14, 12, 8, 0, 0, 0, 0, 13, 6, 0, 0, 0, 0, 0, 4, 14, 1, 0, 0, 0, 0, 0, 14, 7, 0, 0, 0, 0], [0, 0, 4, 12, 13, 1, 0, 0, 0, 0, 4, 16, 16, 5, 0, 0, 0, 0, 9, 16, 10, 0, 0, 0, 0, 8, 16, 16, 11, 4, 0, 0, 0, 0, 4, 8, 16, 16, 7, 0, 0, 0, 0, 0, 2, 14, 14, 0, 0, 0, 0, 4, 15, 16, 11, 0, 0, 0, 5, 16, 14, 8, 0, 0], [0, 0, 3, 12, 8, 1, 0, 0, 0, 0, 14, 16, 16, 15, 1, 0, 0, 3, 15, 2, 1, 12, 4, 0, 0, 6, 9, 0, 0, 7, 8, 0, 0, 7, 8, 0, 0, 5, 8, 0, 0, 4, 12, 0, 0, 9, 6, 0, 0, 0, 15, 11, 9, 16, 2, 0, 0, 0, 3, 11, 15, 7, 0, 0], [0, 0, 7, 11, 12, 14, 2, 0, 0, 8, 16, 9, 4, 3, 0, 0, 0, 10, 15, 5, 0, 0, 0, 0, 0, 3, 12, 16, 14, 4, 0, 0, 0, 0, 0, 2, 13, 16, 2, 0, 0, 0, 0, 0, 0, 15, 9, 0, 0, 0, 2, 4, 8, 15, 9, 0, 0, 0, 10, 16, 13, 8, 0, 0], [0, 0, 1, 7, 12, 3, 0, 0, 0, 4, 16, 12, 12, 10, 0, 0, 0, 14, 9, 0, 11, 8, 0, 0, 0, 7, 5, 0, 15, 4, 0, 0, 0, 0, 0, 2, 14, 7, 0, 0, 0, 0, 0, 0, 2, 13, 9, 0, 0, 0, 5, 10, 4, 0, 14, 5, 0, 0, 1, 9, 15, 16, 16, 8], [0, 0, 2, 14, 5, 0, 0, 0, 0, 0, 9, 12, 0, 0, 0, 0, 0, 1, 15, 1, 0, 0, 0, 0, 0, 3, 15, 0, 0, 0, 0, 0, 0, 6, 16, 16, 16, 13, 1, 0, 0, 2, 16, 8, 4, 7, 11, 0, 0, 0, 12, 11, 1, 8, 11, 0, 0, 0, 3, 12, 16, 15, 4, 0], [0, 2, 15, 13, 0, 0, 0, 0, 0, 12, 16, 16, 3, 0, 0, 0, 0, 5, 13, 16, 4, 0, 0, 0, 0, 0, 8, 16, 4, 0, 0, 0, 0, 0, 8, 16, 4, 0, 0, 0, 0, 0, 12, 16, 0, 0, 0, 0, 0, 0, 16, 16, 16, 14, 6, 0, 0, 1, 16, 16, 16, 12, 7, 0], [0, 0, 10, 11, 4, 0, 0, 0, 0, 0, 10, 15, 13, 13, 1, 0, 0, 0, 8, 11, 0, 14, 4, 0, 0, 0, 0, 13, 15, 13, 0, 0, 0, 1, 11, 16, 16, 0, 0, 0, 0, 1, 15, 3, 9, 10, 0, 0, 0, 0, 14, 6, 15, 10, 0, 0, 0, 0, 8, 14, 7, 1, 0, 0], [0, 0, 12, 16, 14, 8, 0, 0, 0, 7, 16, 10, 14, 16, 0, 0, 0, 4, 16, 11, 14, 16, 4, 0, 0, 0, 5, 14, 16, 16, 8, 0, 0, 0, 0, 0, 0, 16, 8, 0, 0, 0, 0, 0, 4, 16, 6, 0, 0, 2, 12, 9, 16, 15, 1, 0, 0, 1, 9, 16, 14, 3, 0, 0], [0, 0, 5, 16, 14, 1, 0, 0, 0, 0, 14, 14, 14, 9, 0, 0, 0, 1, 15, 6, 1, 12, 1, 0, 0, 3, 16, 1, 0, 10, 4, 0, 0, 5, 16, 5, 0, 11, 2, 0, 0, 2, 16, 2, 3, 16, 0, 0, 0, 0, 11, 13, 14, 12, 0, 0, 0, 0, 3, 15, 15, 5, 0, 0], [0, 0, 7, 16, 14, 3, 0, 0, 0, 0, 9, 14, 11, 15, 0, 0, 0, 0, 1, 5, 0, 15, 5, 0, 0, 0, 0, 0, 0, 16, 5, 0, 0, 0, 0, 0, 3, 16, 4, 0, 0, 0, 0, 1, 12, 14, 1, 0, 0, 0, 5, 12, 16, 16, 14, 1, 0, 0, 8, 16, 14, 10, 13, 3], [0, 0, 5, 13, 16, 16, 6, 0, 0, 0, 12, 6, 4, 13, 9, 0, 0, 0, 0, 0, 1, 15, 3, 0, 0, 0, 0, 2, 10, 13, 2, 0, 0, 2, 12, 16, 16, 12, 5, 0, 0, 1, 6, 9, 11, 0, 0, 0, 0, 0, 0, 15, 1, 0, 0, 0, 0, 0, 6, 11, 0, 0, 0, 0], [0, 0, 0, 0, 12, 5, 0, 0, 0, 0, 0, 2, 16, 12, 0, 0, 0, 0, 1, 12, 16, 11, 0, 0, 0, 2, 12, 16, 16, 10, 0, 0, 0, 6, 11, 5, 15, 6, 0, 0, 0, 0, 0, 1, 16, 9, 0, 0, 0, 0, 0, 2, 16, 11, 0, 0, 0, 0, 0, 3, 16, 8, 0, 0], [0, 0, 1, 13, 16, 14, 4, 0, 0, 2, 11, 8, 4, 11, 7, 0, 0, 6, 16, 3, 3, 13, 2, 0, 0, 0, 9, 14, 14, 4, 0, 0, 0, 0, 7, 16, 10, 0, 0, 0, 0, 0, 12, 10, 16, 1, 0, 0, 0, 0, 11, 10, 15, 4, 0, 0, 0, 0, 1, 14, 15, 1, 0, 0], [0, 0, 4, 13, 11, 7, 0, 0, 0, 0, 14, 16, 13, 16, 2, 0, 0, 5, 16, 4, 0, 5, 7, 0, 0, 8, 14, 0, 0, 4, 8, 0, 0, 6, 9, 0, 0, 4, 8, 0, 0, 2, 14, 1, 0, 8, 6, 0, 0, 0, 13, 12, 9, 15, 2, 0, 0, 0, 3, 16, 12, 5, 0, 0], [0, 0, 6, 16, 16, 16, 10, 0, 0, 0, 13, 15, 9, 6, 0, 0, 0, 6, 16, 4, 0, 0, 0, 0, 0, 12, 15, 4, 2, 0, 0, 0, 0, 8, 16, 16, 16, 15, 1, 0, 0, 0, 6, 8, 9, 16, 4, 0, 0, 0, 1, 3, 13, 15, 1, 0, 0, 0, 7, 16, 15, 3, 0, 0], [0, 0, 10, 16, 16, 10, 1, 0, 0, 4, 16, 11, 11, 16, 3, 0, 0, 1, 9, 1, 10, 15, 1, 0, 0, 0, 0, 5, 16, 10, 0, 0, 0, 0, 0, 0, 7, 15, 10, 0, 0, 0, 0, 0, 0, 7, 16, 0, 0, 2, 12, 7, 4, 14, 15, 1, 0, 0, 11, 16, 16, 15, 4, 0], [0, 2, 11, 13, 11, 2, 0, 0, 0, 7, 12, 4, 13, 8, 0, 0, 0, 6, 13, 5, 14, 13, 0, 0, 0, 0, 3, 11, 9, 11, 5, 0, 0, 0, 0, 0, 0, 8, 8, 0, 0, 0, 0, 0, 0, 3, 11, 0, 0, 0, 2, 0, 1, 9, 10, 0, 0, 1, 15, 16, 16, 14, 2, 0], [0, 0, 2, 9, 14, 16, 15, 0, 0, 3, 16, 13, 8, 10, 16, 0, 0, 1, 2, 0, 0, 11, 9, 0, 0, 0, 3, 11, 12, 16, 11, 0, 0, 0, 12, 14, 16, 12, 2, 0, 0, 0, 0, 3, 16, 3, 0, 0, 0, 0, 0, 7, 10, 0, 0, 0, 0, 0, 0, 14, 5, 0, 0, 0], [0, 0, 0, 5, 16, 9, 0, 0, 0, 0, 1, 13, 16, 6, 0, 0, 0, 0, 13, 16, 16, 4, 0, 0, 0, 5, 15, 16, 16, 5, 0, 0, 0, 0, 0, 10, 16, 7, 0, 0, 0, 0, 0, 9, 16, 8, 0, 0, 0, 0, 0, 9, 16, 13, 0, 0, 0, 0, 0, 5, 14, 9, 0, 0], [0, 0, 5, 15, 12, 4, 0, 0, 0, 2, 15, 8, 11, 16, 4, 0, 0, 8, 9, 0, 6, 16, 4, 0, 0, 8, 8, 0, 2, 10, 8, 0, 0, 8, 7, 0, 0, 13, 5, 0, 0, 2, 14, 0, 0, 16, 2, 0, 0, 0, 14, 8, 11, 10, 0, 0, 0, 0, 4, 13, 14, 0, 0, 0], [0, 0, 7, 14, 12, 1, 0, 0, 0, 7, 14, 5, 8, 10, 0, 0, 0, 8, 11, 1, 7, 10, 0, 0, 0, 1, 9, 16, 15, 4, 0, 0, 0, 0, 1, 14, 14, 12, 0, 0, 0, 0, 7, 11, 0, 12, 7, 0, 0, 0, 11, 5, 0, 11, 8, 0, 0, 0, 4, 14, 16, 12, 1, 0]], "labeled": [48, 80, 90, 115, 56, 76, 66, 8, 67, 81, 19, 64, 16, 94, 45, 20, 119, 60, 43, 25], "labels": [9, 2, 7, 9, 3, 2, 8, 3, 8, 3, 9, 2, 3, 2, 7, 5, 8, 6, 5, 6], "strategy": "hybrid", "batch": 5, "seed": 11, "expected_picks": [96, 36, 26, 7, 105]}