WH route B: m=2 N grid (virgo rerun), gates txt per N, run log
Browse files- results/witsenhausen/route_b_N/arms_m2_s0.json +0 -0
- results/witsenhausen/route_b_N/gates_s0_N1000.txt +14 -0
- results/witsenhausen/route_b_N/gates_s0_N300.txt +14 -0
- results/witsenhausen/route_b_N/gates_s0_N3000.txt +14 -0
- results/witsenhausen/route_b_N/gen_m2_N1000_s0.pth +3 -0
- results/witsenhausen/route_b_N/gen_m2_N3000_s0.pth +3 -0
- results/witsenhausen/route_b_N/gen_m2_N300_s0.pth +2 -2
- results/witsenhausen/route_b_N/wh_rbN_m2_virgo.log +45 -0
results/witsenhausen/route_b_N/arms_m2_s0.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
results/witsenhausen/route_b_N/gates_s0_N1000.txt
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
== honest J (pre-registered gates)
|
| 2 |
+
m 1 (test 200): GEN 58.0% KNN-BR 100.0% KNN-KS 100.0% gap -42.0 pts McNemar gen-only 0 knn-only 84 p 1e-25
|
| 3 |
+
C1 (GEN >= 70%): FAIL
|
| 4 |
+
m 2 (test 200): GEN 10.0% KNN-BR 30.0% KNN-KS 18.0% gap -20.0 pts McNemar gen-only 16 knn-only 56 p 2.4e-06
|
| 5 |
+
m 4 (test 200): GEN 0.0% KNN-BR 5.0% KNN-KS 4.5% gap -5.0 pts McNemar gen-only 0 knn-only 10 p 0.002
|
| 6 |
+
C2 (gap >= 10 pts, p < 0.05): FAIL KILL (KNN-BR >= GEN): True KNN-KS >= GEN (then C2 is not 'generation beats retrieval'): True
|
| 7 |
+
C3: gap(m 4) - gap(m 1) = +37.0 pts, paired bootstrap 95% CI [+29.5, +44.5] over 200 test seeds: PASS
|
| 8 |
+
== exact-Bayes controller 2 (pre-registered diagnostic)
|
| 9 |
+
m 1 (test 200): GEN 87.5% KNN-BR 100.0% KNN-KS 100.0% gap -12.5 pts McNemar gen-only 0 knn-only 25 p 6e-08
|
| 10 |
+
C1 (GEN >= 70%): PASS
|
| 11 |
+
m 2 (test 200): GEN 70.5% KNN-BR 41.5% KNN-KS 29.0% gap +29.0 pts McNemar gen-only 79 knn-only 21 p 4.3e-09
|
| 12 |
+
m 4 (test 200): GEN 24.0% KNN-BR 11.0% KNN-KS 13.0% gap +13.0 pts McNemar gen-only 42 knn-only 16 p 0.00086
|
| 13 |
+
C2 (gap >= 10 pts, p < 0.05): PASS KILL (KNN-BR >= GEN): False KNN-KS >= GEN (then C2 is not 'generation beats retrieval'): False
|
| 14 |
+
C3: gap(m 4) - gap(m 1) = +25.5 pts, paired bootstrap 95% CI [+18.0, +33.5] over 200 test seeds: PASS
|
results/witsenhausen/route_b_N/gates_s0_N300.txt
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
== honest J (pre-registered gates)
|
| 2 |
+
m 1 (test 200): GEN 68.0% KNN-BR 100.0% KNN-KS 100.0% gap -32.0 pts McNemar gen-only 0 knn-only 64 p 1.1e-19
|
| 3 |
+
C1 (GEN >= 70%): FAIL
|
| 4 |
+
m 2 (test 200): GEN 1.5% KNN-BR 20.5% KNN-KS 20.5% gap -19.0 pts McNemar gen-only 2 knn-only 40 p 4.1e-10
|
| 5 |
+
m 4 (test 200): GEN 0.0% KNN-BR 4.0% KNN-KS 5.5% gap -4.0 pts McNemar gen-only 0 knn-only 8 p 0.0078
|
| 6 |
+
C2 (gap >= 10 pts, p < 0.05): FAIL KILL (KNN-BR >= GEN): True KNN-KS >= GEN (then C2 is not 'generation beats retrieval'): True
|
| 7 |
+
C3: gap(m 4) - gap(m 1) = +28.0 pts, paired bootstrap 95% CI [+21.0, +35.0] over 200 test seeds: PASS
|
| 8 |
+
== exact-Bayes controller 2 (pre-registered diagnostic)
|
| 9 |
+
m 1 (test 200): GEN 84.5% KNN-BR 100.0% KNN-KS 100.0% gap -15.5 pts McNemar gen-only 0 knn-only 31 p 9.3e-10
|
| 10 |
+
C1 (GEN >= 70%): PASS
|
| 11 |
+
m 2 (test 200): GEN 54.0% KNN-BR 28.5% KNN-KS 29.0% gap +25.5 pts McNemar gen-only 70 knn-only 19 p 5e-08
|
| 12 |
+
m 4 (test 200): GEN 7.5% KNN-BR 8.0% KNN-KS 14.5% gap -0.5 pts McNemar gen-only 11 knn-only 12 p 1
|
| 13 |
+
C2 (gap >= 10 pts, p < 0.05): FAIL KILL (KNN-BR >= GEN): True KNN-KS >= GEN (then C2 is not 'generation beats retrieval'): True
|
| 14 |
+
C3: gap(m 4) - gap(m 1) = +15.0 pts, paired bootstrap 95% CI [+9.0, +21.0] over 200 test seeds: PASS
|
results/witsenhausen/route_b_N/gates_s0_N3000.txt
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
== honest J (pre-registered gates)
|
| 2 |
+
m 1 (test 200): GEN 49.0% KNN-BR 100.0% KNN-KS 100.0% gap -51.0 pts McNemar gen-only 0 knn-only 102 p 3.9e-31
|
| 3 |
+
C1 (GEN >= 70%): FAIL
|
| 4 |
+
m 2 (test 200): GEN 22.0% KNN-BR 41.0% KNN-KS 16.5% gap -19.0 pts McNemar gen-only 28 knn-only 66 p 0.00011
|
| 5 |
+
m 4 (test 200): GEN 0.5% KNN-BR 5.5% KNN-KS 3.5% gap -5.0 pts McNemar gen-only 1 knn-only 11 p 0.0063
|
| 6 |
+
C2 (gap >= 10 pts, p < 0.05): FAIL KILL (KNN-BR >= GEN): True KNN-KS >= GEN (then C2 is not 'generation beats retrieval'): True
|
| 7 |
+
C3: gap(m 4) - gap(m 1) = +46.0 pts, paired bootstrap 95% CI [+38.5, +53.5] over 200 test seeds: PASS
|
| 8 |
+
== exact-Bayes controller 2 (pre-registered diagnostic)
|
| 9 |
+
m 1 (test 200): GEN 75.0% KNN-BR 100.0% KNN-KS 100.0% gap -25.0 pts McNemar gen-only 0 knn-only 50 p 1.8e-15
|
| 10 |
+
C1 (GEN >= 70%): PASS
|
| 11 |
+
m 2 (test 200): GEN 78.5% KNN-BR 51.0% KNN-KS 26.0% gap +27.5 pts McNemar gen-only 77 knn-only 22 p 2.5e-08
|
| 12 |
+
m 4 (test 200): GEN 52.5% KNN-BR 10.5% KNN-KS 8.5% gap +42.0 pts McNemar gen-only 94 knn-only 10 p 2.9e-18
|
| 13 |
+
C2 (gap >= 10 pts, p < 0.05): PASS KILL (KNN-BR >= GEN): False KNN-KS >= GEN (then C2 is not 'generation beats retrieval'): False
|
| 14 |
+
C3: gap(m 4) - gap(m 1) = +67.0 pts, paired bootstrap 95% CI [+58.5, +75.5] over 200 test seeds: PASS
|
results/witsenhausen/route_b_N/gen_m2_N1000_s0.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f35c6fb747186d0f4e7d30163dbfe0e093249f48aeba63cd4211b7cb013bbb7c
|
| 3 |
+
size 1399966
|
results/witsenhausen/route_b_N/gen_m2_N3000_s0.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:311ea1c7a6f23248a9149a3cfa0f6913197b647984618b8cc6ecc981695636fb
|
| 3 |
+
size 1399966
|
results/witsenhausen/route_b_N/gen_m2_N300_s0.pth
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:094e34a27edad215a87a8157f46c68d1e8c72c47b61fb64025260ba2695aaba2
|
| 3 |
+
size 1399945
|
results/witsenhausen/route_b_N/wh_rbN_m2_virgo.log
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/cercola/matte_server/DiffusionControl/witsenhausen/route_b.py:257: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
|
| 2 |
+
Consider using tensor.detach() first. (Triggered internally at /__w/pytorch/pytorch/torch/csrc/autograd/generated/python_variable_methods.cpp:852.)
|
| 3 |
+
log(f" gen step {it + 1}/{steps}: loss {float(loss):.4f} [{time.time() - t0:.0f}s]")
|
| 4 |
+
gen step 3000/30000: loss 0.2163 [71s]
|
| 5 |
+
gen step 6000/30000: loss 0.2060 [145s]
|
| 6 |
+
gen step 9000/30000: loss 0.2108 [215s]
|
| 7 |
+
gen step 12000/30000: loss 0.2176 [289s]
|
| 8 |
+
gen step 15000/30000: loss 0.2007 [365s]
|
| 9 |
+
gen step 18000/30000: loss 0.1997 [449s]
|
| 10 |
+
gen step 21000/30000: loss 0.2057 [533s]
|
| 11 |
+
gen step 24000/30000: loss 0.2041 [606s]
|
| 12 |
+
gen step 27000/30000: loss 0.2158 [689s]
|
| 13 |
+
gen step 30000/30000: loss 0.1938 [773s]
|
| 14 |
+
gen in-bank (first 50 train rows): within-1% 72% median regret 0.61%
|
| 15 |
+
m 2 N 300: gen within-1% 1.5% (median regret 13.78%, below ref 0%) knn_br within-1% 20.5% (median regret 3.86%, below ref 0%) knn_ks within-1% 20.5% (median regret 7.80%, below ref 0%) knn_map within-1% 1.0% (median regret 33.41%, below ref 0%) aff within-1% 0.0% (median regret 199.17%, below ref 0%)
|
| 16 |
+
McNemar gen vs knn_br: gen-only 2, knn-only 40, p 4.1e-10
|
| 17 |
+
[m 2 N 300 done, 799s]
|
| 18 |
+
gen step 3000/30000: loss 0.2218 [73s]
|
| 19 |
+
gen step 6000/30000: loss 0.2138 [154s]
|
| 20 |
+
gen step 9000/30000: loss 0.2067 [229s]
|
| 21 |
+
gen step 12000/30000: loss 0.2034 [326s]
|
| 22 |
+
gen step 15000/30000: loss 0.1968 [469s]
|
| 23 |
+
gen step 18000/30000: loss 0.1914 [585s]
|
| 24 |
+
gen step 21000/30000: loss 0.2001 [728s]
|
| 25 |
+
gen step 24000/30000: loss 0.2003 [852s]
|
| 26 |
+
gen step 27000/30000: loss 0.2060 [989s]
|
| 27 |
+
gen step 30000/30000: loss 0.1912 [1143s]
|
| 28 |
+
gen in-bank (first 50 train rows): within-1% 38% median regret 1.63%
|
| 29 |
+
m 2 N 1000: gen within-1% 10.0% (median regret 8.58%, below ref 0%) knn_br within-1% 30.0% (median regret 2.03%, below ref 1%) knn_ks within-1% 18.0% (median regret 9.59%, below ref 0%) knn_map within-1% 1.5% (median regret 21.48%, below ref 0%) aff within-1% 0.0% (median regret 199.17%, below ref 0%)
|
| 30 |
+
McNemar gen vs knn_br: gen-only 16, knn-only 56, p 2.4e-06
|
| 31 |
+
[m 2 N 1000 done, 1971s]
|
| 32 |
+
gen step 3000/30000: loss 0.2421 [158s]
|
| 33 |
+
gen step 6000/30000: loss 0.2234 [355s]
|
| 34 |
+
gen step 9000/30000: loss 0.2228 [485s]
|
| 35 |
+
gen step 12000/30000: loss 0.2151 [672s]
|
| 36 |
+
gen step 15000/30000: loss 0.1999 [834s]
|
| 37 |
+
gen step 18000/30000: loss 0.1995 [1005s]
|
| 38 |
+
gen step 21000/30000: loss 0.2072 [1139s]
|
| 39 |
+
gen step 24000/30000: loss 0.1965 [1295s]
|
| 40 |
+
gen step 27000/30000: loss 0.2167 [1417s]
|
| 41 |
+
gen step 30000/30000: loss 0.2003 [1545s]
|
| 42 |
+
gen in-bank (first 50 train rows): within-1% 24% median regret 2.12%
|
| 43 |
+
m 2 N 3000: gen within-1% 22.0% (median regret 5.06%, below ref 0%) knn_br within-1% 41.0% (median regret 1.27%, below ref 1%) knn_ks within-1% 16.5% (median regret 10.71%, below ref 0%) knn_map within-1% 5.0% (median regret 12.63%, below ref 0%) aff within-1% 0.0% (median regret 199.17%, below ref 0%)
|
| 44 |
+
McNemar gen vs knn_br: gen-only 28, knn-only 66, p 0.00011
|
| 45 |
+
[m 2 N 3000 done, 3544s]
|