| # Stage 3 - do tick embeddings add to the shipped pre-match P4 model? |
|
|
| ## Setup |
| - tools/t3_enc.py: per (match,map,player) mean of the frozen encoder z over all ticks of the map, |
| random-projected to 64 dims. Plus 4 behavioural summaries of the same ticks (mean speed, alive |
| fraction, mean z, mean sin(yaw)) so the embedding can be separated from "he was alive longer". |
| - tools/t3_prof.py: causal (expanding) per-player profile, exactly the Phase-1 protocol: |
| p_ = mean over EARLIER maps, m_ = mean over earlier maps on the same map, ordered by match timestamp. |
| - tools/t3_eval_cov.py: base = 88 shipped features, +beh = + 4 behavioural profiles, +emb = + 64 |
| embedding profiles, +both. Expanding-origin by quarter (folds Q>400 rows, first 3 dropped), |
| xgb 2 seeds + logistic .5/.5, and a map-clustered bootstrap on the NON-TIE accuracy delta, |
| which is the shipped gate metric (+0.3pp). |
|
|
| ## Coverage is the whole story |
| Tick data exists for 3.3k of 6182 pairs6 maps. Only 11.4% of the 154,550 pairs have BOTH players |
| covered (17,556 pairs). p1_prof_eval.py on the full set therefore evaluates a treatment that is |
| NaN-imputed for 89% of rows (it reported cov 0.092 and produced no improvement - meaningless). |
| t3_eval_cov.py restricts fitting and scoring to the 17,556 covered pairs, which is the only honest test. |
|
|
| ## Results (covered pairs only, 15,368 scored on the expanding-origin folds) |
|
|
| 14k checkpoint 30k checkpoint |
| base 88 feats acc .5911 auc .6310 base .5911 / .6310 (identical) |
| +4 behavioural acc .5904 auc .6303 -0.12pp [-0.42, +0.17] |
| +64 embedding acc .5897 auc .6321 -0.50pp [-1.02, +0.01] |
| +both acc .5897 auc .6309 -0.44pp [-0.93, +0.07] |
|
|
| Gate is +0.30pp. Every CI excludes the gate and none is positive. VERDICT: FAIL, clearly - and |
| this is now measured twice with two different encoders. |
|
|
| ## Reading |
| On the 11% of pairs where we have tick data the shipped 88-feature model already gets .5911, and a |
| per-player profile of a 192-d SSL embedding adds nothing measurable. This is the same answer Phase 1 |
| gave for every other embedding family (a_gru/b_gru +0.12pp, all others ~0): the per-player information |
| in a pre-match kill pair is already saturated by the aggregate features, and the tick encoder's skill |
| is in *within-round* state prediction (Stage 2, AUC .78) not in stable cross-match player traits. |
|
|
