--- license: mit library_name: pytorch tags: - steam - game-reviews - representation-learning - contrastive-learning - set-encoder --- # I-CE checkpoints The trained set encoders ("towers") of the paper *Invariant Game Representation Learning from Steam Reviews* (AICCC 2026). A tower is a small attention-pooling network of about 0.36 M parameters: a few latent queries cross-attend over the cached sentence embeddings of a game's reviews (Qwen3-Embedding-0.6B, 1,024-d) and an MLP maps the pooled result to one L2-normalized 128-d vector per game. - Code, appendices and supplement: [`Nice9Tian/I2CE-GameRepresention-by-SteamReviews`](https://github.com/Nice9Tian/I2CE-GameRepresention-by-SteamReviews) - The tensors the towers read: [`chironYu/latenarray-i2ce-data`](https://huggingface.co/datasets/chironYu/latenarray-i2ce-data) ## Contents 275 checkpoints, about 593 MiB, and four JSON files that list every one of them: | Listed in | Checkpoints | What they are | |---|---|---| | `rank_ckpt_plan.json` | 166 | the five-fold runs behind the paper's numbers, each at the epoch the validation rank rule of Section 5 selects. One is the DINO-style control of Table A1, trained on fold 0 only; five, CE capped at 2,048 sentences, are the partners of I-CE runs the paper prints and are not printed themselves | | `i2bce_sel_pool.json` | 5 | I + SimCLR on the five folds (Figure 8, Appendix L), selected by the same rule | | `fixed_split_ckpt_plan.json` | 69 | the fixed-split runs: 15 runs at their selected epochs, two of them with an earlier checkpoint kept beside the selected one, and 52 more checkpoints of two runs that the paper reads across their epochs (field `trajectory`) | | `rank_ckpt_plan.json`, field `init_ckpt` | 5 | the BYOL checkpoints the five two-stage runs started from | | `superseded_ckpt_plan.json` | 30 | kept for the record; no number of the paper reads them | An entry of the three plan files gives the arm, the fold, the anchor pack, the selected epoch, the file name and the file's SHA-256; `i2bce_sel_pool.json` is the selection as the evaluation script wrote it, with the rank score of every checkpoint of the run. The same four files are in `contrast_experiment/w9/eval/` of the code repository, where the evaluation script reads them. Every tower that a number or a figure of the paper reads is here, at the checkpoint it is read at. A run saved a checkpoint every 50 epochs and the rank rule chose among them; the ones it did not choose are not here, except for the two runs listed under `trajectory`. The other passages that follow runs over training quote what the workers logged at each checkpoint: the learnable temperature and the view-count and view-size sweeps in Appendix C, and the decline of the views-first anchors after epoch 50 in Supplement A. ## File names ckpt_w9cv_wcle__fold_fp_ep.pt five folds, fold k is the test fold ckpt_w9_wcle__fp_ep.pt the fixed split - `` is the objective, listed in the tables below. - `` is the anchor pack the run trained with, counted in sentences with the store-page prefix included: nothing for 512, `_g1024`, `_g2048`, or `_gpool` for the prefix plus the game's whole 2,048-sentence review pool (a median of 2,068 sentences). The paper calls `_g2048` and `_gpool` alike its 2,048-sentence budget; the main five-fold towers are the `_gpool` runs. - ``: `_wllm` for LLM-rewritten documents as the document view, `_bw` for a run initialized from a BYOL tower, `_w` for review views drawn until they hold at least N sentences (the recipe stops at 16), `_d512` for a run on the current 512-sentence packs (see below). - `_fp`: the strong views were drawn from the full review pool. `ep` is the epoch. ## The five-fold runs Table and figure numbers are those of the paper, its appendices (Tables A1 to A19) and its supplement (Tables S1 to S4). The last column comes from the provenance table of the code repository, `contrast_experiment/w9/Pod/w9_profiles.py`, which cites by cell: a table named for a row reads at least one of the row's runs. That file also names the notebook that trains each cell. | Runs | What they are | Pack | Folds | Read by | |---|---|---|---|---| | `ai2ce_icetf_d512` | CE + anchor-in-I | 512 | 5 | Table 2, Section 6.2, Table A15 | | `ce_cetf_d512`, `i2ce_icetf_d512` | CE; I-CE, the recipe | 512 | 5 | Table 2, Table A2, Table A3, Table A4, Table A5, Table A6, Table A7, Table A11, Table A15, Table A19, Table S1 | | `d1r2_i2ce_icetf_d512`, `d1r4_i2ce_icetf_d512`, `d1r5_i2ce_icetf_d512`, `d1r5_i3ce_icetf_d512`, `d1r6_i2ce_icetf_d512` | I-CE, V = 3; I-CE, V = 5; I-CE, V = 6; I-CE, V = 6 with λ = 3; I-CE, V = 7 | 512 | 0 to 2 | Table A5 | | `i2ce_icetf_w8_d512`, `i2ce_icetf_w32_d512`, `i2ce_icetf_w64_d512` | I-CE, review views of at least 8 sentences; of at least 32; of at least 64 | 512 | 0 to 2 | Table A6 | | `i2ce_icetf_wllm_d512`, `nodoc_i2ce_icetf_d512` | I-CE, LLM-rewritten documents; I-CE without the document view | 512 | 5 | Table A19 | | `i2cet05_icetf_d512`, `i2cet10_icetf_d512` | I-CE, τ = 0.05; I-CE, τ = 0.10 | 512 | 5 | Table A3 | | `i3ce_icetf_d512`, `i4ce_icetf_d512`, `ice_icetf_d512` | I-CE, λ = 3; I-CE, λ = 4; I-CE, λ = 1 | 512 | 0 to 2 | Table A4 | | `slot16i2cemean_icetf_d512`, `slot1i2cemean_icetf_d512`, `slot2i2cemean_icetf_d512`, `slot8i2cemean_icetf_d512` | I-CE, 16 latent queries; I-CE, 1 latent query; I-CE, 2 latent queries; I-CE, 8 latent queries | 512 | 0 to 2 | Table A7 | | `vfai2ce_icetf_d512` | I-CE, views-first anchors | 512 | 5 | Table S1 | | `ce_cetf_g1024`, `i2ce_icetf_g1024` | CE; I-CE, the recipe | 1,024 | 5 | Table A2, Table A11, Table A15 | | `ce_cetf_g2048`, `i2ce_icetf_g2048` | CE; I-CE, the recipe | 2,048 | 5 | Table A3, Table A11 | | `i2ce_icetl_g2048`, `i2cet05_icetf_g2048`, `i2cet10_icetf_g2048` | I-CE, learnable τ; I-CE, τ = 0.05; I-CE, τ = 0.10 | 2,048 | 5 | Table A3 | | `bce_cetf_gpool`, `byol_bytf_gpool`, `epd_v20i10c20_cetf_gpool`, `mq3072i2ce_icetf_gpool` | SimCLR-style; BYOL; VICReg (batch 192); I-CE with an EMA teacher and a memory bank | pool | 5 | Figure 7, Figure 8, Table 2, Section 3.2, Table A1, Table A13, Table A16, Table A17, Table A18, Figure A2, Table S2, Table S3, Table S4 | | `ce_cetf_gpool`, `i2ce_icetf_gpool` | CE; I-CE, the recipe | pool | 5 | Figure 7, Figure 8, Table 2, Section 6.1, Table A1, Table A2, Table A7, Table A13, Table A14, Table A15, Table A16, Table A17, Table A18, Figure A2, Table S2, Table S3, Table S4 | | `slot8i2cemean_icetf_gpool` | I-CE, 8 latent queries | pool | 5 | Table A7 | | `swin168step84loop2i2ce_icetf_gpool`, `swin168step84loop2i2ce_icetf_gpool_bw` | I-CE with the windowed teacher; I-CE, two-stage: BYOL warm start, then the windowed teacher | pool | 5 | Table A13, Table S2, Table S3, Table S4 | | `dino_dntf_gpool` | DINO at its published settings, the control of Section 4.3 | pool | 0 only | Table A1 | I + SimCLR (`i2bce_icetf`, files without a pack or a suffix, five folds) backs Figure 8 and the five-fold readings of Appendix L. The fold-0 run of `i2ce_icetf_g2048` also gives the game vectors of Figure 10. The DINO-style control has one fold, so Table A1 gives its spread over the ten anchor draws; its file is 169 MiB because it carries the two prototype heads (see Format). ## The fixed-split runs | Runs | What they are | Pack | Folds | Read by | |---|---|---|---|---| | `ai25auni2_icetf_d512`, `ai2auni25_icetf_d512`, `ai4auni2_icetf_d512`, `ai6auni2_icetf_d512`, `i4uni2_icetf` | alignment + uniformity aimed at the anchor gallery, the four settings of Section 6.2 (`ai25auni2` is Table 3's a+u @ gallery row); alignment + uniformity aimed at the batch views (Table 3's a+u @ views row) | 512 | fixed split | Table 3, Section 6.2, Appendix L | | `ai2ce_icetf_d512`, `ce_cetf_d512`, `i2ce_icetf_d512` | CE + anchor-in-I; CE; I-CE, the recipe | 512 | fixed split | Table 3, Section 6.2, Table A8 | | `ce_cetf_g2048`, `i2ce_icetf_g2048` | CE; I-CE, the recipe | 2,048 | fixed split | Figure 9, Figure A1, Figure A3, Appendix L | | `bce_cetf`, `i2bce_icetf`, `epd_v20i10c20_cetf` | SimCLR-style; I + SimCLR; VICReg, v/i/c = 20/10/20 (the recipe weights) | 512 | fixed split | Figure 9, Figure A3, Appendix L, Table A8 | | `epd_v25i25c1_cetf`, `epd_v20i10c15_cetf` | VICReg, v/i/c = 25/25/1; VICReg, v/i/c = 20/10/15 | 512 | fixed split | Table A8 | Figure 9, Figure A3 and the neighborhood statistics of Appendix L read six encoders on the fixed split: the frozen embedder, which has no checkpoint, and five towers, namely `i2ce_icetf_g2048` (epoch 1,750), `ce_cetf_g2048` (1,150), `bce_cetf` (1,350), `i2bce_icetf` (1,350) and `epd_v20i10c20_cetf` (100). Two runs have more than their selected checkpoint here, listed under `trajectory` in the plan file: all 40 checkpoints of `i2ce_icetf_g2048`, whose four latent queries Figure A1 draws at every one, and the `bce_cetf` checkpoints from epoch 1,350 to 2,000, the span of the test curve Appendix L describes. Runs that read no anchor in training (`bce_cetf`, `i2bce_icetf`, `i4uni2_icetf` and the three VICReg weightings; the 512 in their rows is only the pack their worker encoded at each checkpoint) were selected after training: each of the run's 40 checkpoints encodes the 4,096-sentence anchor pack of the fixed split and the rank rule chooses on that. Ten of the fifteen runs stopped at 1,000 epochs under their first schedule and were trained on to 2,000 from the weights of epoch 1,000 with a fresh optimizer: those six, CE + anchor-in-I, and three of the gallery-aimed settings (`ai6auni2`, `ai4auni2`, `ai2auni25`). The plan entries say so under `extended`. Two selections moved with the longer runs, CE + anchor-in-I to epoch 1,650 and `ai4auni2` to epoch 1,250; the checkpoints selected before (epochs 1,000 and 900) are kept and named under `previous`. ## Format Every file holds tensors only, so `torch.load(path, weights_only=True)` reads it. Most are the state dict of the tower itself, `SetPoolN(Q)` of `contrast_experiment/w9/Pod/w9_cv_worker.py` (keys `q0`, `attn.*`, `head.*`; `q0` has shape `[1, Q, 128]`, Q = 4 except in the `slot` runs). Four families wrap it: BYOL files hold `{"model", "pred", "target"}`, VICReg files `{"model", "expander"}`, the memory-bank files `{"model", "shadow"}`, and the DINO-style file `{"model", "student", "t_head", "s_head", "center"}`; the tower is `model`. In the DINO-style file `model` is the teacher's tower, the one DINO evaluates and the paper reads, `student` the student's, and the two heads hold the 65,536 prototypes. ```python import sys, torch sys.path[:0] = ["contrast_experiment/w9", "contrast_experiment/w9/Pod"] # from the code repository's root from Pod.w9_cv_worker import SetPoolN sd = torch.load("checkpoints/ckpt_w9cv_wcle_i2ce_icetf_fold0_gpool_fp_ep2000.pt", weights_only=True) sd = sd.get("model", sd) sd = {k: v for k, v in sd.items() if k == "q0" or k.startswith(("attn.", "head."))} tower = SetPoolN(sd["q0"].shape[1]); tower.load_state_dict(sd); tower.eval() # S: [games, sentences, 1024] sentence vectors, each row standardized over its 1,024 dimensions # pad: [games, sentences] bool, True where a row is padding z = tower(S, pad) # [games, 128], L2-normalized ``` ## Scoring them as the paper does The paper scores a five-fold tower against ten independent 4,096-sentence anchor packs per game and reports the mean. From the root of the code repository, with the data folder in place: ```bash hf download chironYu/latenarray-i2ce-checkpoints --local-dir checkpoints export W9_DATA_DIR=/path/to/latenarray-i2ce-data python contrast_experiment/w9/eval/k10_eval.py cells cells.json --ckpt-dir checkpoints python contrast_experiment/w9/eval/k10_eval.py encode cells.json draws.npz python contrast_experiment/w9/eval/k10_eval.py score cells.json draws.npz scores.json python contrast_experiment/w9/eval/k10_eval.py table scores.json ``` `cells` takes the 171 five-fold checkpoints of `rank_ckpt_plan.json` and `i2bce_sel_pool.json`; `--only ` keeps a subset, for example `--only "_d512$"` for the 512-sentence runs. In its table the five families of the main comparison go by the paper's names (`I-CE`, `CE`, `SimCLR`, `VICReg`, `BYOL`) and every other family by its plan key without the fold; a family with one fold, the DINO-style control, is printed with its spread over the ten draws. The fixed split has one command of its own: ```bash python contrast_experiment/w9/eval/fixed_split_eval.py --ckpt-dir checkpoints ``` It reads every run of `fixed_split_ckpt_plan.json` at its selected checkpoint and prints each reading beside the number the paper prints: Table 3, the gallery-aimed settings of Section 6.2 and Table A8 on the one 4,096-sentence anchor draw of the fixed split, the SimCLR-style test curve of Appendix L, the latent queries of Figure A1, and, through `k10_eval.py`, the towers of Figure 9 against the ten draws. A near tie can move a cell by one query of 204 (0.005), and the collapsed a+u @ gallery row by about 0.01. The figures built on these towers are drawn from the checkpoints as well: `python figures/neighborhoods/run_all.py --ckpt-dir checkpoints --data-dir ` makes Figures 8, 9 and A3 and the neighborhood statistics of Appendix L. ## The 512-sentence packs and the `_d512` runs Until 2026-11-07 the 512-sentence anchor packs (`wscan_gal_rev.npz` of the dataset) carried the store-page prefix rows without the per-row standardization every other input receives (row std about 2.9 against 1.0), and the tower has no input normalization. Only the 512 budget was affected; larger packs were standardized. The dataset now carries the corrected packs, and every run whose loss reads the 512-sentence anchors was trained again on them under the paper's recipe. Those runs are named `_d512`, and they are the ones the paper reads. The workers of the code repository give a run on the current packs that name themselves. Runs whose loss never reads the anchors (the SimCLR-style tower, I + SimCLR, BYOL, VICReg and alignment plus uniformity aimed at the batch views) train the same weights on either pack, so their checkpoints were not replaced; only readings that used the 512-sentence packs were redone. Their files therefore carry no `_d512`: the fixed-split ones, and those of the five-fold I + SimCLR. Stripped hit@1 on the ten 4,096-sentence draws, five-fold mean ± std, before and after: | Tower | on the packs as first built | on the corrected packs (`_d512`) | |---|---|---| | I-CE (`i2ce_icetf`) | 0.669 ± 0.031 | 0.715 ± 0.010 | | CE (`ce_cetf`) | 0.638 ± 0.016 | 0.676 ± 0.013 | | CE + anchor-in-I (`ai2ce_icetf`) | 0.663 ± 0.014 | 0.725 ± 0.015 | | I-CE without the document view (`nodoc_i2ce_icetf`) | 0.650 ± 0.022 | 0.683 ± 0.025 | | I-CE, LLM-rewritten documents (`i2ce_icetf`, `_wllm`) | 0.680 ± 0.034 | 0.711 ± 0.009 | | I-CE, τ = 0.05 (`i2cet05_icetf`) | 0.601 ± 0.029 | 0.642 ± 0.026 | | I-CE, τ = 0.10 (`i2cet10_icetf`) | 0.510 ± 0.022 (four folds) | 0.520 ± 0.015 | The checkpoints behind the left column are the superseded files of the next section, except those of CE + anchor-in-I, whose first runs were not uploaded. The other `_d512` families (the sweeps on folds 0 to 2 and the views-first anchors) have no earlier counterpart here. ## Superseded checkpoints `superseded_ckpt_plan.json` lists 30 files that stay here although the paper reads none of them: the 29 five-fold runs at 512 sentences made on the packs as first built, whose names have no `_d512` (I-CE, CE, the two document-view runs and the two temperatures), each with a note naming the run that replaced it, and one fold of a VICReg arm that used the whole training set as its batch, which the paper dropped. The file has the format of a selection file, so `k10_eval.py cells` scores them when it is given the file. The dataset keeps the first packs as `wscan_gal_rev_rawprefix.npz`; the workers read them with `--raw-prefix` and then name the run without the tag. ## Changes - 2026-11-12, later that day. Nine checkpoints were added, the view-size sweep of the appendix's new Table A6: I-CE with each review view drawn until it holds at least 8, 32 or 64 sentences (`i2ce_icetf_fold_w_d512`, folds 0 to 2). `rank_ckpt_plan.json` lists them. The new table moved every later appendix table one place, so the old Tables A6 to A18 are Tables A7 to A19 in the tables above; no earlier entry changed its checkpoint. - 2026-11-12. The code repository gained `figures/neighborhoods/`, the scripts that drew Figures 8, 9 and A3. - 2026-11-11, at night. The code repository gained `fixed_split_eval.py`, which reads the fixed-split checkpoints as the paper does. - 2026-11-11, later that day. 62 checkpoints were added, so that every tower the paper reads is here: the DINO-style control (fold 0), eight fixed-split runs at their selected epochs (CE and I-CE trained at 2,048 sentences, the SimCLR-style tower, I + SimCLR, a+u @ views and the three VICReg weightings), the 52 `trajectory` checkpoints, and epoch 1,250 of the 4:1 gallery-aimed run of Section 6.2 (`ai4auni2_icetf_d512`). That run's entry in `fixed_split_ckpt_plan.json` named epoch 900, its selection before it was trained on to 2,000 epochs; the paper reads epoch 1,250. No other earlier entry changed its checkpoint. - 2026-11-11. The four JSON files above replace the earlier per-campaign lists (`docfix512_ckpt_plan.json`, `c2_docfix512_ckpt_plan.json`, `c3_docfix512_ckpt_plan.json`, `q8_gpool_ckpt_plan.json`); no selection changed. The five BYOL checkpoints the two-stage runs started from were added. The `_gpool` files were renamed from `_g4096`, a cap the review pool never reached; the files are the same. - 2026-11-07 to 2026-11-10. The `_d512` runs were added. ## Citation Zhilong Yu et al. *Invariant Game Representation Learning from Steam Reviews.* 2026 9th Artificial Intelligence and Cloud Computing Conference (AICCC 2026), December 18 to 21, 2026, Tokyo, Japan. ```bibtex @inproceedings{yu2026invariant, author = {Yu, Zhilong and others}, title = {Invariant Game Representation Learning from Steam Reviews}, booktitle = {2026 9th Artificial Intelligence and Cloud Computing Conference (AICCC 2026)}, address = {Tokyo, Japan}, year = {2026} } ``` Code and checkpoints are under the MIT license.