GulkoA commited on
Commit
a131386
·
verified ·
1 Parent(s): 499dc7f

Add the soft-max history benchmark row

Browse files
Files changed (1) hide show
  1. README.md +130 -124
README.md CHANGED
@@ -1,124 +1,130 @@
1
- ---
2
- license: apache-2.0
3
- language:
4
- - en
5
- library_name: sentence-transformers
6
- pipeline_tag: sentence-similarity
7
- base_model: sentence-transformers/all-MiniLM-L6-v2
8
- datasets:
9
- - FronkonGames/steam-games-dataset
10
- tags:
11
- - transformers.js
12
- - onnx
13
- - sentence-transformers
14
- - sentence-similarity
15
- - feature-extraction
16
- - recommendation
17
- - video-games
18
- - steam
19
- ---
20
-
21
- # gamespace.2
22
-
23
- gamespace.2 embeds video games into a 128-dimensional space where games that the same people enjoy
24
- sit close together. It powers the [GameMerge](https://huggingface.co/spaces/GulkoA/gamemerge)
25
- recommender. Output vectors are L2-normalised, so a dot product is the cosine similarity.
26
-
27
- It is [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) fine-tuned on
28
- Steam store data, like [gamespace.1](https://huggingface.co/GulkoA/gamespace.1). It also learns from
29
- player behaviour: pairs of games that the same Steam reviewers engaged with are positive examples.
30
- The model still reads only a game's store text, so it works for any game, including new ones with no
31
- players yet.
32
-
33
- ## Steam next-game benchmark
34
-
35
- The benchmark follows the protocol of SASRec (Kang & McAuley, ICDM 2018) on their Steam review crawl.
36
- Each user's most recent reviewed game is held out and ranked against 100 randomly sampled games, and
37
- also against the whole catalog. Text models score candidates by cosine similarity to the mean of the
38
- user's earlier games. Games no longer on the Steam store have no text to embed, so they were removed
39
- before the five-review filter. That leaves 250,955 users, 10,318 games and 3.05M interactions. The
40
- paper had 334,730 users and 13,047 games. Popularity and SASRec were re-run on this exact split.
41
-
42
- | Model | Learns from user histories | Hit@10, 100 negatives | NDCG@10, 100 negatives | Hit@10, all games | NDCG@10, all games |
43
- |---|---|---|---|---|---|
44
- | SASRec, reproduced | yes | 0.836 | 0.581 | 0.115 | 0.060 |
45
- | Popularity | yes | 0.688 | 0.420 | 0.050 | 0.025 |
46
- | **gamespace.2** | **pairs from training histories** | **0.511** | **0.301** | **0.034** | **0.018** |
47
- | gamespace.2 + CLIP header art | pairs from training histories | 0.520 | 0.309 | 0.037 | 0.019 |
48
- | gamespace.1 | no | 0.438 | 0.251 | 0.027 | 0.014 |
49
- | e5-large-v2 | no | 0.404 | 0.234 | 0.030 | 0.016 |
50
- | Qwen3-Embedding-0.6B | no | 0.336 | 0.185 | 0.018 | 0.010 |
51
- | bge-large-en-v1.5 | no | 0.334 | 0.191 | 0.026 | 0.014 |
52
- | all-MiniLM-L6-v2 | no | 0.306 | 0.162 | 0.017 | 0.010 |
53
- | CLIP header art only | no | 0.288 | 0.158 | 0.014 | 0.007 |
54
-
55
- For reference, the SASRec paper reports 0.873 and 0.631 on its full 13,047-game set.
56
-
57
- gamespace.2 is the best of the text embedding models, ahead of much larger general-purpose embedders.
58
- Sequential recommenders trained on the interaction data remain far stronger at predicting the next
59
- game, because they also learn popularity and order. gamespace.2 trades that for working on any game
60
- from its description alone.
61
-
62
- ## Training
63
-
64
- - **Data.** The [Steam Games Dataset](https://huggingface.co/datasets/FronkonGames/steam-games-dataset)
65
- has 147,352 apps. Soundtracks, demos, playtests, servers, SDKs, and empty records were removed,
66
- leaving 138,920 games.
67
- - **Objective.** Contrastive training uses `CachedMultipleNegativesRankingLoss` with 128 in-batch
68
- negatives. There are three kinds of positive pairs:
69
- - 70,000 pairs are two randomly degraded views of the same game.
70
- - 70,000 pairs are games whose vote-weighted user-tag profiles have a cosine of at least 0.6.
71
- - 66,904 pairs are co-liked games. There are 36,371 distinct pairs, each shown in both orders.
72
- They are mined from the benchmark's training histories only, so no validation or test game is
73
- used. A pair must be shared by at least 5 reviewers, it is scored by normalised PMI, and each
74
- game keeps its 10 strongest partners.
75
- - **Settings.** The projection is `Dense(384→128)`, initialised from PCA. Training runs one epoch at
76
- learning rate 3e-5, with 128-token documents.
77
-
78
- A looser pair set, with at least 3 shared reviewers and 20 partners per game, scored lower on
79
- validation, with 0.477 against 0.503 Hit@10. Pair quality matters more than quantity.
80
-
81
- ## Usage
82
-
83
- The model reads one short document per game. Build it with `docs.py` from this repo, using the same
84
- format as gamespace.1:
85
-
86
- ```
87
- Hollow Knight. Genres: Action, Adventure, Indie. Tags: Metroidvania, Souls-like, ... Modes: Single-player. Developer: Team Cherry. Forge your own path...
88
- ```
89
-
90
- ```python
91
- from sentence_transformers import SentenceTransformer
92
- model = SentenceTransformer("GulkoA/gamespace.2")
93
- vec = model.encode("Hollow Knight. Genres: Action, Adventure, Indie. Tags: Metroidvania, Souls-like.")
94
- ```
95
-
96
- ### In the browser
97
-
98
- `onnx/model_quantized.onnx` is 22 MB, per-channel int8, with cosine 0.995 to the PyTorch model. It
99
- runs with [transformers.js](https://huggingface.co/docs/transformers.js). Apply the 128x384 projection
100
- in `dense.bin`, which holds float32 weights in row-major order followed by the 128 biases, to the
101
- mean-pooled output:
102
-
103
- ```js
104
- import { pipeline } from "@huggingface/transformers";
105
-
106
- const extractor = await pipeline("feature-extraction", "GulkoA/gamespace.2", { dtype: "q8" });
107
- const w = new Float32Array(await (await fetch("https://huggingface.co/GulkoA/gamespace.2/resolve/main/dense.bin")).arrayBuffer());
108
- const { data: x } = await extractor("cozy farming game with fishing", { pooling: "mean", normalize: false });
109
- const y = Array.from({ length: 128 }, (_, i) => w[128 * 384 + i] + x.reduce((s, xj, j) => s + w[i * 384 + j] * xj, 0));
110
- const norm = Math.hypot(...y);
111
- const vec = y.map((v) => v / norm);
112
- ```
113
-
114
- ## Limitations
115
-
116
- - The behaviour pairs come from reviews written between 2010 and January 2018. Newer games benefit
117
- only through what the model generalises from text.
118
- - Steam display names identify reviewers, and different people can share a name.
119
- - Similarity ignores popularity and quality, so obscure games rank alongside well-known ones.
120
-
121
- ## License
122
-
123
- The model is licensed under Apache 2.0, the same as the base model. The store data is MIT-licensed.
124
- The review crawl is from the UCSD Steam datasets (Kang & McAuley, 2018).
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ library_name: sentence-transformers
6
+ pipeline_tag: sentence-similarity
7
+ base_model: sentence-transformers/all-MiniLM-L6-v2
8
+ datasets:
9
+ - FronkonGames/steam-games-dataset
10
+ tags:
11
+ - transformers.js
12
+ - onnx
13
+ - sentence-transformers
14
+ - sentence-similarity
15
+ - feature-extraction
16
+ - recommendation
17
+ - video-games
18
+ - steam
19
+ ---
20
+
21
+ # gamespace.2
22
+
23
+ gamespace.2 embeds video games into a 128-dimensional space where games that the same people enjoy
24
+ sit close together. It powers the [GameMerge](https://huggingface.co/spaces/GulkoA/gamemerge)
25
+ recommender. Output vectors are L2-normalised, so a dot product is the cosine similarity.
26
+
27
+ It is [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) fine-tuned on
28
+ Steam store data, like [gamespace.1](https://huggingface.co/GulkoA/gamespace.1). It also learns from
29
+ player behaviour: pairs of games that the same Steam reviewers engaged with are positive examples.
30
+ The model still reads only a game's store text, so it works for any game, including new ones with no
31
+ players yet.
32
+
33
+ ## Steam next-game benchmark
34
+
35
+ The benchmark follows the protocol of SASRec (Kang & McAuley, ICDM 2018) on their Steam review crawl.
36
+ Each user's most recent reviewed game is held out and ranked against 100 randomly sampled games, and
37
+ also against the whole catalog. Text models score candidates by cosine similarity to the mean of the
38
+ user's earlier games. Games no longer on the Steam store have no text to embed, so they were removed
39
+ before the five-review filter. That leaves 250,955 users, 10,318 games and 3.05M interactions. The
40
+ paper had 334,730 users and 13,047 games. Popularity and SASRec were re-run on this exact split.
41
+
42
+ | Model | Learns from user histories | Hit@10, 100 negatives | NDCG@10, 100 negatives | Hit@10, all games | NDCG@10, all games |
43
+ |---|---|---|---|---|---|
44
+ | SASRec, reproduced | yes | 0.836 | 0.581 | 0.115 | 0.060 |
45
+ | Popularity | yes | 0.688 | 0.420 | 0.050 | 0.025 |
46
+ | **gamespace.2, soft-max over history** | **pairs from training histories** | **0.518** | **0.308** | **0.046** | **0.025** |
47
+ | gamespace.2, mean of history | pairs from training histories | 0.511 | 0.301 | 0.034 | 0.018 |
48
+ | gamespace.2 + CLIP header art | pairs from training histories | 0.520 | 0.309 | 0.037 | 0.019 |
49
+ | gamespace.1 | no | 0.438 | 0.251 | 0.027 | 0.014 |
50
+ | e5-large-v2 | no | 0.404 | 0.234 | 0.030 | 0.016 |
51
+ | Qwen3-Embedding-0.6B | no | 0.336 | 0.185 | 0.018 | 0.010 |
52
+ | bge-large-en-v1.5 | no | 0.334 | 0.191 | 0.026 | 0.014 |
53
+ | all-MiniLM-L6-v2 | no | 0.306 | 0.162 | 0.017 | 0.010 |
54
+ | CLIP header art only | no | 0.288 | 0.158 | 0.014 | 0.007 |
55
+
56
+ For reference, the SASRec paper reports 0.873 and 0.631 on its full 13,047-game set.
57
+
58
+ The soft-max row scores each candidate by a soft maximum (log-mean-exp, temperature 0.1, chosen on
59
+ the validation split) of its cosine similarity to each earlier game, instead of similarity to their
60
+ mean. Mixed histories are not pulled toward one average point, which matters most when ranking the
61
+ whole catalog. GameMerge uses it to combine your picks.
62
+
63
+ gamespace.2 is the best of the text embedding models, ahead of much larger general-purpose embedders.
64
+ Sequential recommenders trained on the interaction data remain far stronger at predicting the next
65
+ game, because they also learn popularity and order. gamespace.2 trades that for working on any game
66
+ from its description alone.
67
+
68
+ ## Training
69
+
70
+ - **Data.** The [Steam Games Dataset](https://huggingface.co/datasets/FronkonGames/steam-games-dataset)
71
+ has 147,352 apps. Soundtracks, demos, playtests, servers, SDKs, and empty records were removed,
72
+ leaving 138,920 games.
73
+ - **Objective.** Contrastive training uses `CachedMultipleNegativesRankingLoss` with 128 in-batch
74
+ negatives. There are three kinds of positive pairs:
75
+ - 70,000 pairs are two randomly degraded views of the same game.
76
+ - 70,000 pairs are games whose vote-weighted user-tag profiles have a cosine of at least 0.6.
77
+ - 66,904 pairs are co-liked games. There are 36,371 distinct pairs, each shown in both orders.
78
+ They are mined from the benchmark's training histories only, so no validation or test game is
79
+ used. A pair must be shared by at least 5 reviewers, it is scored by normalised PMI, and each
80
+ game keeps its 10 strongest partners.
81
+ - **Settings.** The projection is `Dense(384→128)`, initialised from PCA. Training runs one epoch at
82
+ learning rate 3e-5, with 128-token documents.
83
+
84
+ A looser pair set, with at least 3 shared reviewers and 20 partners per game, scored lower on
85
+ validation, with 0.477 against 0.503 Hit@10. Pair quality matters more than quantity.
86
+
87
+ ## Usage
88
+
89
+ The model reads one short document per game. Build it with `docs.py` from this repo, using the same
90
+ format as gamespace.1:
91
+
92
+ ```
93
+ Hollow Knight. Genres: Action, Adventure, Indie. Tags: Metroidvania, Souls-like, ... Modes: Single-player. Developer: Team Cherry. Forge your own path...
94
+ ```
95
+
96
+ ```python
97
+ from sentence_transformers import SentenceTransformer
98
+ model = SentenceTransformer("GulkoA/gamespace.2")
99
+ vec = model.encode("Hollow Knight. Genres: Action, Adventure, Indie. Tags: Metroidvania, Souls-like.")
100
+ ```
101
+
102
+ ### In the browser
103
+
104
+ `onnx/model_quantized.onnx` is 22 MB, per-channel int8, with cosine 0.995 to the PyTorch model. It
105
+ runs with [transformers.js](https://huggingface.co/docs/transformers.js). Apply the 128x384 projection
106
+ in `dense.bin`, which holds float32 weights in row-major order followed by the 128 biases, to the
107
+ mean-pooled output:
108
+
109
+ ```js
110
+ import { pipeline } from "@huggingface/transformers";
111
+
112
+ const extractor = await pipeline("feature-extraction", "GulkoA/gamespace.2", { dtype: "q8" });
113
+ const w = new Float32Array(await (await fetch("https://huggingface.co/GulkoA/gamespace.2/resolve/main/dense.bin")).arrayBuffer());
114
+ const { data: x } = await extractor("cozy farming game with fishing", { pooling: "mean", normalize: false });
115
+ const y = Array.from({ length: 128 }, (_, i) => w[128 * 384 + i] + x.reduce((s, xj, j) => s + w[i * 384 + j] * xj, 0));
116
+ const norm = Math.hypot(...y);
117
+ const vec = y.map((v) => v / norm);
118
+ ```
119
+
120
+ ## Limitations
121
+
122
+ - The behaviour pairs come from reviews written between 2010 and January 2018. Newer games benefit
123
+ only through what the model generalises from text.
124
+ - Steam display names identify reviewers, and different people can share a name.
125
+ - Similarity ignores popularity and quality, so obscure games rank alongside well-known ones.
126
+
127
+ ## License
128
+
129
+ The model is licensed under Apache 2.0, the same as the base model. The store data is MIT-licensed.
130
+ The review crawl is from the UCSD Steam datasets (Kang & McAuley, 2018).