Upload 10 files
Browse files- README.md +31 -0
- RELEASE_README.md +33 -0
- c_big.config.json +32 -0
- c_big_free.config.json +31 -0
- chart_tokens.py +67 -0
- columnist-base.pt +3 -0
- columnist-embed.pt +3 -0
- columnist.py +268 -0
- embed_infer.py +34 -0
- prep_dataset.py +115 -0
README.md
CHANGED
|
@@ -1,3 +1,34 @@
|
|
| 1 |
---
|
| 2 |
license: cc-by-4.0
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: cc-by-4.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags: [osu, osu-mania, rhythm-game, foundation-model]
|
| 5 |
---
|
| 6 |
+
# Columnist
|
| 7 |
+
|
| 8 |
+
A foundation model for osu!mania 4K charts. Pretrained without labels on unlabelled charts, then adapted to specific tasks.
|
| 9 |
+
|
| 10 |
+
## Variants
|
| 11 |
+
| File | Use |
|
| 12 |
+
|---|---|
|
| 13 |
+
| columnist-base | Pretrained encoder (256d, 6 layers) |
|
| 14 |
+
| columnist-embed | Label-free encoder behind the similar-charts search |
|
| 15 |
+
| columnist-joint | Fine-tuned: 31 pattern tags and categories, overall difficulty (stars ≈ 3.886 + 0.140 × score), per-window and per-row difficulty |
|
| 16 |
+
| columnist-lens | Per-note pattern readout used by the player page |
|
| 17 |
+
|
| 18 |
+
## Training
|
| 19 |
+
- Pretraining data: 62,161 unlabelled 4K charts (59,361 unique); chart ids only are released.
|
| 20 |
+
- Fine-tuning labels: community sheet labels, osu! user tags and tournament mappool slots; 8,453 labelled charts.
|
| 21 |
+
- Inputs use millisecond timing and visible scroll only. No snap or BPM-grid information.
|
| 22 |
+
|
| 23 |
+
## Results (5-fold, mean AP, categories / tags)
|
| 24 |
+
Fine-tuned from the base Columnist (5-fold, song-grouped, mean AP over 11 categories / 20 tags):
|
| 25 |
+
- columnist-base fine-tuned for tags and categories: 0.703 / 0.581
|
| 26 |
+
- columnist-joint (tags, categories and difficulty): 0.698 / 0.578
|
| 27 |
+
- Difficulty: Spearman 0.94 against official star rating, mean absolute error 0.33 stars after linear calibration; held-out tournament round ordering 0.95 accuracy.
|
| 28 |
+
The earlier boosted-statistics pipeline scores higher on categories (about 0.75 / 0.60); Columnist trades some of that for one shared network that also serves similarity, difficulty and per-note readout.
|
| 29 |
+
|
| 30 |
+
## Limitations
|
| 31 |
+
4K only; at least 64 rows per chart; long-note patterns are the weakest area; labels come from user tags and mappool slots, not ground truth. Research tool, not an official osu! tool, and not meant for Beatmap Nominator decisions on its own.
|
| 32 |
+
|
| 33 |
+
## Licence and data
|
| 34 |
+
Weights, labels and this card: CC BY 4.0. Code: Apache-2.0. Charts remain the property of their mappers and are not redistributed. Use of osu! data complies with fair research use.
|
RELEASE_README.md
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Columnist pretrained encoders (release folder)
|
| 2 |
+
|
| 3 |
+
Two pretraining checkpoints of the Columnist encoder for 4K osu!mania charts. Both are d=256, 6 layers, 6.28M parameters, 20,000 pretraining steps, no human labels.
|
| 4 |
+
No snap, beat or BPM-grid information is used anywhere.
|
| 5 |
+
|
| 6 |
+
| file | what it is |
|
| 7 |
+
|---|---|
|
| 8 |
+
| `c_big.pt` | pretrained with masked rows + contrastive (sibling difficulties) + next row **+ a `feats` regression target** (a 16-number PCA of hand-built whole-chart features; no human labels). **This is the base of the 0.703 / 0.581 fine-tuned results** (teacher + relaxed labels + length feature). Frozen linear probe, categories / tags AP at 100% labels: 0.662 / 0.534. Retrieval recall@5 0.642 |
|
| 9 |
+
| `c_big_free.pt` | pretrained with masked rows + contrastive + next row only: **purely label-free** (no `feats`, no teacher, no labels). Frozen probe 0.592 / 0.486; retrieval recall@5 **0.744** (the best retrieval of all models) |
|
| 10 |
+
| `*.config.json` | architecture and pretraining settings of each checkpoint |
|
| 11 |
+
| `columnist.py` | the model code (needs torch, numpy); both checkpoints load with it |
|
| 12 |
+
| `embed_infer.py` | `load_encoder(path)` and `embed_chart(enc, tok, gap, sv)` -> a 256-number chart embedding |
|
| 13 |
+
| `chart_tokens.py`, `prep_dataset.py` | a 4K `.osu` file -> the arrays: `parse()` then `chart_rows()`, see below |
|
| 14 |
+
|
| 15 |
+
**Which one for what:** I do not know which checkpoint the Lens `/similar` feature uses; I could only identify these two by their training recipes. For **similarity / retrieval of charts** (finding sibling difficulties of the same song) `c_big_free.pt` is
|
| 16 |
+
clearly the better of the two (recall@5 0.744 against 0.642, MRR 0.583 against 0.513). For a **starting point for fine-tuning on labels** use `c_big.pt` (+0.07 categories AP frozen and about +0.014 after fine-tuning with the teacher).
|
| 17 |
+
|
| 18 |
+
## Loading and using
|
| 19 |
+
```python
|
| 20 |
+
from embed_infer import load_encoder, embed_chart # columnist.py must be importable
|
| 21 |
+
enc = load_encoder("c_big_free.pt") # strict=False: the checkpoint may hold extra optional heads (c_big has a 'feat_head')
|
| 22 |
+
vec = embed_chart(enc, tok, gap, sv) # (256,) numpy; cosine similarity of these vectors = chart similarity
|
| 23 |
+
```
|
| 24 |
+
`embed_chart` equals the repository's `columnist.embed_charts` exactly (checked on 2 charts for each checkpoint: difference 0.0). The embedding is the mean of the encoder outputs over the valid rows of each non-overlapping 512-row window, averaged over the windows weighted by their length.
|
| 25 |
+
For retrieval the evaluation standardised the vectors over the candidate set before taking cosine similarity.
|
| 26 |
+
|
| 27 |
+
## Input arrays of one chart
|
| 28 |
+
* `tok` (L,) int: one value per row (a moment where a note starts or a long note ends; events within 2 ms are merged), 2 bits per column (column c at bits 2c..2c+1): 0 none, 1 tap head, 2 long-note head, 3 long-note tail.
|
| 29 |
+
* `gap` (L,) float: milliseconds since the previous row (0 for the first), clipped to [0, 60000]. `sv` (L,) float: scroll-speed multiplier at the row (1.0 = no change).
|
| 30 |
+
* From a `.osu` file: `mode, keys, notes, sv = chart_tokens.parse(Path(osu))`, then `tok, gap, svrow, stats = prep_dataset.chart_rows(notes, sv)` (use only 4K mania charts with at least 64 rows). Vocabulary: 256 tokens (the 4 columns x 4 states), plus a mask id 256 used in pretraining only.
|
| 31 |
+
* Windows of 512 rows; inference does not mirror the chart (left-right mirroring was a random 50% training augmentation); the model has a local attention window of +-64 rows with a per-head time-decay bias, so it never looks beyond about 64 rows in any layer.
|
| 32 |
+
|
| 33 |
+
No keys, datasets or label files are in this folder.
|
c_big.config.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"file": "c_big.pt",
|
| 3 |
+
"description": "pretrained with masked rows + contrastive (sibling difficulties) + next row + a feats regression target (PCA of hand-built whole-chart features, no human labels); the base of the 0.703 / 0.581 fine-tuned results",
|
| 4 |
+
"architecture": {
|
| 5 |
+
"class": "columnist.Columnist",
|
| 6 |
+
"d": 256,
|
| 7 |
+
"layers": 6,
|
| 8 |
+
"heads": 4,
|
| 9 |
+
"look": false,
|
| 10 |
+
"v3_inputs": false,
|
| 11 |
+
"token_vocabulary": 256,
|
| 12 |
+
"attention": "local +-64 rows with time-decay bias",
|
| 13 |
+
"window_rows": 512,
|
| 14 |
+
"parameters_million": 6.28
|
| 15 |
+
},
|
| 16 |
+
"pretraining": {
|
| 17 |
+
"objectives": "masked,contrastive,next,feats",
|
| 18 |
+
"steps": 20000,
|
| 19 |
+
"batch": 24,
|
| 20 |
+
"lr": 0.0003,
|
| 21 |
+
"mask_p": 0.06,
|
| 22 |
+
"seed": 0,
|
| 23 |
+
"target_file": "data/foundation/feats16.npy"
|
| 24 |
+
},
|
| 25 |
+
"checkpoint_keys": [
|
| 26 |
+
"model",
|
| 27 |
+
"args",
|
| 28 |
+
"step",
|
| 29 |
+
"feat_head"
|
| 30 |
+
],
|
| 31 |
+
"load": "m = columnist.Columnist(d, layers, look=False, v3=False); m.load_state_dict(ck['model'], strict=False)"
|
| 32 |
+
}
|
c_big_free.config.json
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"file": "c_big_free.pt",
|
| 3 |
+
"description": "pretrained with masked rows + contrastive + next row only (no feats target, no labels of any kind); the purely label-free encoder",
|
| 4 |
+
"architecture": {
|
| 5 |
+
"class": "columnist.Columnist",
|
| 6 |
+
"d": 256,
|
| 7 |
+
"layers": 6,
|
| 8 |
+
"heads": 4,
|
| 9 |
+
"look": false,
|
| 10 |
+
"v3_inputs": false,
|
| 11 |
+
"token_vocabulary": 256,
|
| 12 |
+
"attention": "local +-64 rows with time-decay bias",
|
| 13 |
+
"window_rows": 512,
|
| 14 |
+
"parameters_million": 6.28
|
| 15 |
+
},
|
| 16 |
+
"pretraining": {
|
| 17 |
+
"objectives": "masked,contrastive,next",
|
| 18 |
+
"steps": 20000,
|
| 19 |
+
"batch": 24,
|
| 20 |
+
"lr": 0.0003,
|
| 21 |
+
"mask_p": 0.06,
|
| 22 |
+
"seed": 0,
|
| 23 |
+
"target_file": null
|
| 24 |
+
},
|
| 25 |
+
"checkpoint_keys": [
|
| 26 |
+
"model",
|
| 27 |
+
"args",
|
| 28 |
+
"step"
|
| 29 |
+
],
|
| 30 |
+
"load": "m = columnist.Columnist(d, layers, look=False, v3=False); m.load_state_dict(ck['model'], strict=False)"
|
| 31 |
+
}
|
chart_tokens.py
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Read 4K .osu files into a compact, snap-free store and de-duplicate them.
|
| 2 |
+
|
| 3 |
+
python scripts/foundation/chart_tokens.py data/foundation/osu data/foundation/charts.jsonl.gz
|
| 4 |
+
|
| 5 |
+
Per chart we keep only millisecond note times (column, start, hold end) and visible scroll-speed changes
|
| 6 |
+
(red-line BPM and snap data are dropped on purpose). `fingerprint` hashes the column sequence of all notes
|
| 7 |
+
and hold flags, so rate edits and copies share it; `dup_of` names the first chart with the same fingerprint.
|
| 8 |
+
"""
|
| 9 |
+
import gzip
|
| 10 |
+
import hashlib
|
| 11 |
+
import json
|
| 12 |
+
import sys
|
| 13 |
+
from pathlib import Path
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
def parse(path: Path):
|
| 17 |
+
sect, notes, sv, keys, meta = "", [], [], None, {}
|
| 18 |
+
raw = path.read_bytes()
|
| 19 |
+
text = raw.decode("utf-16", errors="replace") if raw[:2] in (b"\xff\xfe", b"\xfe\xff") \
|
| 20 |
+
else raw.decode("utf-8-sig", errors="replace")
|
| 21 |
+
for line in text.splitlines():
|
| 22 |
+
line = line.strip()
|
| 23 |
+
if line.startswith("["):
|
| 24 |
+
sect = line
|
| 25 |
+
continue
|
| 26 |
+
if sect == "[General]" and line.startswith("Mode:"):
|
| 27 |
+
meta["mode"] = int(line.split(":")[1])
|
| 28 |
+
elif sect == "[Difficulty]" and line.startswith("CircleSize:"):
|
| 29 |
+
keys = int(float(line.split(":")[1]))
|
| 30 |
+
elif sect == "[TimingPoints]" and line:
|
| 31 |
+
p = line.split(",")
|
| 32 |
+
if len(p) > 6 and p[6] == "0": # green line: visible scroll speed
|
| 33 |
+
sv.append((round(float(p[0])), round(-100 / float(p[1]), 3)))
|
| 34 |
+
elif sect == "[HitObjects]" and line:
|
| 35 |
+
p = line.split(",")
|
| 36 |
+
t, typ = round(float(p[2])), int(p[3])
|
| 37 |
+
col = min(3, int(float(p[0]) * 4 / 512))
|
| 38 |
+
end = int(float(p[5].split(":")[0])) if typ & 128 else None
|
| 39 |
+
notes.append((col, t, end))
|
| 40 |
+
return meta.get("mode"), keys, notes, sv
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def main(indir: str, out: str):
|
| 44 |
+
seen, n, skipped = {}, 0, []
|
| 45 |
+
with gzip.open(out, "wt") as f:
|
| 46 |
+
for p in sorted(Path(indir).glob("*.osu")):
|
| 47 |
+
try:
|
| 48 |
+
mode, keys, notes, sv = parse(p)
|
| 49 |
+
except ValueError:
|
| 50 |
+
skipped.append(p.stem) # e.g. a NaN hit-object time
|
| 51 |
+
continue
|
| 52 |
+
if mode != 3 or keys != 4 or len(notes) < 100:
|
| 53 |
+
continue
|
| 54 |
+
notes.sort(key=lambda x: (x[1], x[0]))
|
| 55 |
+
if notes[-1][1] - notes[0][1] < 20000:
|
| 56 |
+
continue
|
| 57 |
+
fp = hashlib.md5(bytes(c + (4 if e else 0) for c, _, e in notes)).hexdigest()
|
| 58 |
+
bid = int(p.stem)
|
| 59 |
+
dup = seen.setdefault(fp, bid)
|
| 60 |
+
f.write(json.dumps({"beatmap_id": bid, "fingerprint": fp, "dup_of": None if dup == bid else dup,
|
| 61 |
+
"notes": notes, "sv": sv}) + "\n")
|
| 62 |
+
n += 1
|
| 63 |
+
print(n, "charts,", len(seen), "unique fingerprints;", len(skipped), "unparseable:", skipped[:20])
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
if __name__ == "__main__":
|
| 67 |
+
main(sys.argv[1], sys.argv[2])
|
columnist-base.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4525eac7a2e60078dcfdcbe65c9518cc626af79706baafb39fafc9142a5e5cb3
|
| 3 |
+
size 25152249
|
columnist-embed.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b3ae20e5eaa11f486d8d6dffa8e4d12733ef42074e43ebf262f173e1faec9ccb
|
| 3 |
+
size 25135301
|
columnist.py
ADDED
|
@@ -0,0 +1,268 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Columnist: a small sequence model for 4K osu!mania charts (data helpers + network).
|
| 2 |
+
|
| 3 |
+
A chart is a sequence of ROWS (notes that fall at the same millisecond). Each row has a token for the state of
|
| 4 |
+
the four columns (none / tap / hold head / hold tail -> 4**4 = 256 tokens), the gap in ms to the previous
|
| 5 |
+
row, and the visible scroll speed. No BPM, snap or grid information is used anywhere.
|
| 6 |
+
|
| 7 |
+
Network: row embedding + numeric features -> causal conv stem -> transformer layers with local attention
|
| 8 |
+
(window of rows) and a per-head time penalty (rows far apart in seconds attend less). The same network can run
|
| 9 |
+
bidirectionally (masked-row and contrastive objectives) or causally (next-row objective, per-note use).
|
| 10 |
+
"""
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
import math
|
| 14 |
+
|
| 15 |
+
import os
|
| 16 |
+
|
| 17 |
+
import numpy as np
|
| 18 |
+
import torch
|
| 19 |
+
import torch.nn.functional as F
|
| 20 |
+
from torch import nn
|
| 21 |
+
|
| 22 |
+
N_TOK = 256
|
| 23 |
+
MASK_ID = 256
|
| 24 |
+
N_GAPB = 24
|
| 25 |
+
WINDOW = 64 # rows to each side
|
| 26 |
+
L_CROP = 512
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def gap_bucket(gap_ms: torch.Tensor) -> torch.Tensor:
|
| 30 |
+
return (torch.log2(gap_ms.float() + 1) * 2).floor().clamp(0, N_GAPB - 1).long()
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def mirror_lut() -> torch.Tensor:
|
| 34 |
+
"""token -> token with the four columns reversed. The table covers tokens with the optional held bits (bits 16..19, one per
|
| 35 |
+
column, set by Store when extras are on) and MASK_ID, so lut[tok] works for every kind of token."""
|
| 36 |
+
v = np.arange(1 << 20, dtype=np.int64)
|
| 37 |
+
base, held = v & 0xFFFF, v >> 16
|
| 38 |
+
mb = np.where(base < N_TOK, sum((((base >> (2 * c)) & 3) << (2 * (3 - c))) for c in range(4)), base)
|
| 39 |
+
mh = sum((((held >> c) & 1) << (3 - c)) for c in range(4))
|
| 40 |
+
return torch.from_numpy((mb | (mh << 16)).astype(np.int64))
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
N_LOOK = 6
|
| 44 |
+
VIS_H = (400.0, 800.0, 1600.0) # scroll distance (ms of travel at multiplier 1) that counts as "on screen"
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def look_features(g_ms, sv, valid, causal):
|
| 48 |
+
"""Extra inputs (B,L,6): time to the next row, its log ratio to the previous gap, scroll distance since the previous
|
| 49 |
+
row, and how many following rows are within VIS_H of scroll travel (a proxy for what is on screen). Everything is
|
| 50 |
+
computed from row times and the scroll multiplier; no snap, beat or BPM-grid information. The columns that look ahead
|
| 51 |
+
(0, 1, 3, 4, 5) are zeroed when causal=True so the next-row objective cannot see the answer. g_ms must already be
|
| 52 |
+
zero for masked rows (so a masked row's gap never leaks through its neighbours)."""
|
| 53 |
+
B, L = g_ms.shape
|
| 54 |
+
nxt = torch.cat([g_ms[:, 1:], torch.zeros_like(g_ms[:, :1])], 1) * torch.cat([valid[:, 1:], torch.zeros_like(valid[:, :1])], 1)
|
| 55 |
+
f_next = torch.log1p(nxt) / 6.9
|
| 56 |
+
ratio = (torch.log1p(nxt) - torch.log1p(g_ms)).clamp(-6, 6) / 3
|
| 57 |
+
svc = torch.nan_to_num(sv.float(), nan=1.0, posinf=10.0, neginf=0.1).clamp(0.1, 10.0)
|
| 58 |
+
dx = g_ms * svc * valid
|
| 59 |
+
xv = dx.cumsum(1)
|
| 60 |
+
idx = torch.arange(L, device=g_ms.device)
|
| 61 |
+
x = torch.where(valid, xv, xv.max(1, keepdim=True).values + 1e6 * (1 + idx)[None])
|
| 62 |
+
nval = valid.sum(1, keepdim=True)
|
| 63 |
+
cols = [f_next, ratio, torch.log1p(dx) / 9]
|
| 64 |
+
vis = []
|
| 65 |
+
for H in VIS_H:
|
| 66 |
+
j = torch.searchsorted(x.contiguous(), (x + H).contiguous(), right=True).minimum(nval)
|
| 67 |
+
vis.append(torch.log1p(((j - idx[None] - 1).clamp(min=0) * valid).float()) / 4)
|
| 68 |
+
out = torch.stack(cols + vis, -1)
|
| 69 |
+
if causal:
|
| 70 |
+
keep = torch.tensor([0, 0, 1, 0, 0, 0], dtype=out.dtype, device=out.device)
|
| 71 |
+
out = out * keep
|
| 72 |
+
return out
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
N_V3 = 7
|
| 76 |
+
V3_H = 600.0 # ms of scroll travel that counts as "on screen" (as osu_patterns.deep.tokens.H_MS)
|
| 77 |
+
V3_W = 64 # rows looked back when measuring the speed swing and the number of speed changes on screen
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def v3_features(g_ms, sv, valid, held):
|
| 81 |
+
"""The inputs of the old v3 network that Columnist lacks (no snap, beat or BPM-grid information): the explicit per-column
|
| 82 |
+
'held' state (a hold passes through this row; computed from the whole chart by make_held.py, bits 16..19 of the token),
|
| 83 |
+
how long the row has been on screen, and the speed swing and the number of speed changes among the rows on screen.
|
| 84 |
+
g_ms is zero for masked rows. Returns (B,L,7)."""
|
| 85 |
+
B, L = g_ms.shape
|
| 86 |
+
svc = torch.nan_to_num(sv.float(), nan=1.0, posinf=10.0, neginf=0.1).clamp(0.1, 10.0)
|
| 87 |
+
lsp = torch.log(svc)
|
| 88 |
+
x = (g_ms * svc * valid).cumsum(1)
|
| 89 |
+
t = (g_ms * valid).cumsum(1)
|
| 90 |
+
ar = torch.arange(L, device=g_ms.device)
|
| 91 |
+
j0 = torch.searchsorted(x.contiguous(), (x - V3_H).contiguous(), right=False)
|
| 92 |
+
j0 = torch.minimum(j0, ar[None].expand(B, L))
|
| 93 |
+
vis = torch.log(((t - t.gather(1, j0)) / V3_H).clamp(0.05, 20.0))
|
| 94 |
+
k = torch.arange(V3_W, device=g_ms.device)
|
| 95 |
+
idx = ar[:, None] - k[None, :] # (L,W) rows i-k
|
| 96 |
+
ok = (idx >= 0)[None] & (idx[None] >= j0[:, :, None]) # rows still on screen
|
| 97 |
+
ls = lsp[:, idx.clamp(min=0)] # (B,L,W)
|
| 98 |
+
mx = ls.masked_fill(~ok, -1e4).max(-1).values
|
| 99 |
+
mn = ls.masked_fill(~ok, 1e4).min(-1).values
|
| 100 |
+
swing = (mx - mn).clamp(0, 6) / 2
|
| 101 |
+
chg = (((ls[..., :-1] - ls[..., 1:]).abs() > 0.02) & ok[..., 1:]).sum(-1).float()
|
| 102 |
+
held_f = torch.stack([((held >> c) & 1).float() for c in range(4)], -1)
|
| 103 |
+
return torch.cat([held_f, torch.stack([vis, swing, torch.log1p(chg) / 3], -1)], -1)
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
def row_features(tok, gap, sv, valid, look=False, causal=False, v3=False):
|
| 107 |
+
"""tok (B,L) long (MASK_ID allowed), gap (B,L) ms, sv (B,L) scroll multiplier -> (B,L,14 or 14+N_LOOK) floats, times (B,L) s."""
|
| 108 |
+
held_bits = (tok >> 16) & 15
|
| 109 |
+
tok = tok & 0xFFFF
|
| 110 |
+
masked = tok == MASK_ID
|
| 111 |
+
t = tok.clamp(max=N_TOK - 1)
|
| 112 |
+
states = torch.stack([(t >> (2 * c)) & 3 for c in range(4)], -1) # (B,L,4)
|
| 113 |
+
states = torch.where(masked[..., None], torch.zeros_like(states), states)
|
| 114 |
+
delta = (states == 2).long() - (states == 3).long()
|
| 115 |
+
held_before = (delta.cumsum(1) - delta).clamp(min=0).float().clamp(max=1)
|
| 116 |
+
gap = torch.nan_to_num(gap.float(), nan=0.0, posinf=60000.0)
|
| 117 |
+
g = torch.log1p(gap.clamp(min=0)) / 6.9
|
| 118 |
+
g = torch.where(masked, torch.zeros_like(g), g)
|
| 119 |
+
four = [f(g * (2.0 ** k) * math.pi) for k in range(4) for f in (torch.sin, torch.cos)]
|
| 120 |
+
svf = torch.log(torch.nan_to_num(sv.float(), nan=1.0, posinf=10.0, neginf=0.1).clamp(0.1, 10.0))
|
| 121 |
+
feats = torch.cat([g[..., None], torch.stack(four, -1), svf[..., None], held_before], -1)
|
| 122 |
+
if look:
|
| 123 |
+
gm = torch.where(masked, torch.zeros_like(gap), gap.clamp(min=0))
|
| 124 |
+
feats = torch.cat([feats, look_features(gm, sv, valid, causal)], -1)
|
| 125 |
+
if v3:
|
| 126 |
+
gm3 = torch.where(masked, torch.zeros_like(gap), gap.clamp(min=0))
|
| 127 |
+
feats = torch.cat([feats, v3_features(gm3, sv, valid, held_bits)], -1)
|
| 128 |
+
time = (gap.float().clamp(min=0) * (~masked) * valid).cumsum(1) / 1000.0
|
| 129 |
+
return feats * valid[..., None], time
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
class Block(nn.Module):
|
| 133 |
+
def __init__(self, d, heads, p):
|
| 134 |
+
super().__init__()
|
| 135 |
+
self.h = heads
|
| 136 |
+
self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
|
| 137 |
+
self.qkv = nn.Linear(d, 3 * d)
|
| 138 |
+
self.o = nn.Linear(d, d)
|
| 139 |
+
self.ff = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
|
| 140 |
+
self.drop = nn.Dropout(p)
|
| 141 |
+
|
| 142 |
+
def forward(self, x, bias):
|
| 143 |
+
B, L, D = x.shape
|
| 144 |
+
q, k, v = self.qkv(self.n1(x)).view(B, L, 3, self.h, D // self.h).permute(2, 0, 3, 1, 4)
|
| 145 |
+
a = F.scaled_dot_product_attention(q, k, v, attn_mask=bias.to(q.dtype))
|
| 146 |
+
x = x + self.drop(self.o(a.transpose(1, 2).reshape(B, L, D)))
|
| 147 |
+
return x + self.drop(self.ff(self.n2(x)))
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
class Columnist(nn.Module):
|
| 151 |
+
def __init__(self, d=128, layers=4, heads=4, p=0.1, look=False, v3=False):
|
| 152 |
+
super().__init__()
|
| 153 |
+
self.d, self.heads, self.look, self.v3 = d, heads, look, v3
|
| 154 |
+
self.tok = nn.Embedding(N_TOK + 1, d)
|
| 155 |
+
self.num = nn.Linear(14 + (N_LOOK if look else 0) + (N_V3 if v3 else 0), d)
|
| 156 |
+
self.stem = nn.ModuleList([nn.Conv1d(d, 2 * d, 5) for _ in range(2)]) # left-padded: causal
|
| 157 |
+
self.blocks = nn.ModuleList(Block(d, heads, p) for _ in range(layers))
|
| 158 |
+
self.norm = nn.LayerNorm(d)
|
| 159 |
+
self.register_buffer("slopes", torch.tensor([0.05 * 2 ** (i * 5 / max(1, heads - 1)) for i in range(heads)]))
|
| 160 |
+
self.tok_head = nn.Linear(d, N_TOK)
|
| 161 |
+
self.gap_head = nn.Linear(d, N_GAPB)
|
| 162 |
+
self.proj = nn.Sequential(nn.Linear(d, d), nn.GELU(), nn.Linear(d, 64))
|
| 163 |
+
self.rank_head = nn.Linear(d, 1) # difficulty-order objectives (ladder, rounds)
|
| 164 |
+
self.order_head = nn.Linear(2 * d, 1) # which of two windows comes first
|
| 165 |
+
self.stat_head = nn.Linear(d, 4) # window statistics
|
| 166 |
+
self.rate_head = nn.Linear(2 * d, 1) # log rate ratio between two views of one window
|
| 167 |
+
|
| 168 |
+
def attn_bias(self, time, valid, causal):
|
| 169 |
+
B, L = valid.shape
|
| 170 |
+
idx = torch.arange(L, device=valid.device)
|
| 171 |
+
rel = idx[None, :] - idx[:, None] # j - i
|
| 172 |
+
allowed = rel.abs() <= WINDOW
|
| 173 |
+
if causal:
|
| 174 |
+
allowed = allowed & (rel <= 0)
|
| 175 |
+
allowed = allowed[None] & valid[:, None, :]
|
| 176 |
+
allowed = allowed | torch.eye(L, dtype=torch.bool, device=valid.device)[None]
|
| 177 |
+
dt = (time[:, None, :] - time[:, :, None]).abs() # (B,L,L) seconds
|
| 178 |
+
bias = -self.slopes[None, :, None, None] * dt[:, None]
|
| 179 |
+
return bias.masked_fill(~allowed[:, None], float("-inf"))
|
| 180 |
+
|
| 181 |
+
def forward(self, tok, gap, sv, valid, causal=False):
|
| 182 |
+
feats, time = row_features(tok, gap, sv, valid, self.look, causal, self.v3)
|
| 183 |
+
x = self.tok(tok & 0xFFFF) + self.num(feats)
|
| 184 |
+
h = x.transpose(1, 2)
|
| 185 |
+
for conv in self.stem:
|
| 186 |
+
h = h + F.glu(conv(F.pad(h, (4, 0))), dim=1)
|
| 187 |
+
x = h.transpose(1, 2)
|
| 188 |
+
bias = self.attn_bias(time, valid, causal)
|
| 189 |
+
for blk in self.blocks:
|
| 190 |
+
x = blk(x, bias)
|
| 191 |
+
return self.norm(x) * valid[..., None]
|
| 192 |
+
|
| 193 |
+
@staticmethod
|
| 194 |
+
def pool(h, valid):
|
| 195 |
+
return h.sum(1) / valid.sum(1, keepdim=True).clamp(min=1)
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
class Store:
|
| 199 |
+
"""Memory-mapped chart store written by prep_dataset.py."""
|
| 200 |
+
|
| 201 |
+
def __init__(self, d):
|
| 202 |
+
self.tok = np.load(f"{d}/tok.npy", mmap_mode="r")
|
| 203 |
+
self.gap = np.load(f"{d}/gap.npy", mmap_mode="r")
|
| 204 |
+
self.sv = np.load(f"{d}/sv.npy", mmap_mode="r")
|
| 205 |
+
self.off = np.load(f"{d}/offsets.npy")
|
| 206 |
+
self.held = np.load(f"{d}/held.npy", mmap_mode="r") if os.path.exists(f"{d}/held.npy") else None # per-column held bits (make_held.py)
|
| 207 |
+
self.extras = False # when True, crop() adds the held bits as bits 16..19 of every token
|
| 208 |
+
|
| 209 |
+
def n(self, i):
|
| 210 |
+
return int(self.off[i + 1] - self.off[i])
|
| 211 |
+
|
| 212 |
+
def crop(self, i, start, L=L_CROP):
|
| 213 |
+
a = int(self.off[i]) + start
|
| 214 |
+
b = min(a + L, int(self.off[i + 1]))
|
| 215 |
+
n = b - a
|
| 216 |
+
tok = np.zeros(L, np.int64); gap = np.zeros(L, np.float32); sv = np.ones(L, np.float32)
|
| 217 |
+
tok[:n], gap[:n], sv[:n] = self.tok[a:b], self.gap[a:b], self.sv[a:b]
|
| 218 |
+
if self.extras and self.held is not None:
|
| 219 |
+
tok[:n] |= self.held[a:b].astype(np.int64) << 16
|
| 220 |
+
valid = np.zeros(L, bool); valid[:n] = True
|
| 221 |
+
return tok, gap, sv, valid
|
| 222 |
+
|
| 223 |
+
def random_crop(self, i, rng, L=L_CROP):
|
| 224 |
+
return self.crop(i, int(rng.integers(0, max(1, self.n(i) - L + 1))), L)
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
@torch.no_grad()
|
| 228 |
+
def embed_charts(model, store, ids, device, L=L_CROP, bs=64):
|
| 229 |
+
"""Chart embedding = mean of window embeddings over non-overlapping windows of L rows. Returns (len(ids), d)."""
|
| 230 |
+
model.eval()
|
| 231 |
+
store.extras = bool(getattr(model, "v3", False))
|
| 232 |
+
wins = [(k, s) for k, i in enumerate(ids) for s in range(0, max(1, store.n(i)), L)]
|
| 233 |
+
sums = torch.zeros(len(ids), model.d); cnt = torch.zeros(len(ids))
|
| 234 |
+
for b in range(0, len(wins), bs):
|
| 235 |
+
part = wins[b:b + bs]
|
| 236 |
+
arr = [store.crop(ids[k], s, L) for k, s in part]
|
| 237 |
+
tok, gap, sv, valid = (torch.from_numpy(np.stack(x)).to(device) for x in zip(*arr))
|
| 238 |
+
with torch.autocast(device_type=device.type, enabled=device.type == "cuda"):
|
| 239 |
+
h = model(tok, gap, sv, valid)
|
| 240 |
+
e = Columnist.pool(h.float(), valid).cpu()
|
| 241 |
+
w = valid.sum(1).cpu().float()
|
| 242 |
+
for (k, _), ek, wk in zip(part, e, w):
|
| 243 |
+
sums[k] += ek * wk; cnt[k] += wk
|
| 244 |
+
return (sums / cnt.clamp(min=1)[:, None]).numpy()
|
| 245 |
+
|
| 246 |
+
|
| 247 |
+
def window_stats(tok, gap, valid):
|
| 248 |
+
"""Free regression targets from a window: log rows/s, hold share, mean chord size, mean log gap. (B,4)"""
|
| 249 |
+
t = (tok & 0xFFFF).clamp(max=N_TOK - 1)
|
| 250 |
+
states = torch.stack([(t >> (2 * c)) & 3 for c in range(4)], -1)
|
| 251 |
+
v = valid.float()
|
| 252 |
+
n = v.sum(1).clamp(min=1)
|
| 253 |
+
secs = (torch.nan_to_num(gap.float()) * v).sum(1).clamp(min=1) / 1000
|
| 254 |
+
chord = ((states > 0).sum(-1).float() * v).sum(1) / n
|
| 255 |
+
hold = (((states == 2) | (states == 3)).any(-1).float() * v).sum(1) / n
|
| 256 |
+
lg = (torch.log1p(torch.nan_to_num(gap.float()).clamp(min=0)) * v).sum(1) / n
|
| 257 |
+
return torch.stack([torch.log(n / secs), hold, chord, lg], 1)
|
| 258 |
+
|
| 259 |
+
|
| 260 |
+
def diff_level(name: str) -> int:
|
| 261 |
+
"""Mapper's own difficulty name -> coarse level (1 easy .. 5 expert); 0 if unknown."""
|
| 262 |
+
import re
|
| 263 |
+
n = re.sub(r"[^a-z ]+", " ", str(name).lower())
|
| 264 |
+
for words, lvl in ((("beginner", "easy", "novice"), 1), (("normal", "basic"), 2), (("hard", "advanced"), 3),
|
| 265 |
+
(("insane", "another"), 4), (("expert", "extra", "extreme", "ex", "master", "lunatic"), 5)):
|
| 266 |
+
if any(w in n.split() for w in words):
|
| 267 |
+
return lvl
|
| 268 |
+
return 0
|
embed_infer.py
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Chart embedding with a Columnist pretraining checkpoint (needs columnist.py next to this file).
|
| 2 |
+
|
| 3 |
+
from embed_infer import load_encoder, embed_chart
|
| 4 |
+
enc = load_encoder("c_big.pt")
|
| 5 |
+
vec = embed_chart(enc, tok, gap, sv) # (d,) numpy vector; tok/gap/sv are the arrays of one 4K chart (see README.md)
|
| 6 |
+
"""
|
| 7 |
+
import numpy as np
|
| 8 |
+
import torch
|
| 9 |
+
|
| 10 |
+
import columnist as C
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
def load_encoder(path, device=None):
|
| 14 |
+
dev = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
|
| 15 |
+
ck = torch.load(path, map_location="cpu", weights_only=False)
|
| 16 |
+
a = ck["args"]
|
| 17 |
+
m = C.Columnist(a["d"], a["layers"], look=bool(a.get("look", False)), v3=bool(a.get("v3_inputs", False)))
|
| 18 |
+
m.load_state_dict(ck["model"], strict=False) # the pretraining checkpoints also hold optional heads
|
| 19 |
+
return m.to(dev).eval()
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
@torch.no_grad()
|
| 23 |
+
def embed_chart(enc, tok, gap, sv, L=512):
|
| 24 |
+
"""Mean over the valid rows of the encoder output of each non-overlapping L-row window, averaged over windows weighted by their length."""
|
| 25 |
+
dev = next(enc.parameters()).device
|
| 26 |
+
tok = np.asarray(tok).astype(np.int64); gap = np.clip(np.nan_to_num(np.asarray(gap, dtype=np.float32)), 0, 60000); sv = np.asarray(sv, dtype=np.float32)
|
| 27 |
+
n = len(tok); starts = list(range(0, max(1, n), L)); W = len(starts)
|
| 28 |
+
T = np.zeros((W, L), np.int64); G = np.zeros((W, L), np.float32); S = np.ones((W, L), np.float32); V = np.zeros((W, L), bool)
|
| 29 |
+
for k, s in enumerate(starts):
|
| 30 |
+
m = min(L, n - s); T[k, :m], G[k, :m], S[k, :m], V[k, :m] = tok[s:s + m], gap[s:s + m], sv[s:s + m], True
|
| 31 |
+
T, G, S, V = (torch.from_numpy(x).to(dev) for x in (T, G, S, V))
|
| 32 |
+
e = C.Columnist.pool(enc(T, G, S, V).float(), V) # (W, d)
|
| 33 |
+
w = V.sum(1).float()
|
| 34 |
+
return ((e * w[:, None]).sum(0) / w.sum()).cpu().numpy()
|
prep_dataset.py
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Turn charts.jsonl.gz into the arrays the model trains on, plus split flags and simple chart statistics.
|
| 2 |
+
|
| 3 |
+
python scripts/foundation/prep_dataset.py --tokens data/foundation/charts.jsonl.gz \\
|
| 4 |
+
--listing data/foundation/charts_4k.csv --bench dataset/om-mania-labels/charts.csv \\
|
| 5 |
+
--pairs data/foundation/round_pairs.csv --out data/foundation/store
|
| 6 |
+
|
| 7 |
+
Writes tok.npy (uint8 row token), gap.npy (ms to previous row), sv.npy (scroll multiplier), offsets.npy,
|
| 8 |
+
meta.csv (one line per chart: ids, split flags, statistics, star rating).
|
| 9 |
+
Pretraining uses only charts that are not copies, not in a benchmark song, not in a held-out tournament and
|
| 10 |
+
not in a held-out retrieval set.
|
| 11 |
+
"""
|
| 12 |
+
import argparse
|
| 13 |
+
import gzip
|
| 14 |
+
import hashlib
|
| 15 |
+
import json
|
| 16 |
+
import re
|
| 17 |
+
from pathlib import Path
|
| 18 |
+
|
| 19 |
+
import numpy as np
|
| 20 |
+
import pandas as pd
|
| 21 |
+
|
| 22 |
+
STAT_COLS = ["n_notes", "n_rows", "length_s", "ln_share", "nps_mean", "nps_peak", "chord_mean", "chord3_share",
|
| 23 |
+
"gap_logstd", "gap_logmean", "jack_share", "n_sv", "sv_std", "hold_len_mean"]
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def h(s, mod):
|
| 27 |
+
return int(hashlib.md5(str(s).encode()).hexdigest(), 16) % mod
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def song_key(artist, title):
|
| 31 |
+
return re.sub(r"[^a-z0-9 ]+", "", f"{artist} {title}".lower()).strip()
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def chart_rows(notes, sv):
|
| 35 |
+
"""notes [(col, t, end|None)], sv [(t, mult)] -> tok, gap, svrow, stats."""
|
| 36 |
+
cols = np.array([n[0] for n in notes]); t = np.array([n[1] for n in notes], dtype=np.int64)
|
| 37 |
+
ends = np.array([-1 if n[2] is None else n[2] for n in notes], dtype=np.int64)
|
| 38 |
+
hold = ends >= 0
|
| 39 |
+
et = np.concatenate([t, ends[hold]]); ec = np.concatenate([cols, cols[hold]])
|
| 40 |
+
es = np.concatenate([np.where(hold, 2, 1), np.full(hold.sum(), 3)])
|
| 41 |
+
times, inv = np.unique(et, return_inverse=True)
|
| 42 |
+
states = np.zeros((len(times), 4), np.int64)
|
| 43 |
+
np.maximum.at(states, (inv, ec), es)
|
| 44 |
+
tok = (states * np.array([1, 4, 16, 64])).sum(1)
|
| 45 |
+
gap = np.diff(times, prepend=times[0])
|
| 46 |
+
if sv:
|
| 47 |
+
st = np.array([s[0] for s in sv]); sm = np.array([s[1] for s in sv], dtype=np.float32)
|
| 48 |
+
o = np.argsort(st, kind="stable"); st, sm = st[o], sm[o]
|
| 49 |
+
j = np.searchsorted(st, times, side="right") - 1
|
| 50 |
+
svrow = np.where(j >= 0, sm[np.clip(j, 0, None)], 1.0).astype(np.float32)
|
| 51 |
+
else:
|
| 52 |
+
svrow, sm = np.ones(len(times), np.float32), np.ones(1, np.float32)
|
| 53 |
+
length = max(1.0, (times[-1] - times[0]) / 1000)
|
| 54 |
+
taps = (states > 0).sum(1)
|
| 55 |
+
heads = (states == 1) | (states == 2)
|
| 56 |
+
pt = np.where(heads.any(1), 1, 0)
|
| 57 |
+
sec = np.bincount(((t - t[0]) // 1000).astype(int))
|
| 58 |
+
lg = np.log1p(gap[1:].clip(min=0)) if len(gap) > 1 else np.zeros(1)
|
| 59 |
+
prev_heads = np.vstack([np.zeros((1, 4), bool), heads[:-1]])
|
| 60 |
+
jack = ((heads & prev_heads) .any(1) & (gap < 200)).mean()
|
| 61 |
+
stats = [len(notes), len(times), length, hold.mean(), len(notes) / length, sec.max(),
|
| 62 |
+
(taps[pt == 1].mean() if pt.any() else 0), (taps >= 3).mean(), lg.std(), lg.mean(), jack,
|
| 63 |
+
len(sv), float(np.log(np.clip(sm, 0.1, 10)).std()), (ends[hold] - t[hold]).mean() if hold.any() else 0]
|
| 64 |
+
return tok.astype(np.uint8), gap.clip(0, 60000).astype(np.float16), svrow.astype(np.float16), stats
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def main():
|
| 68 |
+
ap = argparse.ArgumentParser()
|
| 69 |
+
ap.add_argument("--tokens", required=True); ap.add_argument("--listing", required=True)
|
| 70 |
+
ap.add_argument("--bench", required=True); ap.add_argument("--pairs", required=True)
|
| 71 |
+
ap.add_argument("--out", required=True); ap.add_argument("--max-charts", type=int, default=0)
|
| 72 |
+
a = ap.parse_args()
|
| 73 |
+
out = Path(a.out); out.mkdir(parents=True, exist_ok=True)
|
| 74 |
+
toks, gaps, svs, offs, meta, stats = [], [], [], [0], [], []
|
| 75 |
+
with gzip.open(a.tokens, "rt") as f:
|
| 76 |
+
for line in f:
|
| 77 |
+
c = json.loads(line)
|
| 78 |
+
tk, gp, sv, st = chart_rows(c["notes"], c["sv"])
|
| 79 |
+
if len(tk) < 32:
|
| 80 |
+
continue
|
| 81 |
+
toks.append(tk); gaps.append(gp); svs.append(sv); stats.append(st)
|
| 82 |
+
offs.append(offs[-1] + len(tk)); meta.append((c["beatmap_id"], c["fingerprint"], c["dup_of"]))
|
| 83 |
+
if a.max_charts and len(meta) >= a.max_charts:
|
| 84 |
+
break
|
| 85 |
+
np.save(out / "tok.npy", np.concatenate(toks)); np.save(out / "gap.npy", np.concatenate(gaps))
|
| 86 |
+
np.save(out / "sv.npy", np.concatenate(svs)); np.save(out / "offsets.npy", np.array(offs, np.int64))
|
| 87 |
+
m = pd.DataFrame(meta, columns=["beatmap_id", "fingerprint", "dup_of"])
|
| 88 |
+
m = pd.concat([m, pd.DataFrame(stats, columns=STAT_COLS)], axis=1)
|
| 89 |
+
|
| 90 |
+
lst = pd.read_csv(a.listing, usecols=["beatmap_id", "beatmapset_id", "artist", "title", "stars", "mapper_id"])
|
| 91 |
+
lst["song"] = [song_key(x, y) for x, y in zip(lst.artist, lst.title)]
|
| 92 |
+
m = m.merge(lst[["beatmap_id", "beatmapset_id", "song", "stars", "mapper_id"]], on="beatmap_id", how="left")
|
| 93 |
+
m["set_id"] = m.beatmapset_id.fillna(-m.beatmap_id).astype(int)
|
| 94 |
+
m["song"] = m.song.fillna("id" + m.beatmap_id.astype(str))
|
| 95 |
+
bench = pd.read_csv(a.bench)
|
| 96 |
+
bench = bench[bench["keys"] == 4][["beatmap_id", "fold"]].rename(columns={"fold": "bench_fold"})
|
| 97 |
+
m = m.merge(bench, on="beatmap_id", how="left")
|
| 98 |
+
bench_sets = set(m.loc[m.bench_fold.notna(), "set_id"]); bench_songs = set(m.loc[m.bench_fold.notna(), "song"])
|
| 99 |
+
m["in_bench_group"] = m.set_id.isin(bench_sets) | m.song.isin(bench_songs)
|
| 100 |
+
p = pd.read_csv(a.pairs)
|
| 101 |
+
held = {t for t in p.tournament.unique() if h(t, 5) == 0}
|
| 102 |
+
test_ids = set(p.loc[p.tournament.isin(held), ["easy_id", "hard_id"]].to_numpy().ravel())
|
| 103 |
+
m["tournament_test"] = m.beatmap_id.isin(test_ids)
|
| 104 |
+
sizes = m[m.dup_of.isna()].groupby("set_id").size()
|
| 105 |
+
multi = set(sizes[sizes >= 2].index)
|
| 106 |
+
m["retrieval_test"] = m.set_id.isin(multi) & (m.set_id.map(lambda s: h(s, 33)) == 0) & ~m.in_bench_group
|
| 107 |
+
m["pretrain_ok"] = (m.dup_of.isna() & ~m.in_bench_group & ~m.tournament_test & ~m.retrieval_test)
|
| 108 |
+
m.to_csv(out / "meta.csv", index=False)
|
| 109 |
+
print(len(m), "charts;", int(m.pretrain_ok.sum()), "for pretraining;", int(m.bench_fold.notna().sum()), "benchmark;",
|
| 110 |
+
int(m.tournament_test.sum()), "in held-out tournaments;", int(m.retrieval_test.sum()), "in retrieval test sets;",
|
| 111 |
+
f"{offs[-1]} rows")
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
if __name__ == "__main__":
|
| 115 |
+
main()
|