Quenlla commited on
Commit
b3308b3
·
verified ·
1 Parent(s): 864db58

Upload 10 files

Browse files
README.md CHANGED
@@ -1,3 +1,34 @@
1
  ---
2
  license: cc-by-4.0
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: cc-by-4.0
3
+ library_name: pytorch
4
+ tags: [osu, osu-mania, rhythm-game, foundation-model]
5
  ---
6
+ # Columnist
7
+
8
+ A foundation model for osu!mania 4K charts. Pretrained without labels on unlabelled charts, then adapted to specific tasks.
9
+
10
+ ## Variants
11
+ | File | Use |
12
+ |---|---|
13
+ | columnist-base | Pretrained encoder (256d, 6 layers) |
14
+ | columnist-embed | Label-free encoder behind the similar-charts search |
15
+ | columnist-joint | Fine-tuned: 31 pattern tags and categories, overall difficulty (stars ≈ 3.886 + 0.140 × score), per-window and per-row difficulty |
16
+ | columnist-lens | Per-note pattern readout used by the player page |
17
+
18
+ ## Training
19
+ - Pretraining data: 62,161 unlabelled 4K charts (59,361 unique); chart ids only are released.
20
+ - Fine-tuning labels: community sheet labels, osu! user tags and tournament mappool slots; 8,453 labelled charts.
21
+ - Inputs use millisecond timing and visible scroll only. No snap or BPM-grid information.
22
+
23
+ ## Results (5-fold, mean AP, categories / tags)
24
+ Fine-tuned from the base Columnist (5-fold, song-grouped, mean AP over 11 categories / 20 tags):
25
+ - columnist-base fine-tuned for tags and categories: 0.703 / 0.581
26
+ - columnist-joint (tags, categories and difficulty): 0.698 / 0.578
27
+ - Difficulty: Spearman 0.94 against official star rating, mean absolute error 0.33 stars after linear calibration; held-out tournament round ordering 0.95 accuracy.
28
+ The earlier boosted-statistics pipeline scores higher on categories (about 0.75 / 0.60); Columnist trades some of that for one shared network that also serves similarity, difficulty and per-note readout.
29
+
30
+ ## Limitations
31
+ 4K only; at least 64 rows per chart; long-note patterns are the weakest area; labels come from user tags and mappool slots, not ground truth. Research tool, not an official osu! tool, and not meant for Beatmap Nominator decisions on its own.
32
+
33
+ ## Licence and data
34
+ Weights, labels and this card: CC BY 4.0. Code: Apache-2.0. Charts remain the property of their mappers and are not redistributed. Use of osu! data complies with fair research use.
RELEASE_README.md ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Columnist pretrained encoders (release folder)
2
+
3
+ Two pretraining checkpoints of the Columnist encoder for 4K osu!mania charts. Both are d=256, 6 layers, 6.28M parameters, 20,000 pretraining steps, no human labels.
4
+ No snap, beat or BPM-grid information is used anywhere.
5
+
6
+ | file | what it is |
7
+ |---|---|
8
+ | `c_big.pt` | pretrained with masked rows + contrastive (sibling difficulties) + next row **+ a `feats` regression target** (a 16-number PCA of hand-built whole-chart features; no human labels). **This is the base of the 0.703 / 0.581 fine-tuned results** (teacher + relaxed labels + length feature). Frozen linear probe, categories / tags AP at 100% labels: 0.662 / 0.534. Retrieval recall@5 0.642 |
9
+ | `c_big_free.pt` | pretrained with masked rows + contrastive + next row only: **purely label-free** (no `feats`, no teacher, no labels). Frozen probe 0.592 / 0.486; retrieval recall@5 **0.744** (the best retrieval of all models) |
10
+ | `*.config.json` | architecture and pretraining settings of each checkpoint |
11
+ | `columnist.py` | the model code (needs torch, numpy); both checkpoints load with it |
12
+ | `embed_infer.py` | `load_encoder(path)` and `embed_chart(enc, tok, gap, sv)` -> a 256-number chart embedding |
13
+ | `chart_tokens.py`, `prep_dataset.py` | a 4K `.osu` file -> the arrays: `parse()` then `chart_rows()`, see below |
14
+
15
+ **Which one for what:** I do not know which checkpoint the Lens `/similar` feature uses; I could only identify these two by their training recipes. For **similarity / retrieval of charts** (finding sibling difficulties of the same song) `c_big_free.pt` is
16
+ clearly the better of the two (recall@5 0.744 against 0.642, MRR 0.583 against 0.513). For a **starting point for fine-tuning on labels** use `c_big.pt` (+0.07 categories AP frozen and about +0.014 after fine-tuning with the teacher).
17
+
18
+ ## Loading and using
19
+ ```python
20
+ from embed_infer import load_encoder, embed_chart # columnist.py must be importable
21
+ enc = load_encoder("c_big_free.pt") # strict=False: the checkpoint may hold extra optional heads (c_big has a 'feat_head')
22
+ vec = embed_chart(enc, tok, gap, sv) # (256,) numpy; cosine similarity of these vectors = chart similarity
23
+ ```
24
+ `embed_chart` equals the repository's `columnist.embed_charts` exactly (checked on 2 charts for each checkpoint: difference 0.0). The embedding is the mean of the encoder outputs over the valid rows of each non-overlapping 512-row window, averaged over the windows weighted by their length.
25
+ For retrieval the evaluation standardised the vectors over the candidate set before taking cosine similarity.
26
+
27
+ ## Input arrays of one chart
28
+ * `tok` (L,) int: one value per row (a moment where a note starts or a long note ends; events within 2 ms are merged), 2 bits per column (column c at bits 2c..2c+1): 0 none, 1 tap head, 2 long-note head, 3 long-note tail.
29
+ * `gap` (L,) float: milliseconds since the previous row (0 for the first), clipped to [0, 60000]. `sv` (L,) float: scroll-speed multiplier at the row (1.0 = no change).
30
+ * From a `.osu` file: `mode, keys, notes, sv = chart_tokens.parse(Path(osu))`, then `tok, gap, svrow, stats = prep_dataset.chart_rows(notes, sv)` (use only 4K mania charts with at least 64 rows). Vocabulary: 256 tokens (the 4 columns x 4 states), plus a mask id 256 used in pretraining only.
31
+ * Windows of 512 rows; inference does not mirror the chart (left-right mirroring was a random 50% training augmentation); the model has a local attention window of +-64 rows with a per-head time-decay bias, so it never looks beyond about 64 rows in any layer.
32
+
33
+ No keys, datasets or label files are in this folder.
c_big.config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "file": "c_big.pt",
3
+ "description": "pretrained with masked rows + contrastive (sibling difficulties) + next row + a feats regression target (PCA of hand-built whole-chart features, no human labels); the base of the 0.703 / 0.581 fine-tuned results",
4
+ "architecture": {
5
+ "class": "columnist.Columnist",
6
+ "d": 256,
7
+ "layers": 6,
8
+ "heads": 4,
9
+ "look": false,
10
+ "v3_inputs": false,
11
+ "token_vocabulary": 256,
12
+ "attention": "local +-64 rows with time-decay bias",
13
+ "window_rows": 512,
14
+ "parameters_million": 6.28
15
+ },
16
+ "pretraining": {
17
+ "objectives": "masked,contrastive,next,feats",
18
+ "steps": 20000,
19
+ "batch": 24,
20
+ "lr": 0.0003,
21
+ "mask_p": 0.06,
22
+ "seed": 0,
23
+ "target_file": "data/foundation/feats16.npy"
24
+ },
25
+ "checkpoint_keys": [
26
+ "model",
27
+ "args",
28
+ "step",
29
+ "feat_head"
30
+ ],
31
+ "load": "m = columnist.Columnist(d, layers, look=False, v3=False); m.load_state_dict(ck['model'], strict=False)"
32
+ }
c_big_free.config.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "file": "c_big_free.pt",
3
+ "description": "pretrained with masked rows + contrastive + next row only (no feats target, no labels of any kind); the purely label-free encoder",
4
+ "architecture": {
5
+ "class": "columnist.Columnist",
6
+ "d": 256,
7
+ "layers": 6,
8
+ "heads": 4,
9
+ "look": false,
10
+ "v3_inputs": false,
11
+ "token_vocabulary": 256,
12
+ "attention": "local +-64 rows with time-decay bias",
13
+ "window_rows": 512,
14
+ "parameters_million": 6.28
15
+ },
16
+ "pretraining": {
17
+ "objectives": "masked,contrastive,next",
18
+ "steps": 20000,
19
+ "batch": 24,
20
+ "lr": 0.0003,
21
+ "mask_p": 0.06,
22
+ "seed": 0,
23
+ "target_file": null
24
+ },
25
+ "checkpoint_keys": [
26
+ "model",
27
+ "args",
28
+ "step"
29
+ ],
30
+ "load": "m = columnist.Columnist(d, layers, look=False, v3=False); m.load_state_dict(ck['model'], strict=False)"
31
+ }
chart_tokens.py ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Read 4K .osu files into a compact, snap-free store and de-duplicate them.
2
+
3
+ python scripts/foundation/chart_tokens.py data/foundation/osu data/foundation/charts.jsonl.gz
4
+
5
+ Per chart we keep only millisecond note times (column, start, hold end) and visible scroll-speed changes
6
+ (red-line BPM and snap data are dropped on purpose). `fingerprint` hashes the column sequence of all notes
7
+ and hold flags, so rate edits and copies share it; `dup_of` names the first chart with the same fingerprint.
8
+ """
9
+ import gzip
10
+ import hashlib
11
+ import json
12
+ import sys
13
+ from pathlib import Path
14
+
15
+
16
+ def parse(path: Path):
17
+ sect, notes, sv, keys, meta = "", [], [], None, {}
18
+ raw = path.read_bytes()
19
+ text = raw.decode("utf-16", errors="replace") if raw[:2] in (b"\xff\xfe", b"\xfe\xff") \
20
+ else raw.decode("utf-8-sig", errors="replace")
21
+ for line in text.splitlines():
22
+ line = line.strip()
23
+ if line.startswith("["):
24
+ sect = line
25
+ continue
26
+ if sect == "[General]" and line.startswith("Mode:"):
27
+ meta["mode"] = int(line.split(":")[1])
28
+ elif sect == "[Difficulty]" and line.startswith("CircleSize:"):
29
+ keys = int(float(line.split(":")[1]))
30
+ elif sect == "[TimingPoints]" and line:
31
+ p = line.split(",")
32
+ if len(p) > 6 and p[6] == "0": # green line: visible scroll speed
33
+ sv.append((round(float(p[0])), round(-100 / float(p[1]), 3)))
34
+ elif sect == "[HitObjects]" and line:
35
+ p = line.split(",")
36
+ t, typ = round(float(p[2])), int(p[3])
37
+ col = min(3, int(float(p[0]) * 4 / 512))
38
+ end = int(float(p[5].split(":")[0])) if typ & 128 else None
39
+ notes.append((col, t, end))
40
+ return meta.get("mode"), keys, notes, sv
41
+
42
+
43
+ def main(indir: str, out: str):
44
+ seen, n, skipped = {}, 0, []
45
+ with gzip.open(out, "wt") as f:
46
+ for p in sorted(Path(indir).glob("*.osu")):
47
+ try:
48
+ mode, keys, notes, sv = parse(p)
49
+ except ValueError:
50
+ skipped.append(p.stem) # e.g. a NaN hit-object time
51
+ continue
52
+ if mode != 3 or keys != 4 or len(notes) < 100:
53
+ continue
54
+ notes.sort(key=lambda x: (x[1], x[0]))
55
+ if notes[-1][1] - notes[0][1] < 20000:
56
+ continue
57
+ fp = hashlib.md5(bytes(c + (4 if e else 0) for c, _, e in notes)).hexdigest()
58
+ bid = int(p.stem)
59
+ dup = seen.setdefault(fp, bid)
60
+ f.write(json.dumps({"beatmap_id": bid, "fingerprint": fp, "dup_of": None if dup == bid else dup,
61
+ "notes": notes, "sv": sv}) + "\n")
62
+ n += 1
63
+ print(n, "charts,", len(seen), "unique fingerprints;", len(skipped), "unparseable:", skipped[:20])
64
+
65
+
66
+ if __name__ == "__main__":
67
+ main(sys.argv[1], sys.argv[2])
columnist-base.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4525eac7a2e60078dcfdcbe65c9518cc626af79706baafb39fafc9142a5e5cb3
3
+ size 25152249
columnist-embed.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b3ae20e5eaa11f486d8d6dffa8e4d12733ef42074e43ebf262f173e1faec9ccb
3
+ size 25135301
columnist.py ADDED
@@ -0,0 +1,268 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Columnist: a small sequence model for 4K osu!mania charts (data helpers + network).
2
+
3
+ A chart is a sequence of ROWS (notes that fall at the same millisecond). Each row has a token for the state of
4
+ the four columns (none / tap / hold head / hold tail -> 4**4 = 256 tokens), the gap in ms to the previous
5
+ row, and the visible scroll speed. No BPM, snap or grid information is used anywhere.
6
+
7
+ Network: row embedding + numeric features -> causal conv stem -> transformer layers with local attention
8
+ (window of rows) and a per-head time penalty (rows far apart in seconds attend less). The same network can run
9
+ bidirectionally (masked-row and contrastive objectives) or causally (next-row objective, per-note use).
10
+ """
11
+ from __future__ import annotations
12
+
13
+ import math
14
+
15
+ import os
16
+
17
+ import numpy as np
18
+ import torch
19
+ import torch.nn.functional as F
20
+ from torch import nn
21
+
22
+ N_TOK = 256
23
+ MASK_ID = 256
24
+ N_GAPB = 24
25
+ WINDOW = 64 # rows to each side
26
+ L_CROP = 512
27
+
28
+
29
+ def gap_bucket(gap_ms: torch.Tensor) -> torch.Tensor:
30
+ return (torch.log2(gap_ms.float() + 1) * 2).floor().clamp(0, N_GAPB - 1).long()
31
+
32
+
33
+ def mirror_lut() -> torch.Tensor:
34
+ """token -> token with the four columns reversed. The table covers tokens with the optional held bits (bits 16..19, one per
35
+ column, set by Store when extras are on) and MASK_ID, so lut[tok] works for every kind of token."""
36
+ v = np.arange(1 << 20, dtype=np.int64)
37
+ base, held = v & 0xFFFF, v >> 16
38
+ mb = np.where(base < N_TOK, sum((((base >> (2 * c)) & 3) << (2 * (3 - c))) for c in range(4)), base)
39
+ mh = sum((((held >> c) & 1) << (3 - c)) for c in range(4))
40
+ return torch.from_numpy((mb | (mh << 16)).astype(np.int64))
41
+
42
+
43
+ N_LOOK = 6
44
+ VIS_H = (400.0, 800.0, 1600.0) # scroll distance (ms of travel at multiplier 1) that counts as "on screen"
45
+
46
+
47
+ def look_features(g_ms, sv, valid, causal):
48
+ """Extra inputs (B,L,6): time to the next row, its log ratio to the previous gap, scroll distance since the previous
49
+ row, and how many following rows are within VIS_H of scroll travel (a proxy for what is on screen). Everything is
50
+ computed from row times and the scroll multiplier; no snap, beat or BPM-grid information. The columns that look ahead
51
+ (0, 1, 3, 4, 5) are zeroed when causal=True so the next-row objective cannot see the answer. g_ms must already be
52
+ zero for masked rows (so a masked row's gap never leaks through its neighbours)."""
53
+ B, L = g_ms.shape
54
+ nxt = torch.cat([g_ms[:, 1:], torch.zeros_like(g_ms[:, :1])], 1) * torch.cat([valid[:, 1:], torch.zeros_like(valid[:, :1])], 1)
55
+ f_next = torch.log1p(nxt) / 6.9
56
+ ratio = (torch.log1p(nxt) - torch.log1p(g_ms)).clamp(-6, 6) / 3
57
+ svc = torch.nan_to_num(sv.float(), nan=1.0, posinf=10.0, neginf=0.1).clamp(0.1, 10.0)
58
+ dx = g_ms * svc * valid
59
+ xv = dx.cumsum(1)
60
+ idx = torch.arange(L, device=g_ms.device)
61
+ x = torch.where(valid, xv, xv.max(1, keepdim=True).values + 1e6 * (1 + idx)[None])
62
+ nval = valid.sum(1, keepdim=True)
63
+ cols = [f_next, ratio, torch.log1p(dx) / 9]
64
+ vis = []
65
+ for H in VIS_H:
66
+ j = torch.searchsorted(x.contiguous(), (x + H).contiguous(), right=True).minimum(nval)
67
+ vis.append(torch.log1p(((j - idx[None] - 1).clamp(min=0) * valid).float()) / 4)
68
+ out = torch.stack(cols + vis, -1)
69
+ if causal:
70
+ keep = torch.tensor([0, 0, 1, 0, 0, 0], dtype=out.dtype, device=out.device)
71
+ out = out * keep
72
+ return out
73
+
74
+
75
+ N_V3 = 7
76
+ V3_H = 600.0 # ms of scroll travel that counts as "on screen" (as osu_patterns.deep.tokens.H_MS)
77
+ V3_W = 64 # rows looked back when measuring the speed swing and the number of speed changes on screen
78
+
79
+
80
+ def v3_features(g_ms, sv, valid, held):
81
+ """The inputs of the old v3 network that Columnist lacks (no snap, beat or BPM-grid information): the explicit per-column
82
+ 'held' state (a hold passes through this row; computed from the whole chart by make_held.py, bits 16..19 of the token),
83
+ how long the row has been on screen, and the speed swing and the number of speed changes among the rows on screen.
84
+ g_ms is zero for masked rows. Returns (B,L,7)."""
85
+ B, L = g_ms.shape
86
+ svc = torch.nan_to_num(sv.float(), nan=1.0, posinf=10.0, neginf=0.1).clamp(0.1, 10.0)
87
+ lsp = torch.log(svc)
88
+ x = (g_ms * svc * valid).cumsum(1)
89
+ t = (g_ms * valid).cumsum(1)
90
+ ar = torch.arange(L, device=g_ms.device)
91
+ j0 = torch.searchsorted(x.contiguous(), (x - V3_H).contiguous(), right=False)
92
+ j0 = torch.minimum(j0, ar[None].expand(B, L))
93
+ vis = torch.log(((t - t.gather(1, j0)) / V3_H).clamp(0.05, 20.0))
94
+ k = torch.arange(V3_W, device=g_ms.device)
95
+ idx = ar[:, None] - k[None, :] # (L,W) rows i-k
96
+ ok = (idx >= 0)[None] & (idx[None] >= j0[:, :, None]) # rows still on screen
97
+ ls = lsp[:, idx.clamp(min=0)] # (B,L,W)
98
+ mx = ls.masked_fill(~ok, -1e4).max(-1).values
99
+ mn = ls.masked_fill(~ok, 1e4).min(-1).values
100
+ swing = (mx - mn).clamp(0, 6) / 2
101
+ chg = (((ls[..., :-1] - ls[..., 1:]).abs() > 0.02) & ok[..., 1:]).sum(-1).float()
102
+ held_f = torch.stack([((held >> c) & 1).float() for c in range(4)], -1)
103
+ return torch.cat([held_f, torch.stack([vis, swing, torch.log1p(chg) / 3], -1)], -1)
104
+
105
+
106
+ def row_features(tok, gap, sv, valid, look=False, causal=False, v3=False):
107
+ """tok (B,L) long (MASK_ID allowed), gap (B,L) ms, sv (B,L) scroll multiplier -> (B,L,14 or 14+N_LOOK) floats, times (B,L) s."""
108
+ held_bits = (tok >> 16) & 15
109
+ tok = tok & 0xFFFF
110
+ masked = tok == MASK_ID
111
+ t = tok.clamp(max=N_TOK - 1)
112
+ states = torch.stack([(t >> (2 * c)) & 3 for c in range(4)], -1) # (B,L,4)
113
+ states = torch.where(masked[..., None], torch.zeros_like(states), states)
114
+ delta = (states == 2).long() - (states == 3).long()
115
+ held_before = (delta.cumsum(1) - delta).clamp(min=0).float().clamp(max=1)
116
+ gap = torch.nan_to_num(gap.float(), nan=0.0, posinf=60000.0)
117
+ g = torch.log1p(gap.clamp(min=0)) / 6.9
118
+ g = torch.where(masked, torch.zeros_like(g), g)
119
+ four = [f(g * (2.0 ** k) * math.pi) for k in range(4) for f in (torch.sin, torch.cos)]
120
+ svf = torch.log(torch.nan_to_num(sv.float(), nan=1.0, posinf=10.0, neginf=0.1).clamp(0.1, 10.0))
121
+ feats = torch.cat([g[..., None], torch.stack(four, -1), svf[..., None], held_before], -1)
122
+ if look:
123
+ gm = torch.where(masked, torch.zeros_like(gap), gap.clamp(min=0))
124
+ feats = torch.cat([feats, look_features(gm, sv, valid, causal)], -1)
125
+ if v3:
126
+ gm3 = torch.where(masked, torch.zeros_like(gap), gap.clamp(min=0))
127
+ feats = torch.cat([feats, v3_features(gm3, sv, valid, held_bits)], -1)
128
+ time = (gap.float().clamp(min=0) * (~masked) * valid).cumsum(1) / 1000.0
129
+ return feats * valid[..., None], time
130
+
131
+
132
+ class Block(nn.Module):
133
+ def __init__(self, d, heads, p):
134
+ super().__init__()
135
+ self.h = heads
136
+ self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
137
+ self.qkv = nn.Linear(d, 3 * d)
138
+ self.o = nn.Linear(d, d)
139
+ self.ff = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
140
+ self.drop = nn.Dropout(p)
141
+
142
+ def forward(self, x, bias):
143
+ B, L, D = x.shape
144
+ q, k, v = self.qkv(self.n1(x)).view(B, L, 3, self.h, D // self.h).permute(2, 0, 3, 1, 4)
145
+ a = F.scaled_dot_product_attention(q, k, v, attn_mask=bias.to(q.dtype))
146
+ x = x + self.drop(self.o(a.transpose(1, 2).reshape(B, L, D)))
147
+ return x + self.drop(self.ff(self.n2(x)))
148
+
149
+
150
+ class Columnist(nn.Module):
151
+ def __init__(self, d=128, layers=4, heads=4, p=0.1, look=False, v3=False):
152
+ super().__init__()
153
+ self.d, self.heads, self.look, self.v3 = d, heads, look, v3
154
+ self.tok = nn.Embedding(N_TOK + 1, d)
155
+ self.num = nn.Linear(14 + (N_LOOK if look else 0) + (N_V3 if v3 else 0), d)
156
+ self.stem = nn.ModuleList([nn.Conv1d(d, 2 * d, 5) for _ in range(2)]) # left-padded: causal
157
+ self.blocks = nn.ModuleList(Block(d, heads, p) for _ in range(layers))
158
+ self.norm = nn.LayerNorm(d)
159
+ self.register_buffer("slopes", torch.tensor([0.05 * 2 ** (i * 5 / max(1, heads - 1)) for i in range(heads)]))
160
+ self.tok_head = nn.Linear(d, N_TOK)
161
+ self.gap_head = nn.Linear(d, N_GAPB)
162
+ self.proj = nn.Sequential(nn.Linear(d, d), nn.GELU(), nn.Linear(d, 64))
163
+ self.rank_head = nn.Linear(d, 1) # difficulty-order objectives (ladder, rounds)
164
+ self.order_head = nn.Linear(2 * d, 1) # which of two windows comes first
165
+ self.stat_head = nn.Linear(d, 4) # window statistics
166
+ self.rate_head = nn.Linear(2 * d, 1) # log rate ratio between two views of one window
167
+
168
+ def attn_bias(self, time, valid, causal):
169
+ B, L = valid.shape
170
+ idx = torch.arange(L, device=valid.device)
171
+ rel = idx[None, :] - idx[:, None] # j - i
172
+ allowed = rel.abs() <= WINDOW
173
+ if causal:
174
+ allowed = allowed & (rel <= 0)
175
+ allowed = allowed[None] & valid[:, None, :]
176
+ allowed = allowed | torch.eye(L, dtype=torch.bool, device=valid.device)[None]
177
+ dt = (time[:, None, :] - time[:, :, None]).abs() # (B,L,L) seconds
178
+ bias = -self.slopes[None, :, None, None] * dt[:, None]
179
+ return bias.masked_fill(~allowed[:, None], float("-inf"))
180
+
181
+ def forward(self, tok, gap, sv, valid, causal=False):
182
+ feats, time = row_features(tok, gap, sv, valid, self.look, causal, self.v3)
183
+ x = self.tok(tok & 0xFFFF) + self.num(feats)
184
+ h = x.transpose(1, 2)
185
+ for conv in self.stem:
186
+ h = h + F.glu(conv(F.pad(h, (4, 0))), dim=1)
187
+ x = h.transpose(1, 2)
188
+ bias = self.attn_bias(time, valid, causal)
189
+ for blk in self.blocks:
190
+ x = blk(x, bias)
191
+ return self.norm(x) * valid[..., None]
192
+
193
+ @staticmethod
194
+ def pool(h, valid):
195
+ return h.sum(1) / valid.sum(1, keepdim=True).clamp(min=1)
196
+
197
+
198
+ class Store:
199
+ """Memory-mapped chart store written by prep_dataset.py."""
200
+
201
+ def __init__(self, d):
202
+ self.tok = np.load(f"{d}/tok.npy", mmap_mode="r")
203
+ self.gap = np.load(f"{d}/gap.npy", mmap_mode="r")
204
+ self.sv = np.load(f"{d}/sv.npy", mmap_mode="r")
205
+ self.off = np.load(f"{d}/offsets.npy")
206
+ self.held = np.load(f"{d}/held.npy", mmap_mode="r") if os.path.exists(f"{d}/held.npy") else None # per-column held bits (make_held.py)
207
+ self.extras = False # when True, crop() adds the held bits as bits 16..19 of every token
208
+
209
+ def n(self, i):
210
+ return int(self.off[i + 1] - self.off[i])
211
+
212
+ def crop(self, i, start, L=L_CROP):
213
+ a = int(self.off[i]) + start
214
+ b = min(a + L, int(self.off[i + 1]))
215
+ n = b - a
216
+ tok = np.zeros(L, np.int64); gap = np.zeros(L, np.float32); sv = np.ones(L, np.float32)
217
+ tok[:n], gap[:n], sv[:n] = self.tok[a:b], self.gap[a:b], self.sv[a:b]
218
+ if self.extras and self.held is not None:
219
+ tok[:n] |= self.held[a:b].astype(np.int64) << 16
220
+ valid = np.zeros(L, bool); valid[:n] = True
221
+ return tok, gap, sv, valid
222
+
223
+ def random_crop(self, i, rng, L=L_CROP):
224
+ return self.crop(i, int(rng.integers(0, max(1, self.n(i) - L + 1))), L)
225
+
226
+
227
+ @torch.no_grad()
228
+ def embed_charts(model, store, ids, device, L=L_CROP, bs=64):
229
+ """Chart embedding = mean of window embeddings over non-overlapping windows of L rows. Returns (len(ids), d)."""
230
+ model.eval()
231
+ store.extras = bool(getattr(model, "v3", False))
232
+ wins = [(k, s) for k, i in enumerate(ids) for s in range(0, max(1, store.n(i)), L)]
233
+ sums = torch.zeros(len(ids), model.d); cnt = torch.zeros(len(ids))
234
+ for b in range(0, len(wins), bs):
235
+ part = wins[b:b + bs]
236
+ arr = [store.crop(ids[k], s, L) for k, s in part]
237
+ tok, gap, sv, valid = (torch.from_numpy(np.stack(x)).to(device) for x in zip(*arr))
238
+ with torch.autocast(device_type=device.type, enabled=device.type == "cuda"):
239
+ h = model(tok, gap, sv, valid)
240
+ e = Columnist.pool(h.float(), valid).cpu()
241
+ w = valid.sum(1).cpu().float()
242
+ for (k, _), ek, wk in zip(part, e, w):
243
+ sums[k] += ek * wk; cnt[k] += wk
244
+ return (sums / cnt.clamp(min=1)[:, None]).numpy()
245
+
246
+
247
+ def window_stats(tok, gap, valid):
248
+ """Free regression targets from a window: log rows/s, hold share, mean chord size, mean log gap. (B,4)"""
249
+ t = (tok & 0xFFFF).clamp(max=N_TOK - 1)
250
+ states = torch.stack([(t >> (2 * c)) & 3 for c in range(4)], -1)
251
+ v = valid.float()
252
+ n = v.sum(1).clamp(min=1)
253
+ secs = (torch.nan_to_num(gap.float()) * v).sum(1).clamp(min=1) / 1000
254
+ chord = ((states > 0).sum(-1).float() * v).sum(1) / n
255
+ hold = (((states == 2) | (states == 3)).any(-1).float() * v).sum(1) / n
256
+ lg = (torch.log1p(torch.nan_to_num(gap.float()).clamp(min=0)) * v).sum(1) / n
257
+ return torch.stack([torch.log(n / secs), hold, chord, lg], 1)
258
+
259
+
260
+ def diff_level(name: str) -> int:
261
+ """Mapper's own difficulty name -> coarse level (1 easy .. 5 expert); 0 if unknown."""
262
+ import re
263
+ n = re.sub(r"[^a-z ]+", " ", str(name).lower())
264
+ for words, lvl in ((("beginner", "easy", "novice"), 1), (("normal", "basic"), 2), (("hard", "advanced"), 3),
265
+ (("insane", "another"), 4), (("expert", "extra", "extreme", "ex", "master", "lunatic"), 5)):
266
+ if any(w in n.split() for w in words):
267
+ return lvl
268
+ return 0
embed_infer.py ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Chart embedding with a Columnist pretraining checkpoint (needs columnist.py next to this file).
2
+
3
+ from embed_infer import load_encoder, embed_chart
4
+ enc = load_encoder("c_big.pt")
5
+ vec = embed_chart(enc, tok, gap, sv) # (d,) numpy vector; tok/gap/sv are the arrays of one 4K chart (see README.md)
6
+ """
7
+ import numpy as np
8
+ import torch
9
+
10
+ import columnist as C
11
+
12
+
13
+ def load_encoder(path, device=None):
14
+ dev = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
15
+ ck = torch.load(path, map_location="cpu", weights_only=False)
16
+ a = ck["args"]
17
+ m = C.Columnist(a["d"], a["layers"], look=bool(a.get("look", False)), v3=bool(a.get("v3_inputs", False)))
18
+ m.load_state_dict(ck["model"], strict=False) # the pretraining checkpoints also hold optional heads
19
+ return m.to(dev).eval()
20
+
21
+
22
+ @torch.no_grad()
23
+ def embed_chart(enc, tok, gap, sv, L=512):
24
+ """Mean over the valid rows of the encoder output of each non-overlapping L-row window, averaged over windows weighted by their length."""
25
+ dev = next(enc.parameters()).device
26
+ tok = np.asarray(tok).astype(np.int64); gap = np.clip(np.nan_to_num(np.asarray(gap, dtype=np.float32)), 0, 60000); sv = np.asarray(sv, dtype=np.float32)
27
+ n = len(tok); starts = list(range(0, max(1, n), L)); W = len(starts)
28
+ T = np.zeros((W, L), np.int64); G = np.zeros((W, L), np.float32); S = np.ones((W, L), np.float32); V = np.zeros((W, L), bool)
29
+ for k, s in enumerate(starts):
30
+ m = min(L, n - s); T[k, :m], G[k, :m], S[k, :m], V[k, :m] = tok[s:s + m], gap[s:s + m], sv[s:s + m], True
31
+ T, G, S, V = (torch.from_numpy(x).to(dev) for x in (T, G, S, V))
32
+ e = C.Columnist.pool(enc(T, G, S, V).float(), V) # (W, d)
33
+ w = V.sum(1).float()
34
+ return ((e * w[:, None]).sum(0) / w.sum()).cpu().numpy()
prep_dataset.py ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Turn charts.jsonl.gz into the arrays the model trains on, plus split flags and simple chart statistics.
2
+
3
+ python scripts/foundation/prep_dataset.py --tokens data/foundation/charts.jsonl.gz \\
4
+ --listing data/foundation/charts_4k.csv --bench dataset/om-mania-labels/charts.csv \\
5
+ --pairs data/foundation/round_pairs.csv --out data/foundation/store
6
+
7
+ Writes tok.npy (uint8 row token), gap.npy (ms to previous row), sv.npy (scroll multiplier), offsets.npy,
8
+ meta.csv (one line per chart: ids, split flags, statistics, star rating).
9
+ Pretraining uses only charts that are not copies, not in a benchmark song, not in a held-out tournament and
10
+ not in a held-out retrieval set.
11
+ """
12
+ import argparse
13
+ import gzip
14
+ import hashlib
15
+ import json
16
+ import re
17
+ from pathlib import Path
18
+
19
+ import numpy as np
20
+ import pandas as pd
21
+
22
+ STAT_COLS = ["n_notes", "n_rows", "length_s", "ln_share", "nps_mean", "nps_peak", "chord_mean", "chord3_share",
23
+ "gap_logstd", "gap_logmean", "jack_share", "n_sv", "sv_std", "hold_len_mean"]
24
+
25
+
26
+ def h(s, mod):
27
+ return int(hashlib.md5(str(s).encode()).hexdigest(), 16) % mod
28
+
29
+
30
+ def song_key(artist, title):
31
+ return re.sub(r"[^a-z0-9 ]+", "", f"{artist} {title}".lower()).strip()
32
+
33
+
34
+ def chart_rows(notes, sv):
35
+ """notes [(col, t, end|None)], sv [(t, mult)] -> tok, gap, svrow, stats."""
36
+ cols = np.array([n[0] for n in notes]); t = np.array([n[1] for n in notes], dtype=np.int64)
37
+ ends = np.array([-1 if n[2] is None else n[2] for n in notes], dtype=np.int64)
38
+ hold = ends >= 0
39
+ et = np.concatenate([t, ends[hold]]); ec = np.concatenate([cols, cols[hold]])
40
+ es = np.concatenate([np.where(hold, 2, 1), np.full(hold.sum(), 3)])
41
+ times, inv = np.unique(et, return_inverse=True)
42
+ states = np.zeros((len(times), 4), np.int64)
43
+ np.maximum.at(states, (inv, ec), es)
44
+ tok = (states * np.array([1, 4, 16, 64])).sum(1)
45
+ gap = np.diff(times, prepend=times[0])
46
+ if sv:
47
+ st = np.array([s[0] for s in sv]); sm = np.array([s[1] for s in sv], dtype=np.float32)
48
+ o = np.argsort(st, kind="stable"); st, sm = st[o], sm[o]
49
+ j = np.searchsorted(st, times, side="right") - 1
50
+ svrow = np.where(j >= 0, sm[np.clip(j, 0, None)], 1.0).astype(np.float32)
51
+ else:
52
+ svrow, sm = np.ones(len(times), np.float32), np.ones(1, np.float32)
53
+ length = max(1.0, (times[-1] - times[0]) / 1000)
54
+ taps = (states > 0).sum(1)
55
+ heads = (states == 1) | (states == 2)
56
+ pt = np.where(heads.any(1), 1, 0)
57
+ sec = np.bincount(((t - t[0]) // 1000).astype(int))
58
+ lg = np.log1p(gap[1:].clip(min=0)) if len(gap) > 1 else np.zeros(1)
59
+ prev_heads = np.vstack([np.zeros((1, 4), bool), heads[:-1]])
60
+ jack = ((heads & prev_heads) .any(1) & (gap < 200)).mean()
61
+ stats = [len(notes), len(times), length, hold.mean(), len(notes) / length, sec.max(),
62
+ (taps[pt == 1].mean() if pt.any() else 0), (taps >= 3).mean(), lg.std(), lg.mean(), jack,
63
+ len(sv), float(np.log(np.clip(sm, 0.1, 10)).std()), (ends[hold] - t[hold]).mean() if hold.any() else 0]
64
+ return tok.astype(np.uint8), gap.clip(0, 60000).astype(np.float16), svrow.astype(np.float16), stats
65
+
66
+
67
+ def main():
68
+ ap = argparse.ArgumentParser()
69
+ ap.add_argument("--tokens", required=True); ap.add_argument("--listing", required=True)
70
+ ap.add_argument("--bench", required=True); ap.add_argument("--pairs", required=True)
71
+ ap.add_argument("--out", required=True); ap.add_argument("--max-charts", type=int, default=0)
72
+ a = ap.parse_args()
73
+ out = Path(a.out); out.mkdir(parents=True, exist_ok=True)
74
+ toks, gaps, svs, offs, meta, stats = [], [], [], [0], [], []
75
+ with gzip.open(a.tokens, "rt") as f:
76
+ for line in f:
77
+ c = json.loads(line)
78
+ tk, gp, sv, st = chart_rows(c["notes"], c["sv"])
79
+ if len(tk) < 32:
80
+ continue
81
+ toks.append(tk); gaps.append(gp); svs.append(sv); stats.append(st)
82
+ offs.append(offs[-1] + len(tk)); meta.append((c["beatmap_id"], c["fingerprint"], c["dup_of"]))
83
+ if a.max_charts and len(meta) >= a.max_charts:
84
+ break
85
+ np.save(out / "tok.npy", np.concatenate(toks)); np.save(out / "gap.npy", np.concatenate(gaps))
86
+ np.save(out / "sv.npy", np.concatenate(svs)); np.save(out / "offsets.npy", np.array(offs, np.int64))
87
+ m = pd.DataFrame(meta, columns=["beatmap_id", "fingerprint", "dup_of"])
88
+ m = pd.concat([m, pd.DataFrame(stats, columns=STAT_COLS)], axis=1)
89
+
90
+ lst = pd.read_csv(a.listing, usecols=["beatmap_id", "beatmapset_id", "artist", "title", "stars", "mapper_id"])
91
+ lst["song"] = [song_key(x, y) for x, y in zip(lst.artist, lst.title)]
92
+ m = m.merge(lst[["beatmap_id", "beatmapset_id", "song", "stars", "mapper_id"]], on="beatmap_id", how="left")
93
+ m["set_id"] = m.beatmapset_id.fillna(-m.beatmap_id).astype(int)
94
+ m["song"] = m.song.fillna("id" + m.beatmap_id.astype(str))
95
+ bench = pd.read_csv(a.bench)
96
+ bench = bench[bench["keys"] == 4][["beatmap_id", "fold"]].rename(columns={"fold": "bench_fold"})
97
+ m = m.merge(bench, on="beatmap_id", how="left")
98
+ bench_sets = set(m.loc[m.bench_fold.notna(), "set_id"]); bench_songs = set(m.loc[m.bench_fold.notna(), "song"])
99
+ m["in_bench_group"] = m.set_id.isin(bench_sets) | m.song.isin(bench_songs)
100
+ p = pd.read_csv(a.pairs)
101
+ held = {t for t in p.tournament.unique() if h(t, 5) == 0}
102
+ test_ids = set(p.loc[p.tournament.isin(held), ["easy_id", "hard_id"]].to_numpy().ravel())
103
+ m["tournament_test"] = m.beatmap_id.isin(test_ids)
104
+ sizes = m[m.dup_of.isna()].groupby("set_id").size()
105
+ multi = set(sizes[sizes >= 2].index)
106
+ m["retrieval_test"] = m.set_id.isin(multi) & (m.set_id.map(lambda s: h(s, 33)) == 0) & ~m.in_bench_group
107
+ m["pretrain_ok"] = (m.dup_of.isna() & ~m.in_bench_group & ~m.tournament_test & ~m.retrieval_test)
108
+ m.to_csv(out / "meta.csv", index=False)
109
+ print(len(m), "charts;", int(m.pretrain_ok.sum()), "for pretraining;", int(m.bench_fold.notna().sum()), "benchmark;",
110
+ int(m.tournament_test.sum()), "in held-out tournaments;", int(m.retrieval_test.sum()), "in retrieval test sets;",
111
+ f"{offs[-1]} rows")
112
+
113
+
114
+ if __name__ == "__main__":
115
+ main()