| --- |
| license: cc-by-sa-4.0 |
| base_model: Qwen/Qwen3-1.7B |
| library_name: transformers |
| pipeline_tag: text-generation |
| language: |
| - en |
| tags: |
| - music |
| - playlist |
| - recommendation |
| - generative-retrieval |
| - semantic-ids |
| - qwen3 |
| --- |
| |
| # juke-1.7b: type anything, get a playlist |
|
|
| **juke** is a 1.7B language model that turns any prompt into a playlist of real songs, like *"bossa nova brunch"*, |
| *"sad midwest emo for a long drive"*, *"yttrium exoplanet"* or *"my cat's tax return"*. It can **think out loud like |
| a DJ** first, then give a title, a one-line idea, and a numbered tracklist with a short reason for each song. |
|
|
| > **v1.1 (recommended):** repeats inside a playlist cut from 5% to 1.8% by a **weight edit, not retraining**. Four |
| > "copy-suppression" attention heads were made 2Γ louder (about 0.03% of the weights). See [What's new in v1.1](#whats-new-in-v11). |
|
|
| It is a **single model**: there is no search index, database or extra code. The model has memorized a catalog of |
| **229,042 songs**, and each song has a short address it can write, for example `<a_67><b_10><c_0>` = Brian Eno β An Ending. |
|
|
| ``` |
| > yttrium exoplanet |
| |
| <think> |
| I'm reading "yttrium exoplanet" as a cosmic object with a metallic, distant, slightly surreal feel... ambient, |
| post-rock, dream pop and spacious electronic pieces... opening with distance, moving through shimmer and gravity, |
| then landing in a hushed, suspended afterimage. |
| </think> |
| |
| Yttrium Orbit |
| Logic: Spacious, reflective tracks that feel like orbiting a cold planet. |
| 1. <a_67><b_10><c_0> Brian Eno β An Ending β Soft light fading into deep space |
| 2. <a_12><b_1><c_0> Tangerine Dream β Phaedra β Classic cosmic drift and shimmer |
| 3. <a_67><b_68><c_0> Vangelis β Alpha β Bright, floating, otherworldly calm |
| 4. <a_9><b_27><c_0> Boards of Canada β Dayvan Cowboy β Warm distance with a metallic sheen |
| ... |
| ``` |
|
|
| ## Quick start (π€ transformers) |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| import re |
| |
| repo = "asharrm/juke-1.7b-dev" |
| tok = AutoTokenizer.from_pretrained(repo, subfolder="v1.1") |
| model = AutoModelForCausalLM.from_pretrained(repo, subfolder="v1.1", torch_dtype="auto", device_map="auto") |
| |
| prompt = "sad midwest emo for a long drive" |
| text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, |
| add_generation_prompt=True, enable_thinking=True) # False = no DJ monologue |
| ids = tok(text, return_tensors="pt").to(model.device) |
| out = model.generate(**ids, max_new_tokens=700, do_sample=False) |
| reply = tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=False).replace("<|im_end|>", "") |
| |
| # hide the song addresses for display |
| print(re.sub(r"(<[abc]_\d+>){3}\s*", "", reply)) |
| ``` |
|
|
| **Thinking on or off:** `enable_thinking=True` makes the model write a short plan (60β120 words) and a reason per |
| track, which suits a "talking DJ". `False` gives just title, logic and tracks. In our tests, thinking on produced |
| more varied playlists. |
|
|
| **Removing repeats:** about 2% of tracks (v1.1) are repeats of a song already in the same list (see Evaluation). Drop |
| any line whose `<a_><b_><c_>` address already appeared, or ask for a few extra tracks and de-duplicate. |
|
|
| ## llama.cpp / GGUF |
|
|
| A ready Q8_0 file is in this repo: `gguf/juke-1.7b-v1.1-q8_0.gguf` (1.8 GB). |
|
|
| ```bash |
| huggingface-cli download asharrm/juke-1.7b-dev gguf/juke-1.7b-v1.1-q8_0.gguf --local-dir . |
| llama-server -m gguf/juke-1.7b-v1.1-q8_0.gguf -ngl 99 |
| ``` |
|
|
| Or convert yourself with llama.cpp's `convert_hf_to_gguf.py` (the 768 song-address tokens convert as normal added tokens): |
|
|
| ```bash |
| python convert_hf_to_gguf.py final/ --outtype q8_0 --outfile juke-1.7b-q8_0.gguf |
| llama-server -m juke-1.7b-q8_0.gguf -ngl 99 |
| ``` |
|
|
| Use raw completion with the Qwen3 chat format. For thinking off, add an empty think block: |
| `<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n`. |
| A Q8_0 GGUF (1.8 GB) runs at ~146 tokens/s on a PS5 GPU (Vulkan), which is about 1.5 s per playlist. |
| |
| ## Output format |
| |
| ``` |
| <think>β¦planβ¦</think> (thinking mode only) |
| |
| <Playlist title> |
| Logic: <one sentence> |
| 1. <a_x><b_y><c_z> Artist β Title β <reason> (reasons in thinking mode only) |
| 2. β¦ |
| ``` |
| |
| Other trained prompt styles: |
| - `/tracks <playlist name>` β just a tracklist, in the style of real user playlists |
| - `/tag shoegaze` β 10 songs carrying that genre/tag |
| - `<song> <a_x><b_y><c_z>` β `Artist β Title` (catalog lookup) |
| - `<code> Artist β Title` β `<a_x><b_y><c_z>` (reverse lookup) |
| |
| ## How song addresses work |
| |
| Every catalog song gets a 3-token address. Songs that sound alike share a prefix: |
| |
| - `a` = one of 256 broad areas, e.g. shoegaze, bossa nova or modern country (~900 songs each) |
| - `b` = a small sub-style inside the area (~9 songs each) |
| - `c` = the song's position inside the sub-style (most popular first) |
| |
| The addresses come from a balanced k-means tree over song vectors built from **playlist co-occurrence** (which songs |
| people put together) plus **genre/style tags** and an artist signal. Writing the address first means the model |
| "walks into the right neighbourhood" before naming the song, the same idea as Spotify's Text2Tracks and other |
| semantic-ID generative retrieval work. |
| |
| ## Training |
| |
| - **Base:** Qwen3-1.7B, full fine-tune, 768 new address tokens plus `<song>` and `<code>`. |
| - **Data:** 579k examples, 39M tokens per pass, five tasks: |
| |
| | Task | Examples | What it teaches | |
| |---|---|---| |
| | Memorization | 458k | address β exact song name, every catalog song, both directions | |
| | Teacher playlists | 26.5k | prompt β title, logic, tracks; 10.9k with a thinking plan and per-track reasons | |
| | Odd prompts | 2k | absurd and abstract prompts with committed interpretations | |
| | Real playlists | 79k | real user playlist names β their tracks | |
| | Tags | 14.6k | genre/style tag β songs | |
| |
| Teacher playlists, thinking traces and odd-prompt answers were written by GPT-5.4-mini. Every track was checked |
| against the catalog, and any song not in it was dropped. |
| - **Schedule:** 8 rounds. Every round contains all memorization examples plus a sampled share of the other tasks. |
| 2048-token packed sequences, AdamW, peak learning rate 1e-4 with cosine decay, bf16 autocast with fp32 master |
| weights. |
| - **Compute:** about 4.5 h on one RTX PRO 6000 (96 GB). Total GPU cost for the project was about $10. |
| - **Versions in this repo:** `v1.1/` (recommended: `final/` plus the head edit below), `final/` (end of round 8), and the |
| `pass7/` and `pass6/` checkpoints. |
| |
| ## What's new in v1.1: fixing repeats by editing the weights |
| |
| v1.0 (`final/`) sometimes listed the same song twice in one playlist. About **5% of tracks were repeats**, e.g. |
| *Leonard Cohen β Everybody Knows* three times in "dark cabaret heartbreak". The usual fix is more fine-tuning. Instead |
| we found the circuit responsible and **edited it directly: no gradient steps, no new data, about 0.03% of the weights**. |
| |
| ### 1. Collect the behaviour |
| Greedy-decode 400 playlists from training prompts that the evaluation does not use. Label every list item as |
| **repeat** (its 3-token song address already appeared earlier in the list) or **fresh**: 239 repeats and 3,692 fresh. |
| |
| ### 2. Where is "about to repeat" visible? (correlational) |
| At the position that predicts each item's first address token, take the residual stream at every layer. The |
| difference of class means gives a direction, scored on a held-out half by Cohen's *d*: |
| |
| | layers | 1β9 | 10β15 | **16β19** | 20β21 | 22β28 | |
| |---|---|---|---|---|---| |
| | Cohen's *d* | 0.6β1.1 | 0.4β0.95 | **1.4β1.6** | 1.05β1.3 | 0.4β0.9 | |
| |
| The decision is most legible in the middle-late layers. Correlation is not causation, so we went one step further. |
| |
| ### 3. Which units cause it? (attribution patching) |
| For each list item, target = log p(the item's 3 address tokens). For every MLP neuron (`down_proj` input) and every |
| attention head (its slice of the `o_proj` input), attribution = activation Γ βtarget/βactivation, summed over the |
| 3 predicting positions. Then: |
|
|
| repeat score = mean attribution on **repeated** items β mean attribution on **fresh** items |
|
|
| The largest scores were *negative*, meaning units that push repeats down. These are **copy-suppression heads** that look |
| back at songs already listed and lower their probability: |
|
|
| | head | repeat score | |
| |---|---| |
| | L22 h8 | β0.068 | |
| | L22 h7 | β0.046 | |
| | L26 h7 | β0.042 | |
| | L16 h14 | β0.040 | |
| | (most repeat-promoting: L15 h6) | +0.028 | |
|
|
| ### 4. Test edits causally, against controls |
| Each candidate edit was applied to a fresh copy of the weights and scored by teacher forcing on held-out playlists |
| (eval prompts). Ξ is the change in mean log-probability of repeated vs. fresh items; we want Ξrepeat βͺ Ξfresh β 0. |
|
|
| | edit | Ξ repeat | Ξ fresh | | |
| |---|---|---|---| |
| | 8 / 32 / 128 random neurons β 0 (control) | 0.00 / 0.00 / β0.03 | 0.00 / 0.00 / β0.01 | control is flat β
| |
| | top 8 / 32 / 128 repeat-promoting neurons β 0 | β0.33 / β0.91 / β1.90 | β0.27 / β0.65 / β1.49 | hurts everything β | |
| | remove the probe direction (layers 15β19) | β0.41 | β0.41 | not selective β | |
| | boost top anti-repeat neurons Γ1.5β2 | β0.06 to β0.21 | β0.03 to β0.11 | too weak | |
| | top 2 / 4 repeat-promoting heads β 0 | β0.49 / β0.82 | β0.26 / β0.53 | partly selective | |
| | **4 copy-suppression heads Γ1.5** | **β0.68** | **β0.25** | selective β
| |
| | **4 copy-suppression heads Γ2** | **β1.64** | **β0.60** | **chosen** β
| |
| | 4 copy-suppression heads Γ3 | β3.81 | β1.51 | starts to damage | |
|
|
| ### 5. The edit |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM |
| m = AutoModelForCausalLM.from_pretrained("asharrm/juke-1.7b-dev", subfolder="final", torch_dtype=torch.bfloat16) |
| hd = m.config.head_dim # 128 |
| with torch.no_grad(): |
| for layer, head in [(22, 8), (22, 7), (26, 7), (16, 14)]: |
| m.model.layers[layer].self_attn.o_proj.weight[:, head * hd:(head + 1) * hd] *= 2.0 |
| m.save_pretrained("juke-1.7b-v1.1") |
| ``` |
| That is 4 Γ 128 columns of `o_proj`: 1.05M of 1.7B weights, about **0.03%**. |
|
|
| ### 6. Result (greedy decoding, 110 prompts, real generation) |
|
|
| | metric | v1.0 `final/` | Γ1.5 | **v1.1 (Γ2)** | |
| |---|---|---|---| |
| | repeated tracks inside a playlist | 5.0% | 2.9% | **1.8%** | |
| | song addresses that exist | 99.9% | 100% | 99.9% | |
| | song name matches its address | 96.6% | 97.1% | **98.0%** | |
| | distinct songs across 40 unrelated prompts | 90.9% | 91.7% | 93.0% | |
| | overlap with teacher playlists (trained prompts) | 7.2% | 5.9% | 5.6% | |
|
|
| Before and after: |
|
|
| | prompt | v1.0 | v1.1 | |
| |---|---|---| |
| | dark cabaret heartbreak | Leonard Cohen β Everybody Knows Γ3 | The Dresden Dolls, Nellie McKay, Cold Cave, no repeats | |
| | wistful indie rock from the 2000s | The Postal Service β Sleeping In Γ3 | Jimmy Eat World, Bright Eyes, Death Cab for Cutie, Dashboard Confessional, The Shinsβ¦ | |
| | weird punk songs | The Adverts Γ2 | Misfits, Bad Brains, Black Flag, Television, Dead Kennedys, The Undertones | |
|
|
| **Side effect:** where the model knows few songs in a genre, it now drifts to a neighbouring style more often instead of |
| repeating. For example, "banda" gave cumbia and mambo. The edit removes an obstacle; it cannot add songs the model never |
| learned, which matches the weight-space-programming finding below. |
|
|
| ### Why this worked |
| The training data never contained repeats, so the model learned a copy-suppression mechanism. It was simply too quiet, |
| and greedy decoding sometimes let a strongly-recalled song win twice. Turning up a mechanism that already exists is a |
| targeted fix. Switching off neurons that merely *correlate* with repeats damaged everything. |
|
|
| Inspired by: *Neural Networks Are Already Clean* (Cammarata, Goodfire, 2026). The "Evil Qwen" single-neuron reproductions |
| (HedwigAI, 2026), where the most active neurons were not the causal ones, so rank by causal attribution and check random |
| controls. Copy-suppression heads catalogued in neuron-lens (stdlogits). *Sparks of Weight-Space Programming* (Takehi, |
| Teoh, Zhang & Khattab, MIT CSAIL, 2026). Scripts: `repeat_direction.py`, `repeat_attrib.py`, `repeat_edit_test.py`, |
| `repeat_apply.py` in the juke repo. |
|
|
| ## Evaluation (v1.0 `final/`, greedy, 110 prompts; v1.1 numbers above) |
|
|
| | Metric | Result | |
| |---|---| |
| | Song addresses that exist in the catalog | 99.9% | |
| | Song name matches its address | 96.6% | |
| | Address β exact name (300 random catalog songs) | 92.7% | |
| | Name β exact address | 63.3% | |
| | Repeated tracks inside a playlist | 5% | |
| | Songs shared across 40 unrelated prompts | 91% distinct, none in 3+ playlists | |
| | Overlap with the teacher's playlist (trained / unseen prompts) | 7% / 2.5%, so it is not copying | |
|
|
| ## Limitations |
|
|
| - **Catalog:** 229k songs, mostly Western music popular in English-language playlist data (Spotify playlists from the mid-2010s |
| and ListenBrainz playlists up to 2026). Newer and regional music is thin. The model can only name songs it memorized. |
| - **Hallucinations:** about 3% of track lines have a name that doesn't match its address, and these are sometimes |
| half-remembered titles. Check addresses against your own catalog if exactness matters. |
| - **Repeats:** about 2% of tracks in v1.1 (5% in v1.0). De-duplicate (see above). |
| - **Taste:** the model inherits the biases of its playlist data and its teacher model, including over-represented |
| artists and genres. |
| - **Reasons are explanations, not ground truth.** Per-track reasons can stretch to justify a pick. |
|
|
| ## Data sources and licence |
|
|
| The model weights are released under **CC BY-SA 4.0**: you may use, share and adapt them, including commercially, |
| if you give credit and share adaptations under the same licence. The base model, Qwen3-1.7B, is Apache-2.0. |
|
|
| Data used to build the catalog, song addresses and training examples: |
|
|
| | Source | Licence | Used for | |
| |---|---|---| |
| | Spotify playlist dataset (Pichl, Zangerle & Specht; Zenodo 2594557) | CC BY 4.0 | real playlists, co-occurrence | |
| | ListenBrainz playlists, MusicBrainz | CC0 | real playlists, catalog verification | |
| | Discogs data dumps, AcousticBrainz | CC0 | genres and styles | |
| | Wikipedia / Wikidata | CC BY-SA 4.0 / CC0 | genre vocabulary, artist genres | |
| | Last.fm dataset (Million Song Dataset) | research / non-commercial terms | song tags (genre tag task, address grouping) | |
| | Art of the Mix 2011 | research | popularity counts | |
| | GPT-5.4-mini outputs | OpenAI terms | teacher playlists, thinking traces | |
|
|
| β οΈ **Caveat:** Last.fm and Art of the Mix data come with research-use terms. They shaped the tag task and the song |
| grouping, but none of their text is reproduced. If you need a strictly commercial-clean model, the planned fix is to |
| rebuild tags from CC0/CC-BY sources only. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{juke2026, |
| title = {juke: a standalone prompt-to-playlist model with semantic song addresses}, |
| author = {asharrm}, |
| year = {2026}, |
| url = {https://huggingface.co/asharrm/juke-1.7b-dev} |
| } |
| ``` |
|
|
| Related work: Text2Tracks (Palumbo et al., 2025), TIGER / semantic IDs for generative retrieval (Rajput et al., 2023). |
|
|
