--- license: cc-by-sa-4.0 base_model: Qwen/Qwen3-1.7B library_name: transformers pipeline_tag: text-generation language: - en tags: - music - playlist - recommendation - generative-retrieval - semantic-ids - qwen3 --- # juke-1.7b: type anything, get a playlist **juke** is a 1.7B language model that turns any prompt into a playlist of real songs, like *"bossa nova brunch"*, *"sad midwest emo for a long drive"*, *"yttrium exoplanet"* or *"my cat's tax return"*. It can **think out loud like a DJ** first, then give a title, a one-line idea, and a numbered tracklist with a short reason for each song. > **v1.1 (recommended):** repeats inside a playlist cut from 5% to 1.8% by a **weight edit, not retraining**. Four > "copy-suppression" attention heads were made 2× louder (about 0.03% of the weights). See [What's new in v1.1](#whats-new-in-v11). It is a **single model**: there is no search index, database or extra code. The model has memorized a catalog of **229,042 songs**, and each song has a short address it can write, for example `` = Brian Eno – An Ending. ``` > yttrium exoplanet I'm reading "yttrium exoplanet" as a cosmic object with a metallic, distant, slightly surreal feel... ambient, post-rock, dream pop and spacious electronic pieces... opening with distance, moving through shimmer and gravity, then landing in a hushed, suspended afterimage. Yttrium Orbit Logic: Spacious, reflective tracks that feel like orbiting a cold planet. 1. Brian Eno – An Ending — Soft light fading into deep space 2. Tangerine Dream – Phaedra — Classic cosmic drift and shimmer 3. Vangelis – Alpha — Bright, floating, otherworldly calm 4. Boards of Canada – Dayvan Cowboy — Warm distance with a metallic sheen ... ``` ## Quick start (🤗 transformers) ```python from transformers import AutoModelForCausalLM, AutoTokenizer import re repo = "asharrm/juke-1.7b-dev" tok = AutoTokenizer.from_pretrained(repo, subfolder="v1.1") model = AutoModelForCausalLM.from_pretrained(repo, subfolder="v1.1", torch_dtype="auto", device_map="auto") prompt = "sad midwest emo for a long drive" text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True, enable_thinking=True) # False = no DJ monologue ids = tok(text, return_tensors="pt").to(model.device) out = model.generate(**ids, max_new_tokens=700, do_sample=False) reply = tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=False).replace("<|im_end|>", "") # hide the song addresses for display print(re.sub(r"(<[abc]_\d+>){3}\s*", "", reply)) ``` **Thinking on or off:** `enable_thinking=True` makes the model write a short plan (60–120 words) and a reason per track, which suits a "talking DJ". `False` gives just title, logic and tracks. In our tests, thinking on produced more varied playlists. **Removing repeats:** about 2% of tracks (v1.1) are repeats of a song already in the same list (see Evaluation). Drop any line whose `` address already appeared, or ask for a few extra tracks and de-duplicate. ## llama.cpp / GGUF A ready Q8_0 file is in this repo: `gguf/juke-1.7b-v1.1-q8_0.gguf` (1.8 GB). ```bash huggingface-cli download asharrm/juke-1.7b-dev gguf/juke-1.7b-v1.1-q8_0.gguf --local-dir . llama-server -m gguf/juke-1.7b-v1.1-q8_0.gguf -ngl 99 ``` Or convert yourself with llama.cpp's `convert_hf_to_gguf.py` (the 768 song-address tokens convert as normal added tokens): ```bash python convert_hf_to_gguf.py final/ --outtype q8_0 --outfile juke-1.7b-q8_0.gguf llama-server -m juke-1.7b-q8_0.gguf -ngl 99 ``` Use raw completion with the Qwen3 chat format. For thinking off, add an empty think block: `<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n\n\n\n\n`. A Q8_0 GGUF (1.8 GB) runs at ~146 tokens/s on a PS5 GPU (Vulkan), which is about 1.5 s per playlist. ## Output format ``` …plan… (thinking mode only) Logic: 1. Artist – Title — (reasons in thinking mode only) 2. … ``` Other trained prompt styles: - `/tracks ` → just a tracklist, in the style of real user playlists - `/tag shoegaze` → 10 songs carrying that genre/tag - ` ` → `Artist – Title` (catalog lookup) - ` Artist – Title` → `` (reverse lookup) ## How song addresses work Every catalog song gets a 3-token address. Songs that sound alike share a prefix: - `a` = one of 256 broad areas, e.g. shoegaze, bossa nova or modern country (~900 songs each) - `b` = a small sub-style inside the area (~9 songs each) - `c` = the song's position inside the sub-style (most popular first) The addresses come from a balanced k-means tree over song vectors built from **playlist co-occurrence** (which songs people put together) plus **genre/style tags** and an artist signal. Writing the address first means the model "walks into the right neighbourhood" before naming the song, the same idea as Spotify's Text2Tracks and other semantic-ID generative retrieval work. ## Training - **Base:** Qwen3-1.7B, full fine-tune, 768 new address tokens plus `` and ``. - **Data:** 579k examples, 39M tokens per pass, five tasks: | Task | Examples | What it teaches | |---|---|---| | Memorization | 458k | address ↔ exact song name, every catalog song, both directions | | Teacher playlists | 26.5k | prompt → title, logic, tracks; 10.9k with a thinking plan and per-track reasons | | Odd prompts | 2k | absurd and abstract prompts with committed interpretations | | Real playlists | 79k | real user playlist names → their tracks | | Tags | 14.6k | genre/style tag → songs | Teacher playlists, thinking traces and odd-prompt answers were written by GPT-5.4-mini. Every track was checked against the catalog, and any song not in it was dropped. - **Schedule:** 8 rounds. Every round contains all memorization examples plus a sampled share of the other tasks. 2048-token packed sequences, AdamW, peak learning rate 1e-4 with cosine decay, bf16 autocast with fp32 master weights. - **Compute:** about 4.5 h on one RTX PRO 6000 (96 GB). Total GPU cost for the project was about $10. - **Versions in this repo:** `v1.1/` (recommended: `final/` plus the head edit below), `final/` (end of round 8), and the `pass7/` and `pass6/` checkpoints. ## What's new in v1.1: fixing repeats by editing the weights v1.0 (`final/`) sometimes listed the same song twice in one playlist. About **5% of tracks were repeats**, e.g. *Leonard Cohen – Everybody Knows* three times in "dark cabaret heartbreak". The usual fix is more fine-tuning. Instead we found the circuit responsible and **edited it directly: no gradient steps, no new data, about 0.03% of the weights**. ### 1. Collect the behaviour Greedy-decode 400 playlists from training prompts that the evaluation does not use. Label every list item as **repeat** (its 3-token song address already appeared earlier in the list) or **fresh**: 239 repeats and 3,692 fresh. ### 2. Where is "about to repeat" visible? (correlational) At the position that predicts each item's first address token, take the residual stream at every layer. The difference of class means gives a direction, scored on a held-out half by Cohen's *d*: | layers | 1–9 | 10–15 | **16–19** | 20–21 | 22–28 | |---|---|---|---|---|---| | Cohen's *d* | 0.6–1.1 | 0.4–0.95 | **1.4–1.6** | 1.05–1.3 | 0.4–0.9 | The decision is most legible in the middle-late layers. Correlation is not causation, so we went one step further. ### 3. Which units cause it? (attribution patching) For each list item, target = log p(the item's 3 address tokens). For every MLP neuron (`down_proj` input) and every attention head (its slice of the `o_proj` input), attribution = activation × ∂target/∂activation, summed over the 3 predicting positions. Then: repeat score = mean attribution on **repeated** items − mean attribution on **fresh** items The largest scores were *negative*, meaning units that push repeats down. These are **copy-suppression heads** that look back at songs already listed and lower their probability: | head | repeat score | |---|---| | L22 h8 | −0.068 | | L22 h7 | −0.046 | | L26 h7 | −0.042 | | L16 h14 | −0.040 | | (most repeat-promoting: L15 h6) | +0.028 | ### 4. Test edits causally, against controls Each candidate edit was applied to a fresh copy of the weights and scored by teacher forcing on held-out playlists (eval prompts). Δ is the change in mean log-probability of repeated vs. fresh items; we want Δrepeat ≪ Δfresh ≈ 0. | edit | Δ repeat | Δ fresh | | |---|---|---|---| | 8 / 32 / 128 random neurons → 0 (control) | 0.00 / 0.00 / −0.03 | 0.00 / 0.00 / −0.01 | control is flat ✅ | | top 8 / 32 / 128 repeat-promoting neurons → 0 | −0.33 / −0.91 / −1.90 | −0.27 / −0.65 / −1.49 | hurts everything ❌ | | remove the probe direction (layers 15–19) | −0.41 | −0.41 | not selective ❌ | | boost top anti-repeat neurons ×1.5–2 | −0.06 to −0.21 | −0.03 to −0.11 | too weak | | top 2 / 4 repeat-promoting heads → 0 | −0.49 / −0.82 | −0.26 / −0.53 | partly selective | | **4 copy-suppression heads ×1.5** | **−0.68** | **−0.25** | selective ✅ | | **4 copy-suppression heads ×2** | **−1.64** | **−0.60** | **chosen** ✅ | | 4 copy-suppression heads ×3 | −3.81 | −1.51 | starts to damage | ### 5. The edit ```python import torch from transformers import AutoModelForCausalLM m = AutoModelForCausalLM.from_pretrained("asharrm/juke-1.7b-dev", subfolder="final", torch_dtype=torch.bfloat16) hd = m.config.head_dim # 128 with torch.no_grad(): for layer, head in [(22, 8), (22, 7), (26, 7), (16, 14)]: m.model.layers[layer].self_attn.o_proj.weight[:, head * hd:(head + 1) * hd] *= 2.0 m.save_pretrained("juke-1.7b-v1.1") ``` That is 4 × 128 columns of `o_proj`: 1.05M of 1.7B weights, about **0.03%**. ### 6. Result (greedy decoding, 110 prompts, real generation) | metric | v1.0 `final/` | ×1.5 | **v1.1 (×2)** | |---|---|---|---| | repeated tracks inside a playlist | 5.0% | 2.9% | **1.8%** | | song addresses that exist | 99.9% | 100% | 99.9% | | song name matches its address | 96.6% | 97.1% | **98.0%** | | distinct songs across 40 unrelated prompts | 90.9% | 91.7% | 93.0% | | overlap with teacher playlists (trained prompts) | 7.2% | 5.9% | 5.6% | Before and after: | prompt | v1.0 | v1.1 | |---|---|---| | dark cabaret heartbreak | Leonard Cohen – Everybody Knows ×3 | The Dresden Dolls, Nellie McKay, Cold Cave, no repeats | | wistful indie rock from the 2000s | The Postal Service – Sleeping In ×3 | Jimmy Eat World, Bright Eyes, Death Cab for Cutie, Dashboard Confessional, The Shins… | | weird punk songs | The Adverts ×2 | Misfits, Bad Brains, Black Flag, Television, Dead Kennedys, The Undertones | **Side effect:** where the model knows few songs in a genre, it now drifts to a neighbouring style more often instead of repeating. For example, "banda" gave cumbia and mambo. The edit removes an obstacle; it cannot add songs the model never learned, which matches the weight-space-programming finding below. ### Why this worked The training data never contained repeats, so the model learned a copy-suppression mechanism. It was simply too quiet, and greedy decoding sometimes let a strongly-recalled song win twice. Turning up a mechanism that already exists is a targeted fix. Switching off neurons that merely *correlate* with repeats damaged everything. Inspired by: *Neural Networks Are Already Clean* (Cammarata, Goodfire, 2026). The "Evil Qwen" single-neuron reproductions (HedwigAI, 2026), where the most active neurons were not the causal ones, so rank by causal attribution and check random controls. Copy-suppression heads catalogued in neuron-lens (stdlogits). *Sparks of Weight-Space Programming* (Takehi, Teoh, Zhang & Khattab, MIT CSAIL, 2026). Scripts: `repeat_direction.py`, `repeat_attrib.py`, `repeat_edit_test.py`, `repeat_apply.py` in the juke repo. ## Evaluation (v1.0 `final/`, greedy, 110 prompts; v1.1 numbers above) | Metric | Result | |---|---| | Song addresses that exist in the catalog | 99.9% | | Song name matches its address | 96.6% | | Address → exact name (300 random catalog songs) | 92.7% | | Name → exact address | 63.3% | | Repeated tracks inside a playlist | 5% | | Songs shared across 40 unrelated prompts | 91% distinct, none in 3+ playlists | | Overlap with the teacher's playlist (trained / unseen prompts) | 7% / 2.5%, so it is not copying | ## Limitations - **Catalog:** 229k songs, mostly Western music popular in English-language playlist data (Spotify playlists from the mid-2010s and ListenBrainz playlists up to 2026). Newer and regional music is thin. The model can only name songs it memorized. - **Hallucinations:** about 3% of track lines have a name that doesn't match its address, and these are sometimes half-remembered titles. Check addresses against your own catalog if exactness matters. - **Repeats:** about 2% of tracks in v1.1 (5% in v1.0). De-duplicate (see above). - **Taste:** the model inherits the biases of its playlist data and its teacher model, including over-represented artists and genres. - **Reasons are explanations, not ground truth.** Per-track reasons can stretch to justify a pick. ## Data sources and licence The model weights are released under **CC BY-SA 4.0**: you may use, share and adapt them, including commercially, if you give credit and share adaptations under the same licence. The base model, Qwen3-1.7B, is Apache-2.0. Data used to build the catalog, song addresses and training examples: | Source | Licence | Used for | |---|---|---| | Spotify playlist dataset (Pichl, Zangerle & Specht; Zenodo 2594557) | CC BY 4.0 | real playlists, co-occurrence | | ListenBrainz playlists, MusicBrainz | CC0 | real playlists, catalog verification | | Discogs data dumps, AcousticBrainz | CC0 | genres and styles | | Wikipedia / Wikidata | CC BY-SA 4.0 / CC0 | genre vocabulary, artist genres | | Last.fm dataset (Million Song Dataset) | research / non-commercial terms | song tags (genre tag task, address grouping) | | Art of the Mix 2011 | research | popularity counts | | GPT-5.4-mini outputs | OpenAI terms | teacher playlists, thinking traces | ⚠️ **Caveat:** Last.fm and Art of the Mix data come with research-use terms. They shaped the tag task and the song grouping, but none of their text is reproduced. If you need a strictly commercial-clean model, the planned fix is to rebuild tags from CC0/CC-BY sources only. ## Citation ```bibtex @misc{juke2026, title = {juke: a standalone prompt-to-playlist model with semantic song addresses}, author = {asharrm}, year = {2026}, url = {https://huggingface.co/asharrm/juke-1.7b-dev} } ``` Related work: Text2Tracks (Palumbo et al., 2025), TIGER / semantic IDs for generative retrieval (Rajput et al., 2023).