Blame for README.md

---
license: cc-by-sa-4.0
base_model: Qwen/Qwen3-1.7B
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- music
- playlist
- recommendation
- generative-retrieval
- semantic-ids
- qwen3
---

# juke-1.7b: type anything, get a playlist

**juke** is a 1.7B language model that turns any prompt into a playlist of real songs, like *"bossa nova brunch"*,
*"sad midwest emo for a long drive"*, *"yttrium exoplanet"* or *"my cat's tax return"*. It can **think out loud like
a DJ** first, then give a title, a one-line idea, and a numbered tracklist with a short reason for each song.

> **v1.1 (recommended):** repeats inside a playlist cut from 5% to 1.8% by a **weight edit, not retraining**. Four
> "copy-suppression" attention heads were made 2× louder (about 0.03% of the weights). See [What's new in v1.1](#whats-new-in-v11).

It is a **single model**: there is no search index, database or extra code. The model has memorized a catalog of
**229,042 songs**, and each song has a short address it can write, for example `<a_67><b_10><c_0>` = Brian Eno – An Ending.

```
> yttrium exoplanet

<think>
I'm reading "yttrium exoplanet" as a cosmic object with a metallic, distant, slightly surreal feel... ambient,
post-rock, dream pop and spacious electronic pieces... opening with distance, moving through shimmer and gravity,
then landing in a hushed, suspended afterimage.
</think>

Yttrium Orbit
Logic: Spacious, reflective tracks that feel like orbiting a cold planet.
1. <a_67><b_10><c_0> Brian Eno – An Ending — Soft light fading into deep space
2. <a_12><b_1><c_0> Tangerine Dream – Phaedra — Classic cosmic drift and shimmer
3. <a_67><b_68><c_0> Vangelis – Alpha — Bright, floating, otherworldly calm
4. <a_9><b_27><c_0> Boards of Canada – Dayvan Cowboy — Warm distance with a metallic sheen
...
```

## Quick start (🤗 transformers)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import re

repo = "asharrm/juke-1.7b-dev"
tok = AutoTokenizer.from_pretrained(repo, subfolder="v1.1")
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="v1.1", torch_dtype="auto", device_map="auto")

prompt = "sad midwest emo for a long drive"
text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,
                               add_generation_prompt=True, enable_thinking=True)   # False = no DJ monologue
ids = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=700, do_sample=False)
reply = tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=False).replace("<|im_end|>", "")

# hide the song addresses for display
print(re.sub(r"(<[abc]_\d+>){3}\s*", "", reply))
```

**Thinking on or off:** `enable_thinking=True` makes the model write a short plan (60–120 words) and a reason per
track, which suits a "talking DJ". `False` gives just title, logic and tracks. In our tests, thinking on produced
more varied playlists.

**Removing repeats:** about 2% of tracks (v1.1) are repeats of a song already in the same list (see Evaluation). Drop
any line whose `<a_><b_><c_>` address already appeared, or ask for a few extra tracks and de-duplicate.

## llama.cpp / GGUF

A ready Q8_0 file is in this repo: `gguf/juke-1.7b-v1.1-q8_0.gguf` (1.8 GB).

```bash
huggingface-cli download asharrm/juke-1.7b-dev gguf/juke-1.7b-v1.1-q8_0.gguf --local-dir .
llama-server -m gguf/juke-1.7b-v1.1-q8_0.gguf -ngl 99
```

Or convert yourself with llama.cpp's `convert_hf_to_gguf.py` (the 768 song-address tokens convert as normal added tokens):

```bash
python convert_hf_to_gguf.py final/ --outtype q8_0 --outfile juke-1.7b-q8_0.gguf
llama-server -m juke-1.7b-q8_0.gguf -ngl 99
```

Use raw completion with the Qwen3 chat format. For thinking off, add an empty think block:
`<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n`.
A Q8_0 GGUF (1.8 GB) runs at ~146 tokens/s on a PS5 GPU (Vulkan), which is about 1.5 s per playlist.

## Output format

```
<think>…plan…</think>            (thinking mode only)

<Playlist title>
Logic: <one sentence>
1. <a_x><b_y><c_z> Artist – Title — <reason>      (reasons in thinking mode only)
2. …
```

Other trained prompt styles:
- `/tracks <playlist name>` → just a tracklist, in the style of real user playlists
- `/tag shoegaze` → 10 songs carrying that genre/tag
- `<song> <a_x><b_y><c_z>` → `Artist – Title` (catalog lookup)
- `<code> Artist – Title` → `<a_x><b_y><c_z>` (reverse lookup)

## How song addresses work

Every catalog song gets a 3-token address. Songs that sound alike share a prefix:

- `a` = one of 256 broad areas, e.g. shoegaze, bossa nova or modern country (~900 songs each)
- `b` = a small sub-style inside the area (~9 songs each)
- `c` = the song's position inside the sub-style (most popular first)

The addresses come from a balanced k-means tree over song vectors built from **playlist co-occurrence** (which songs
people put together) plus **genre/style tags** and an artist signal. Writing the address first means the model
"walks into the right neighbourhood" before naming the song, the same idea as Spotify's Text2Tracks and other
semantic-ID generative retrieval work.

## Training

- **Base:** Qwen3-1.7B, full fine-tune, 768 new address tokens plus `<song>` and `<code>`.
- **Data:** 579k examples, 39M tokens per pass, five tasks:

  | Task | Examples | What it teaches |
  |---|---|---|
  | Memorization | 458k | address ↔ exact song name, every catalog song, both directions |
  | Teacher playlists | 26.5k | prompt → title, logic, tracks; 10.9k with a thinking plan and per-track reasons |
  | Odd prompts | 2k | absurd and abstract prompts with committed interpretations |
  | Real playlists | 79k | real user playlist names → their tracks |
  | Tags | 14.6k | genre/style tag → songs |

  Teacher playlists, thinking traces and odd-prompt answers were written by GPT-5.4-mini. Every track was checked
  against the catalog, and any song not in it was dropped.
- **Schedule:** 8 rounds. Every round contains all memorization examples plus a sampled share of the other tasks.
  2048-token packed sequences, AdamW, peak learning rate 1e-4 with cosine decay, bf16 autocast with fp32 master
  weights.
- **Compute:** about 4.5 h on one RTX PRO 6000 (96 GB). Total GPU cost for the project was about $10.
- **Versions in this repo:** `v1.1/` (recommended: `final/` plus the head edit below), `final/` (end of round 8), and the
  `pass7/` and `pass6/` checkpoints.

## What's new in v1.1: fixing repeats by editing the weights

v1.0 (`final/`) sometimes listed the same song twice in one playlist. About **5% of tracks were repeats**, e.g.
*Leonard Cohen – Everybody Knows* three times in "dark cabaret heartbreak". The usual fix is more fine-tuning. Instead
we found the circuit responsible and **edited it directly: no gradient steps, no new data, about 0.03% of the weights**.

### 1. Collect the behaviour
Greedy-decode 400 playlists from training prompts that the evaluation does not use. Label every list item as
**repeat** (its 3-token song address already appeared earlier in the list) or **fresh**: 239 repeats and 3,692 fresh.

### 2. Where is "about to repeat" visible? (correlational)
At the position that predicts each item's first address token, take the residual stream at every layer. The
difference of class means gives a direction, scored on a held-out half by Cohen's *d*:

| layers | 1–9 | 10–15 | **16–19** | 20–21 | 22–28 |
|---|---|---|---|---|---|
| Cohen's *d* | 0.6–1.1 | 0.4–0.95 | **1.4–1.6** | 1.05–1.3 | 0.4–0.9 |

The decision is most legible in the middle-late layers. Correlation is not causation, so we went one step further.

### 3. Which units cause it? (attribution patching)
For each list item, target = log p(the item's 3 address tokens). For every MLP neuron (`down_proj` input) and every
attention head (its slice of the `o_proj` input), attribution = activation × ∂target/∂activation, summed over the
3 predicting positions. Then:

  repeat score = mean attribution on **repeated** items − mean attribution on **fresh** items

The largest scores were *negative*, meaning units that push repeats down. These are **copy-suppression heads** that look
back at songs already listed and lower their probability:

| head | repeat score |
|---|---|
| L22 h8 | −0.068 |
| L22 h7 | −0.046 |
| L26 h7 | −0.042 |
| L16 h14 | −0.040 |
| (most repeat-promoting: L15 h6) | +0.028 |

### 4. Test edits causally, against controls
Each candidate edit was applied to a fresh copy of the weights and scored by teacher forcing on held-out playlists
(eval prompts). Δ is the change in mean log-probability of repeated vs. fresh items; we want Δrepeat ≪ Δfresh ≈ 0.

| edit | Δ repeat | Δ fresh | |
|---|---|---|---|
| 8 / 32 / 128 random neurons → 0 (control) | 0.00 / 0.00 / −0.03 | 0.00 / 0.00 / −0.01 | control is flat ✅ |
| top 8 / 32 / 128 repeat-promoting neurons → 0 | −0.33 / −0.91 / −1.90 | −0.27 / −0.65 / −1.49 | hurts everything ❌ |
| remove the probe direction (layers 15–19) | −0.41 | −0.41 | not selective ❌ |
| boost top anti-repeat neurons ×1.5–2 | −0.06 to −0.21 | −0.03 to −0.11 | too weak |
| top 2 / 4 repeat-promoting heads → 0 | −0.49 / −0.82 | −0.26 / −0.53 | partly selective |
| **4 copy-suppression heads ×1.5** | **−0.68** | **−0.25** | selective ✅ |
| **4 copy-suppression heads ×2** | **−1.64** | **−0.60** | **chosen** ✅ |
| 4 copy-suppression heads ×3 | −3.81 | −1.51 | starts to damage |

### 5. The edit
```python
import torch
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("asharrm/juke-1.7b-dev", subfolder="final", torch_dtype=torch.bfloat16)
hd = m.config.head_dim                                   # 128
with torch.no_grad():
    for layer, head in [(22, 8), (22, 7), (26, 7), (16, 14)]:
        m.model.layers[layer].self_attn.o_proj.weight[:, head * hd:(head + 1) * hd] *= 2.0
m.save_pretrained("juke-1.7b-v1.1")
```
That is 4 × 128 columns of `o_proj`: 1.05M of 1.7B weights, about **0.03%**.

### 6. Result (greedy decoding, 110 prompts, real generation)

| metric | v1.0 `final/` | ×1.5 | **v1.1 (×2)** |
|---|---|---|---|
| repeated tracks inside a playlist | 5.0% | 2.9% | **1.8%** |
| song addresses that exist | 99.9% | 100% | 99.9% |
| song name matches its address | 96.6% | 97.1% | **98.0%** |
| distinct songs across 40 unrelated prompts | 90.9% | 91.7% | 93.0% |
| overlap with teacher playlists (trained prompts) | 7.2% | 5.9% | 5.6% |

Before and after:

| prompt | v1.0 | v1.1 |
|---|---|---|
| dark cabaret heartbreak | Leonard Cohen – Everybody Knows ×3 | The Dresden Dolls, Nellie McKay, Cold Cave, no repeats |
| wistful indie rock from the 2000s | The Postal Service – Sleeping In ×3 | Jimmy Eat World, Bright Eyes, Death Cab for Cutie, Dashboard Confessional, The Shins… |
| weird punk songs | The Adverts ×2 | Misfits, Bad Brains, Black Flag, Television, Dead Kennedys, The Undertones |

**Side effect:** where the model knows few songs in a genre, it now drifts to a neighbouring style more often instead of
repeating. For example, "banda" gave cumbia and mambo. The edit removes an obstacle; it cannot add songs the model never
learned, which matches the weight-space-programming finding below.

### Why this worked
The training data never contained repeats, so the model learned a copy-suppression mechanism. It was simply too quiet,
and greedy decoding sometimes let a strongly-recalled song win twice. Turning up a mechanism that already exists is a
targeted fix. Switching off neurons that merely *correlate* with repeats damaged everything.

Inspired by: *Neural Networks Are Already Clean* (Cammarata, Goodfire, 2026). The "Evil Qwen" single-neuron reproductions
(HedwigAI, 2026), where the most active neurons were not the causal ones, so rank by causal attribution and check random
controls. Copy-suppression heads catalogued in neuron-lens (stdlogits). *Sparks of Weight-Space Programming* (Takehi,
Teoh, Zhang & Khattab, MIT CSAIL, 2026). Scripts: `repeat_direction.py`, `repeat_attrib.py`, `repeat_edit_test.py`,
`repeat_apply.py` in the juke repo.

## Evaluation (v1.0 `final/`, greedy, 110 prompts; v1.1 numbers above)

| Metric | Result |
|---|---|
| Song addresses that exist in the catalog | 99.9% |
| Song name matches its address | 96.6% |
| Address → exact name (300 random catalog songs) | 92.7% |
| Name → exact address | 63.3% |
| Repeated tracks inside a playlist | 5% |
| Songs shared across 40 unrelated prompts | 91% distinct, none in 3+ playlists |
| Overlap with the teacher's playlist (trained / unseen prompts) | 7% / 2.5%, so it is not copying |

## Limitations

- **Catalog:** 229k songs, mostly Western music popular in English-language playlist data (Spotify playlists from the mid-2010s
  and ListenBrainz playlists up to 2026). Newer and regional music is thin. The model can only name songs it memorized.
- **Hallucinations:** about 3% of track lines have a name that doesn't match its address, and these are sometimes
  half-remembered titles. Check addresses against your own catalog if exactness matters.
- **Repeats:** about 2% of tracks in v1.1 (5% in v1.0). De-duplicate (see above).
- **Taste:** the model inherits the biases of its playlist data and its teacher model, including over-represented
  artists and genres.
- **Reasons are explanations, not ground truth.** Per-track reasons can stretch to justify a pick.

## Data sources and licence

The model weights are released under **CC BY-SA 4.0**: you may use, share and adapt them, including commercially,
if you give credit and share adaptations under the same licence. The base model, Qwen3-1.7B, is Apache-2.0.

Data used to build the catalog, song addresses and training examples:

| Source | Licence | Used for |
|---|---|---|
| Spotify playlist dataset (Pichl, Zangerle & Specht; Zenodo 2594557) | CC BY 4.0 | real playlists, co-occurrence |
| ListenBrainz playlists, MusicBrainz | CC0 | real playlists, catalog verification |
| Discogs data dumps, AcousticBrainz | CC0 | genres and styles |
| Wikipedia / Wikidata | CC BY-SA 4.0 / CC0 | genre vocabulary, artist genres |
| Last.fm dataset (Million Song Dataset) | research / non-commercial terms | song tags (genre tag task, address grouping) |
| Art of the Mix 2011 | research | popularity counts |
| GPT-5.4-mini outputs | OpenAI terms | teacher playlists, thinking traces |

⚠️ **Caveat:** Last.fm and Art of the Mix data come with research-use terms. They shaped the tag task and the song
grouping, but none of their text is reproduced. If you need a strictly commercial-clean model, the planned fix is to
rebuild tags from CC0/CC-BY sources only.

## Citation

```bibtex
@misc{juke2026,
  title  = {juke: a standalone prompt-to-playlist model with semantic song addresses},
  author = {asharrm},
  year   = {2026},
  url    = {https://huggingface.co/asharrm/juke-1.7b-dev}
}
```

Related work: Text2Tracks (Palumbo et al., 2025), TIGER / semantic IDs for generative retrieval (Rajput et al., 2023).