| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - slmkit |
| - llama |
| - from-scratch |
| - abc-music |
| - music |
| - abc-notation |
| - folk-music |
| - symbolic-music |
| --- |
| |
| # abc-folk:3 |
|
|
| A 864,256-parameter Llama-style language model trained from scratch with |
| [slmkit](https://github.com/m87techlabs/slmkit) for the `abc_music` project. It is a **fine-tuned (SFT) model**: it answers requests in the format it was fine-tuned on. |
|
|
| ## How to prompt it |
|
|
| Generation starts from the prompt and stops at `<eos>` (token ID 1). An example prompt, exactly |
| as the evaluation used it: |
|
|
| ``` |
| % a reel in D major, 2/2 time |
| ``` |
|
|
| Serve it and ask over HTTP. A copy downloaded from the Hugging Face Hub goes in |
| `$SLM_HOME/models/abc-folk/3/` first; that is where `slm serve` looks for `abc-folk:3`. |
|
|
| ```bash |
| uv run slm serve --model abc-folk:3 |
| curl -s localhost:8000/generate -H 'content-type: application/json' \ |
| -d '{"prompt": "% a reel in D major, 2/2 time\n", "seed": 0}' |
| ``` |
|
|
| Or load it with Hugging Face `transformers`; the directory is a standard `LlamaForCausalLM` |
| checkpoint: |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| path = "$SLM_HOME/models/abc-folk/3" # or its Hugging Face Hub repo ID, if it was published there |
| tok = AutoTokenizer.from_pretrained(path) |
| model = AutoModelForCausalLM.from_pretrained(path) |
| ``` |
|
|
| ## Model |
|
|
| | | | |
| |---|---| |
| | architecture | `LlamaForCausalLM` (RMSNorm, RoPE, SwiGLU, tied embeddings) | |
| | parameters | 864,256 (853,120 non-embedding) | |
| | vocabulary | 87 tokens | |
| | context | 512 tokens | |
| | weights | `model.safetensors`, fp32, 3.5 MB | |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | run | `run-9b890db6dced` (abc-baseline-sft), checkpoint `best` at step 1,000 | |
| | tokens seen (this stage) | 16,384,000 | |
| | validation loss | 1.1092 | |
| | GPU-hours (this stage) | 0.006 | |
| | tokenizer | `tk-1030590a89ca` | |
| | code | slmkit `c34aed6` | |
|
|
| Full lineage back to the raw data: `uv run slm lineage abc-folk:3`. |
|
|
| ## Evaluation |
|
|
| From eval report `ev-0ddc32e2a9`: 3 sampling seeds × 200 samples, prompted with the project's **sft** prompts. Mean ± standard deviation across seeds. |
|
|
| | metric | score | |
| |---|---| |
| | ended | 0.982 ± 0.006 | |
| | length | 259.345 ± 4.707 | |
| | plays | 0.730 ± 0.048 | |
| | bar_accuracy | 0.851 ± 0.023 | |
| | ends_on_tonic | 0.340 ± 0.026 | |
| | novelty | 0.998 ± 0.001 | |
| |
| Metric definitions: the project's README and `docs/concepts/evaluation.md`. |
| |
| ## Verification |
| |
| Checked before it was published, over 94 tokens: slmkit reloads the files with max |
| |Δlogit| = 0.0e+00 (identical). `transformers` 5.17.0 loads them with max |Δlogit| = 0.0e+00 and identical token IDs. |
| |
| ## Data and licence |
| |
| Trained on 4,248 tunes in ABC notation from three **public-domain** tune books: Ryan's Mammoth |
| Collection (1883), O'Neill's Music of Ireland (1903) and Aird's Selection of Airs (1778–1803), as |
| transcribed by volunteers and shipped in the [`music21`](https://github.com/cuthbertLab/music21) |
| package (BSD-3-Clause). |
| |
| **One open question, stated plainly.** The printed books are public domain, but the transcription |
| files carry no licence of their own. Faithful transcriptions of public-domain scores are generally |
| not treated as new creative work, but that is an assumption, not a grant. If a transcriber or |
| rights-holder objects, the collection is removed, the model retrained, and this version withdrawn. |
| The Session's data was deliberately *not* used: its licence forbids training language models. |
| |
| The weights are released under Apache-2.0, the same licence as slmkit. |
| |
| More detail: [`projects/abc_music/README.md`](https://github.com/m87techlabs/slmkit/blob/main/projects/abc_music/README.md) in the slmkit repository. |
| |
| ## Limitations |
| |
| - **Context of 512 tokens.** slmkit's sampler keeps only the last 512 tokens as |
| context; positions beyond that were never trained. |
| - **Fixed vocabulary of 87 tokens.** Text the tokenizer has no token for becomes |
| `<unk>`, which the model learned nothing about. `slm serve` reports how many prompt tokens that |
| affected. |
| - **Small and narrow.** It knows only its training domain. The scores above are averages; |
| individual samples can be wrong, so check them with the project's graders. |
| |
