| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: text-to-audio |
| tags: |
| - text-to-music |
| - music-generation |
| - encodec |
| - transformer |
| - pytorch |
| --- |
| |
| # Instrumenta |
|
|
| Instrumenta is a small text-to-music model trained from scratch on a single consumer GPU (RTX 4060, 8GB VRAM). It generates short instrumental audio clips from text prompts describing genre, mood, key, tempo, and instrumentation. |
|
|
| This is a hobby/personal project, not a research release or a commercial product. It is shared as-is. |
|
|
| ## Model summary |
|
|
| - **Architecture:** autoregressive Transformer decoder over EnCodec discrete audio tokens, using a delay pattern across codebooks (MusicGen-style) and cross-attention to a frozen T5 text encoder for prompt conditioning. |
| - **Audio tokenizer:** pretrained EnCodec (frozen, used only to tokenize/detokenize audio — not fine-tuned). |
| - **Text encoder:** T5-small (frozen). |
| - **Trainable parameters:** ~30M (the Transformer decoder only; EnCodec and T5 are frozen and not counted). |
| - **Model dimensions:** 448 embedding dim, 8 attention heads, 8 layers. |
| - **Codebooks:** 4 (from EnCodec), 1024 tokens per codebook vocabulary. |
| - **Sample rate:** 24 kHz. |
| - **Output length:** 10-second clips (750 frames at 75 frames/sec). |
| - **Conditioning:** free-text prompt, with classifier-free guidance (CFG) supported at inference time. |
|
|
| ## Training data |
|
|
| Trained on a personal/self-assembled dataset of roughly 10,000 short instrumental audio clips with paired text captions (genre, mood, key, tempo, instrumentation), plus a small held-out validation split. The dataset is not publicly released alongside this model. |
|
|
| Because the dataset is on the small side for training a generative audio model from scratch, expect more variance and less fidelity than larger models trained on much bigger datasets (e.g. MusicGen, Stable Audio). This project intentionally traded dataset/model scale for something trainable end-to-end on a single 8GB GPU. |
|
|
| ## Training procedure |
|
|
| - **Phase 1:** continued training of Instrumenta-1 for another 100~ epochs using a better learning rate and many optimizations to ensure much better results than Instrumenta-1 |
| - **Precision:** mixed precision (FP16 autocast + gradient scaling) for the trainable Transformer; EnCodec and T5 run in their pretrained precision. |
| - **Hardware:** single NVIDIA RTX 4060 (8GB VRAM). |
|
|
| No objective benchmark scores (e.g. FAD, CLAP score) have been computed for this model yet. Any quality claims below are based on informal listening, not a scored evaluation. |
|
|
| ## Files in this repository |
|
|
| | File | Description | |
| |---|---| |
| | `instrumenta_best.pt` | Checkpoint with the lowest validation loss reached during training. | |
| | `instrumenta_last.pt` | Final checkpoint at the end of training (last completed epoch). | |
|
|
| Both checkpoints are full training checkpoints — in addition to model weights, they also contain optimizer/scaler state, epoch/step counters, and the training config, so they can be used to resume training as well as for inference. |
|
|
| To use only the model weights for inference, load the `"model"` key from the checkpoint dictionary. |
|
|
| ## Usage |
|
|
| This repository contains only weights, not a packaged inference pipeline. To generate audio you need the accompanying model code (Transformer architecture, delay-pattern encode/decode, EnCodec + T5 loading) — this model is not yet wrapped for the `transformers`/`diffusers` `pipeline()` API. |
|
|
| ```python |
| import torch |
| |
| checkpoint = torch.load("instrumenta_best.pt", map_location="cpu") |
| model_state_dict = checkpoint["model"] |
| config = checkpoint["config"] # training configuration used for this checkpoint |
| ``` |
|
|
| Load `model_state_dict` into the matching model architecture from the training code, then use it together with the frozen EnCodec and T5-small encoders to condition on a text prompt and autoregressively generate audio tokens, which EnCodec decodes back into a waveform. |
|
|
| ### Example prompts the model was trained on |
|
|
| Prompts describe genre/mood, key, tempo, and instrumentation, e.g.: |
|
|
| > "a punchy dark instrumental arrangement, ambient-style, in D minor, 86 bpm, no vocals" |
|
|
| ## Limitations |
|
|
| - **Small training dataset.** ~10k clips is modest for training a generative audio model from scratch; expect inconsistent quality across prompts, and weaker performance on prompt combinations underrepresented in training. |
| - **Short, fixed output length.** Generates fixed 10-second clips; no long-form structure (intro/verse/chorus) is modeled. |
| - **No formal evaluation.** No FAD/CLAP/KL scores or human evaluation have been run. Quality assessment so far is informal listening only. |
| - **English-language prompts only** (T5-small was used as-is, not fine-tuned on non-English captions). |
| - **Not intended for production or commercial use** without further evaluation. |
|
|
| ## License |
|
|
| Apache 2.0. Note that this model was built using pretrained EnCodec and T5-small as frozen components at both training and inference time — check their respective licenses if redistributing a pipeline that includes them. |
|
|
| ## Acknowledgements |
|
|
| Built on top of pretrained [EnCodec](https://huggingface.co/facebook/encodec_24khz) (audio tokenizer) and [T5-small](https://huggingface.co/google-t5/t5-small) (text encoder), both used frozen. Architecture inspired by MusicGen-style delay-pattern autoregressive audio generation. |
