| --- |
| license: apache-2.0 |
| tags: |
| - audio |
| - speech |
| - language-model |
| - auristream |
| library_name: transformers |
| --- |
| |
| # AuriStream - Speech Language Model |
|
|
| **AuriStream** is a speech language model by **Greta Tuckute** and **Klemen Kotar**. |
|
|
| This repository contains the shared model code for AuriStream models. |
|
|
| ## Overview |
|
|
| AuriStream is a GPT-like transformer model for cochlear token prediction with optional |
| multi-token prediction (MTP) heads. |
|
|
| This model predicts cochlear tokens from a tokenizer such as [WavCochCausalV8192](https://huggingface.co/TuKoResearch/WavCochCausalV8192). |
|
|
| ## Usage |
|
|
| This repository is not meant to be used directly. Instead, use one of the checkpoint |
| repositories that reference this base code: |
|
|
| - [AuriStream7B_40Pred_BigAudioDataset_500k](https://huggingface.co/TuKoResearch/AuriStream7B_40Pred_BigAudioDataset_500k) |
|
|
| To load a checkpoint: |
|
|
| ```python |
| from transformers import AutoModel, AutoConfig |
| |
| model = AutoModel.from_pretrained( |
| "TuKoResearch/AuriStream7B_40Pred_BigAudioDataset_500k", |
| trust_remote_code=True, |
| ) |
| ``` |
|
|
| ## Model Architecture |
|
|
| The AuriStream model includes: |
| - RMSNorm for layer normalization |
| - Rotary Position Embeddings (RoPE) |
| - SiLU activation in MLP layers |
| - Multi-token prediction heads |
|
|
| ## Configuration Options |
|
|
| | Parameter | Description | Default | |
| |-----------|-------------|---------| |
| | `vocab_size` | Number of cochlear tokens | 8192 | |
| | `n_embd` | Hidden dimension | 768 | |
| | `n_layer` | Number of transformer layers | 12 | |
| | `n_head` | Number of attention heads | 12 | |
| | `n_pred_steps` | Number of prediction steps (MTP) | 1 | |
|
|
| ## Files |
|
|
| - `configuration_auristream.py` - Configuration class |
| - `modeling_auristream.py` - Model implementation |
|
|
| ## Tokenizer |
|
|
| This model uses cochlear tokens from [WavCochCausalV8192](https://huggingface.co/TuKoResearch/WavCochCausalV8192). |
|
|
