AuriStream100M_40Pred_BigAudioDataset_500k-randinit

AuriStream is a speech language model by Greta Tuckute and Klemen Kotar.

This model predicts cochlear tokens from a tokenizer such as WavCochCausalV8192.

Native training step-zero initialization for the 100M 40-prediction model. This exactly uses origin seed 11101994 and historical source commit c3ef0ac189b893a321158d6df554eca3148ccd76, matching the start of W&B run 5x3xfsf4. The weights are untrained FP32 values produced before XLA/FSDP wrapping.

Model Details

Parameter Value
Parameters ~0.34B
Layers 12
Hidden Size 768
Attention Heads 12
Vocab Size 8192
Prediction Steps 40

Usage

from transformers import AutoModel, AutoConfig

# Load with trust_remote_code for custom model
model = AutoModel.from_pretrained(
    "TuKoResearch/AuriStream100M_40Pred_BigAudioDataset_500k-randinit",
    trust_remote_code=True,
)

# Or load config first
config = AutoConfig.from_pretrained("TuKoResearch/AuriStream100M_40Pred_BigAudioDataset_500k-randinit", trust_remote_code=True)

Base Model Code

This checkpoint uses shared model code from TuKoResearch/AuriStream-base.

Tokenizer

This model uses cochlear tokens from WavCochCausalV8192.

Downloads last month
29
Safetensors
Model size
0.3B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support