| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: transformers |
| tags: |
| - chat |
| - mamba |
| - experimental |
| - mamba2 |
| - hybrid |
| - pytorch |
| - slm |
| --- |
| |
|
|
| # Pebble-50M-Chat-beta |
|
|
| Pebble-50M-Chat-beta is the chat-tuned version of Pebble-50M-beta, an experimental 50M-parameter language model designed to test how a larger Pebble architecture performs with a 16,384-token vocabulary and 16,384-token context window. |
|
|
| The base model underperformed Pebble-25M and, on some evaluations, Pebble-10M. Pebble-50M-Chat-beta was subsequently fine-tuned to improve its ability to follow instructions and engage in conversational interactions. |
|
|
| ## Model Details |
|
|
| * **Architecture:** Hybrid Mamba2 / Transformer |
| * **Block Pattern:** 3 Mamba2 blocks : 1 Attention block (repeating) |
| * **Parameters:** ~49,334,448 (50M) |
| * **Hidden Dimension:** 768 |
| * **Layers:** 8 (6 Mamba2, 2 Attention) |
| * **Vocab Size:** 16,384 (Custom Byte-Level BPE) |
| * **Context Length:** 16,384 |
| * **Pretraining Tokens:** ~25,000,000,000 (~25 Billion) |
| * **Fine-tuning Tokens:** ~250,000,000 (~250 Million) |
| * **Fine-tuning Dataset:** Smol-SmolTalk |
| * **Optimizer:** Muon (for 2D hidden weights) + AdamW (for embeddings, norms, and scalars) |
| * **Precision:** fp32 master weights with bf16 autocast |
|
|
| ## Dataset Sources |
|
|
| The base model was trained on a 25B-token subset of the following datasets: |
|
|
| | Dataset | Token Allocation | Share | |
| | ------------- | ----------------: | -------: | |
| | FineWeb-Edu | 7.50 billion | 30% | |
| | DCLM | 5.00 billion | 20% | |
| | Cosmopedia-v2 | 3.75 billion | 15% | |
| | FineMath-4+ | 3.75 billion | 15% | |
| | FinePhrase | 3.00 billion | 12% | |
| | NPset | 2.00 billion | 8% | |
| | **Total** | **25.00 billion** | **100%** | |
|
|
| ## Fine-Tuning |
|
|
| Pebble-50M-Chat-beta was fine-tuned on approximately 250M tokens from **Smol-SmolTalk** to improve conversational ability and instruction following. |
|
|
| ## Benchmarks |
|
|
| The original benchmark logs for the base model were lost, so exact evaluation results are unavailable. |
|
|
| The chat model is primarily intended for conversational use and should not be directly compared with the base model on benchmarks without considering the effects of fine-tuning. |
|
|
| ## Usage |
|
|
| Pebble-50M-Chat-beta does not require the `mamba-ssm` library and is intended to be usable with standard PyTorch-based inference implementations. |
|
|
| It may run on CUDA GPUs, AMD GPUs, Intel GPUs, and CPUs depending on the inference framework and available hardware acceleration. |
|
|
| ## Status |
|
|
| This is a **beta/experimental model**. It is primarily intended for research, experimentation, and conversational use. |
|
|
| ## License |
|
|
| Apache 2.0 |
