---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- pebble
- base-model
- mamba
- mamba2
- hybrid
- pytorch
- custom-architecture
---

# Pebble-50M-beta

Pebble-50M-beta is an experimental 50M-parameter language model designed to test how a larger Pebble architecture performs with a 16,384-token vocabulary and 16,384-token context window.

Despite having roughly twice the parameters of Pebble-25M, Pebble-50M-beta underperformed Pebble-25M and, on some evaluations, Pebble-10M. This model is therefore primarily useful as an experimental result rather than as the strongest Pebble model.

## Model Details

* **Architecture:** Hybrid Mamba2 / Transformer
* **Block Pattern:** 3 Mamba2 blocks : 1 Attention block (repeating)
* **Parameters:** ~49,334,448 (50M)
* **Hidden Dimension:** 768
* **Layers:** 8 (6 Mamba2, 2 Attention)
* **Vocab Size:** 16,384 (Custom Byte-Level BPE)
* **Context Length:** 16,384
* **Training Tokens:** ~25,000,000,000 (~25 Billion)
* **Optimizer:** Muon (for 2D hidden weights) + AdamW (for embeddings, norms, and scalars)
* **Precision:** fp32 master weights with bf16 autocast

## Dataset Sources

The model was trained on a 25B-token subset of the following datasets:

| Dataset       |  Token Allocation |    Share |
| ------------- | ----------------: | -------: |
| FineWeb-Edu   |      7.50 billion |      30% |
| DCLM          |      5.00 billion |      20% |
| Cosmopedia-v2 |      3.75 billion |      15% |
| FineMath-4+   |      3.75 billion |      15% |
| FinePhrase    |      3.00 billion |      12% |
| NPset         |      2.00 billion |       8% |
| **Total**     | **25.00 billion** | **100%** |

## Benchmarks

The original benchmark logs for this model were lost, so exact evaluation results are unavailable.

Qualitatively, Pebble-50M-beta underperformed Pebble-25M and, on some evaluations, Pebble-10M.

## Usage

Pebble-50M-beta does not require the `mamba-ssm` library and is intended to be usable with standard PyTorch-based inference implementations.

It may run on CUDA GPUs, AMD GPUs, Intel GPUs, and CPUs depending on the inference framework and available hardware acceleration.

## Status

This is a **beta/experimental model**. It is primarily intended for research and experimentation with the Pebble architecture.

## License

Apache 2.0