File size: 2,351 Bytes
3a25f23 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | ---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- pebble
- base-model
- mamba
- mamba2
- hybrid
- pytorch
- custom-architecture
---
# Pebble-50M-beta
Pebble-50M-beta is an experimental 50M-parameter language model designed to test how a larger Pebble architecture performs with a 16,384-token vocabulary and 16,384-token context window.
Despite having roughly twice the parameters of Pebble-25M, Pebble-50M-beta underperformed Pebble-25M and, on some evaluations, Pebble-10M. This model is therefore primarily useful as an experimental result rather than as the strongest Pebble model.
## Model Details
* **Architecture:** Hybrid Mamba2 / Transformer
* **Block Pattern:** 3 Mamba2 blocks : 1 Attention block (repeating)
* **Parameters:** ~49,334,448 (50M)
* **Hidden Dimension:** 768
* **Layers:** 8 (6 Mamba2, 2 Attention)
* **Vocab Size:** 16,384 (Custom Byte-Level BPE)
* **Context Length:** 16,384
* **Training Tokens:** ~25,000,000,000 (~25 Billion)
* **Optimizer:** Muon (for 2D hidden weights) + AdamW (for embeddings, norms, and scalars)
* **Precision:** fp32 master weights with bf16 autocast
## Dataset Sources
The model was trained on a 25B-token subset of the following datasets:
| Dataset | Token Allocation | Share |
| ------------- | ----------------: | -------: |
| FineWeb-Edu | 7.50 billion | 30% |
| DCLM | 5.00 billion | 20% |
| Cosmopedia-v2 | 3.75 billion | 15% |
| FineMath-4+ | 3.75 billion | 15% |
| FinePhrase | 3.00 billion | 12% |
| NPset | 2.00 billion | 8% |
| **Total** | **25.00 billion** | **100%** |
## Benchmarks
The original benchmark logs for this model were lost, so exact evaluation results are unavailable.
Qualitatively, Pebble-50M-beta underperformed Pebble-25M and, on some evaluations, Pebble-10M.
## Usage
Pebble-50M-beta does not require the `mamba-ssm` library and is intended to be usable with standard PyTorch-based inference implementations.
It may run on CUDA GPUs, AMD GPUs, Intel GPUs, and CPUs depending on the inference framework and available hardware acceleration.
## Status
This is a **beta/experimental model**. It is primarily intended for research and experimentation with the Pebble architecture.
## License
Apache 2.0 |