Hoglet-33 commited on
Commit
3a25f23
·
verified ·
1 Parent(s): aea62d0

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +68 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ pipeline_tag: text-generation
6
+ library_name: transformers
7
+ tags:
8
+ - pebble
9
+ - base-model
10
+ - mamba
11
+ - mamba2
12
+ - hybrid
13
+ - pytorch
14
+ - custom-architecture
15
+ ---
16
+
17
+ # Pebble-50M-beta
18
+
19
+ Pebble-50M-beta is an experimental 50M-parameter language model designed to test how a larger Pebble architecture performs with a 16,384-token vocabulary and 16,384-token context window.
20
+
21
+ Despite having roughly twice the parameters of Pebble-25M, Pebble-50M-beta underperformed Pebble-25M and, on some evaluations, Pebble-10M. This model is therefore primarily useful as an experimental result rather than as the strongest Pebble model.
22
+
23
+ ## Model Details
24
+
25
+ * **Architecture:** Hybrid Mamba2 / Transformer
26
+ * **Block Pattern:** 3 Mamba2 blocks : 1 Attention block (repeating)
27
+ * **Parameters:** ~49,334,448 (50M)
28
+ * **Hidden Dimension:** 768
29
+ * **Layers:** 8 (6 Mamba2, 2 Attention)
30
+ * **Vocab Size:** 16,384 (Custom Byte-Level BPE)
31
+ * **Context Length:** 16,384
32
+ * **Training Tokens:** ~25,000,000,000 (~25 Billion)
33
+ * **Optimizer:** Muon (for 2D hidden weights) + AdamW (for embeddings, norms, and scalars)
34
+ * **Precision:** fp32 master weights with bf16 autocast
35
+
36
+ ## Dataset Sources
37
+
38
+ The model was trained on a 25B-token subset of the following datasets:
39
+
40
+ | Dataset | Token Allocation | Share |
41
+ | ------------- | ----------------: | -------: |
42
+ | FineWeb-Edu | 7.50 billion | 30% |
43
+ | DCLM | 5.00 billion | 20% |
44
+ | Cosmopedia-v2 | 3.75 billion | 15% |
45
+ | FineMath-4+ | 3.75 billion | 15% |
46
+ | FinePhrase | 3.00 billion | 12% |
47
+ | NPset | 2.00 billion | 8% |
48
+ | **Total** | **25.00 billion** | **100%** |
49
+
50
+ ## Benchmarks
51
+
52
+ The original benchmark logs for this model were lost, so exact evaluation results are unavailable.
53
+
54
+ Qualitatively, Pebble-50M-beta underperformed Pebble-25M and, on some evaluations, Pebble-10M.
55
+
56
+ ## Usage
57
+
58
+ Pebble-50M-beta does not require the `mamba-ssm` library and is intended to be usable with standard PyTorch-based inference implementations.
59
+
60
+ It may run on CUDA GPUs, AMD GPUs, Intel GPUs, and CPUs depending on the inference framework and available hardware acceleration.
61
+
62
+ ## Status
63
+
64
+ This is a **beta/experimental model**. It is primarily intended for research and experimentation with the Pebble architecture.
65
+
66
+ ## License
67
+
68
+ Apache 2.0