File size: 1,827 Bytes
95cb500 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | ---
license: apache-2.0
language:
- code
- en
pipeline_tag: text-generation
tags:
- causal-lm
- code
- custom-code
- partially-looped
- n-gram
---
# BananaMind 2.1 MiniCoder
BananaMind 2.1 MiniCoder is an under-25M-parameter code language model.
- 24,949,999 total parameters
- 19,950,029 Transformer parameters
- 4,999,970 parameters in causal bigram/four-gram memory
- BananaMind 2 Nano's 8,192-token tokenizer
- 13 physical layers and 18 effective passes
- execution: `L1 → L2 → L3 → L4 → L5 → L6 → L7 → L8 → L9 → L5 → L6 → L7 → L8 → L9 → L10 → L11 → L12 → L13`
- the complete L5-L9 middle stack shares weights across its two passes
- 4,096-token context
The n-gram module has independent 51,699-entry bigram and four-gram hash
tables, each with 48-dimensional values. Their concatenated representation is
projected to the 384-wide residual stream. It is injected through separate
learned gates at the beginning of both middle-stack passes.
## Training data
The exact 30B-token streamed mixture is:
| Source | Tokens | Share |
|---|---:|---:|
| [The Stack v3 train](https://huggingface.co/datasets/HuggingFaceCode/stack-v3-train) | 22.5B | 75% |
| [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | 7.5B | 25% |
Stack v3 is streamed as repository-ordered source files. Vendored files are
skipped, while repository path, file path, and detected language are included
in the training text. FineWeb-Edu supplies prose, naming, comments, and general
language knowledge.
Checkpoints are uploaded every 5% with safetensors, tokenizer files, metrics,
pinned dataset revisions, exact source-token accounting, and optimizer state.
## Launch
```bash
./launch_training_hf_job.sh 4 fresh
./launch_training_hf_job.sh 4 resume
./launch_training_hf_job.sh 8 resume
```
|