File size: 4,370 Bytes
b9622a9 e55fc31 b9622a9 5eb8463 b9622a9 c00d3d3 b9622a9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 |
---
license: apache-2.0
language:
- en
tags:
- qwen3
- gast
- causal-lm
- base-model
- text-generation
- gguf
- small-language-model
library_name: transformers
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
- Salesforce/wikitext
---
<div align="center">
<h1 style="font-size: 3rem; margin: 12px 0 6px 0; font-weight: 800; letter-spacing: 1.5px;">Gast-Mini</h1>
<p>
<img src="https://img.shields.io/badge/Parameters-7.07M-blue.svg" alt="Parameters" />
<img src="https://img.shields.io/badge/Context-512_tokens-orange.svg" alt="Context" />
<img src="https://img.shields.io/badge/Tokens-9B-green.svg" alt="Tokens" />
</p>
<p>Base causal language model with 7.07M parameters, trained from scratch.</p>
</div>
---
## Architecture
| Field | Value |
|---|---|
| Parameters | 7.07M |
| Layers | 16 |
| Hidden size | 192 |
| Attention | GQA (3 Query / 1 KV head, head_dim 64) |
| Intermediate size | 512 SwiGLU |
| Norm | RMSNorm (eps 1e-6) + per-head QK-Norm |
| RoPE theta | 10000.0 |
| Context length | 512 |
| Vocab size | 4000 |
| Tokenizer | Custom BPE (byte-level) |
| Embeddings | Tied (`lm_head` == `embed_tokens`) |
| HF Layout | `Qwen3ForCausalLM` |
## Training & Corpus
| Parameter | Value |
|---|---|
| Optimizer | AdamW (beta1=0.9, beta2=0.95, wd=0.1) |
| LR | 2e-3 (flat, 500-step warmup) |
| Batch | 196,608 tokens/step |
| Precision | fp16 |
| Tokens seen | ~9B |
### Pretraining Data
| Source | Tokens | License |
|---|---:|---|
| FineWeb-Edu (sample/10BT) | ~8.85B | ODC-By 1.0 |
| WikiText-103-raw-v1 | ~0.17B | CC BY-SA 4.0 |
| **Total** | **~9B** | |
## Samples
Generated with `llama.cpp`, `temperature=0.85, top_p=0.92, top_k=50, repeat_penalty=1.15`.
> **Prompt:** Artificial intelligence is
>
> Artificial intelligence is the most efficient, and it is difficult to obtain data from human beings. It has no clearly defined or explicit identity, but only a firm assumption of a common law. The Digital Literature Society, Digital Literature Society, ABA, and Reference Library Society have been compiled to address the data collection needs of the digital literacy system. They provide advanced technical assistance in a variety of applications, including on-line communication with computers, and are now referred to as Dig
> **Prompt:** Photosynthesis is the process by which
>
> Photosynthesis is the process by which DNA can transfer a protein to a cell with the same function of its function. The most common function of a cell is to transfer it from one cell to another, and then to use it for a given cell or genome. The key step in creating a healthy organism is to create a bridge between living organisms that can survive. This process involves the formation of new cells in which the cells are made up of new cells. The cell contains only two cells, each one being formed into two sections, with an additional two cells having a different cell shape
> **Prompt:** The theory of relativity states that
>
> The theory of relativity states that the stars are all uniform in shape. For example, the star can be measured by a gravitational force, which is a small amount of mass that can be measured in terms of grams per kilogram. Similarly, the star is used to calculate its mass: This theory states that the grams of the stars are equal to 1,000,000,000 and their masses are equal to 2,500,000,000 and the masses are equal to 12,000,000.
## Usage
### llama.cpp
```bash
./llama-cli -m gast-7m-f16.gguf -p "The history of the Roman Empire" -n 128 --temp 0.85 --top-k 50 --top-p 0.92 --repeat-penalty 1.15
```
### llama-cpp-python
```python
from llama_cpp import Llama
llm = Llama(model_path="gast-7m-f16.gguf", n_ctx=512)
out = llm(
"The history of the Roman Empire",
max_tokens=120,
temperature=0.85,
top_p=0.92,
top_k=50,
repeat_penalty=1.15,
)
print(out["choices"][0]["text"])
```
## Limitations
- English only.
- Base model, not instruction tuned, no chat template.
- Trained on ~9B tokens; not factual, barely coherent beyond a few words.
## License
Apache 2.0.
## Credits
- FineWeb-Edu - Hugging Face (ODC-By 1.0)
- WikiText-103 - Salesforce (CC BY-SA 4.0)
- llama.cpp - GGUF tooling
- Hugging Face
## Citation
```bibtex
@misc{gastmini,
title = {Gast-Mini},
author = {Meridian-MRM},
howpublished = {https://huggingface.co/Meridian-MRM/Gast-Mini}
}
```
|