| --- |
| language: |
| - en |
| license: mit |
| tags: |
| - llama |
| - agron |
| - code |
| - cybersecurity |
| - domain-pretrained |
| - text-generation |
| pipeline_tag: text-generation |
| --- |
| |
| # Agron (domain-pretrained: programming languages + cybersecurity) |
|
|
| Agron is a ~50.3M-parameter LLaMA-style model. The weights in this repository are |
| the result of **one full epoch** of domain continued pretraining over code, |
| cybersecurity, and WikiText replay, started from the previously published Agron |
| checkpoint. |
|
|
| ## Current promoted weights |
|
|
| | Field | Value | |
| |---|---| |
| | Steps | 7,246 (1 full epoch) | |
| | Tokens | 118.7M | |
| | Batch size | 16 x 2 grad accum = 32 | |
| | Sequence length | 512 | |
| | LR | 3e-4 (cosine, warmup 100) | |
| | Optimizer | AdamW (fused) | |
| | Hardware | 1x Tesla T4 (fp16) | |
| | Training time | ~61 min | |
| | Avg train loss | 1.319 | |
|
|
| Full metrics: [`report.json`](report.json), [`eval_history.json`](eval_history.json), |
| [`eval_before.json`](eval_before.json). |
|
|
| ## Pretraining corpus |
| Mixed on-the-fly from flat token streams: |
| - **Code** (majority): `codeparrot/github-code` filtered to common source extensions (.py, .js, .ts, .java, .c, .cpp, .go, .rs, .cs, .sh, .sql, ...), ~150M chars |
| - **Cybersecurity**: `Trendyol/Trendyol-Cybersecurity-Instruction-Tuning-Dataset` (system/user/assistant), ~120M chars |
| - **Replay**: `Salesforce/wikitext-2-raw-v1`, ~15M chars (prevents catastrophic forgetting) |
|
|
| Corpus built by [`prepare_domain_corpus.py`](https://github.com/Njap-png/Agron/blob/main/prepare_domain_corpus.py). |
|
|
| ## Held-out evaluation |
|
|
| | Source | Before (loss/PPL) | After (loss/PPL) | Δ | |
| |---|---|---|---| |
| | code | 1.453 / 4.28 | 1.245 / 3.47 | -0.208 | |
| | cyber | 1.358 / 3.89 | 1.064 / 2.90 | -0.294 | |
| | replay (wikitext) | 5.626 / 277.6 | 5.024 / 152.0 | -0.602 | |
|
|
|  |
|
|
| ## Usage |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| tok = AutoTokenizer.from_pretrained("teddyoas/Agron") |
| model = AutoModelForCausalLM.from_pretrained("teddyoas/Agron") # promoted 1-epoch weights |
| ``` |
|
|
| ## Files |
| - `model.safetensors`, `config.json`, `generation_config.json` |
| - `tokenizer.json`, `tokenizer_config.json` |
| - `domain_pretraining_curve.png` |
| - `eval_history.json`, `eval_before.json`, `report.json`, `samples.json` |
|
|
