File size: 2,254 Bytes
6300644 74c9056 6300644 74c9056 2d2a140 74c9056 2d2a140 74c9056 2d2a140 74c9056 2d2a140 74c9056 2d2a140 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | ---
language:
- en
license: mit
tags:
- llama
- agron
- code
- cybersecurity
- domain-pretrained
- text-generation
pipeline_tag: text-generation
---
# Agron (domain-pretrained: programming languages + cybersecurity)
Agron is a ~50.3M-parameter LLaMA-style model. The weights in this repository are
the result of **one full epoch** of domain continued pretraining over code,
cybersecurity, and WikiText replay, started from the previously published Agron
checkpoint.
## Current promoted weights
| Field | Value |
|---|---|
| Steps | 7,246 (1 full epoch) |
| Tokens | 118.7M |
| Batch size | 16 x 2 grad accum = 32 |
| Sequence length | 512 |
| LR | 3e-4 (cosine, warmup 100) |
| Optimizer | AdamW (fused) |
| Hardware | 1x Tesla T4 (fp16) |
| Training time | ~61 min |
| Avg train loss | 1.319 |
Full metrics: [`report.json`](report.json), [`eval_history.json`](eval_history.json),
[`eval_before.json`](eval_before.json).
## Pretraining corpus
Mixed on-the-fly from flat token streams:
- **Code** (majority): `codeparrot/github-code` filtered to common source extensions (.py, .js, .ts, .java, .c, .cpp, .go, .rs, .cs, .sh, .sql, ...), ~150M chars
- **Cybersecurity**: `Trendyol/Trendyol-Cybersecurity-Instruction-Tuning-Dataset` (system/user/assistant), ~120M chars
- **Replay**: `Salesforce/wikitext-2-raw-v1`, ~15M chars (prevents catastrophic forgetting)
Corpus built by [`prepare_domain_corpus.py`](https://github.com/Njap-png/Agron/blob/main/prepare_domain_corpus.py).
## Held-out evaluation
| Source | Before (loss/PPL) | After (loss/PPL) | Δ |
|---|---|---|---|
| code | 1.453 / 4.28 | 1.245 / 3.47 | -0.208 |
| cyber | 1.358 / 3.89 | 1.064 / 2.90 | -0.294 |
| replay (wikitext) | 5.626 / 277.6 | 5.024 / 152.0 | -0.602 |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("teddyoas/Agron")
model = AutoModelForCausalLM.from_pretrained("teddyoas/Agron") # promoted 1-epoch weights
```
## Files
- `model.safetensors`, `config.json`, `generation_config.json`
- `tokenizer.json`, `tokenizer_config.json`
- `domain_pretraining_curve.png`
- `eval_history.json`, `eval_before.json`, `report.json`, `samples.json`
|