---
language:
- en
license: mit
tags:
- llama
- agron
- code
- cybersecurity
- domain-pretrained
- text-generation
pipeline_tag: text-generation
---

# Agron (domain-pretrained: programming languages + cybersecurity)

Agron is a ~50.3M-parameter LLaMA-style model. The weights in this repository are
the result of **one full epoch** of domain continued pretraining over code,
cybersecurity, and WikiText replay, started from the previously published Agron
checkpoint.

## Current promoted weights

| Field | Value |
|---|---|
| Steps | 7,246 (1 full epoch) |
| Tokens | 118.7M |
| Batch size | 16 x 2 grad accum = 32 |
| Sequence length | 512 |
| LR | 3e-4 (cosine, warmup 100) |
| Optimizer | AdamW (fused) |
| Hardware | 1x Tesla T4 (fp16) |
| Training time | ~61 min |
| Avg train loss | 1.319 |

Full metrics: [`report.json`](report.json), [`eval_history.json`](eval_history.json),
[`eval_before.json`](eval_before.json).

## Pretraining corpus
Mixed on-the-fly from flat token streams:
- **Code** (majority): `codeparrot/github-code` filtered to common source extensions (.py, .js, .ts, .java, .c, .cpp, .go, .rs, .cs, .sh, .sql, ...), ~150M chars
- **Cybersecurity**: `Trendyol/Trendyol-Cybersecurity-Instruction-Tuning-Dataset` (system/user/assistant), ~120M chars  
- **Replay**: `Salesforce/wikitext-2-raw-v1`, ~15M chars (prevents catastrophic forgetting)

Corpus built by [`prepare_domain_corpus.py`](https://github.com/Njap-png/Agron/blob/main/prepare_domain_corpus.py).

## Held-out evaluation

| Source | Before (loss/PPL) | After (loss/PPL) | Δ |
|---|---|---|---|
| code | 1.453 / 4.28 | 1.245 / 3.47 | -0.208 |
| cyber | 1.358 / 3.89 | 1.064 / 2.90 | -0.294 |
| replay (wikitext) | 5.626 / 277.6 | 5.024 / 152.0 | -0.602 |

![Domain pretraining curves](domain_pretraining_curve.png)

## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("teddyoas/Agron")
model = AutoModelForCausalLM.from_pretrained("teddyoas/Agron")  # promoted 1-epoch weights
```

## Files
- `model.safetensors`, `config.json`, `generation_config.json`
- `tokenizer.json`, `tokenizer_config.json`
- `domain_pretraining_curve.png`
- `eval_history.json`, `eval_before.json`, `report.json`, `samples.json`