README.md

metadata

language:
  - en
license: mit
tags:
  - llama
  - agron
  - code
  - cybersecurity
  - domain-pretrained
  - text-generation
pipeline_tag: text-generation

Agron (domain-pretrained: programming languages + cybersecurity)

Agron is a ~50.3M-parameter LLaMA-style model. The weights in this repository are the result of one full epoch of domain continued pretraining over code, cybersecurity, and WikiText replay, started from the previously published Agron checkpoint.

Current promoted weights

Field	Value
Steps	7,246 (1 full epoch)
Tokens	118.7M
Batch size	16 x 2 grad accum = 32
Sequence length	512
LR	3e-4 (cosine, warmup 100)
Optimizer	AdamW (fused)
Hardware	1x Tesla T4 (fp16)
Training time	~61 min
Avg train loss	1.319

Full metrics: report.json, eval_history.json, eval_before.json.

Pretraining corpus

Mixed on-the-fly from flat token streams:

• Code (majority): codeparrot/github-code filtered to common source extensions (.py, .js, .ts, .java, .c, .cpp, .go, .rs, .cs, .sh, .sql, ...), ~150M chars
• Cybersecurity: Trendyol/Trendyol-Cybersecurity-Instruction-Tuning-Dataset (system/user/assistant), ~120M chars
• Replay: Salesforce/wikitext-2-raw-v1, ~15M chars (prevents catastrophic forgetting)

Corpus built by prepare_domain_corpus.py.

Held-out evaluation

Source	Before (loss/PPL)	After (loss/PPL)	Δ
code	1.453 / 4.28	1.245 / 3.47	-0.208
cyber	1.358 / 3.89	1.064 / 2.90	-0.294
replay (wikitext)	5.626 / 277.6	5.024 / 152.0	-0.602

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("teddyoas/Agron")
model = AutoModelForCausalLM.from_pretrained("teddyoas/Agron")  # promoted 1-epoch weights

Files

• model.safetensors, config.json, generation_config.json
• tokenizer.json, tokenizer_config.json
• domain_pretraining_curve.png
• eval_history.json, eval_before.json, report.json, samples.json